2024-09-16T01:37:27.592 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T01:37:27.614 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:37:27.707 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490 branch: squid description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906490' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - but it is still running - objects unfound and apparently lost - MDS_SLOW_METADATA_IO - MDS_TRIM sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkw4PX9rR7/EB1JahGQ2oaSXLILjhfMwVUH/+K8MD0CwCwrscYyqeMbC2nQvi7e4V9ylyHnbSpGle900WpmsRo= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIX1L+j0gd1wROxVfqPy7JRwhfMkjIyjTt06IByBlwoC9Rixjqd6FOHk+LHzACj/ByIvt2asPf6M2CvoQxkB/Pk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - thrashosds: null - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T01:37:27.708 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T01:37:27.709 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T01:37:27.709 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T01:37:27.711 INFO:teuthology.task.internal:Checking packages... 2024-09-16T01:37:27.731 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T01:37:27.731 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T01:37:27.731 INFO:teuthology.packaging:ref: None 2024-09-16T01:37:27.731 INFO:teuthology.packaging:tag: None 2024-09-16T01:37:27.732 INFO:teuthology.packaging:branch: squid 2024-09-16T01:37:27.732 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:37:27.732 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T01:37:27.971 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T01:37:27.972 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T01:37:27.978 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T01:37:27.978 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T01:37:28.020 INFO:teuthology.task.internal:Saving configuration 2024-09-16T01:37:28.037 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T01:37:28.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T01:37:28.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:32:21.456632', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkw4PX9rR7/EB1JahGQ2oaSXLILjhfMwVUH/+K8MD0CwCwrscYyqeMbC2nQvi7e4V9ylyHnbSpGle900WpmsRo='} 2024-09-16T01:37:28.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:32:21.454874', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIX1L+j0gd1wROxVfqPy7JRwhfMkjIyjTt06IByBlwoC9Rixjqd6FOHk+LHzACj/ByIvt2asPf6M2CvoQxkB/Pk='} 2024-09-16T01:37:28.089 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T01:37:28.094 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-16T01:37:28.095 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-09-16T01:37:28.095 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T01:37:28.172 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8b95bb3490>, signals=[15]) 2024-09-16T01:37:28.172 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T01:37:28.178 INFO:teuthology.task.internal:Opening connections... 2024-09-16T01:37:28.178 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:37:28.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:37:28.252 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:37:28.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:37:28.327 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T01:37:28.333 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-09-16T01:37:28.351 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-09-16T01:37:28.352 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-09-16T01:37:28.410 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-09-16T01:37:28.410 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-09-16T01:37:28.410 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-09-16T01:37:28.410 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T01:37:28.411 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T01:37:28.412 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-09-16T01:37:28.436 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-09-16T01:37:28.454 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-09-16T01:37:28.454 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-09-16T01:37:28.513 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T01:37:28.514 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T01:37:28.514 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T01:37:28.514 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-09-16T01:37:28.536 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T01:37:28.545 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T01:37:28.588 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T01:37:28.588 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:37:28.592 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:37:28.609 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T01:37:28.646 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T01:37:28.647 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:37:28.649 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:37:28.764 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T01:37:28.822 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T01:37:28.836 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-16T01:37:28.836 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-09-16T01:37:28.836 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-09-16T01:37:28.837 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:37:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:37:28.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-16T01:37:29.035 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-16T01:37:29.035 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-16T01:37:29.035 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-16T01:37:29.054 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-503.el9.x86_64 2024-09-16T01:37:29.054 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-16T01:37:29.054 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-16T01:37:29.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum upgrade -y linux-firmware 2024-09-16T01:37:31.411 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-16T01:37:35.412 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 20 MB 00:02 2024-09-16T01:37:40.453 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.5 MB 00:01 2024-09-16T01:37:41.946 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-16T01:37:43.025 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T01:37:48.398 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-16T01:37:50.521 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-09-16T01:37:50.522 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-09-16T01:37:50.522 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-09-16T01:37:50.621 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-16T01:37:50.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-16T01:37:50.778 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-16T01:37:50.779 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-16T01:37:50.781 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi146.front.sepia.ceph.com to complete... 2024-09-16T01:37:59.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16 01:37:59 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-16T01:37:59.458 INFO:teuthology.orchestra.run.smithi146.stderr:FINISHED --2024-09-16 01:37:59-- 2024-09-16T01:37:59.458 INFO:teuthology.orchestra.run.smithi146.stderr:Total wall clock time: 8.6s 2024-09-16T01:37:59.458 INFO:teuthology.orchestra.run.smithi146.stderr:Downloaded: 1 files, 343M in 8.2s (41.7 MB/s) 2024-09-16T01:37:59.461 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-16T01:37:59.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-16T01:38:01.582 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying... ######################################## 2024-09-16T01:38:01.718 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing... ######################################## 2024-09-16T01:38:01.846 INFO:teuthology.orchestra.run.smithi146.stdout:Updating / installing... 2024-09-16T01:38:09.895 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-16T01:38:49.451 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-16T01:38:49.452 DEBUG:teuthology.orchestra.run.smithi146:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-16T01:38:49.535 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-16T01:38:49.536 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-16T01:38:49.537 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-16T01:38:49.538 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-16T01:38:49.539 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-16T01:38:49.540 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-16T01:38:49.541 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-16T01:38:49.542 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-16T01:38:49.543 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-16T01:38:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-16T01:38:49.545 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-16T01:38:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-16T01:38:49.547 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-16T01:38:49.548 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-16T01:38:49.549 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-16T01:38:49.550 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-16T01:38:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-16T01:38:49.552 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-16T01:38:49.553 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-16T01:38:49.554 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-16T01:38:49.555 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-16T01:38:49.556 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-16T01:38:49.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-16T01:38:49.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-16T01:38:49.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-16T01:38:49.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-16T01:38:49.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-16T01:38:49.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-16T01:38:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-16T01:38:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-16T01:38:49.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-16T01:38:49.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-16T01:38:49.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-16T01:38:49.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-16T01:38:49.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-16T01:38:49.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-16T01:38:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-16T01:38:49.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-16T01:38:49.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-16T01:38:49.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-16T01:38:49.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-16T01:38:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-16T01:38:49.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-16T01:38:49.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-16T01:38:49.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-16T01:38:49.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-16T01:38:49.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-16T01:38:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-16T01:38:49.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-16T01:38:49.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-16T01:38:49.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-16T01:38:49.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-16T01:38:49.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-16T01:38:49.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-16T01:38:49.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-16T01:38:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-16T01:38:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-16T01:38:49.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-16T01:38:49.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-16T01:38:49.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-16T01:38:49.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-16T01:38:49.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-16T01:38:49.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-16T01:38:49.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-16T01:38:49.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-16T01:38:49.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-16T01:38:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-16T01:38:49.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-16T01:38:49.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-16T01:38:49.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-16T01:38:49.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-16T01:38:49.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-16T01:38:49.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-16T01:38:49.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-16T01:38:49.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-16T01:38:49.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-16T01:38:49.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-16T01:38:49.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-16T01:38:49.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-16T01:38:49.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-16T01:38:49.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-16T01:38:49.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-16T01:38:49.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-16T01:38:49.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-16T01:38:49.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-16T01:38:49.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-16T01:38:49.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-16T01:38:49.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-16T01:38:49.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-16T01:38:49.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-16T01:38:49.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-16T01:38:49.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-16T01:38:49.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-16T01:38:49.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-16T01:38:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-16T01:38:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-16T01:38:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-16T01:38:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-16T01:38:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-16T01:38:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-16T01:38:49.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-16T01:38:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-16T01:38:49.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-16T01:38:49.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-16T01:38:49.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-16T01:38:49.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-16T01:38:49.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-16T01:38:49.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-16T01:38:49.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-16T01:38:49.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-16T01:38:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-16T01:38:49.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-16T01:38:49.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-16T01:38:49.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-16T01:38:49.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-16T01:38:49.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-16T01:38:49.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-16T01:38:49.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-16T01:38:49.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-16T01:38:49.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-16T01:38:49.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-16T01:38:49.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-16T01:38:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-16T01:38:49.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-16T01:38:49.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-16T01:38:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-16T01:38:49.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-16T01:38:49.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-16T01:38:49.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-16T01:38:49.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-16T01:38:49.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-16T01:38:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-16T01:38:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-16T01:38:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-16T01:38:49.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-16T01:38:49.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-16T01:38:49.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-16T01:38:49.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-16T01:38:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-16T01:38:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-16T01:38:49.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-16T01:38:49.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-16T01:38:49.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-16T01:38:49.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-16T01:38:49.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-16T01:38:49.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-16T01:38:49.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-16T01:38:49.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-16T01:38:49.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-16T01:38:49.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-16T01:38:49.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-16T01:38:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-16T01:38:49.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-16T01:38:49.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-16T01:38:49.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-16T01:38:49.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-16T01:38:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-16T01:38:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-16T01:38:49.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-16T01:38:49.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-16T01:38:49.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-16T01:38:49.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-16T01:38:49.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-16T01:38:49.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-16T01:38:49.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-16T01:38:49.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-16T01:38:49.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-16T01:38:49.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-16T01:38:49.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-16T01:38:49.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-16T01:38:49.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-16T01:38:49.686 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-16T01:38:49.687 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-16T01:38:49.692 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.692 DEBUG:teuthology.orchestra.run.smithi146:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-16T01:38:49.715 INFO:teuthology.orchestra.run.smithi146.stdout:postinstall scriptlet (using /bin/sh): 2024-09-16T01:38:49.715 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout:for file in vmlinuz System.map config; do 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout: fi 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-16T01:38:49.716 INFO:teuthology.orchestra.run.smithi146.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.717 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-09-16T01:38:49.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-09-16T01:38:49.798 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-09-16T01:38:49.798 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-09-16T01:38:49.798 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-09-16T01:38:49.798 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 82.el9 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8274670 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-09-16T01:38:49.799 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:38:49.800 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T01:38:49.801 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T01:38:49.801 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:49.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T01:38:50.393 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-09-16T01:38:51.811 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T01:38:51.847 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-09-16T01:38:51.849 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-09-16T01:38:51.864 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.sfPfAQam8I 2024-09-16T01:38:51.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sfPfAQam8I 2024-09-16T01:38:51.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.sfPfAQam8I 2024-09-16T01:38:52.083 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.sfPfAQam8I is 6KB 2024-09-16T01:38:52.131 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.sfPfAQam8I 2024-09-16T01:38:52.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T01:38:52.209 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T01:38:52.209 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T01:38:52.209 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T01:38:52.209 INFO:teuthology.orchestra.run.smithi146.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-16T01:38:52.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:38:52.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-09-16T01:39:22.363 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-16T01:39:22.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:39:22.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:39:40.733 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-16T01:39:49.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:39:49.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:39:52.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-16T01:40:04.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:40:04.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:41:04.802 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T01:41:19.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:41:19.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:41:20.130 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-16T01:41:20.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:41:20.654 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-16T01:41:20.654 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-16T01:41:20.688 INFO:teuthology.orchestra.run.smithi146.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:41:20.688 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-16T01:41:20.689 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T01:41:20.689 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T01:41:21.690 DEBUG:teuthology.parallel:result is None 2024-09-16T01:41:21.690 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T01:41:21.698 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T01:41:21.698 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T01:41:21.703 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T01:41:21.737 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T01:41:21.743 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T01:41:21.791 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T01:41:21.792 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T01:41:21.796 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T01:41:21.894 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T01:41:21.900 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T01:41:21.900 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:41:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:21.917 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T01:41:21.922 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T01:41:22.023 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:41:22.025 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:41:22.043 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:41:22.053 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:41:22.056 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T01:41:22.063 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T01:41:22.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T01:41:22.067 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T01:41:22.121 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T01:41:22.131 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T01:41:22.131 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T01:41:22.136 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T01:41:22.166 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:41:22.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:41:22.349 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:41:22.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:41:22.483 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:41:22.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T01:41:22.538 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:41:22.584 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:41:22.638 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:41:22.721 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:41:22.766 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:41:22.766 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T01:41:22.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-09-16T01:41:22.837 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-09-16T01:41:22.910 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:41:22.911 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:41:23.341 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T01:41:23.377 INFO:teuthology.task.internal:Starting timer... 2024-09-16T01:41:23.377 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T01:41:23.412 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T01:41:23.460 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T01:41:23.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-09-16T01:41:23.548 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-09-16T01:41:23.551 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-09-16T01:41:23.612 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-09-16T01:41:23.614 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T01:41:23.614 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-09-16T01:41:23.658 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-09-16T01:41:23.659 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-09-16T01:41:23.682 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-09-16T01:41:23.682 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi146.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2024-09-16T01:41:23.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:41:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:23.757 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:41:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:23.785 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T01:41:23.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-09-16T01:41:23.848 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-09-16T01:41:23.894 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T01:41:23.904 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T01:41:23.916 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T01:41:23.917 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi146.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T01:46:05.082 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-09-16T01:46:05.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:46:05.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:46:05.191 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-16T01:46:05.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-16T01:46:05.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-16T01:46:05.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:46:05.370 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-09-16T01:46:05.448 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-09-16T01:46:05.449 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T01:46:05.459 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T01:46:05.459 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T01:46:05.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:46:05.463 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T01:46:05.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:46:05.519 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T01:46:05.520 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T01:46:05.533 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T01:46:05.549 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T01:46:05.557 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-09-16T01:46:05.568 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-09-16T01:46:05.583 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T01:46:05.597 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T01:46:05.600 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-09-16T01:46:05.624 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-09-16T01:46:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:46:05.654 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T01:46:05.655 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:46:05.655 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-09-16T01:46:05.655 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.655 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.655 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.655 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T01:46:05.778 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:46:05.784 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:46:05.784 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-09-16T01:46:05.784 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.784 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.784 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.785 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:46:05.785 INFO:teuthology.run_tasks:Running task install... 2024-09-16T01:46:05.793 DEBUG:teuthology.task.install:project ceph 2024-09-16T01:46:05.793 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T01:46:05.794 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T01:46:05.794 INFO:teuthology.task.install:Using flavor: default 2024-09-16T01:46:05.799 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T01:46:05.799 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-16T01:46:05.799 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T01:46:05.800 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:46:05.802 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T01:46:05.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:46:05.944 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T01:46:05.945 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T01:46:05.950 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T01:46:05.950 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T01:46:06.053 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T01:46:06.053 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:46:06.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T01:46:06.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T01:46:06.060 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:46:06.060 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T01:46:06.086 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T01:46:06.086 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T01:46:06.110 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T01:46:06.110 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T01:46:06.152 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T01:46:06.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T01:46:06.227 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T01:46:06.286 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T01:46:06.291 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-09-16T01:46:06.292 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-09-16T01:46:06.341 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-09-16T01:46:06.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-09-16T01:46:06.526 INFO:teuthology.orchestra.run.smithi161.stdout:45 files removed 2024-09-16T01:46:06.552 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T01:46:06.615 INFO:teuthology.orchestra.run.smithi146.stdout:45 files removed 2024-09-16T01:46:06.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T01:46:07.111 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 297 kB/s | 83 kB 00:00 2024-09-16T01:46:07.325 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 298 kB/s | 83 kB 00:00 2024-09-16T01:46:07.378 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 52 kB/s | 12 kB 00:00 2024-09-16T01:46:07.608 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T01:46:07.646 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-16T01:46:07.867 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-16T01:46:08.326 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.2 MB 00:00 2024-09-16T01:46:09.337 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-16T01:46:12.721 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 20 MB 00:02 2024-09-16T01:46:12.977 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 20 MB 00:03 2024-09-16T01:46:16.922 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-16T01:46:18.261 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.5 MB 00:01 2024-09-16T01:46:18.414 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-16T01:46:19.444 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T01:46:19.903 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-16T01:46:20.966 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T01:46:24.925 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T01:46:27.211 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:46:27.211 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:46:27.223 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T01:46:27.223 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T01:46:27.224 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T01:46:27.225 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T01:46:27.227 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T01:46:27.229 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T01:46:27.230 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T01:46:27.280 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-16T01:46:27.293 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-09-16T01:46:27.313 INFO:teuthology.orchestra.run.smithi161.stdout:====================================================================================== 2024-09-16T01:46:27.313 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout:====================================================================================== 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T01:46:27.314 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T01:46:27.315 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T01:46:27.316 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T01:46:27.317 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T01:46:27.318 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T01:46:27.319 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T01:46:27.320 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T01:46:27.321 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T01:46:27.322 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T01:46:27.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T01:46:27.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T01:46:27.325 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout:====================================================================================== 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout:Install 148 Packages 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 8 Packages 2024-09-16T01:46:27.326 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:46:27.327 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 226 M 2024-09-16T01:46:27.327 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-09-16T01:46:28.134 INFO:teuthology.orchestra.run.smithi161.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 44 kB/s | 6.5 kB 00:00 2024-09-16T01:46:28.369 INFO:teuthology.orchestra.run.smithi161.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.8 MB/s | 896 kB 00:00 2024-09-16T01:46:28.451 INFO:teuthology.orchestra.run.smithi161.stdout:(3/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 12 MB/s | 5.4 MB 00:00 2024-09-16T01:46:28.485 INFO:teuthology.orchestra.run.smithi161.stdout:(4/156): ceph-immutable-object-cache-19.1.1-326 1.2 MB/s | 145 kB 00:00 2024-09-16T01:46:28.569 INFO:teuthology.orchestra.run.smithi161.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 20 MB/s | 2.4 MB 00:00 2024-09-16T01:46:28.702 INFO:teuthology.orchestra.run.smithi161.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 7.0 MB/s | 1.5 MB 00:00 2024-09-16T01:46:28.786 INFO:teuthology.orchestra.run.smithi161.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 22 MB/s | 4.7 MB 00:00 2024-09-16T01:46:29.294 INFO:teuthology.orchestra.run.smithi161.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 16 MB/s | 21 MB 00:01 2024-09-16T01:46:29.428 INFO:teuthology.orchestra.run.smithi161.stdout:(9/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 17 MB/s | 11 MB 00:00 2024-09-16T01:46:29.453 INFO:teuthology.orchestra.run.smithi161.stdout:(10/156): ceph-selinux-19.1.1-326.g8921c8ef.el9 159 kB/s | 25 kB 00:00 2024-09-16T01:46:29.528 INFO:teuthology.orchestra.run.smithi161.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.0 MB/s | 228 kB 00:00 2024-09-16T01:46:29.595 INFO:teuthology.orchestra.run.smithi161.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 405 kB/s | 27 kB 00:00 2024-09-16T01:46:29.671 INFO:teuthology.orchestra.run.smithi161.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 423 kB/s | 32 kB 00:00 2024-09-16T01:46:29.784 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:46:29.785 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:46:29.801 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T01:46:29.801 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T01:46:29.801 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T01:46:29.801 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T01:46:29.802 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T01:46:29.805 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T01:46:29.806 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T01:46:29.875 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-09-16T01:46:29.897 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-09-16T01:46:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-09-16T01:46:29.897 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-09-16T01:46:29.897 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-09-16T01:46:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T01:46:29.898 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T01:46:29.899 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-09-16T01:46:29.900 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T01:46:29.901 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T01:46:29.902 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T01:46:29.903 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T01:46:29.904 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T01:46:29.905 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T01:46:29.906 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T01:46:29.907 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T01:46:29.908 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T01:46:29.909 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T01:46:29.910 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T01:46:29.911 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:46:29.912 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-09-16T01:46:29.912 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-09-16T01:46:29.912 INFO:teuthology.orchestra.run.smithi146.stdout:Install 148 Packages 2024-09-16T01:46:29.912 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 8 Packages 2024-09-16T01:46:29.912 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:46:29.912 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 226 M 2024-09-16T01:46:29.913 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-09-16T01:46:29.913 INFO:teuthology.orchestra.run.smithi161.stdout:(14/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 14 MB/s | 17 MB 00:01 2024-09-16T01:46:29.938 INFO:teuthology.orchestra.run.smithi161.stdout:(15/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 2.8 MB/s | 751 kB 00:00 2024-09-16T01:46:29.971 INFO:teuthology.orchestra.run.smithi161.stdout:(16/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.7 MB/s | 163 kB 00:00 2024-09-16T01:46:29.997 INFO:teuthology.orchestra.run.smithi161.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 2.1 MB/s | 127 kB 00:00 2024-09-16T01:46:30.030 INFO:teuthology.orchestra.run.smithi161.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 8.4 MB/s | 506 kB 00:00 2024-09-16T01:46:30.081 INFO:teuthology.orchestra.run.smithi161.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 903 kB/s | 45 kB 00:00 2024-09-16T01:46:30.172 INFO:teuthology.orchestra.run.smithi161.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.5 MB/s | 136 kB 00:00 2024-09-16T01:46:30.273 INFO:teuthology.orchestra.run.smithi161.stdout:(21/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 19 MB/s | 5.3 MB 00:00 2024-09-16T01:46:30.298 INFO:teuthology.orchestra.run.smithi161.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.3 MB/s | 161 kB 00:00 2024-09-16T01:46:30.332 INFO:teuthology.orchestra.run.smithi161.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 5.4 MB/s | 320 kB 00:00 2024-09-16T01:46:30.357 INFO:teuthology.orchestra.run.smithi161.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 5.0 MB/s | 300 kB 00:00 2024-09-16T01:46:30.382 INFO:teuthology.orchestra.run.smithi161.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T01:46:30.416 INFO:teuthology.orchestra.run.smithi161.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.4 MB/s | 85 kB 00:00 2024-09-16T01:46:30.475 INFO:teuthology.orchestra.run.smithi161.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.9 MB/s | 170 kB 00:00 2024-09-16T01:46:30.541 INFO:teuthology.orchestra.run.smithi161.stdout:(28/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 20 MB/s | 3.1 MB 00:00 2024-09-16T01:46:30.567 INFO:teuthology.orchestra.run.smithi161.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 347 kB/s | 32 kB 00:00 2024-09-16T01:46:30.600 INFO:teuthology.orchestra.run.smithi161.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.4 MB/s | 146 kB 00:00 2024-09-16T01:46:30.792 INFO:teuthology.orchestra.run.smithi161.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 16 MB/s | 3.6 MB 00:00 2024-09-16T01:46:30.824 INFO:teuthology.orchestra.run.smithi146.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 26 kB/s | 6.5 kB 00:00 2024-09-16T01:46:30.876 INFO:teuthology.orchestra.run.smithi161.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 2.9 MB/s | 249 kB 00:00 2024-09-16T01:46:30.951 INFO:teuthology.orchestra.run.smithi161.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 659 kB/s | 49 kB 00:00 2024-09-16T01:46:31.043 INFO:teuthology.orchestra.run.smithi161.stdout:(34/156): ceph-mgr-diskprediction-local-19.1.1- 17 MB/s | 7.4 MB 00:00 2024-09-16T01:46:31.060 INFO:teuthology.orchestra.run.smithi161.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 151 kB/s | 16 kB 00:00 2024-09-16T01:46:31.102 INFO:teuthology.orchestra.run.smithi161.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.8 MB/s | 285 kB 00:00 2024-09-16T01:46:31.136 INFO:teuthology.orchestra.run.smithi161.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 10 MB/s | 765 kB 00:00 2024-09-16T01:46:31.161 INFO:teuthology.orchestra.run.smithi161.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 311 kB/s | 18 kB 00:00 2024-09-16T01:46:31.214 INFO:teuthology.orchestra.run.smithi146.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 2.2 MB/s | 896 kB 00:00 2024-09-16T01:46:31.270 INFO:teuthology.orchestra.run.smithi161.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 597 kB/s | 80 kB 00:00 2024-09-16T01:46:31.307 INFO:teuthology.orchestra.run.smithi146.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.6 MB/s | 145 kB 00:00 2024-09-16T01:46:31.695 INFO:teuthology.orchestra.run.smithi161.stdout:(40/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 21 MB/s | 49 MB 00:02 2024-09-16T01:46:31.729 INFO:teuthology.orchestra.run.smithi161.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 417 kB/s | 191 kB 00:00 2024-09-16T01:46:31.754 INFO:teuthology.orchestra.run.smithi161.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 801 kB 00:00 2024-09-16T01:46:31.796 INFO:teuthology.orchestra.run.smithi161.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T01:46:31.846 INFO:teuthology.orchestra.run.smithi161.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.2 MB 00:00 2024-09-16T01:46:31.871 INFO:teuthology.orchestra.run.smithi161.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-09-16T01:46:31.890 INFO:teuthology.orchestra.run.smithi146.stdout:(4/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 16 MB/s | 21 MB 00:01 2024-09-16T01:46:31.896 INFO:teuthology.orchestra.run.smithi161.stdout:(46/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T01:46:31.930 INFO:teuthology.orchestra.run.smithi161.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-16T01:46:31.949 INFO:teuthology.orchestra.run.smithi146.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 3.7 MB/s | 2.4 MB 00:00 2024-09-16T01:46:31.955 INFO:teuthology.orchestra.run.smithi161.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-16T01:46:32.049 INFO:teuthology.orchestra.run.smithi146.stdout:(6/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 3.7 MB/s | 5.4 MB 00:01 2024-09-16T01:46:32.099 INFO:teuthology.orchestra.run.smithi146.stdout:(7/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 7.3 MB/s | 1.5 MB 00:00 2024-09-16T01:46:32.500 INFO:teuthology.orchestra.run.smithi146.stdout:(8/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 27 MB/s | 11 MB 00:00 2024-09-16T01:46:32.541 INFO:teuthology.orchestra.run.smithi146.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 609 kB/s | 25 kB 00:00 2024-09-16T01:46:32.633 INFO:teuthology.orchestra.run.smithi146.stdout:(10/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-09-16T01:46:32.656 INFO:teuthology.orchestra.run.smithi161.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 153 kB/s | 107 kB 00:00 2024-09-16T01:46:32.700 INFO:teuthology.orchestra.run.smithi146.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.4 MB/s | 228 kB 00:00 2024-09-16T01:46:32.742 INFO:teuthology.orchestra.run.smithi146.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 656 kB/s | 27 kB 00:00 2024-09-16T01:46:32.798 INFO:teuthology.orchestra.run.smithi161.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 765 kB/s | 689 kB 00:00 2024-09-16T01:46:32.801 INFO:teuthology.orchestra.run.smithi146.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 545 kB/s | 32 kB 00:00 2024-09-16T01:46:32.840 INFO:teuthology.orchestra.run.smithi161.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-09-16T01:46:32.951 INFO:teuthology.orchestra.run.smithi146.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 4.9 MB/s | 751 kB 00:00 2024-09-16T01:46:33.035 INFO:teuthology.orchestra.run.smithi146.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 1.9 MB/s | 163 kB 00:00 2024-09-16T01:46:33.118 INFO:teuthology.orchestra.run.smithi146.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 1.5 MB/s | 127 kB 00:00 2024-09-16T01:46:33.198 INFO:teuthology.orchestra.run.smithi161.stdout:(52/156): elfutils-libelf-devel-0.191-4.el9.x86 40 kB/s | 22 kB 00:00 2024-09-16T01:46:33.285 INFO:teuthology.orchestra.run.smithi146.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 3.0 MB/s | 506 kB 00:00 2024-09-16T01:46:33.390 INFO:teuthology.orchestra.run.smithi161.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 59 kB/s | 33 kB 00:00 2024-09-16T01:46:33.488 INFO:teuthology.orchestra.run.smithi146.stdout:(18/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 12 MB/s | 17 MB 00:01 2024-09-16T01:46:33.528 INFO:teuthology.orchestra.run.smithi146.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 1.1 MB/s | 45 kB 00:00 2024-09-16T01:46:33.590 INFO:teuthology.orchestra.run.smithi161.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 396 kB/s | 313 kB 00:00 2024-09-16T01:46:33.603 INFO:teuthology.orchestra.run.smithi146.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.8 MB/s | 136 kB 00:00 2024-09-16T01:46:33.703 INFO:teuthology.orchestra.run.smithi146.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.6 MB/s | 161 kB 00:00 2024-09-16T01:46:33.770 INFO:teuthology.orchestra.run.smithi146.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.7 MB/s | 320 kB 00:00 2024-09-16T01:46:33.862 INFO:teuthology.orchestra.run.smithi146.stdout:(23/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 9.3 MB/s | 5.3 MB 00:00 2024-09-16T01:46:33.874 INFO:teuthology.orchestra.run.smithi161.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 38 kB/s | 18 kB 00:00 2024-09-16T01:46:33.887 INFO:teuthology.orchestra.run.smithi146.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 2.5 MB/s | 300 kB 00:00 2024-09-16T01:46:33.913 INFO:teuthology.orchestra.run.smithi146.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T01:46:33.963 INFO:teuthology.orchestra.run.smithi146.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.1 MB/s | 85 kB 00:00 2024-09-16T01:46:34.057 INFO:teuthology.orchestra.run.smithi161.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 24 kB/s | 11 kB 00:00 2024-09-16T01:46:34.333 INFO:teuthology.orchestra.run.smithi161.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-09-16T01:46:34.422 INFO:teuthology.orchestra.run.smithi146.stdout:(27/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 26 MB/s | 49 MB 00:01 2024-09-16T01:46:34.447 INFO:teuthology.orchestra.run.smithi146.stdout:(28/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 352 kB/s | 170 kB 00:00 2024-09-16T01:46:34.472 INFO:teuthology.orchestra.run.smithi146.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 639 kB/s | 32 kB 00:00 2024-09-16T01:46:34.497 INFO:teuthology.orchestra.run.smithi146.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.9 MB/s | 146 kB 00:00 2024-09-16T01:46:34.564 INFO:teuthology.orchestra.run.smithi146.stdout:(31/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 4.8 MB/s | 3.1 MB 00:00 2024-09-16T01:46:34.600 INFO:teuthology.orchestra.run.smithi161.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 221 kB/s | 160 kB 00:00 2024-09-16T01:46:34.625 INFO:teuthology.orchestra.run.smithi161.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 80 kB/s | 45 kB 00:00 2024-09-16T01:46:34.648 INFO:teuthology.orchestra.run.smithi146.stdout:(32/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 20 MB/s | 3.6 MB 00:00 2024-09-16T01:46:34.673 INFO:teuthology.orchestra.run.smithi146.stdout:(33/156): ceph-mgr-modules-core-19.1.1-326.g892 2.2 MB/s | 249 kB 00:00 2024-09-16T01:46:34.698 INFO:teuthology.orchestra.run.smithi146.stdout:(34/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 990 kB/s | 49 kB 00:00 2024-09-16T01:46:34.724 INFO:teuthology.orchestra.run.smithi146.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 328 kB/s | 16 kB 00:00 2024-09-16T01:46:34.757 INFO:teuthology.orchestra.run.smithi146.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.8 MB/s | 285 kB 00:00 2024-09-16T01:46:34.799 INFO:teuthology.orchestra.run.smithi146.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 440 kB/s | 18 kB 00:00 2024-09-16T01:46:34.833 INFO:teuthology.orchestra.run.smithi146.stdout:(38/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 6.9 MB/s | 765 kB 00:00 2024-09-16T01:46:34.966 INFO:teuthology.orchestra.run.smithi146.stdout:(39/156): ceph-mgr-diskprediction-local-19.1.1- 16 MB/s | 7.4 MB 00:00 2024-09-16T01:46:35.167 INFO:teuthology.orchestra.run.smithi161.stdout:(60/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 45 kB/s | 24 kB 00:00 2024-09-16T01:46:35.192 INFO:teuthology.orchestra.run.smithi161.stdout:(61/156): librdkafka-1.6.1-102.el9.x86_64.rpm 772 kB/s | 662 kB 00:00 2024-09-16T01:46:35.467 INFO:teuthology.orchestra.run.smithi161.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 670 kB/s | 581 kB 00:00 2024-09-16T01:46:35.525 INFO:teuthology.orchestra.run.smithi146.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 110 kB/s | 80 kB 00:00 2024-09-16T01:46:35.750 INFO:teuthology.orchestra.run.smithi146.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 244 kB/s | 191 kB 00:00 2024-09-16T01:46:35.792 INFO:teuthology.orchestra.run.smithi161.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-09-16T01:46:35.801 INFO:teuthology.orchestra.run.smithi146.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 828 kB/s | 801 kB 00:00 2024-09-16T01:46:35.976 INFO:teuthology.orchestra.run.smithi161.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 373 kB/s | 292 kB 00:00 2024-09-16T01:46:36.193 INFO:teuthology.orchestra.run.smithi161.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 260 kB/s | 188 kB 00:00 2024-09-16T01:46:36.293 INFO:teuthology.orchestra.run.smithi146.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 330 kB/s | 253 kB 00:00 2024-09-16T01:46:36.360 INFO:teuthology.orchestra.run.smithi161.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 68 kB/s | 38 kB 00:00 2024-09-16T01:46:36.526 INFO:teuthology.orchestra.run.smithi146.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 147 kB/s | 106 kB 00:00 2024-09-16T01:46:36.727 INFO:teuthology.orchestra.run.smithi146.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2024-09-16T01:46:37.010 INFO:teuthology.orchestra.run.smithi146.stdout:(46/156): python3-pycparser-2.20-6.el9.noarch.r 189 kB/s | 135 kB 00:00 2024-09-16T01:46:37.193 INFO:teuthology.orchestra.run.smithi161.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 4.1 MB/s | 4.9 MB 00:01 2024-09-16T01:46:37.252 INFO:teuthology.orchestra.run.smithi146.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 175 kB/s | 125 kB 00:00 2024-09-16T01:46:37.360 INFO:teuthology.orchestra.run.smithi161.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:00 2024-09-16T01:46:37.444 INFO:teuthology.orchestra.run.smithi161.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-09-16T01:46:37.460 INFO:teuthology.orchestra.run.smithi146.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 293 kB/s | 215 kB 00:00 2024-09-16T01:46:37.744 INFO:teuthology.orchestra.run.smithi146.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 939 kB/s | 689 kB 00:00 2024-09-16T01:46:37.861 INFO:teuthology.orchestra.run.smithi146.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 186 kB/s | 22 kB 00:00 2024-09-16T01:46:37.920 INFO:teuthology.orchestra.run.smithi146.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 233 kB/s | 107 kB 00:00 2024-09-16T01:46:37.978 INFO:teuthology.orchestra.run.smithi146.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-09-16T01:46:38.029 INFO:teuthology.orchestra.run.smithi146.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-16T01:46:38.094 INFO:teuthology.orchestra.run.smithi161.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 187 kB/s | 137 kB 00:00 2024-09-16T01:46:38.095 INFO:teuthology.orchestra.run.smithi146.stdout:(54/156): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-09-16T01:46:38.129 INFO:teuthology.orchestra.run.smithi146.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-09-16T01:46:38.136 INFO:teuthology.orchestra.run.smithi161.stdout:(71/156): postgresql-contrib-13.16-1.el9.x86_64 966 kB/s | 910 kB 00:00 2024-09-16T01:46:38.213 INFO:teuthology.orchestra.run.smithi146.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-09-16T01:46:38.305 INFO:teuthology.orchestra.run.smithi146.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.2 MB/s | 3.0 MB 00:00 2024-09-16T01:46:38.338 INFO:teuthology.orchestra.run.smithi146.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-09-16T01:46:38.363 INFO:teuthology.orchestra.run.smithi146.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 684 kB/s | 160 kB 00:00 2024-09-16T01:46:38.464 INFO:teuthology.orchestra.run.smithi146.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-16T01:46:38.489 INFO:teuthology.orchestra.run.smithi146.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 196 kB/s | 24 kB 00:00 2024-09-16T01:46:38.531 INFO:teuthology.orchestra.run.smithi146.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 3.0 MB/s | 581 kB 00:00 2024-09-16T01:46:38.581 INFO:teuthology.orchestra.run.smithi146.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 406 kB/s | 47 kB 00:00 2024-09-16T01:46:38.620 INFO:teuthology.orchestra.run.smithi161.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 4.9 MB/s | 5.8 MB 00:01 2024-09-16T01:46:38.648 INFO:teuthology.orchestra.run.smithi146.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-16T01:46:38.690 INFO:teuthology.orchestra.run.smithi146.stdout:(65/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-16T01:46:38.715 INFO:teuthology.orchestra.run.smithi146.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 287 kB/s | 38 kB 00:00 2024-09-16T01:46:38.811 INFO:teuthology.orchestra.run.smithi161.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 129 kB/s | 87 kB 00:00 2024-09-16T01:46:38.857 INFO:teuthology.orchestra.run.smithi146.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-16T01:46:39.000 INFO:teuthology.orchestra.run.smithi146.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 6.3 MB/s | 910 kB 00:00 2024-09-16T01:46:39.078 INFO:teuthology.orchestra.run.smithi161.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-09-16T01:46:39.092 INFO:teuthology.orchestra.run.smithi146.stdout:(69/156): openblas-openmp-0.3.26-2.el9.x86_64.r 11 MB/s | 4.9 MB 00:00 2024-09-16T01:46:39.125 INFO:teuthology.orchestra.run.smithi146.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.1 MB/s | 137 kB 00:00 2024-09-16T01:46:39.200 INFO:teuthology.orchestra.run.smithi146.stdout:(71/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-09-16T01:46:39.259 INFO:teuthology.orchestra.run.smithi146.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-09-16T01:46:39.309 INFO:teuthology.orchestra.run.smithi146.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 806 kB/s | 87 kB 00:00 2024-09-16T01:46:39.426 INFO:teuthology.orchestra.run.smithi146.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 525 kB/s | 61 kB 00:00 2024-09-16T01:46:39.445 INFO:teuthology.orchestra.run.smithi161.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 97 kB/s | 61 kB 00:00 2024-09-16T01:46:39.510 INFO:teuthology.orchestra.run.smithi146.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-09-16T01:46:39.543 INFO:teuthology.orchestra.run.smithi146.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-16T01:46:39.627 INFO:teuthology.orchestra.run.smithi146.stdout:(77/156): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-16T01:46:39.652 INFO:teuthology.orchestra.run.smithi146.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-09-16T01:46:39.669 INFO:teuthology.orchestra.run.smithi146.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 204 kB/s | 26 kB 00:00 2024-09-16T01:46:39.736 INFO:teuthology.orchestra.run.smithi146.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-09-16T01:46:39.770 INFO:teuthology.orchestra.run.smithi146.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-16T01:46:39.786 INFO:teuthology.orchestra.run.smithi146.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 297 kB/s | 35 kB 00:00 2024-09-16T01:46:39.820 INFO:teuthology.orchestra.run.smithi161.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 6.0 MB 00:01 2024-09-16T01:46:39.846 INFO:teuthology.orchestra.run.smithi161.stdout:(77/156): python3-devel-3.9.19-8.el9.x86_64.rpm 319 kB/s | 245 kB 00:00 2024-09-16T01:46:39.887 INFO:teuthology.orchestra.run.smithi146.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-09-16T01:46:39.912 INFO:teuthology.orchestra.run.smithi146.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-09-16T01:46:40.037 INFO:teuthology.orchestra.run.smithi146.stdout:(85/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-16T01:46:40.054 INFO:teuthology.orchestra.run.smithi146.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 298 kB/s | 42 kB 00:00 2024-09-16T01:46:40.079 INFO:teuthology.orchestra.run.smithi146.stdout:(87/156): python3-packaging-20.9-5.el9.noarch.r 402 kB/s | 77 kB 00:00 2024-09-16T01:46:40.155 INFO:teuthology.orchestra.run.smithi146.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-09-16T01:46:40.180 INFO:teuthology.orchestra.run.smithi146.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-16T01:46:40.205 INFO:teuthology.orchestra.run.smithi146.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-09-16T01:46:40.221 INFO:teuthology.orchestra.run.smithi161.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 320 kB/s | 248 kB 00:00 2024-09-16T01:46:40.256 INFO:teuthology.orchestra.run.smithi146.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 507 kB/s | 51 kB 00:00 2024-09-16T01:46:40.281 INFO:teuthology.orchestra.run.smithi146.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 534 kB/s | 54 kB 00:00 2024-09-16T01:46:40.346 INFO:teuthology.orchestra.run.smithi161.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 38 kB/s | 19 kB 00:00 2024-09-16T01:46:40.372 INFO:teuthology.orchestra.run.smithi146.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-09-16T01:46:40.389 INFO:teuthology.orchestra.run.smithi146.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 407 kB/s | 44 kB 00:00 2024-09-16T01:46:40.422 INFO:teuthology.orchestra.run.smithi161.stdout:(80/156): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-09-16T01:46:40.490 INFO:teuthology.orchestra.run.smithi146.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-16T01:46:40.515 INFO:teuthology.orchestra.run.smithi146.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 287 kB/s | 36 kB 00:00 2024-09-16T01:46:40.599 INFO:teuthology.orchestra.run.smithi146.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-09-16T01:46:40.632 INFO:teuthology.orchestra.run.smithi146.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-16T01:46:40.708 INFO:teuthology.orchestra.run.smithi146.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 532 kB/s | 58 kB 00:00 2024-09-16T01:46:40.733 INFO:teuthology.orchestra.run.smithi146.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 957 kB/s | 96 kB 00:00 2024-09-16T01:46:40.939 INFO:teuthology.orchestra.run.smithi161.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 240 kB/s | 172 kB 00:00 2024-09-16T01:46:40.942 INFO:teuthology.orchestra.run.smithi146.stdout:(101/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-16T01:46:40.966 INFO:teuthology.orchestra.run.smithi146.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 192 kB/s | 45 kB 00:00 2024-09-16T01:46:40.972 INFO:teuthology.orchestra.run.smithi161.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-09-16T01:46:41.007 INFO:teuthology.orchestra.run.smithi146.stdout:(103/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 213 kB/s | 64 kB 00:00 2024-09-16T01:46:41.069 INFO:teuthology.orchestra.run.smithi146.stdout:(104/156): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-09-16T01:46:41.109 INFO:teuthology.orchestra.run.smithi146.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 5.3 MB/s | 548 kB 00:00 2024-09-16T01:46:41.143 INFO:teuthology.orchestra.run.smithi146.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 613 kB/s | 19 kB 00:00 2024-09-16T01:46:41.176 INFO:teuthology.orchestra.run.smithi146.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.8 MB/s | 308 kB 00:00 2024-09-16T01:46:41.209 INFO:teuthology.orchestra.run.smithi146.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 809 kB/s | 26 kB 00:00 2024-09-16T01:46:41.234 INFO:teuthology.orchestra.run.smithi146.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T01:46:41.268 INFO:teuthology.orchestra.run.smithi146.stdout:(110/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.8 MB/s | 863 kB 00:00 2024-09-16T01:46:41.293 INFO:teuthology.orchestra.run.smithi146.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-16T01:46:41.352 INFO:teuthology.orchestra.run.smithi146.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-16T01:46:41.386 INFO:teuthology.orchestra.run.smithi146.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 839 kB 00:00 2024-09-16T01:46:41.411 INFO:teuthology.orchestra.run.smithi146.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-16T01:46:41.436 INFO:teuthology.orchestra.run.smithi146.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-16T01:46:41.462 INFO:teuthology.orchestra.run.smithi146.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 565 kB/s | 14 kB 00:00 2024-09-16T01:46:41.487 INFO:teuthology.orchestra.run.smithi146.stdout:(117/156): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-09-16T01:46:41.512 INFO:teuthology.orchestra.run.smithi146.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-09-16T01:46:41.546 INFO:teuthology.orchestra.run.smithi146.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-16T01:46:41.556 INFO:teuthology.orchestra.run.smithi161.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-16T01:46:41.571 INFO:teuthology.orchestra.run.smithi146.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-16T01:46:41.596 INFO:teuthology.orchestra.run.smithi146.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-16T01:46:41.659 INFO:teuthology.orchestra.run.smithi146.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.2 MB/s | 144 kB 00:00 2024-09-16T01:46:41.681 INFO:teuthology.orchestra.run.smithi146.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 530 kB/s | 11 kB 00:00 2024-09-16T01:46:41.697 INFO:teuthology.orchestra.run.smithi146.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 1.0 MB/s | 18 kB 00:00 2024-09-16T01:46:41.714 INFO:teuthology.orchestra.run.smithi146.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 1.3 MB/s | 23 kB 00:00 2024-09-16T01:46:41.714 INFO:teuthology.orchestra.run.smithi161.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 300 kB/s | 222 kB 00:00 2024-09-16T01:46:41.731 INFO:teuthology.orchestra.run.smithi146.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 1.2 MB/s | 19 kB 00:00 2024-09-16T01:46:41.739 INFO:teuthology.orchestra.run.smithi161.stdout:(85/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 313 kB/s | 250 kB 00:00 2024-09-16T01:46:41.748 INFO:teuthology.orchestra.run.smithi146.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 1.1 MB/s | 20 kB 00:00 2024-09-16T01:46:41.806 INFO:teuthology.orchestra.run.smithi146.stdout:(128/156): libarrow-9.0.0-11.el9.x86_64.rpm 6.7 MB/s | 4.4 MB 00:00 2024-09-16T01:46:41.823 INFO:teuthology.orchestra.run.smithi146.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 2.8 MB/s | 46 kB 00:00 2024-09-16T01:46:41.848 INFO:teuthology.orchestra.run.smithi146.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-16T01:46:41.865 INFO:teuthology.orchestra.run.smithi146.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 3.4 MB/s | 58 kB 00:00 2024-09-16T01:46:41.916 INFO:teuthology.orchestra.run.smithi146.stdout:(132/156): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-09-16T01:46:41.932 INFO:teuthology.orchestra.run.smithi146.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 1.0 MB/s | 16 kB 00:00 2024-09-16T01:46:41.966 INFO:teuthology.orchestra.run.smithi146.stdout:(134/156): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-09-16T01:46:41.983 INFO:teuthology.orchestra.run.smithi146.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-16T01:46:42.008 INFO:teuthology.orchestra.run.smithi146.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-09-16T01:46:42.025 INFO:teuthology.orchestra.run.smithi146.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-09-16T01:46:42.051 INFO:teuthology.orchestra.run.smithi146.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-09-16T01:46:42.140 INFO:teuthology.orchestra.run.smithi146.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 518 kB/s | 59 kB 00:00 2024-09-16T01:46:42.168 INFO:teuthology.orchestra.run.smithi146.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-09-16T01:46:42.185 INFO:teuthology.orchestra.run.smithi146.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.6 MB/s | 74 kB 00:00 2024-09-16T01:46:42.210 INFO:teuthology.orchestra.run.smithi146.stdout:(142/156): python3-routes-2.5.1-5.el9.noarch.rp 936 kB/s | 188 kB 00:00 2024-09-16T01:46:42.235 INFO:teuthology.orchestra.run.smithi146.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T01:46:42.252 INFO:teuthology.orchestra.run.smithi146.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T01:46:42.265 INFO:teuthology.orchestra.run.smithi161.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 109 kB/s | 77 kB 00:00 2024-09-16T01:46:42.303 INFO:teuthology.orchestra.run.smithi146.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-09-16T01:46:42.324 INFO:teuthology.orchestra.run.smithi161.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 69 kB/s | 42 kB 00:00 2024-09-16T01:46:42.328 INFO:teuthology.orchestra.run.smithi146.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-16T01:46:42.353 INFO:teuthology.orchestra.run.smithi146.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-09-16T01:46:42.404 INFO:teuthology.orchestra.run.smithi146.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T01:46:42.565 INFO:teuthology.orchestra.run.smithi161.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 326 kB/s | 269 kB 00:00 2024-09-16T01:46:42.621 INFO:teuthology.orchestra.run.smithi146.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 12 MB/s | 3.2 MB 00:00 2024-09-16T01:46:42.696 INFO:teuthology.orchestra.run.smithi146.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 9.3 MB/s | 3.4 MB 00:00 2024-09-16T01:46:42.991 INFO:teuthology.orchestra.run.smithi161.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 219 kB/s | 159 kB 00:00 2024-09-16T01:46:43.099 INFO:teuthology.orchestra.run.smithi161.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 360 kB/s | 279 kB 00:00 2024-09-16T01:46:43.158 INFO:teuthology.orchestra.run.smithi161.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 86 kB/s | 51 kB 00:00 2024-09-16T01:46:43.263 INFO:teuthology.orchestra.run.smithi146.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 72 kB/s | 41 kB 00:00 2024-09-16T01:46:43.413 INFO:teuthology.orchestra.run.smithi146.stdout:(152/156): NetworkManager-1.49.90-1.el9.x86_64. 2.3 MB/s | 2.3 MB 00:01 2024-09-16T01:46:43.591 INFO:teuthology.orchestra.run.smithi161.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 89 kB/s | 54 kB 00:00 2024-09-16T01:46:43.663 INFO:teuthology.orchestra.run.smithi146.stdout:(153/156): NetworkManager-libnm-1.49.90-1.el9.x 1.7 MB/s | 1.8 MB 00:01 2024-09-16T01:46:43.725 INFO:teuthology.orchestra.run.smithi161.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 73 kB/s | 42 kB 00:00 2024-09-16T01:46:44.038 INFO:teuthology.orchestra.run.smithi146.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 321 kB/s | 249 kB 00:00 2024-09-16T01:46:44.175 INFO:teuthology.orchestra.run.smithi161.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 76 kB/s | 44 kB 00:00 2024-09-16T01:46:44.388 INFO:teuthology.orchestra.run.smithi146.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 1.4 MB/s | 1.4 MB 00:00 2024-09-16T01:46:44.675 INFO:teuthology.orchestra.run.smithi161.stdout:(95/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-16T01:46:44.689 INFO:teuthology.orchestra.run.smithi146.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.4 MB/s | 2.4 MB 00:01 2024-09-16T01:46:44.695 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:46:44.696 INFO:teuthology.orchestra.run.smithi146.stdout:Total 15 MB/s | 226 MB 00:14 2024-09-16T01:46:44.701 INFO:teuthology.orchestra.run.smithi161.stdout:(96/156): qatlib-24.02.0-1.el9.x86_64.rpm 226 kB/s | 221 kB 00:00 2024-09-16T01:46:44.868 INFO:teuthology.orchestra.run.smithi161.stdout:(97/156): qatlib-service-24.02.0-1.el9.x86_64.r 52 kB/s | 36 kB 00:00 2024-09-16T01:46:45.293 INFO:teuthology.orchestra.run.smithi161.stdout:(98/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 77 kB/s | 47 kB 00:00 2024-09-16T01:46:45.485 INFO:teuthology.orchestra.run.smithi161.stdout:(99/156): socat-1.7.4.1-6.el9.x86_64.rpm 388 kB/s | 304 kB 00:00 2024-09-16T01:46:45.568 INFO:teuthology.orchestra.run.smithi161.stdout:(100/156): uuid-1.6.2-55.el9.x86_64.rpm 82 kB/s | 58 kB 00:00 2024-09-16T01:46:46.010 INFO:teuthology.orchestra.run.smithi161.stdout:(101/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 134 kB/s | 96 kB 00:00 2024-09-16T01:46:46.119 INFO:teuthology.orchestra.run.smithi161.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 207 kB/s | 22 kB 00:00 2024-09-16T01:46:46.136 INFO:teuthology.orchestra.run.smithi161.stdout:(103/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 98 kB/s | 64 kB 00:00 2024-09-16T01:46:46.152 INFO:teuthology.orchestra.run.smithi161.stdout:(104/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 78 kB/s | 45 kB 00:00 2024-09-16T01:46:46.220 INFO:teuthology.orchestra.run.smithi161.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 6.5 MB/s | 548 kB 00:00 2024-09-16T01:46:46.245 INFO:teuthology.orchestra.run.smithi161.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-16T01:46:46.270 INFO:teuthology.orchestra.run.smithi161.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.6 MB/s | 308 kB 00:00 2024-09-16T01:46:46.295 INFO:teuthology.orchestra.run.smithi161.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T01:46:46.321 INFO:teuthology.orchestra.run.smithi161.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T01:46:46.346 INFO:teuthology.orchestra.run.smithi161.stdout:(110/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-09-16T01:46:46.371 INFO:teuthology.orchestra.run.smithi161.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T01:46:46.430 INFO:teuthology.orchestra.run.smithi161.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-16T01:46:46.472 INFO:teuthology.orchestra.run.smithi161.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 839 kB 00:00 2024-09-16T01:46:46.497 INFO:teuthology.orchestra.run.smithi161.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-16T01:46:46.531 INFO:teuthology.orchestra.run.smithi161.stdout:(115/156): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-09-16T01:46:46.556 INFO:teuthology.orchestra.run.smithi161.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 549 kB/s | 32 kB 00:00 2024-09-16T01:46:46.581 INFO:teuthology.orchestra.run.smithi161.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 282 kB/s | 14 kB 00:00 2024-09-16T01:46:46.648 INFO:teuthology.orchestra.run.smithi161.stdout:(118/156): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T01:46:46.673 INFO:teuthology.orchestra.run.smithi161.stdout:(119/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-09-16T01:46:46.698 INFO:teuthology.orchestra.run.smithi161.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-09-16T01:46:46.724 INFO:teuthology.orchestra.run.smithi161.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-16T01:46:46.728 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-09-16T01:46:46.749 INFO:teuthology.orchestra.run.smithi161.stdout:(122/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-09-16T01:46:46.774 INFO:teuthology.orchestra.run.smithi161.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-16T01:46:46.801 INFO:teuthology.orchestra.run.smithi161.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-16T01:46:46.825 INFO:teuthology.orchestra.run.smithi161.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-09-16T01:46:46.850 INFO:teuthology.orchestra.run.smithi161.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-09-16T01:46:46.876 INFO:teuthology.orchestra.run.smithi161.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-09-16T01:46:46.901 INFO:teuthology.orchestra.run.smithi161.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-09-16T01:46:46.909 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-09-16T01:46:46.909 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-09-16T01:46:46.934 INFO:teuthology.orchestra.run.smithi161.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-09-16T01:46:46.968 INFO:teuthology.orchestra.run.smithi161.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-09-16T01:46:46.994 INFO:teuthology.orchestra.run.smithi161.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-09-16T01:46:47.019 INFO:teuthology.orchestra.run.smithi161.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-09-16T01:46:47.220 INFO:teuthology.orchestra.run.smithi161.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 82 kB/s | 16 kB 00:00 2024-09-16T01:46:47.245 INFO:teuthology.orchestra.run.smithi161.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-09-16T01:46:47.270 INFO:teuthology.orchestra.run.smithi161.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-09-16T01:46:47.353 INFO:teuthology.orchestra.run.smithi161.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 479 kB/s | 28 kB 00:00 2024-09-16T01:46:47.353 INFO:teuthology.orchestra.run.smithi161.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-09-16T01:46:47.354 INFO:teuthology.orchestra.run.smithi161.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-09-16T01:46:47.397 INFO:teuthology.orchestra.run.smithi161.stdout:(139/156): python3-grpcio-1.46.7-10.el9.x86_64. 2.9 MB/s | 2.0 MB 00:00 2024-09-16T01:46:47.422 INFO:teuthology.orchestra.run.smithi161.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-09-16T01:46:47.447 INFO:teuthology.orchestra.run.smithi161.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-09-16T01:46:47.472 INFO:teuthology.orchestra.run.smithi161.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 987 kB/s | 74 kB 00:00 2024-09-16T01:46:47.497 INFO:teuthology.orchestra.run.smithi161.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T01:46:47.522 INFO:teuthology.orchestra.run.smithi161.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T01:46:47.548 INFO:teuthology.orchestra.run.smithi161.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-09-16T01:46:47.573 INFO:teuthology.orchestra.run.smithi161.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-16T01:46:47.606 INFO:teuthology.orchestra.run.smithi161.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-16T01:46:47.707 INFO:teuthology.orchestra.run.smithi161.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-16T01:46:47.866 INFO:teuthology.orchestra.run.smithi161.stdout:(149/156): librados2-19.1.1-326.g8921c8ef.el9.x 12 MB/s | 3.4 MB 00:00 2024-09-16T01:46:47.924 INFO:teuthology.orchestra.run.smithi161.stdout:(150/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 10 MB/s | 3.2 MB 00:00 2024-09-16T01:46:47.974 INFO:teuthology.orchestra.run.smithi161.stdout:(151/156): NetworkManager-1.49.90-1.el9.x86_64. 8.7 MB/s | 2.3 MB 00:00 2024-09-16T01:46:47.991 INFO:teuthology.orchestra.run.smithi161.stdout:(152/156): NetworkManager-team-1.49.90-1.el9.x8 609 kB/s | 41 kB 00:00 2024-09-16T01:46:48.017 INFO:teuthology.orchestra.run.smithi161.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 5.8 MB/s | 249 kB 00:00 2024-09-16T01:46:48.059 INFO:teuthology.orchestra.run.smithi161.stdout:(154/156): NetworkManager-libnm-1.49.90-1.el9.x 9.4 MB/s | 1.8 MB 00:00 2024-09-16T01:46:48.126 INFO:teuthology.orchestra.run.smithi161.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 22 MB/s | 2.4 MB 00:00 2024-09-16T01:46:48.167 INFO:teuthology.orchestra.run.smithi161.stdout:(156/156): openssl-3.2.2-6.el9.x86_64.rpm 7.7 MB/s | 1.4 MB 00:00 2024-09-16T01:46:48.173 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:46:48.173 INFO:teuthology.orchestra.run.smithi161.stdout:Total 11 MB/s | 226 MB 00:20 2024-09-16T01:46:49.241 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-09-16T01:46:49.242 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-09-16T01:46:49.392 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-09-16T01:46:49.544 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-09-16T01:46:49.544 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-09-16T01:46:51.398 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-09-16T01:46:51.399 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-09-16T01:46:55.893 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-09-16T01:46:56.018 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T01:46:56.068 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T01:46:56.134 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T01:46:56.178 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T01:46:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T01:46:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T01:46:56.636 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T01:46:56.736 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:46:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:46:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:46:57.042 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:46:57.075 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:46:57.293 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T01:46:57.366 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T01:46:57.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:46:57.515 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:46:57.541 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:46:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:46:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:46:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:46:57.851 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-09-16T01:46:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:46:57.979 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T01:46:58.026 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T01:46:58.087 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T01:46:58.122 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T01:46:58.140 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:46:58.147 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T01:46:58.190 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T01:46:58.311 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T01:46:58.448 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:46:58.536 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T01:46:58.542 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:46:58.631 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:46:58.753 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:46:58.782 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:46:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:46:58.873 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:46:58.911 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:46:58.975 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T01:46:59.060 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T01:46:59.092 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T01:46:59.117 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T01:46:59.132 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T01:46:59.153 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:46:59.176 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T01:46:59.256 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:46:59.257 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T01:46:59.287 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:46:59.312 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T01:46:59.350 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:46:59.350 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T01:46:59.380 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:46:59.396 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T01:46:59.402 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:46:59.435 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T01:46:59.469 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T01:46:59.557 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T01:46:59.598 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:46:59.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T01:46:59.656 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T01:46:59.702 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T01:46:59.726 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:46:59.783 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T01:46:59.818 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T01:46:59.847 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T01:46:59.862 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T01:46:59.882 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T01:46:59.939 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:46:59.972 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T01:47:00.016 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:47:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T01:47:00.117 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T01:47:00.246 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:47:00.276 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T01:47:00.337 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T01:47:00.362 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T01:47:00.453 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T01:47:00.496 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T01:47:00.532 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T01:47:00.605 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T01:47:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T01:47:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T01:47:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T01:47:00.765 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T01:47:00.793 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T01:47:00.863 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T01:47:00.924 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T01:47:00.954 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T01:47:00.969 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T01:47:00.990 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T01:47:01.067 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T01:47:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T01:47:01.100 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T01:47:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T01:47:01.143 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T01:47:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T01:47:01.172 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T01:47:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T01:47:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T01:47:01.258 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T01:47:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T01:47:01.369 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T01:47:01.436 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T01:47:01.520 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T01:47:01.572 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T01:47:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T01:47:01.859 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T01:47:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T01:47:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T01:47:02.015 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T01:47:02.073 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T01:47:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T01:47:02.131 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T01:47:02.184 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T01:47:02.193 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T01:47:02.222 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T01:47:02.234 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T01:47:02.264 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T01:47:02.302 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T01:47:02.345 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T01:47:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T01:47:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T01:47:02.494 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T01:47:02.496 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T01:47:02.547 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T01:47:02.602 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T01:47:02.613 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T01:47:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T01:47:02.683 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T01:47:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T01:47:02.723 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T01:47:02.732 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T01:47:02.759 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T01:47:02.789 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T01:47:02.806 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T01:47:02.842 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T01:47:02.885 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T01:47:02.913 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T01:47:02.927 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T01:47:03.063 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T01:47:03.065 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T01:47:03.112 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T01:47:03.154 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T01:47:03.264 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T01:47:03.303 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T01:47:03.343 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T01:47:03.369 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T01:47:03.394 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T01:47:03.555 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T01:47:03.647 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T01:47:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T01:47:03.684 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T01:47:03.726 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T01:47:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T01:47:03.756 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T01:47:03.767 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T01:47:03.790 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T01:47:03.806 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T01:47:03.829 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T01:47:03.844 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T01:47:03.861 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T01:47:03.879 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T01:47:03.890 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T01:47:03.903 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T01:47:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T01:47:03.954 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T01:47:03.966 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T01:47:03.987 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T01:47:04.008 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T01:47:04.043 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T01:47:04.085 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T01:47:04.130 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T01:47:04.145 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T01:47:04.173 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T01:47:04.181 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T01:47:04.204 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T01:47:04.307 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T01:47:04.334 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T01:47:04.364 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T01:47:04.402 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T01:47:05.110 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T01:47:05.152 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T01:47:05.194 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T01:47:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T01:47:05.293 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T01:47:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T01:47:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T01:47:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T01:47:05.394 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T01:47:05.482 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T01:47:05.504 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T01:47:05.513 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T01:47:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T01:47:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T01:47:05.699 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T01:47:05.715 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T01:47:05.728 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T01:47:05.758 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T01:47:05.806 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T01:47:06.006 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T01:47:06.038 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T01:47:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T01:47:06.455 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T01:47:06.560 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T01:47:06.976 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T01:47:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T01:47:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T01:47:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T01:47:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T01:47:07.950 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T01:47:08.260 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T01:47:08.305 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:47:08.429 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:47:08.460 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T01:47:08.496 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T01:47:08.680 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T01:47:08.775 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T01:47:08.812 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T01:47:08.845 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T01:47:08.977 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T01:47:09.008 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:47:09.080 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:47:09.110 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T01:47:09.177 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T01:47:09.214 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:47:09.327 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:47:09.387 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T01:47:09.442 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T01:47:09.966 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T01:47:10.025 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:47:10.116 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:47:10.144 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T01:47:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:47:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:47:12.031 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:47:12.068 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:47:12.159 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:47:12.160 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T01:47:12.160 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T01:47:12.160 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:12.194 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:47:12.854 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:47:12.899 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:47:13.529 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:47:13.569 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:47:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:47:13.706 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T01:47:13.706 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T01:47:13.706 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:13.745 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:47:23.859 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:47:23.859 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-09-16T01:47:23.859 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-09-16T01:47:23.860 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-09-16T01:47:23.860 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-09-16T01:47:23.860 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-09-16T01:47:23.860 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-09-16T01:47:23.860 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-09-16T01:47:23.860 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:24.028 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:47:24.044 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:47:24.044 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:24.044 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T01:47:24.044 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:25.522 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:47:25.554 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:47:25.554 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:25.555 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T01:47:25.555 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:47:25.555 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:47:25.555 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:25.799 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:47:25.828 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:47:25.829 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:25.829 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T01:47:25.829 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:47:25.829 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:47:25.829 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:26.276 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:47:26.299 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:47:26.299 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:26.299 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T01:47:26.299 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:47:26.299 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:47:26.299 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:26.414 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T01:47:26.470 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T01:47:26.490 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:47:26.490 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-09-16T01:47:26.490 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-09-16T01:47:26.490 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-09-16T01:47:26.491 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-09-16T01:47:26.491 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-09-16T01:47:26.491 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-09-16T01:47:26.491 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-09-16T01:47:26.491 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:26.525 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T01:47:26.698 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:47:26.710 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T01:47:26.735 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:47:26.735 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:26.735 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T01:47:26.735 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:26.759 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T01:47:26.845 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T01:47:26.933 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T01:47:27.073 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T01:47:27.118 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T01:47:27.149 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T01:47:27.177 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T01:47:27.631 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:47:27.676 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:47:27.707 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T01:47:27.769 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T01:47:27.799 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:47:28.326 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:47:28.383 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:47:28.383 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:28.383 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T01:47:28.383 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:47:28.384 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:47:28.384 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:28.626 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:47:28.677 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:47:28.677 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:28.677 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T01:47:28.677 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:47:28.677 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:47:28.677 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:28.846 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:47:28.884 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:47:29.152 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:47:29.202 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:47:29.202 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:29.202 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T01:47:29.202 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:47:29.202 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:47:29.203 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:29.335 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T01:47:29.395 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T01:47:29.460 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T01:47:29.687 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T01:47:29.739 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T01:47:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T01:47:29.846 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:47:29.876 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:47:29.925 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T01:47:29.978 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:47:30.098 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T01:47:30.146 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T01:47:30.171 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T01:47:30.178 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:47:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:47:30.202 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:30.202 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T01:47:30.202 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:47:30.202 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:47:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:30.211 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T01:47:30.236 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T01:47:30.254 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:47:30.265 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:47:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T01:47:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T01:47:30.777 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:47:30.828 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:47:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T01:47:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T01:47:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:47:31.352 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T01:47:31.381 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:47:31.405 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:47:31.405 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:31.405 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T01:47:31.405 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:47:31.406 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:47:31.406 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:31.449 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:47:31.471 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:47:31.472 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:31.472 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T01:47:31.472 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:31.524 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:47:31.547 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:47:31.548 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:31.548 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T01:47:31.548 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:47:31.548 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:47:31.548 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:31.850 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:47:31.873 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:47:31.873 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:31.873 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T01:47:31.873 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:47:31.873 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:47:31.873 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:32.188 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:47:32.227 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:47:33.314 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:47:33.380 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:47:33.519 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:47:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T01:47:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:47:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:47:33.798 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:33.798 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T01:47:33.798 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:47:33.798 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:47:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:33.855 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:47:33.882 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:47:33.989 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T01:47:34.055 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T01:47:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T01:47:35.082 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:47:35.134 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:47:35.135 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:35.135 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T01:47:35.135 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:47:35.135 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:47:35.135 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:35.191 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:47:35.241 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:47:35.241 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:35.241 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T01:47:35.242 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:35.300 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:47:35.384 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:47:35.384 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:35.384 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T01:47:35.384 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:47:35.385 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:47:35.385 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:35.725 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:47:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:47:35.777 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:47:35.777 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T01:47:35.777 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:47:35.777 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:47:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:36.411 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T01:47:36.452 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T01:47:36.501 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T01:47:36.542 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T01:47:36.569 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T01:47:36.750 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T01:47:36.791 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T01:47:36.821 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T01:47:36.851 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T01:47:36.897 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T01:47:37.531 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:47:37.570 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:47:37.590 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:47:38.435 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T01:47:38.524 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T01:47:38.565 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T01:47:38.647 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T01:47:38.677 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T01:47:38.703 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T01:47:38.737 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T01:47:38.737 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:47:38.829 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:47:38.829 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:47:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:47:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T01:47:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:47:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:47:39.132 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:47:39.132 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:47:39.225 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:47:39.287 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:47:40.705 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T01:47:40.746 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T01:47:40.790 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T01:47:40.819 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T01:47:40.855 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T01:47:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T01:47:41.134 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T01:47:41.164 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T01:47:41.204 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T01:47:41.248 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T01:47:42.000 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:47:42.046 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:47:42.087 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:47:43.368 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T01:47:43.475 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T01:47:43.530 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T01:47:43.640 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T01:47:43.678 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T01:47:43.702 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T01:47:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T01:47:43.748 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:47:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:47:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:47:43.964 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:47:43.988 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T01:47:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:47:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:47:44.250 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:47:44.250 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:47:44.360 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:47:44.433 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:47:44.645 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:47:44.645 INFO:teuthology.orchestra.run.smithi146.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-16T01:47:44.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:47:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T01:47:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:47:44.767 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T01:47:44.768 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T01:47:44.769 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T01:47:44.770 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T01:47:44.771 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T01:47:44.772 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T01:47:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T01:47:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T01:47:44.775 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T01:47:44.776 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T01:47:44.777 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T01:47:44.778 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T01:47:44.779 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T01:47:44.779 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T01:47:44.779 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T01:47:44.779 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T01:47:44.779 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T01:47:44.779 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.377 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.378 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T01:47:47.379 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.380 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T01:47:47.381 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T01:47:47.382 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.383 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T01:47:47.384 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T01:47:47.385 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T01:47:47.386 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T01:47:47.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.388 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:47.389 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-09-16T01:47:47.804 DEBUG:teuthology.parallel:result is None 2024-09-16T01:47:50.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T01:47:50.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T01:47:50.335 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T01:47:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T01:47:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T01:47:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T01:47:50.336 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T01:47:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T01:47:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T01:47:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T01:47:50.349 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T01:47:50.350 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T01:47:50.351 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T01:47:50.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T01:47:50.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T01:47:50.354 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T01:47:50.355 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T01:47:50.356 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T01:47:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T01:47:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T01:47:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T01:47:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T01:47:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T01:47:50.357 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T01:47:50.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T01:47:50.359 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T01:47:50.360 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T01:47:50.361 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T01:47:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:47:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T01:47:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T01:47:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T01:47:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T01:47:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T01:47:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T01:47:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T01:47:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T01:47:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T01:47:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T01:47:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T01:47:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:52.556 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-09-16T01:47:53.179 DEBUG:teuthology.parallel:result is None 2024-09-16T01:47:53.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:47:53.324 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T01:47:53.368 INFO:teuthology.orchestra.run.smithi146.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T01:47:53.369 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T01:47:53.369 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T01:47:53.371 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:47:53.506 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T01:47:53.533 INFO:teuthology.orchestra.run.smithi161.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T01:47:53.533 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T01:47:53.533 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T01:47:53.535 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T01:47:53.535 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:53.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T01:47:53.594 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:53.594 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T01:47:53.626 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T01:47:53.626 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:53.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T01:47:53.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T01:47:53.795 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:53.795 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T01:47:53.824 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T01:47:53.893 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T01:47:53.894 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:53.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T01:47:53.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T01:47:54.056 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:54.056 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T01:47:54.085 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T01:47:54.154 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T01:47:54.154 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:54.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T01:47:54.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T01:47:54.311 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:54.357 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T01:47:54.387 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T01:47:54.455 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T01:47:54.542 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T01:47:54.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-09-16T01:47:54.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2024-09-16T01:47:54.601 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T01:47:54.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T01:47:54.606 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T01:47:54.657 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T01:47:54.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T01:47:54.662 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T01:47:54.729 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-16T01:47:54.729 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-16T01:47:54.729 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-16T01:47:54.730 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:54.730 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-16T01:47:54.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-16T01:47:54.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-16T01:47:54.991 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:54.991 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-16T01:47:55.008 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-16T01:47:55.124 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-16T01:47:55.156 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T01:47:55.156 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T01:47:55.156 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490', 'branch': 'squid', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906490', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3393, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '294/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDkw4PX9rR7/EB1JahGQ2oaSXLILjhfMwVUH/+K8MD0CwCwrscYyqeMbC2nQvi7e4V9ylyHnbSpGle900WpmsRo=', 'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIX1L+j0gd1wROxVfqPy7JRwhfMkjIyjTt06IByBlwoC9Rixjqd6FOHk+LHzACj/ByIvt2asPf6M2CvoQxkB/Pk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'thrashosds': None}, {'workunit': {'clients': {'all': ['suites/iozone.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:24:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T01:47:55.157 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T01:47:55.161 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T01:47:55.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T01:47:55.236 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T01:47:55.288 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:55.288 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:47:55.323 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:47:55.323 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-09-16T01:47:55.398 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:47:55.398 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:55.398 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-09-16T01:47:55.398 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:55.399 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:55.399 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-16 01:47:45.013403926 +0000 2024-09-16T01:47:55.399 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-16 01:45:41.188643635 +0000 2024-09-16T01:47:55.399 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-16 01:45:41.189643620 +0000 2024-09-16T01:47:55.399 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-16 01:45:41.188643635 +0000 2024-09-16T01:47:55.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:47:55.491 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-16T01:47:55.492 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-16T01:47:55.492 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00033912 s, 1.5 MB/s 2024-09-16T01:47:55.494 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:47:55.530 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-09-16T01:47:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-16 01:47:45.014403911 +0000 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-16 01:45:41.644636790 +0000 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-16 01:45:41.644636790 +0000 2024-09-16T01:47:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-16 01:45:41.644636790 +0000 2024-09-16T01:47:55.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:47:55.699 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-16T01:47:55.699 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-16T01:47:55.699 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00024856 s, 2.1 MB/s 2024-09-16T01:47:55.702 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:47:55.738 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-09-16T01:47:55.814 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:47:55.814 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:55.814 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-09-16T01:47:55.814 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:55.815 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:55.815 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-16 01:47:45.015403896 +0000 2024-09-16T01:47:55.815 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-16 01:45:42.128629525 +0000 2024-09-16T01:47:55.815 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-16 01:45:42.128629525 +0000 2024-09-16T01:47:55.815 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-16 01:45:42.128629525 +0000 2024-09-16T01:47:55.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:47:55.907 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-16T01:47:55.907 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-16T01:47:55.907 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00026726 s, 1.9 MB/s 2024-09-16T01:47:55.909 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:47:55.946 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-16 01:47:45.015403896 +0000 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-16 01:45:42.590622590 +0000 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-16 01:45:42.590622590 +0000 2024-09-16T01:47:56.021 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-16 01:45:42.590622590 +0000 2024-09-16T01:47:56.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:47:56.113 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-16T01:47:56.113 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-16T01:47:56.113 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000305636 s, 1.7 MB/s 2024-09-16T01:47:56.116 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:47:56.152 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:47:56.152 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:56.152 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:47:56.171 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:47:56.171 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-09-16T01:47:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:47:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:56.229 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T01:47:56.229 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:56.230 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:56.230 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-16 01:47:39.622001181 +0000 2024-09-16T01:47:56.230 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-16 01:45:06.339772847 +0000 2024-09-16T01:47:56.230 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-16 01:45:06.339772847 +0000 2024-09-16T01:47:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-16 01:45:06.339772847 +0000 2024-09-16T01:47:56.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:47:56.295 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-16T01:47:56.296 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-16T01:47:56.296 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000216942 s, 2.4 MB/s 2024-09-16T01:47:56.297 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:47:56.355 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-16 01:47:39.622001181 +0000 2024-09-16T01:47:56.413 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-16 01:45:06.663766002 +0000 2024-09-16T01:47:56.414 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-16 01:45:06.663766002 +0000 2024-09-16T01:47:56.414 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-16 01:45:06.663766002 +0000 2024-09-16T01:47:56.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:47:56.478 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-16T01:47:56.478 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-16T01:47:56.478 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000243474 s, 2.1 MB/s 2024-09-16T01:47:56.479 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:47:56.537 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-09-16T01:47:56.594 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:47:56.594 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-16 01:47:39.623001160 +0000 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-16 01:45:06.980759305 +0000 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-16 01:45:06.980759305 +0000 2024-09-16T01:47:56.595 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-16 01:45:06.980759305 +0000 2024-09-16T01:47:56.595 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:47:56.659 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-16T01:47:56.660 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-16T01:47:56.660 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00021457 s, 2.4 MB/s 2024-09-16T01:47:56.661 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:47:56.718 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-09-16T01:47:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:47:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:47:56.774 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-16T01:47:56.774 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:47:56.775 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:47:56.775 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-09-16 01:47:39.623001160 +0000 2024-09-16T01:47:56.775 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-09-16 01:45:07.297752607 +0000 2024-09-16T01:47:56.775 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-09-16 01:45:07.297752607 +0000 2024-09-16T01:47:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-09-16 01:45:07.297752607 +0000 2024-09-16T01:47:56.775 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:47:56.838 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-09-16T01:47:56.838 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-09-16T01:47:56.838 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000208874 s, 2.5 MB/s 2024-09-16T01:47:56.839 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:47:56.896 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:47:56.897 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T01:47:56.897 INFO:tasks.ceph:Generating config... 2024-09-16T01:47:56.898 INFO:tasks.ceph:[client] client mount timeout = 600 2024-09-16T01:47:56.898 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-09-16T01:47:56.898 INFO:tasks.ceph:[client] debug client = 20 2024-09-16T01:47:56.898 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T01:47:56.898 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-09-16T01:47:56.898 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-09-16T01:47:56.899 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] debug mds = 20 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] debug ms = 1 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds debug frag = True 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-09-16T01:47:56.899 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-09-16T01:47:56.900 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T01:47:56.901 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T01:47:56.902 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T01:47:56.902 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-09-16T01:47:56.902 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T01:47:56.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T01:47:56.995 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T01:47:57.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T01:47:57.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T01:47:57.143 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.146'), ('mon.b', '172.21.15.161'), ('mon.c', '[v2:172.21.15.161:3301,v1:172.21.15.161:6790]')] 2024-09-16T01:47:57.143 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.146,172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-09-16T01:47:57.144 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:57.144 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T01:47:57.179 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.146 --add b 172.21.15.161 --addv c '[v2:172.21.15.161:3301,v1:172.21.15.161:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:47:57.296 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:47:57.296 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: generated fsid 4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d 2024-09-16T01:47:57.296 INFO:teuthology.orchestra.run.smithi146.stdout:setting min_mon_release = quincy 2024-09-16T01:47:57.296 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 0 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:fsid 4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:last_changed 2024-09-16T01:47:57.296028+0000 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:created 2024-09-16T01:47:57.296028+0000 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:min_mon_release 17 (quincy) 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:election_strategy: 1 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.b 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:2: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.c 2024-09-16T01:47:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T01:47:57.299 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T01:47:57.334 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d... 2024-09-16T01:47:57.335 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T01:47:57.378 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T01:47:57.428 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2024-09-16T01:47:57.429 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2024-09-16T01:47:57.430 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2024-09-16T01:47:57.431 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = "172.21.15.146,172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790]" 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on skipped message = False 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = 4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2024-09-16T01:47:57.432 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug trim objects = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.433 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore block size = 96636764160 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore compression mode = aggressive 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore fsck on mount = True 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 20 2024-09-16T01:47:57.434 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 20 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 10 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd full ratio = 0.9 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd nearfull ratio = 0.8 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: osd blocked scrub grace period = 3600 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: osd failsafe full ratio = 0.95 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: osd max markdown count = 1000 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: osd objectstore = bluestore 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: osd op complaint time = 180 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2024-09-16T01:47:57.435 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-09-16T01:47:57.436 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T01:47:57.437 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon down mkfs grace = 300 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon op complaint time = 120 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on pool no app = False 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-09-16T01:47:57.438 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: client mount timeout = 600 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: client_shutdown_timeout = 120 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 20 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: rados mon op timeout = 900 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout: rados osd op timeout = 900 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.b] 2024-09-16T01:47:57.439 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.c] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.a] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.c] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.e] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.b] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.d] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.f] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout:[mds] 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds = 20 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds balancer = 20 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout: mds bal fragment size max = 10000 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout: mds bal merge size = 5 2024-09-16T01:47:57.440 INFO:teuthology.orchestra.run.smithi161.stdout: mds bal split bits = 3 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds bal split size = 100 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds debug frag = True 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds debug scatterstat = True 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds oft prefetch dirfrags = True 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds op complaint time = 180 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds verify scatter = True 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds_dir_prefetch = True 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: mds_sleep_rank_change = 5000000.0 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: osd op complaint time = 180 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: rados mon op timeout = 900 2024-09-16T01:47:57.441 INFO:teuthology.orchestra.run.smithi161.stdout: rados osd op timeout = 900 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-09-16T01:47:57.471 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.472 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T01:47:57.473 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.146,172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790]" 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on skipped message = False 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d 2024-09-16T01:47:57.474 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-09-16T01:47:57.475 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-09-16T01:47:57.476 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression mode = aggressive 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 20 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 20 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-09-16T01:47:57.477 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 10 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: osd blocked scrub grace period = 3600 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: osd max markdown count = 1000 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-09-16T01:47:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-09-16T01:47:57.479 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T01:47:57.480 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: mon down mkfs grace = 300 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2024-09-16T01:47:57.481 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: client_shutdown_timeout = 120 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-09-16T01:47:57.482 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.e] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.f] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout: mds bal fragment size max = 10000 2024-09-16T01:47:57.483 INFO:teuthology.orchestra.run.smithi146.stdout: mds bal merge size = 5 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds bal split bits = 3 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds bal split size = 100 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds oft prefetch dirfrags = True 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds_dir_prefetch = True 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: mds_sleep_rank_change = 5000000.0 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-09-16T01:47:57.484 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-09-16T01:47:57.496 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T01:47:57.496 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T01:47:57.593 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T01:47:57.593 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:57.593 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T01:47:57.628 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:57.628 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T01:47:57.703 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:47:57.703 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:57.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T01:47:57.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T01:47:57.820 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:47:57.820 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:47:57.895 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:47:57.896 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:57.896 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T01:47:57.896 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T01:47:57.937 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:47:57.937 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:47:57.995 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T01:47:57.995 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T01:47:57.995 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T01:47:58.100 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T01:47:58.106 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T01:47:58.170 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T01:47:58.173 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T01:47:58.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-16T01:47:58.277 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-16T01:47:58.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-16T01:47:58.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-16T01:47:58.442 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-16T01:47:58.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-16T01:47:58.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-09-16T01:47:58.606 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-09-16T01:47:58.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-09-16T01:47:58.667 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-16T01:47:58.727 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-16T01:47:58.729 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-16T01:47:58.756 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-16T01:47:58.850 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-16T01:47:58.853 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-16T01:47:58.878 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-09-16T01:47:58.969 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-09-16T01:47:58.972 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-09-16T01:47:58.996 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T01:47:58.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T01:47:59.079 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T01:47:59.106 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T01:47:59.106 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T01:47:59.106 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T01:47:59.161 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:47:59.161 INFO:tasks.ceph:role: osd.0 2024-09-16T01:47:59.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:47:59.161 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T01:47:59.223 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:47:59.224 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:47:59.626 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-16T01:47:59.630 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-16T01:47:59.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T01:47:59.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T01:47:59.836 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T01:47:59.895 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:47:59.895 INFO:tasks.ceph:role: osd.1 2024-09-16T01:47:59.895 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:47:59.895 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T01:47:59.960 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:47:59.960 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:47:59.960 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:47:59.961 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:00.314 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-16T01:48:00.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-16T01:48:00.319 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T01:48:00.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T01:48:00.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T01:48:00.529 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:48:00.529 INFO:tasks.ceph:role: osd.2 2024-09-16T01:48:00.529 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:48:00.529 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:48:00.592 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:00.974 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-16T01:48:00.979 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-16T01:48:00.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T01:48:01.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T01:48:01.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T01:48:01.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:48:01.196 INFO:tasks.ceph:role: osd.3 2024-09-16T01:48:01.196 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:48:01.196 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:48:01.260 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:48:01.261 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:48:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:48:01.261 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:01.677 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-16T01:48:01.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-16T01:48:01.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T01:48:01.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T01:48:01.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:01.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:01.980+0000 7fcec09b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T01:48:01.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:01.981+0000 7fcec09b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T01:48:01.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:01.981+0000 7fcec09b2640 -1 bdev(0x562da4985800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:01.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:01.982+0000 7fcec09b2640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T01:48:05.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T01:48:05.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:05.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:05.169+0000 7f3f5ae0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T01:48:05.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:05.169+0000 7f3f5ae0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T01:48:05.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:05.169+0000 7f3f5ae0f640 -1 bdev(0x564b60227800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:05.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:05.170+0000 7f3f5ae0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T01:48:08.173 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T01:48:08.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:08.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:08.320+0000 7ff14aa0b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T01:48:08.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:08.320+0000 7ff14aa0b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T01:48:08.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:08.320+0000 7ff14aa0b640 -1 bdev(0x5646f00ff800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:08.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:08.320+0000 7ff14aa0b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T01:48:11.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T01:48:11.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:11.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:11.468+0000 7f1dd2134640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T01:48:11.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:11.468+0000 7f1dd2134640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T01:48:11.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:11.468+0000 7f1dd2134640 -1 bdev(0x55dfd8669800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:11.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:11.469+0000 7f1dd2134640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T01:48:14.497 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T01:48:14.558 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T01:48:14.589 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:48:14.590 INFO:tasks.ceph:role: osd.4 2024-09-16T01:48:14.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:48:14.590 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T01:48:14.666 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:48:14.667 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:15.172 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-09-16T01:48:15.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-09-16T01:48:15.175 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T01:48:15.295 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T01:48:15.335 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T01:48:15.411 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:48:15.412 INFO:tasks.ceph:role: osd.5 2024-09-16T01:48:15.412 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:48:15.412 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T01:48:15.493 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:48:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:48:15.493 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:48:15.494 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:15.950 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-09-16T01:48:15.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-09-16T01:48:15.953 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T01:48:16.038 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T01:48:16.069 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T01:48:16.134 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:48:16.134 INFO:tasks.ceph:role: osd.6 2024-09-16T01:48:16.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:48:16.134 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T01:48:16.206 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:48:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:48:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:48:16.207 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:16.693 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-09-16T01:48:16.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-09-16T01:48:16.695 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T01:48:16.837 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T01:48:16.869 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T01:48:16.934 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:48:16.935 INFO:tasks.ceph:role: osd.7 2024-09-16T01:48:16.935 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:48:16.935 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T01:48:17.007 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:48:17.007 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:48:17.007 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:48:17.007 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:48:17.007 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:48:17.008 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:48:17.008 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:48:17.008 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:48:17.008 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:48:17.008 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:48:17.514 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-09-16T01:48:17.516 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-09-16T01:48:17.516 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T01:48:17.640 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T01:48:17.674 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:17.774+0000 7f0b2820d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T01:48:17.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:17.774+0000 7f0b2820d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T01:48:17.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:17.774+0000 7f0b2820d640 -1 bdev(0x560162d61800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:17.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:17.774+0000 7f0b2820d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T01:48:21.179 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T01:48:21.209 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:21.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:21.298+0000 7fb6daab0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T01:48:21.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:21.298+0000 7fb6daab0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T01:48:21.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:21.298+0000 7fb6daab0640 -1 bdev(0x558921a3f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:21.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:21.298+0000 7fb6daab0640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T01:48:24.457 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T01:48:24.487 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:24.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:24.577+0000 7fdad4cb4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T01:48:24.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:24.577+0000 7fdad4cb4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T01:48:24.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:24.577+0000 7fdad4cb4640 -1 bdev(0x55fe21beb800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:24.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:24.577+0000 7fdad4cb4640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T01:48:25.157 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:48:25.162 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:48:27.495 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T01:48:27.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:27.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:27.610+0000 7f77c9902640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T01:48:27.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:27.611+0000 7f77c9902640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T01:48:27.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:27.611+0000 7f77c9902640 -1 bdev(0x55d8159f7800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T01:48:27.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:27.611+0000 7f77c9902640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T01:48:30.533 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T01:48:30.563 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T01:48:30.564 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:30.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T01:48:30.627 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:30.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-16T01:48:30.726 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:30.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-16T01:48:30.824 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:30.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-09-16T01:48:30.919 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:30.919 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T01:48:31.015 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:31.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T01:48:31.111 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:31.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T01:48:31.208 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:31.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T01:48:31.303 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.304 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T01:48:31.334 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.335 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-16T01:48:31.403 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-16T01:48:31.473 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.473 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-09-16T01:48:31.541 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.541 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T01:48:31.608 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.608 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T01:48:31.674 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T01:48:31.741 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:31.741 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T01:48:31.806 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:31.807 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T01:48:31.842 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T01:48:31.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T01:48:31.887 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T01:48:31.910 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.y] 2024-09-16T01:48:31.910 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudm3Fv0BRAAD0bhRH31e92DV9wGZWXKBQ== 2024-09-16T01:48:31.910 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.a] 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudmkiV/EBAATgyOTPLvL97FR6Kp9mgs/A== 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.c] 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudmEF1hGhAArk3S2EEY6vb5iw10D9JsIQ== 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.e] 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudmFn8aJBAArR95xBoAgczD6ofsUcccpQ== 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBRjudmVzt3OhAAVh9FoC/jX0w/cGMFBk3P6g== 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBVjudmfGQjChAAe+Tdh6tYQ0uk+kyBI2D/Tw== 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBYjudm9CodExAAY9yE1QHANq/1dfoaZndV0w== 2024-09-16T01:48:31.911 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.3] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBbjudmM4QDHBAAr5Lxpv+W/+BFepSwTDa+pQ== 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudmqVImChAA+ZD8xudPHYrOcUVXNE+maA== 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.b] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudmutlSKxAA+k6W+BUrWzZvBdu4FFcziw== 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.d] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudmJRivMhAAhUZFwwym3TdKfQHMQfK1Mw== 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.f] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBOjudm4gXGORAAl2s+VZ4xApSfu/Jykk6N/A== 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.4] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBhjudmE38wLhAA69mlZw8Uu0wb0lo4tkrXbQ== 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.5] 2024-09-16T01:48:31.912 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBljudm64bWERAAzFeZS3ixcyHE7QGe+Unz+A== 2024-09-16T01:48:31.913 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.6] 2024-09-16T01:48:31.913 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBojudmpER2IhAAQgeJ5EHy6cIZx7n4E/RN9A== 2024-09-16T01:48:31.913 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.7] 2024-09-16T01:48:31.913 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBrjudmSZiDJBAA97jL+Gjj22mF1hPd0lcHzA== 2024-09-16T01:48:31.913 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-09-16T01:48:31.913 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBPjudm+2uzBBAAXfD6TpQUC5hjIXK95A/Qhw== 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudm3Fv0BRAAD0bhRH31e92DV9wGZWXKBQ== 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudmkiV/EBAATgyOTPLvL97FR6Kp9mgs/A== 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudmEF1hGhAArk3S2EEY6vb5iw10D9JsIQ== 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.e] 2024-09-16T01:48:31.937 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudmFn8aJBAArR95xBoAgczD6ofsUcccpQ== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBRjudmVzt3OhAAVh9FoC/jX0w/cGMFBk3P6g== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBVjudmfGQjChAAe+Tdh6tYQ0uk+kyBI2D/Tw== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBYjudm9CodExAAY9yE1QHANq/1dfoaZndV0w== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBbjudmM4QDHBAAr5Lxpv+W/+BFepSwTDa+pQ== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudmqVImChAA+ZD8xudPHYrOcUVXNE+maA== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudmutlSKxAA+k6W+BUrWzZvBdu4FFcziw== 2024-09-16T01:48:31.938 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudmJRivMhAAhUZFwwym3TdKfQHMQfK1Mw== 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.f] 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBOjudm4gXGORAAl2s+VZ4xApSfu/Jykk6N/A== 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBhjudmE38wLhAA69mlZw8Uu0wb0lo4tkrXbQ== 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBljudm64bWERAAzFeZS3ixcyHE7QGe+Unz+A== 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBojudmpER2IhAAQgeJ5EHy6cIZx7n4E/RN9A== 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-09-16T01:48:31.939 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBrjudmSZiDJBAA97jL+Gjj22mF1hPd0lcHzA== 2024-09-16T01:48:31.940 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-09-16T01:48:31.940 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBPjudm+2uzBBAAXfD6TpQUC5hjIXK95A/Qhw== 2024-09-16T01:48:31.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:48:31.985 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:48:32.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.079 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.171 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.175 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.272 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.369 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.560 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.653 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:32.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:48:32.746 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:48:32.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.841 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:32.937 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:33.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:33.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:48:33.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.131 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.227 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.318 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.408 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.412 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:48:33.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:48:33.507 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:48:33.599 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T01:48:33.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T01:48:33.655 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T01:48:33.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T01:48:33.820 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T01:48:33.846 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T01:48:33.942 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T01:48:33.967 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T01:48:34.033 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T01:48:34.128 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T01:48:34.154 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:34.159 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:48:34.209 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T01:48:34.209 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T01:48:34.210 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T01:48:34.237 INFO:tasks.ceph.mon.a:Started 2024-09-16T01:48:34.237 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T01:48:34.237 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T01:48:34.251 INFO:tasks.ceph.mon.b:Started 2024-09-16T01:48:34.251 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T01:48:34.252 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T01:48:34.254 INFO:tasks.ceph.mon.c:Started 2024-09-16T01:48:34.254 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T01:48:34.254 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T01:48:34.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T01:48:34.258 INFO:tasks.ceph.mgr.y:Started 2024-09-16T01:48:34.258 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T01:48:34.258 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T01:48:34.261 INFO:tasks.ceph.mgr.x:Started 2024-09-16T01:48:34.261 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T01:48:34.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T01:48:34.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.418+0000 7f791e600640 1 Processor -- start 2024-09-16T01:48:34.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.418+0000 7f791e600640 1 -- start start 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791e600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79180d0340 0x7f791815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79180cc260 0x7f79180c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791e600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79180b6150 con 0x7f79180d0720 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f79180cb020 con 0x7f79180cc260 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791e600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f791815a940 con 0x7f79180d0340 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791d200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f79180d0340 0x7f791815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791dc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79180cc260 0x7f79180c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:34.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f7917e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f79180d0720 0x7f79180c6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34390/0 (socket says 172.21.15.146:34390) 2024-09-16T01:48:34.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f7917e00640 1 -- 172.21.15.146:0/4278786022 learned_addr learned my addr 172.21.15.146:0/4278786022 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:34.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791d200640 1 -- 172.21.15.146:0/4278786022 >> v2:172.21.15.146:3300/0 conn(0x7f79180cc260 msgr2=0x7f79180c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:34.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791d200640 1 --2- 172.21.15.146:0/4278786022 >> v2:172.21.15.146:3300/0 conn(0x7f79180cc260 0x7f79180c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791d200640 1 -- 172.21.15.146:0/4278786022 >> v1:172.21.15.161:6789/0 conn(0x7f79180d0720 legacy=0x7f79180c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:34.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.419+0000 7f791d200640 1 -- 172.21.15.146:0/4278786022 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79180cb270 con 0x7f79180d0340 2024-09-16T01:48:34.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.421+0000 7f791d200640 1 --2- 172.21.15.146:0/4278786022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180d0340 0x7f791815b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79180cdca0 tx=0x7f790c02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad37d08f21de3534 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:34.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.422+0000 7f7917400640 1 -- 172.21.15.146:0/4278786022 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c037070 con 0x7f79180d0340 2024-09-16T01:48:34.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.422+0000 7f7917400640 1 -- 172.21.15.146:0/4278786022 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c032440 con 0x7f79180d0340 2024-09-16T01:48:34.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.422+0000 7f7917400640 1 -- 172.21.15.146:0/4278786022 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c032810 con 0x7f79180d0340 2024-09-16T01:48:34.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 -- 172.21.15.146:0/4278786022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180d0340 msgr2=0x7f791815b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:34.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 --2- 172.21.15.146:0/4278786022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180d0340 0x7f791815b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79180cdca0 tx=0x7f790c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 -- 172.21.15.146:0/4278786022 shutdown_connections 2024-09-16T01:48:34.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 --2- 172.21.15.146:0/4278786022 >> v2:172.21.15.146:3300/0 conn(0x7f79180cc260 0x7f79180c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 --2- 172.21.15.146:0/4278786022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180d0340 0x7f791815b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 -- 172.21.15.146:0/4278786022 >> 172.21.15.146:0/4278786022 conn(0x7f791806e0d0 msgr2=0x7f791806e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:34.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 -- 172.21.15.146:0/4278786022 shutdown_connections 2024-09-16T01:48:34.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.423+0000 7f791e600640 1 -- 172.21.15.146:0/4278786022 wait complete. 2024-09-16T01:48:34.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.425+0000 7f791e600640 1 Processor -- start 2024-09-16T01:48:34.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.425+0000 7f791e600640 1 -- start start 2024-09-16T01:48:34.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 0x7f79180b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:34.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791e600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f79180d0340 0x7f79181e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:34.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79180d0720 0x7f79181f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f79180cb550 con 0x7f79180d0720 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791e600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f791815aec0 con 0x7f79180cc260 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791e600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f79180c6a10 con 0x7f79180d0340 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791d200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 0x7f79180b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79180d0720 0x7f79181f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f7917e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f79180d0340 0x7f79181e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791d200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 0x7f79180b55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54264/0 (socket says 172.21.15.146:54264) 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.426+0000 7f791d200640 1 -- 172.21.15.146:0/1052478661 learned_addr learned my addr 172.21.15.146:0/1052478661 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791d200640 1 -- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f79180d0340 msgr2=0x7f79181e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791d200640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f79180d0340 0x7f79181e52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791d200640 1 -- 172.21.15.146:0/1052478661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79180d0720 msgr2=0x7f79181f3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:34.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791d200640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79180d0720 0x7f79181f3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791d200640 1 -- 172.21.15.146:0/1052478661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79181e1c80 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791d200640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 0x7f79180b55b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f790c02a790 tx=0x7f790c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c037040 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f790c042070 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c030960 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79180b5af0 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79181e2200 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.427+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f790c030b00 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.428+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f790c047bd0 con 0x7f79180cc260 2024-09-16T01:48:34.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.428+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f791806de80 con 0x7f79180cc260 2024-09-16T01:48:34.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.431+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f791806de80 con 0x7f79180cc260 2024-09-16T01:48:34.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.496+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f79180c4260 con 0x7f79180cc260 2024-09-16T01:48:34.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.502+0000 7f7914c00640 1 -- 172.21.15.146:0/1052478661 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f79180c4260 con 0x7f79180cc260 2024-09-16T01:48:34.504 INFO:teuthology.orchestra.run.smithi146.stderr:adjusted tunables profile to default 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 msgr2=0x7f79180b55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 0x7f79180b55b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f790c02a790 tx=0x7f790c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 shutdown_connections 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79180d0720 0x7f79181f3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f79180d0340 0x7f79181e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 --2- 172.21.15.146:0/1052478661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79180cc260 0x7f79180b55b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 >> 172.21.15.146:0/1052478661 conn(0x7f791806e0d0 msgr2=0x7f79180c47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.503+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 shutdown_connections 2024-09-16T01:48:34.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:34.504+0000 7f791e600640 1 -- 172.21.15.146:0/1052478661 wait complete. 2024-09-16T01:48:34.518 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:34.516+0000 7f5115bd6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:48:34.522 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T01:48:34.522 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T01:48:34.522 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:34.522 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T01:48:34.549 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:34.547+0000 7f15a0f30040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:48:34.560 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:34.558+0000 7f5115bd6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:48:34.579 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:34.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T01:48:34.601 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:34.597+0000 7f15a0f30040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:48:34.658 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:34.658 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T01:48:34.699 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:34.697+0000 7f5115bd6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:48:34.750 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:34.749+0000 7f5115bd6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:48:34.754 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:48:34.754 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T01:48:34.782 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:34.780+0000 7f15a0f30040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:48:34.838 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:34.838 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T01:48:34.844 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:34.842+0000 7f15a0f30040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:48:34.864 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:34.864 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T01:48:34.912 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:34.911+0000 7f5115bd6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:48:34.929 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:34.929 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T01:48:34.994 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-09-16T01:48:34.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T01:48:35.028 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:35.026+0000 7f5115bd6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:48:35.036 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:35.034+0000 7f15a0f30040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:48:35.052 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new e7758ddf-75ff-43c7-abae-3647b170f7c0 0 2024-09-16T01:48:35.161 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:35.159+0000 7f15a0f30040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:48:35.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.190+0000 7f20ed9f6640 1 Processor -- start 2024-09-16T01:48:35.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.190+0000 7f20ed9f6640 1 -- start start 2024-09-16T01:48:35.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20ed9f6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20e80cc480 0x7f20e815e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20ed9f6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e815eba0 0x7f20e8165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20ed9f6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e815ba20 con 0x7f20e80cc0a0 2024-09-16T01:48:35.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20ed9f6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f20e815e1d0 con 0x7f20e80cc480 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20ed9f6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f20e815f030 con 0x7f20e815eba0 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f20e80cc0a0 0x7f20e80d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52060/0 (socket says 172.21.15.161:52060) 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e6ffd640 1 -- 172.21.15.161:0/2453954683 learned_addr learned my addr 172.21.15.161:0/2453954683 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e77fe640 1 --2- 172.21.15.161:0/2453954683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e815eba0 0x7f20e8165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e67fc640 1 --2- 172.21.15.161:0/2453954683 >> v2:172.21.15.146:3300/0 conn(0x7f20e80cc480 0x7f20e815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e5ffb640 1 -- 172.21.15.161:0/2453954683 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240342970 0 0) 0x7f20e815ba20 con 0x7f20e80cc0a0 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e5ffb640 1 -- 172.21.15.161:0/2453954683 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d4003520 con 0x7f20e80cc0a0 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e77fe640 1 -- 172.21.15.161:0/2453954683 >> v2:172.21.15.146:3300/0 conn(0x7f20e80cc480 msgr2=0x7f20e815e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e77fe640 1 --2- 172.21.15.161:0/2453954683 >> v2:172.21.15.146:3300/0 conn(0x7f20e80cc480 0x7f20e815e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e77fe640 1 -- 172.21.15.161:0/2453954683 >> v1:172.21.15.161:6789/0 conn(0x7f20e80cc0a0 legacy=0x7f20e80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.191+0000 7f20e77fe640 1 -- 172.21.15.161:0/2453954683 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e815aea0 con 0x7f20e815eba0 2024-09-16T01:48:35.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.192+0000 7f20e77fe640 1 --2- 172.21.15.161:0/2453954683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e815eba0 0x7f20e8165430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009d90 tx=0x7f20e002a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0a9e2f76056b30 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.192+0000 7f20e5ffb640 1 -- 172.21.15.161:0/2453954683 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0007740 con 0x7f20e815eba0 2024-09-16T01:48:35.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.192+0000 7f20e5ffb640 1 -- 172.21.15.161:0/2453954683 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e00078e0 con 0x7f20e815eba0 2024-09-16T01:48:35.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.192+0000 7f20e5ffb640 1 -- 172.21.15.161:0/2453954683 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0007cb0 con 0x7f20e815eba0 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 -- 172.21.15.161:0/2453954683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e815eba0 msgr2=0x7f20e8165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/2453954683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e815eba0 0x7f20e8165430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009d90 tx=0x7f20e002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 -- 172.21.15.161:0/2453954683 shutdown_connections 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/2453954683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e815eba0 0x7f20e8165430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/2453954683 >> v2:172.21.15.146:3300/0 conn(0x7f20e80cc480 0x7f20e815e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 -- 172.21.15.161:0/2453954683 >> 172.21.15.161:0/2453954683 conn(0x7f20e806d560 msgr2=0x7f20e8073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 -- 172.21.15.161:0/2453954683 shutdown_connections 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.193+0000 7f20ed9f6640 1 -- 172.21.15.161:0/2453954683 wait complete. 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.194+0000 7f20ed9f6640 1 Processor -- start 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.194+0000 7f20ed9f6640 1 -- start start 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 0x7f20e815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e80cc480 0x7f20e814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20e815eba0 0x7f20e814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20e815b740 con 0x7f20e80cc480 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20e815b460 con 0x7f20e815eba0 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f20e80cc8a0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 0x7f20e815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 0x7f20e815d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37530/0 (socket says 172.21.15.161:37530) 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 -- 172.21.15.161:0/1263938199 learned_addr learned my addr 172.21.15.161:0/1263938199 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20e815eba0 0x7f20e814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e67fc640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e80cc480 0x7f20e814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 -- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20e815eba0 msgr2=0x7f20e814d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20e815eba0 0x7f20e814d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 -- 172.21.15.161:0/1263938199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e80cc480 msgr2=0x7f20e814d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e80cc480 0x7f20e814d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 -- 172.21.15.161:0/1263938199 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e80cf840 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e67fc640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e80cc480 0x7f20e814d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20e6ffd640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 0x7f20e815d600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20d800c390 tx=0x7f20d800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8017070 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d8005ce0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e80cfb20 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d80072f0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.195+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e80d00e0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.196+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f20d801a070 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.196+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f20d8012740 con 0x7f20e80cc0a0 2024-09-16T01:48:35.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.196+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e80d05d0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.199+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f20d8021020 con 0x7f20e80cc0a0 2024-09-16T01:48:35.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.266+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", "id": 0} v 0) -- 0x7f20e80cbde0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.272+0000 7f20dffff640 1 -- 172.21.15.161:0/1263938199 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f20d80075b0 con 0x7f20e80cc0a0 2024-09-16T01:48:35.274 INFO:teuthology.orchestra.run.smithi161.stdout:0 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 msgr2=0x7f20e815d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 0x7f20e815d600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20d800c390 tx=0x7f20d800c850 comp rx=0 tx=0).stop 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 shutdown_connections 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20e815eba0 0x7f20e814d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20e80cc480 0x7f20e814d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 --2- 172.21.15.161:0/1263938199 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20e80cc0a0 0x7f20e815d600 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 >> 172.21.15.161:0/1263938199 conn(0x7f20e806d560 msgr2=0x7f20e8154390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 shutdown_connections 2024-09-16T01:48:35.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.273+0000 7f20ed9f6640 1 -- 172.21.15.161:0/1263938199 wait complete. 2024-09-16T01:48:35.285 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new a065e77e-26f0-4245-abe8-e36c2fb4436b 1 2024-09-16T01:48:35.343 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:35.341+0000 7f5115bd6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:48:35.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.383+0000 7f4c023b0640 1 Processor -- start 2024-09-16T01:48:35.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.383+0000 7f4c023b0640 1 -- start start 2024-09-16T01:48:35.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c023b0640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc0ccfa0 0x7f4bfc0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c023b0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4bfc06d920 0x7f4bfc163070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c023b0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc06d520 con 0x7f4bfc0cdce0 2024-09-16T01:48:35.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c023b0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4bfc0c78e0 con 0x7f4bfc06d920 2024-09-16T01:48:35.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c023b0640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0c7e20 con 0x7f4bfc0ccfa0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc0ccfa0 0x7f4bfc0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4bfc0cdce0 0x7f4bfc15bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52072/0 (socket says 172.21.15.161:52072) 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfb7fe640 1 -- 172.21.15.161:0/1434961943 learned_addr learned my addr 172.21.15.161:0/1434961943 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c0094d640 1 --2- 172.21.15.161:0/1434961943 >> v2:172.21.15.146:3300/0 conn(0x7f4bfc06d920 0x7f4bfc163070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1434961943 >> v2:172.21.15.146:3300/0 conn(0x7f4bfc06d920 msgr2=0x7f4bfc163070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfaffd640 1 -- 172.21.15.161:0/1434961943 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906494818 0 0) 0x7f4bec001110 con 0x7f4bfc0cdce0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfbfff640 1 --2- 172.21.15.161:0/1434961943 >> v2:172.21.15.146:3300/0 conn(0x7f4bfc06d920 0x7f4bfc163070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1434961943 >> v1:172.21.15.161:6789/0 conn(0x7f4bfc0cdce0 legacy=0x7f4bfc15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1434961943 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc0c7b40 con 0x7f4bfc0ccfa0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.385+0000 7f4c0094d640 1 --2- 172.21.15.161:0/1434961943 >> v2:172.21.15.146:3300/0 conn(0x7f4bfc06d920 0x7f4bfc163070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.386+0000 7f4bfbfff640 1 --2- 172.21.15.161:0/1434961943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc0ccfa0 0x7f4bfc0cd380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4be40097a0 tx=0x7f4be402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5d6688f877f13d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.386+0000 7f4bfaffd640 1 -- 172.21.15.161:0/1434961943 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4037070 con 0x7f4bfc0ccfa0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.386+0000 7f4bfaffd640 1 -- 172.21.15.161:0/1434961943 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be4032440 con 0x7f4bfc0ccfa0 2024-09-16T01:48:35.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.386+0000 7f4bfaffd640 1 -- 172.21.15.161:0/1434961943 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4032810 con 0x7f4bfc0ccfa0 2024-09-16T01:48:35.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 -- 172.21.15.161:0/1434961943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc0ccfa0 msgr2=0x7f4bfc0cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1434961943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc0ccfa0 0x7f4bfc0cd380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4be40097a0 tx=0x7f4be402a210 comp rx=0 tx=0).stop 2024-09-16T01:48:35.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 -- 172.21.15.161:0/1434961943 shutdown_connections 2024-09-16T01:48:35.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1434961943 >> v2:172.21.15.146:3300/0 conn(0x7f4bfc06d920 0x7f4bfc163070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1434961943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc0ccfa0 0x7f4bfc0cd380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 -- 172.21.15.161:0/1434961943 >> 172.21.15.161:0/1434961943 conn(0x7f4bfc0728b0 msgr2=0x7f4bfc072c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 -- 172.21.15.161:0/1434961943 shutdown_connections 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.387+0000 7f4c023b0640 1 -- 172.21.15.161:0/1434961943 wait complete. 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.388+0000 7f4c023b0640 1 Processor -- start 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.388+0000 7f4c023b0640 1 -- start start 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c023b0640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 0x7f4bfc1e5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c023b0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bfc0ccfa0 0x7f4bfc15aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c023b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bfc0cdce0 0x7f4bfc14c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c023b0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc06c270 con 0x7f4bfc0ccfa0 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c023b0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc06bf90 con 0x7f4bfc0cdce0 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c023b0640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0735e0 con 0x7f4bfc06d920 2024-09-16T01:48:35.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 0x7f4bfc1e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 0x7f4bfc1e5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37538/0 (socket says 172.21.15.161:37538) 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1122104412 learned_addr learned my addr 172.21.15.161:0/1122104412 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4c0094d640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bfc0cdce0 0x7f4bfc14c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfb7fe640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bfc0ccfa0 0x7f4bfc15aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bfc0cdce0 msgr2=0x7f4bfc14c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bfc0cdce0 0x7f4bfc14c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1122104412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bfc0ccfa0 msgr2=0x7f4bfc15aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bfc0ccfa0 0x7f4bfc15aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 -- 172.21.15.161:0/1122104412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc1f1b40 con 0x7f4bfc06d920 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfb7fe640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bfc0ccfa0 0x7f4bfc15aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.389+0000 7f4bfbfff640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 0x7f4bfc1e5750 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4be40098c0 tx=0x7f4be40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4041070 con 0x7f4bfc06d920 2024-09-16T01:48:35.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be403c870 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4037040 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc153d90 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc154330 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4be4044070 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.390+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f4be403d3f0 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.391+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bfc0694c0 con 0x7f4bfc06d920 2024-09-16T01:48:35.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.393+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4bfc0694c0 con 0x7f4bfc06d920 2024-09-16T01:48:35.397 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:35.396+0000 7f5115bd6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:48:35.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.463+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", "id": 1} v 0) -- 0x7f4bfc06c5e0 con 0x7f4bfc06d920 2024-09-16T01:48:35.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.466+0000 7f4bf8ff9640 1 -- 172.21.15.161:0/1122104412 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f4be4046450 con 0x7f4bfc06d920 2024-09-16T01:48:35.468 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 msgr2=0x7f4bfc1e5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 0x7f4bfc1e5750 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4be40098c0 tx=0x7f4be40046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 shutdown_connections 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bfc0cdce0 0x7f4bfc14c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bfc0ccfa0 0x7f4bfc15aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 --2- 172.21.15.161:0/1122104412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bfc06d920 0x7f4bfc1e5750 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 >> 172.21.15.161:0/1122104412 conn(0x7f4bfc0728b0 msgr2=0x7f4bfc152530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 shutdown_connections 2024-09-16T01:48:35.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.467+0000 7f4c023b0640 1 -- 172.21.15.161:0/1122104412 wait complete. 2024-09-16T01:48:35.479 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new fefb6d8f-6799-4344-a977-85ab983ec0cb 2 2024-09-16T01:48:35.548 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:35.546+0000 7f15a0f30040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:48:35.557 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:35.555+0000 7f5115bd6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:48:35.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.577+0000 7f02718c3640 1 Processor -- start 2024-09-16T01:48:35.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.577+0000 7f02718c3640 1 -- start start 2024-09-16T01:48:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f02718c3640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 0x7f026c0caa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f02718c3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 0x7f026c0cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f02718c3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026c15b970 con 0x7f026c067d50 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f02718c3640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f026c15b430 con 0x7f026c06bc10 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f02718c3640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f026c0cc5e0 con 0x7f026c0c7a80 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f026b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 0x7f026c0cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f026affd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 0x7f026c0caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f026affd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 0x7f026c0caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37548/0 (socket says 172.21.15.161:37548) 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f026affd640 1 -- 172.21.15.161:0/268157252 learned_addr learned my addr 172.21.15.161:0/268157252 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f026b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 0x7f026c0cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54940/0 (socket says 172.21.15.161:54940) 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.578+0000 7f026a7fc640 1 --1- 172.21.15.161:0/268157252 >> v1:172.21.15.161:6789/0 conn(0x7f026c067d50 0x7f026c0c7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52076/0 (socket says 172.21.15.161:52076) 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f026affd640 1 -- 172.21.15.161:0/268157252 >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 msgr2=0x7f026c0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f026affd640 1 --2- 172.21.15.161:0/268157252 >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 0x7f026c0cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f026affd640 1 -- 172.21.15.161:0/268157252 >> v1:172.21.15.161:6789/0 conn(0x7f026c067d50 legacy=0x7f026c0c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f026affd640 1 -- 172.21.15.161:0/268157252 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c0c8570 con 0x7f026c0c7a80 2024-09-16T01:48:35.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f0269ffb640 1 -- 172.21.15.161:0/268157252 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649993161 0 0) 0x7f026c15b970 con 0x7f026c067d50 2024-09-16T01:48:35.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f026b7fe640 1 --2- 172.21.15.161:0/268157252 >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 0x7f026c0cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:35.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f026affd640 1 --2- 172.21.15.161:0/268157252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 0x7f026c0caa50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f025c009870 tx=0x7f025c02a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa718a335e8d0969 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f0269ffb640 1 -- 172.21.15.161:0/268157252 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c037070 con 0x7f026c0c7a80 2024-09-16T01:48:35.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f0269ffb640 1 -- 172.21.15.161:0/268157252 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025c02ae50 con 0x7f026c0c7a80 2024-09-16T01:48:35.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.579+0000 7f0269ffb640 1 -- 172.21.15.161:0/268157252 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c032860 con 0x7f026c0c7a80 2024-09-16T01:48:35.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 -- 172.21.15.161:0/268157252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 msgr2=0x7f026c0caa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 --2- 172.21.15.161:0/268157252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 0x7f026c0caa50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f025c009870 tx=0x7f025c02a300 comp rx=0 tx=0).stop 2024-09-16T01:48:35.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 -- 172.21.15.161:0/268157252 shutdown_connections 2024-09-16T01:48:35.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 --2- 172.21.15.161:0/268157252 >> v2:172.21.15.161:3300/0 conn(0x7f026c06bc10 0x7f026c0cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 --2- 172.21.15.161:0/268157252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c0c7a80 0x7f026c0caa50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 -- 172.21.15.161:0/268157252 >> 172.21.15.161:0/268157252 conn(0x7f026c0c9220 msgr2=0x7f026c0c9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 -- 172.21.15.161:0/268157252 shutdown_connections 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.580+0000 7f02718c3640 1 -- 172.21.15.161:0/268157252 wait complete. 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.581+0000 7f02718c3640 1 Processor -- start 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.581+0000 7f02718c3640 1 -- start start 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f02718c3640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 0x7f026c14ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f02718c3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f026c06bc10 0x7f026c1597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f02718c3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f026c0c7a80 0x7f026c159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f02718c3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f026c0cb5c0 con 0x7f026c06bc10 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f02718c3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f026c0cb2e0 con 0x7f026c0c7a80 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f02718c3640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f026c0cb8a0 con 0x7f026c067d50 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f026affd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 0x7f026c14ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f026affd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 0x7f026c14ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37550/0 (socket says 172.21.15.161:37550) 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f026b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f026c0c7a80 0x7f026c159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.582+0000 7f026affd640 1 -- 172.21.15.161:0/2615463731 learned_addr learned my addr 172.21.15.161:0/2615463731 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026a7fc640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f026c06bc10 0x7f026c1597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026b7fe640 1 -- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 msgr2=0x7f026c14ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026b7fe640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 0x7f026c14ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026b7fe640 1 -- 172.21.15.161:0/2615463731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f026c06bc10 msgr2=0x7f026c1597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026b7fe640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f026c06bc10 0x7f026c1597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026b7fe640 1 -- 172.21.15.161:0/2615463731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c15a700 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026affd640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 0x7f026c14ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026a7fc640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f026c06bc10 0x7f026c1597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f026b7fe640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f026c0c7a80 0x7f026c159d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f026400ef90 tx=0x7f026400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264018030 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026400cd30 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026c15a9e0 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264003bb0 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026c1f0850 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0264005d10 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.583+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f026401d880 con 0x7f026c0c7a80 2024-09-16T01:48:35.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.584+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f026c1f0360 con 0x7f026c0c7a80 2024-09-16T01:48:35.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.586+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0264022070 con 0x7f026c0c7a80 2024-09-16T01:48:35.609 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:35.607+0000 7f15a0f30040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:48:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.655+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", "id": 2} v 0) -- 0x7f026c0cc080 con 0x7f026c0c7a80 2024-09-16T01:48:35.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.661+0000 7f0263fff640 1 -- 172.21.15.161:0/2615463731 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f0264010040 con 0x7f026c0c7a80 2024-09-16T01:48:35.662 INFO:teuthology.orchestra.run.smithi161.stdout:2 2024-09-16T01:48:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f026c0c7a80 msgr2=0x7f026c159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f026c0c7a80 0x7f026c159d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f026400ef90 tx=0x7f026400c490 comp rx=0 tx=0).stop 2024-09-16T01:48:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 shutdown_connections 2024-09-16T01:48:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f026c0c7a80 0x7f026c159d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f026c06bc10 0x7f026c1597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 --2- 172.21.15.161:0/2615463731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f026c067d50 0x7f026c14ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 >> 172.21.15.161:0/2615463731 conn(0x7f026c0c9220 msgr2=0x7f026c06c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 shutdown_connections 2024-09-16T01:48:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.662+0000 7f02718c3640 1 -- 172.21.15.161:0/2615463731 wait complete. 2024-09-16T01:48:35.674 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 9d1a2f96-efcb-4a73-b5db-ec5e2a63122a 3 2024-09-16T01:48:35.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.773+0000 7fdedb54b640 1 Processor -- start 2024-09-16T01:48:35.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.773+0000 7fdedb54b640 1 -- start start 2024-09-16T01:48:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fdedb54b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fded4152a10 0x7fded4071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fdedb54b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 0x7fded4072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fdedb54b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fded40ce610 con 0x7fded40c3520 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fdedb54b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fded40ce8f0 con 0x7fded4152a10 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fdedb54b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fded40cebd0 con 0x7fded40cbc90 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded8ae6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 0x7fded4072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded9ae8640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fded40c3520 0x7fded415bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52088/0 (socket says 172.21.15.161:52088) 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded9ae8640 1 -- 172.21.15.161:0/1177203360 learned_addr learned my addr 172.21.15.161:0/1177203360 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded8ae6640 1 --2- 172.21.15.161:0/1177203360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 0x7fded4072060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37558/0 (socket says 172.21.15.161:37558) 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded92e7640 1 --2- 172.21.15.161:0/1177203360 >> v2:172.21.15.161:3300/0 conn(0x7fded4152a10 0x7fded4071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded8ae6640 1 -- 172.21.15.161:0/1177203360 >> v2:172.21.15.161:3300/0 conn(0x7fded4152a10 msgr2=0x7fded4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded8ae6640 1 --2- 172.21.15.161:0/1177203360 >> v2:172.21.15.161:3300/0 conn(0x7fded4152a10 0x7fded4071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded8ae6640 1 -- 172.21.15.161:0/1177203360 >> v1:172.21.15.161:6789/0 conn(0x7fded40c3520 legacy=0x7fded415bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded8ae6640 1 -- 172.21.15.161:0/1177203360 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded40ceeb0 con 0x7fded40cbc90 2024-09-16T01:48:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.774+0000 7fded92e7640 1 --2- 172.21.15.161:0/1177203360 >> v2:172.21.15.161:3300/0 conn(0x7fded4152a10 0x7fded4071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.775+0000 7fded8ae6640 1 --2- 172.21.15.161:0/1177203360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 0x7fded4072060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdec40097a0 tx=0x7fdec402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dd3ccf9a11d9eed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.775+0000 7fdecbfff640 1 -- 172.21.15.161:0/1177203360 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4037070 con 0x7fded40cbc90 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.775+0000 7fdecbfff640 1 -- 172.21.15.161:0/1177203360 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec4032440 con 0x7fded40cbc90 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.775+0000 7fdecbfff640 1 -- 172.21.15.161:0/1177203360 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4032810 con 0x7fded40cbc90 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.775+0000 7fdedb54b640 1 -- 172.21.15.161:0/1177203360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 msgr2=0x7fded4072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.775+0000 7fdedb54b640 1 --2- 172.21.15.161:0/1177203360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 0x7fded4072060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdec40097a0 tx=0x7fdec402a210 comp rx=0 tx=0).stop 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.776+0000 7fdedb54b640 1 -- 172.21.15.161:0/1177203360 shutdown_connections 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.776+0000 7fdedb54b640 1 --2- 172.21.15.161:0/1177203360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded40cbc90 0x7fded4072060 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.776+0000 7fdedb54b640 1 --2- 172.21.15.161:0/1177203360 >> v2:172.21.15.161:3300/0 conn(0x7fded4152a10 0x7fded4071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.776+0000 7fdedb54b640 1 -- 172.21.15.161:0/1177203360 >> 172.21.15.161:0/1177203360 conn(0x7fded406bc10 msgr2=0x7fded406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.776+0000 7fdedb54b640 1 -- 172.21.15.161:0/1177203360 shutdown_connections 2024-09-16T01:48:35.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.776+0000 7fdedb54b640 1 -- 172.21.15.161:0/1177203360 wait complete. 2024-09-16T01:48:35.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 Processor -- start 2024-09-16T01:48:35.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 -- start start 2024-09-16T01:48:35.779 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:35.777+0000 7f15a0f30040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:48:35.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fded40c3520 0x7fded41f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 0x7fded41f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded4152a10 0x7fded4154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fded40c9080 con 0x7fded40c3520 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fded40c8f00 con 0x7fded40cbc90 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.777+0000 7fdedb54b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fded4152df0 con 0x7fded4152a10 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 0x7fded41f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 0x7fded41f22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54964/0 (socket says 172.21.15.161:54964) 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded9ae8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded4152a10 0x7fded4154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 -- 172.21.15.161:0/2837744573 learned_addr learned my addr 172.21.15.161:0/2837744573 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded92e7640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fded40c3520 0x7fded41f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 -- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded4152a10 msgr2=0x7fded4154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded4152a10 0x7fded4154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 -- 172.21.15.161:0/2837744573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fded40c3520 msgr2=0x7fded41f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fded40c3520 0x7fded41f1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 -- 172.21.15.161:0/2837744573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded41560d0 con 0x7fded40cbc90 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fded8ae6640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 0x7fded41f22d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdec4035040 tx=0x7fdec40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4037040 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec4030500 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded40c7bf0 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.778+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec4030910 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.779+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded40c81b0 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.779+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdec4044020 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.779+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fdec403d420 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.779+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded4071b20 con 0x7fded40cbc90 2024-09-16T01:48:35.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.781+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fdec403bb60 con 0x7fded40cbc90 2024-09-16T01:48:35.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.850+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", "id": 3} v 0) -- 0x7fded414fde0 con 0x7fded40cbc90 2024-09-16T01:48:35.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.855+0000 7fdec9ffb640 1 -- 172.21.15.161:0/2837744573 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fdec403bde0 con 0x7fded40cbc90 2024-09-16T01:48:35.857 INFO:teuthology.orchestra.run.smithi161.stdout:3 2024-09-16T01:48:35.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.856+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 msgr2=0x7fded41f22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.856+0000 7fdedb54b640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 0x7fded41f22d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdec4035040 tx=0x7fdec40046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 shutdown_connections 2024-09-16T01:48:35.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fded4152a10 0x7fded4154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fded40cbc90 0x7fded41f22d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 --2- 172.21.15.161:0/2837744573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fded40c3520 0x7fded41f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 >> 172.21.15.161:0/2837744573 conn(0x7fded406bc10 msgr2=0x7fded4071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 shutdown_connections 2024-09-16T01:48:35.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.857+0000 7fdedb54b640 1 -- 172.21.15.161:0/2837744573 wait complete. 2024-09-16T01:48:35.869 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c 4 2024-09-16T01:48:35.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.967+0000 7fdab1d5f640 1 Processor -- start 2024-09-16T01:48:35.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.967+0000 7fdab1d5f640 1 -- start start 2024-09-16T01:48:35.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdab1d5f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdaac15b440 0x7fdaac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdab1d5f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaac06c5a0 con 0x7fdaac158140 2024-09-16T01:48:35.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdab1d5f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaac0c7a80 con 0x7fdaac0c6d00 2024-09-16T01:48:35.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdab1d5f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdaac0c8040 con 0x7fdaac15b440 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdaabfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdaac0c6d00 0x7fdaac1652e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52102/0 (socket says 172.21.15.161:52102) 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdaabfff640 1 -- 172.21.15.161:0/929568787 learned_addr learned my addr 172.21.15.161:0/929568787 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.968+0000 7fdaa3fff640 1 --2- 172.21.15.161:0/929568787 >> v2:172.21.15.146:3300/0 conn(0x7fdaac15b440 0x7fdaac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016579179 0 0) 0x7fdaac0c7a80 con 0x7fdaac0c6d00 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda98003520 con 0x7fdaac0c6d00 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 115634531 0 0) 0x7fda98003520 con 0x7fdaac0c6d00 2024-09-16T01:48:35.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 >> v2:172.21.15.146:3300/0 conn(0x7fdaac15b440 msgr2=0x7fdaac06e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 --2- 172.21.15.161:0/929568787 >> v2:172.21.15.146:3300/0 conn(0x7fdaac15b440 0x7fdaac06e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 >> v1:172.21.15.146:6789/0 conn(0x7fdaac158140 legacy=0x7fdaac06cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac0c7d60 con 0x7fdaac0c6d00 2024-09-16T01:48:35.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fdaa40019f0 con 0x7fdaac0c6d00 2024-09-16T01:48:35.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 >> v1:172.21.15.161:6789/0 conn(0x7fdaac0c6d00 legacy=0x7fdaac1652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda98003a00 0x7fda98003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda980051c0 0x7fda9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac0c8040 con 0x7fda980051c0 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda9800c0a0 con 0x7fda98003a00 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fda9800c2a0 con 0x7fda98004800 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaab7fe640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda98003a00 0x7fda98003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaa3fff640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaabfff640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda980051c0 0x7fda9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaab7fe640 1 -- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda98004800 msgr2=0x7fda98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaab7fe640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaab7fe640 1 -- 172.21.15.161:0/929568787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda980051c0 msgr2=0x7fda9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:35.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaab7fe640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda980051c0 0x7fda9800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaab7fe640 1 -- 172.21.15.161:0/929568787 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda98003520 con 0x7fda98003a00 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaa3fff640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaabfff640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda980051c0 0x7fda9800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.969+0000 7fdaabfff640 1 -- 172.21.15.161:0/929568787 reap_dead start 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.970+0000 7fdaab7fe640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda98003a00 0x7fda98003e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fda9c001750 tx=0x7fda98005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0a815355373e87c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.970+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c001370 con 0x7fda98003a00 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.970+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c0052d0 con 0x7fda98003a00 2024-09-16T01:48:35.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.970+0000 7fdaaaffd640 1 -- 172.21.15.161:0/929568787 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c02abc0 con 0x7fda98003a00 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 -- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda98003a00 msgr2=0x7fda98003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda98003a00 0x7fda98003e20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fda9c001750 tx=0x7fda98005830 comp rx=0 tx=0).stop 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 -- 172.21.15.161:0/929568787 shutdown_connections 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 --2- 172.21.15.161:0/929568787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda98003a00 0x7fda98003e20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 -- 172.21.15.161:0/929568787 >> 172.21.15.161:0/929568787 conn(0x7fdaac0728b0 msgr2=0x7fdaac072c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 -- 172.21.15.161:0/929568787 shutdown_connections 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.971+0000 7fdab1d5f640 1 -- 172.21.15.161:0/929568787 wait complete. 2024-09-16T01:48:35.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.972+0000 7fdab1d5f640 1 Processor -- start 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.972+0000 7fdab1d5f640 1 -- start start 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda98003a00 0x7fdaac0c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 0x7fdaac15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdaac0cc8b0 0x7fdaac1f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac0735e0 con 0x7fda98003a00 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac0c0b20 con 0x7fdaac0cc470 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdaac0c0ca0 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaa3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 0x7fdaac15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdaac0cc8b0 0x7fdaac1f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaa3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 0x7fdaac15a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54978/0 (socket says 172.21.15.161:54978) 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaa3fff640 1 -- 172.21.15.161:0/54878103 learned_addr learned my addr 172.21.15.161:0/54878103 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:35.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaab7fe640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda98003a00 0x7fdaac0c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 -- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 msgr2=0x7fdaac15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 0x7fdaac15a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 -- 172.21.15.161:0/54878103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda98003a00 msgr2=0x7fdaac0c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda98003a00 0x7fdaac0c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 -- 172.21.15.161:0/54878103 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac0c3330 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaa3fff640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 0x7fdaac15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaab7fe640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda98003a00 0x7fdaac0c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaabfff640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdaac0cc8b0 0x7fdaac1f0d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdaac06be10 tx=0x7fdaa400e530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400eda0 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa4014430 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.973+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaac1f3fd0 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.974+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4014820 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.974+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaac1f4590 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.974+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdaa40149c0 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.974+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fdaa40133e0 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.974+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaac0c4f10 con 0x7fdaac0cc8b0 2024-09-16T01:48:35.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:35.976+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fdaa4014df0 con 0x7fdaac0cc8b0 2024-09-16T01:48:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.045+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", "id": 4} v 0) -- 0x7fdaac0c83f0 con 0x7fdaac0cc8b0 2024-09-16T01:48:36.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.051+0000 7fdaa8ff9640 1 -- 172.21.15.161:0/54878103 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fdaa401d3f0 con 0x7fdaac0cc8b0 2024-09-16T01:48:36.053 INFO:teuthology.orchestra.run.smithi161.stdout:4 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdaac0cc8b0 msgr2=0x7fdaac1f0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdaac0cc8b0 0x7fdaac1f0d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdaac06be10 tx=0x7fdaa400e530 comp rx=0 tx=0).stop 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 shutdown_connections 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdaac0cc8b0 0x7fdaac1f0d60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdaac0cc470 0x7fdaac15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 --2- 172.21.15.161:0/54878103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda98003a00 0x7fdaac0c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 >> 172.21.15.161:0/54878103 conn(0x7fdaac0728b0 msgr2=0x7fdaac06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 shutdown_connections 2024-09-16T01:48:36.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.052+0000 7fdab1d5f640 1 -- 172.21.15.161:0/54878103 wait complete. 2024-09-16T01:48:36.064 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 11562dd5-d537-4d88-a282-e8687caebad6 5 2024-09-16T01:48:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.162+0000 7fa4ec52c640 1 Processor -- start 2024-09-16T01:48:36.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.162+0000 7fa4ec52c640 1 -- start start 2024-09-16T01:48:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ec52c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e40caa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ec52c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e415b970 con 0x7fa4e4067d50 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ec52c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e40c8570 con 0x7fa4e406bc10 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ec52c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e415b430 con 0x7fa4e40c7a80 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ea2c8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e40caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4eaac9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa4e406bc10 0x7fa4e41547c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:52114/0 (socket says 172.21.15.161:52114) 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4eaac9640 1 -- 172.21.15.161:0/408829883 learned_addr learned my addr 172.21.15.161:0/408829883 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4e92c6640 1 -- 172.21.15.161:0/408829883 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2007984294 0 0) 0x7fa4e40c8570 con 0x7fa4e406bc10 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ea2c8640 1 -- 172.21.15.161:0/408829883 >> v1:172.21.15.161:6789/0 conn(0x7fa4e406bc10 legacy=0x7fa4e41547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ea2c8640 1 -- 172.21.15.161:0/408829883 >> v1:172.21.15.146:6789/0 conn(0x7fa4e4067d50 legacy=0x7fa4e40c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:36.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.163+0000 7fa4ea2c8640 1 -- 172.21.15.161:0/408829883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e415b690 con 0x7fa4e40c7a80 2024-09-16T01:48:36.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.164+0000 7fa4ea2c8640 1 --2- 172.21.15.161:0/408829883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e40caa50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00097d0 tx=0x7fa4e002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5346436106cb790e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.164+0000 7fa4e92c6640 1 -- 172.21.15.161:0/408829883 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0037070 con 0x7fa4e40c7a80 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.164+0000 7fa4e92c6640 1 -- 172.21.15.161:0/408829883 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e0032440 con 0x7fa4e40c7a80 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.164+0000 7fa4e92c6640 1 -- 172.21.15.161:0/408829883 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0032810 con 0x7fa4e40c7a80 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 -- 172.21.15.161:0/408829883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 msgr2=0x7fa4e40caa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 --2- 172.21.15.161:0/408829883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e40caa50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00097d0 tx=0x7fa4e002a250 comp rx=0 tx=0).stop 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 -- 172.21.15.161:0/408829883 shutdown_connections 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 --2- 172.21.15.161:0/408829883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e40caa50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 -- 172.21.15.161:0/408829883 >> 172.21.15.161:0/408829883 conn(0x7fa4e40c9220 msgr2=0x7fa4e40c9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 -- 172.21.15.161:0/408829883 shutdown_connections 2024-09-16T01:48:36.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.165+0000 7fa4ec52c640 1 -- 172.21.15.161:0/408829883 wait complete. 2024-09-16T01:48:36.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.166+0000 7fa4ec52c640 1 Processor -- start 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.166+0000 7fa4ec52c640 1 -- start start 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e4067d50 0x7fa4e41f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 0x7fa4e4159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e4155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e40cb8a0 con 0x7fa4e4067d50 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e40cb5c0 con 0x7fa4e406bc10 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e4154ed0 con 0x7fa4e40c7a80 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e4155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4e9ac7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 0x7fa4e4159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4e9ac7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 0x7fa4e4159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54984/0 (socket says 172.21.15.161:54984) 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e4155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37604/0 (socket says 172.21.15.161:37604) 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4e9ac7640 1 -- 172.21.15.161:0/2004763622 learned_addr learned my addr 172.21.15.161:0/2004763622 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ea2c8640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e4067d50 0x7fa4e41f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 -- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 msgr2=0x7fa4e4159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 0x7fa4e4159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 -- 172.21.15.161:0/2004763622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e4067d50 msgr2=0x7fa4e41f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e4067d50 0x7fa4e41f40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 -- 172.21.15.161:0/2004763622 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e4156770 con 0x7fa4e40c7a80 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4e9ac7640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 0x7fa4e4159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ea2c8640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e4067d50 0x7fa4e41f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4eaac9640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e4155330 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa4e406c080 tx=0x7fa4dc00cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc018070 con 0x7fa4e40c7a80 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc004d60 con 0x7fa4e40c7a80 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e4156a50 con 0x7fa4e40c7a80 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.167+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00a880 con 0x7fa4e40c7a80 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.168+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e415aa80 con 0x7fa4e40c7a80 2024-09-16T01:48:36.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.168+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4dc01b070 con 0x7fa4e40c7a80 2024-09-16T01:48:36.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.168+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fa4dc007760 con 0x7fa4e40c7a80 2024-09-16T01:48:36.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.168+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e4154150 con 0x7fa4e40c7a80 2024-09-16T01:48:36.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.170+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fa4dc0070d0 con 0x7fa4e40c7a80 2024-09-16T01:48:36.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.241+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", "id": 5} v 0) -- 0x7fa4e4153e70 con 0x7fa4e40c7a80 2024-09-16T01:48:36.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.246+0000 7fa4daffd640 1 -- 172.21.15.161:0/2004763622 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fa4dc0072b0 con 0x7fa4e40c7a80 2024-09-16T01:48:36.248 INFO:teuthology.orchestra.run.smithi161.stdout:5 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 msgr2=0x7fa4e4155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e4155330 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa4e406c080 tx=0x7fa4dc00cb60 comp rx=0 tx=0).stop 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 shutdown_connections 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4e40c7a80 0x7fa4e4155330 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4e406bc10 0x7fa4e4159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 --2- 172.21.15.161:0/2004763622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e4067d50 0x7fa4e41f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 >> 172.21.15.161:0/2004763622 conn(0x7fa4e40c9220 msgr2=0x7fa4e40c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 shutdown_connections 2024-09-16T01:48:36.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.247+0000 7fa4ec52c640 1 -- 172.21.15.161:0/2004763622 wait complete. 2024-09-16T01:48:36.259 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new b31adf01-6768-49cf-a232-127b9d0d2dc8 6 2024-09-16T01:48:36.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.357+0000 7fca89ffd640 1 Processor -- start 2024-09-16T01:48:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.357+0000 7fca89ffd640 1 -- start start 2024-09-16T01:48:36.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca89ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 0x7fca8406e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca89ffd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8406c5a0 con 0x7fca84158140 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca89ffd640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca840c7a80 con 0x7fca840c6d00 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca89ffd640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fca840c8040 con 0x7fca8415b440 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca7bfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 0x7fca8406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca7bfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 0x7fca8406e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37620/0 (socket says 172.21.15.161:37620) 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca7bfff640 1 -- 172.21.15.161:0/90202444 learned_addr learned my addr 172.21.15.161:0/90202444 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca7bfff640 1 -- 172.21.15.161:0/90202444 >> v1:172.21.15.161:6789/0 conn(0x7fca840c6d00 legacy=0x7fca841652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca82ffd640 1 -- 172.21.15.161:0/90202444 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3745293797 0 0) 0x7fca840c7a80 con 0x7fca840c6d00 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca7bfff640 1 -- 172.21.15.161:0/90202444 >> v1:172.21.15.146:6789/0 conn(0x7fca84158140 legacy=0x7fca8406cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.358+0000 7fca7bfff640 1 -- 172.21.15.161:0/90202444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca840c7d60 con 0x7fca8415b440 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.359+0000 7fca7bfff640 1 --2- 172.21.15.161:0/90202444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 0x7fca8406e770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0098a0 tx=0x7fca6c02a320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39da10c54f9ea83b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.359+0000 7fca82ffd640 1 -- 172.21.15.161:0/90202444 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c037070 con 0x7fca8415b440 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.359+0000 7fca82ffd640 1 -- 172.21.15.161:0/90202444 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c032640 con 0x7fca8415b440 2024-09-16T01:48:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.359+0000 7fca82ffd640 1 -- 172.21.15.161:0/90202444 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c032a10 con 0x7fca8415b440 2024-09-16T01:48:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 -- 172.21.15.161:0/90202444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 msgr2=0x7fca8406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 --2- 172.21.15.161:0/90202444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 0x7fca8406e770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0098a0 tx=0x7fca6c02a320 comp rx=0 tx=0).stop 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 -- 172.21.15.161:0/90202444 shutdown_connections 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 --2- 172.21.15.161:0/90202444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca8415b440 0x7fca8406e770 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 -- 172.21.15.161:0/90202444 >> 172.21.15.161:0/90202444 conn(0x7fca840728b0 msgr2=0x7fca84072c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 -- 172.21.15.161:0/90202444 shutdown_connections 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.360+0000 7fca89ffd640 1 -- 172.21.15.161:0/90202444 wait complete. 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.361+0000 7fca89ffd640 1 Processor -- start 2024-09-16T01:48:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.361+0000 7fca89ffd640 1 -- start start 2024-09-16T01:48:36.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca89ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 0x7fca84070370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca89ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca84158140 0x7fca840c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca89ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8415b440 0x7fca840c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca89ffd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fca840735e0 con 0x7fca84158140 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca89ffd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fca840c0b20 con 0x7fca8415b440 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca89ffd640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fca840c0ca0 con 0x7fca840c6d00 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca837fe640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 0x7fca84070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8415b440 0x7fca840c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca837fe640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 0x7fca84070370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37636/0 (socket says 172.21.15.161:37636) 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca837fe640 1 -- 172.21.15.161:0/2684760004 learned_addr learned my addr 172.21.15.161:0/2684760004 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca7bfff640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca84158140 0x7fca840c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 -- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 msgr2=0x7fca84070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 0x7fca84070370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 -- 172.21.15.161:0/2684760004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca84158140 msgr2=0x7fca840c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca84158140 0x7fca840c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 -- 172.21.15.161:0/2684760004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca841e5e40 con 0x7fca8415b440 2024-09-16T01:48:36.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca837fe640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 0x7fca84070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.362+0000 7fca83fff640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8415b440 0x7fca840c97f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c6d0 tx=0x7fca7c00cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c003b70 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca7c003d30 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca84153c50 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c0052b0 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca84154210 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fca7c005660 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.363+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fca7c01bc20 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.364+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca840c4f10 con 0x7fca8415b440 2024-09-16T01:48:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.366+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fca7c01b630 con 0x7fca8415b440 2024-09-16T01:48:36.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.437+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", "id": 6} v 0) -- 0x7fca840c83f0 con 0x7fca8415b440 2024-09-16T01:48:36.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.442+0000 7fca80ff9640 1 -- 172.21.15.161:0/2684760004 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fca7c01b8e0 con 0x7fca8415b440 2024-09-16T01:48:36.445 INFO:teuthology.orchestra.run.smithi161.stdout:6 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8415b440 msgr2=0x7fca840c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8415b440 0x7fca840c97f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c6d0 tx=0x7fca7c00cb90 comp rx=0 tx=0).stop 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 shutdown_connections 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8415b440 0x7fca840c97f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca84158140 0x7fca840c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 --2- 172.21.15.161:0/2684760004 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840c6d00 0x7fca84070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 >> 172.21.15.161:0/2684760004 conn(0x7fca840728b0 msgr2=0x7fca841e6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 shutdown_connections 2024-09-16T01:48:36.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.444+0000 7fca89ffd640 1 -- 172.21.15.161:0/2684760004 wait complete. 2024-09-16T01:48:36.459 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 67c95c28-b600-430c-9474-a902cf46a73b 7 2024-09-16T01:48:36.535 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:36.533+0000 7f5115bd6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:48:36.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.557+0000 7fa0cbe5a640 1 Processor -- start 2024-09-16T01:48:36.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.557+0000 7fa0cbe5a640 1 -- start start 2024-09-16T01:48:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0cbe5a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 0x7fa0c40cc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0cbe5a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0c40d02d0 0x7fa0c40d0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0cbe5a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c40d0dd0 con 0x7fa0c415e0d0 2024-09-16T01:48:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0cbe5a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c415bb40 con 0x7fa0c40d02d0 2024-09-16T01:48:36.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0cbe5a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c40cb5f0 con 0x7fa0c4154de0 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 0x7fa0c40cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 0x7fa0c40cc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37642/0 (socket says 172.21.15.161:37642) 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 -- 172.21.15.161:0/670823858 learned_addr learned my addr 172.21.15.161:0/670823858 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 -- 172.21.15.161:0/670823858 >> v2:172.21.15.146:3300/0 conn(0x7fa0c40d02d0 msgr2=0x7fa0c40d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 --2- 172.21.15.161:0/670823858 >> v2:172.21.15.146:3300/0 conn(0x7fa0c40d02d0 0x7fa0c40d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 -- 172.21.15.161:0/670823858 >> v1:172.21.15.146:6789/0 conn(0x7fa0c415e0d0 legacy=0x7fa0c415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.558+0000 7fa0c9bf6640 1 -- 172.21.15.161:0/670823858 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c415b580 con 0x7fa0c4154de0 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.559+0000 7fa0c9bf6640 1 --2- 172.21.15.161:0/670823858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 0x7fa0c40cc090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009870 tx=0x7fa0c002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154dc447f3ebec51 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.559+0000 7fa0c8bf4640 1 -- 172.21.15.161:0/670823858 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0037070 con 0x7fa0c4154de0 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.559+0000 7fa0c8bf4640 1 -- 172.21.15.161:0/670823858 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c0032440 con 0x7fa0c4154de0 2024-09-16T01:48:36.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.559+0000 7fa0c8bf4640 1 -- 172.21.15.161:0/670823858 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0032810 con 0x7fa0c4154de0 2024-09-16T01:48:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/670823858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 msgr2=0x7fa0c40cc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/670823858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 0x7fa0c40cc090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009870 tx=0x7fa0c002a210 comp rx=0 tx=0).stop 2024-09-16T01:48:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/670823858 shutdown_connections 2024-09-16T01:48:36.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/670823858 >> v2:172.21.15.146:3300/0 conn(0x7fa0c40d02d0 0x7fa0c40d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/670823858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c4154de0 0x7fa0c40cc090 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/670823858 >> 172.21.15.161:0/670823858 conn(0x7fa0c406cab0 msgr2=0x7fa0c406e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/670823858 shutdown_connections 2024-09-16T01:48:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.560+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/670823858 wait complete. 2024-09-16T01:48:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.561+0000 7fa0cbe5a640 1 Processor -- start 2024-09-16T01:48:36.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.561+0000 7fa0cbe5a640 1 -- start start 2024-09-16T01:48:36.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0cbe5a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c40d02d0 0x7fa0c414b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0cbe5a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 0x7fa0c41f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0cbe5a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 0x7fa0c41e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0cbe5a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c40cbb70 con 0x7fa0c40d02d0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0cbe5a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c40cb890 con 0x7fa0c4154de0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0cbe5a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c415b270 con 0x7fa0c415e0d0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 0x7fa0c41e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0c93f5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 0x7fa0c41f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 0x7fa0c41e3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:37650/0 (socket says 172.21.15.161:37650) 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0c93f5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 0x7fa0c41f4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:55010/0 (socket says 172.21.15.161:55010) 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 -- 172.21.15.161:0/3445852039 learned_addr learned my addr 172.21.15.161:0/3445852039 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 -- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 msgr2=0x7fa0c41f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 0x7fa0c41f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0c9bf6640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c40d02d0 0x7fa0c414b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 -- 172.21.15.161:0/3445852039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c40d02d0 msgr2=0x7fa0c414b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c40d02d0 0x7fa0c414b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 -- 172.21.15.161:0/3445852039 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c41f5410 con 0x7fa0c415e0d0 2024-09-16T01:48:36.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0c93f5640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 0x7fa0c41f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0c9bf6640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c40d02d0 0x7fa0c414b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ca3f7640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 0x7fa0c41e3ed0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00b970 tx=0x7fa0bc00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.562+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc019070 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0bc00cb60 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c4158d90 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc014440 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c4158ab0 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa0bc01c070 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fa0bc01da10 con 0x7fa0c415e0d0 2024-09-16T01:48:36.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.563+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c41551c0 con 0x7fa0c415e0d0 2024-09-16T01:48:36.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.567+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fa0bc023070 con 0x7fa0c415e0d0 2024-09-16T01:48:36.592 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:36.591+0000 7f5115bd6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:48:36.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.637+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", "id": 7} v 0) -- 0x7fa0c41e6fa0 con 0x7fa0c415e0d0 2024-09-16T01:48:36.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.648+0000 7fa0ba7fc640 1 -- 172.21.15.161:0/3445852039 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fa0bc01dce0 con 0x7fa0c415e0d0 2024-09-16T01:48:36.650 INFO:teuthology.orchestra.run.smithi161.stdout:7 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 msgr2=0x7fa0c41e3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 0x7fa0c41e3ed0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00b970 tx=0x7fa0bc00be30 comp rx=0 tx=0).stop 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 shutdown_connections 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c415e0d0 0x7fa0c41e3ed0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154de0 0x7fa0c41f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 --2- 172.21.15.161:0/3445852039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c40d02d0 0x7fa0c414b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 >> 172.21.15.161:0/3445852039 conn(0x7fa0c406cab0 msgr2=0x7fa0c40cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 shutdown_connections 2024-09-16T01:48:36.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-09-16T01:48:36.649+0000 7fa0cbe5a640 1 -- 172.21.15.161:0/3445852039 wait complete. 2024-09-16T01:48:36.661 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T01:48:36.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T01:48:36.667 INFO:tasks.ceph.osd.0:Started 2024-09-16T01:48:36.667 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T01:48:36.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T01:48:36.672 INFO:tasks.ceph.osd.1:Started 2024-09-16T01:48:36.672 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T01:48:36.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T01:48:36.676 INFO:tasks.ceph.osd.2:Started 2024-09-16T01:48:36.677 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T01:48:36.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T01:48:36.682 INFO:tasks.ceph.osd.3:Started 2024-09-16T01:48:36.682 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T01:48:36.682 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T01:48:36.684 INFO:tasks.ceph.osd.4:Started 2024-09-16T01:48:36.684 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T01:48:36.684 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T01:48:36.687 INFO:tasks.ceph.osd.5:Started 2024-09-16T01:48:36.687 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T01:48:36.687 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T01:48:36.689 INFO:tasks.ceph.osd.6:Started 2024-09-16T01:48:36.689 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T01:48:36.689 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T01:48:36.691 INFO:tasks.ceph.osd.7:Started 2024-09-16T01:48:36.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:48:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.893+0000 7f698aa00640 1 Processor -- start 2024-09-16T01:48:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.893+0000 7f698aa00640 1 -- start start 2024-09-16T01:48:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.894+0000 7f698aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f698415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.894+0000 7f698aa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69840cb2f0 0x7f69840cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.894+0000 7f698aa00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698415b1d0 con 0x7f698415e9d0 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.894+0000 7f698aa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f69840cbd00 con 0x7f69840cb2f0 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.894+0000 7f698aa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f698415ba70 con 0x7f69840cc0c0 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69840cb2f0 0x7f69840cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69840cb2f0 0x7f69840cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40248/0 (socket says 172.21.15.146:40248) 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 -- 172.21.15.146:0/3535975042 learned_addr learned my addr 172.21.15.146:0/3535975042 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983e00640 1 --2- 172.21.15.146:0/3535975042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f698415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6988a00640 1 --1- 172.21.15.146:0/3535975042 >> v1:172.21.15.161:6789/0 conn(0x7f698415e9d0 0x7f69840cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34410/0 (socket says 172.21.15.146:34410) 2024-09-16T01:48:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 -- 172.21.15.146:0/3535975042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 msgr2=0x7f698415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 --2- 172.21.15.146:0/3535975042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f698415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 -- 172.21.15.146:0/3535975042 >> v1:172.21.15.161:6789/0 conn(0x7f698415e9d0 legacy=0x7f69840cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.895+0000 7f6983400640 1 -- 172.21.15.146:0/3535975042 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698415b4b0 con 0x7f69840cb2f0 2024-09-16T01:48:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.896+0000 7f6983400640 1 --2- 172.21.15.146:0/3535975042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69840cb2f0 0x7f69840cc4a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6970009870 tx=0x7f697002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31aa21596a3f269d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.896+0000 7f6982a00640 1 -- 172.21.15.146:0/3535975042 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970037070 con 0x7f69840cb2f0 2024-09-16T01:48:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.896+0000 7f6982a00640 1 -- 172.21.15.146:0/3535975042 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6970032440 con 0x7f69840cb2f0 2024-09-16T01:48:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.896+0000 7f6982a00640 1 -- 172.21.15.146:0/3535975042 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970032810 con 0x7f69840cb2f0 2024-09-16T01:48:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 -- 172.21.15.146:0/3535975042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69840cb2f0 msgr2=0x7f69840cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 --2- 172.21.15.146:0/3535975042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69840cb2f0 0x7f69840cc4a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6970009870 tx=0x7f697002a250 comp rx=0 tx=0).stop 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 -- 172.21.15.146:0/3535975042 shutdown_connections 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 --2- 172.21.15.146:0/3535975042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69840cb2f0 0x7f69840cc4a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 --2- 172.21.15.146:0/3535975042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f698415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 -- 172.21.15.146:0/3535975042 >> 172.21.15.146:0/3535975042 conn(0x7f698406d560 msgr2=0x7f698406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 -- 172.21.15.146:0/3535975042 shutdown_connections 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.897+0000 7f698aa00640 1 -- 172.21.15.146:0/3535975042 wait complete. 2024-09-16T01:48:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.899+0000 7f698aa00640 1 Processor -- start 2024-09-16T01:48:36.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.899+0000 7f698aa00640 1 -- start start 2024-09-16T01:48:36.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f698aa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69840cb2f0 0x7f69840c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f698aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f69841e4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f698aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f698415e9d0 0x7f698415bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f698aa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f698415e1d0 con 0x7f698415e9d0 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f698aa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f698415e540 con 0x7f69840cb2f0 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f698aa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f69840c5560 con 0x7f69840cc0c0 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f69841e4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69840cb2f0 0x7f69840c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f69841e4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47392/0 (socket says 172.21.15.146:47392) 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6988a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f698415e9d0 0x7f698415bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 -- 172.21.15.146:0/3472889479 learned_addr learned my addr 172.21.15.146:0/3472889479 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 -- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69840cb2f0 msgr2=0x7f69840c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69840cb2f0 0x7f69840c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 -- 172.21.15.146:0/3472889479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f698415e9d0 msgr2=0x7f698415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.900+0000 7f6983400640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f698415e9d0 0x7f698415bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f6983400640 1 -- 172.21.15.146:0/3472889479 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69840c8230 con 0x7f69840cc0c0 2024-09-16T01:48:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f6983e00640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69840cb2f0 0x7f69840c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f6988a00640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f698415e9d0 0x7f698415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f6983400640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f69841e4300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f697002a290 tx=0x7f69700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970037040 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6970042070 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6984158650 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970030880 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.901+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69841f3910 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.902+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6970030a20 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.902+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6970030d10 con 0x7f69840cc0c0 2024-09-16T01:48:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.903+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69840cbea0 con 0x7f69840cc0c0 2024-09-16T01:48:36.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.905+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f697004c020 con 0x7f69840cc0c0 2024-09-16T01:48:36.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.966+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69841f46d0 con 0x7f69840cc0c0 2024-09-16T01:48:36.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.967+0000 7f697be00640 1 -- 172.21.15.146:0/3472889479 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f6970047460 con 0x7f69840cc0c0 2024-09-16T01:48:36.969 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:48:36.969 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":10,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:36.639012+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.968+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 msgr2=0x7f69841e4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.968+0000 7f698aa00640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f69841e4300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f697002a290 tx=0x7f69700046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.968+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 shutdown_connections 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.968+0000 7f698aa00640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f698415e9d0 0x7f698415bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.968+0000 7f698aa00640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f69840cc0c0 0x7f69841e4300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.969+0000 7f698aa00640 1 --2- 172.21.15.146:0/3472889479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f69840cb2f0 0x7f69840c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.969+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 >> 172.21.15.146:0/3472889479 conn(0x7f698406d560 msgr2=0x7f69840ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.969+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 shutdown_connections 2024-09-16T01:48:36.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:36.969+0000 7f698aa00640 1 -- 172.21.15.146:0/3472889479 wait complete. 2024-09-16T01:48:36.986 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T01:48:36.986 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T01:48:36.990 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:36.988+0000 7f15a0f30040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:48:37.002 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.001+0000 7f5115bd6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:48:37.053 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.051+0000 7f15a0f30040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:48:37.090 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.089+0000 7f5115bd6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:48:37.199 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.198+0000 7f5115bd6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:48:37.245 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.244+0000 7f5115bd6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:48:37.360 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.359+0000 7f5115bd6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:48:37.413 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.412+0000 7f5115bd6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:48:37.460 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.458+0000 7f5115bd6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:48:37.491 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.489+0000 7f15a0f30040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:48:37.573 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.571+0000 7f5115bd6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:48:37.587 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.585+0000 7f15a0f30040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:48:37.624 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.623+0000 7f5115bd6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:48:37.677 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.676+0000 7f5115bd6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:48:37.689 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:48:37.688+0000 7ff6185db640 -1 Falling back to public interface 2024-09-16T01:48:37.708 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.706+0000 7f15a0f30040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:48:37.747 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.746+0000 7f5115bd6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:48:37.764 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.762+0000 7f15a0f30040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:48:37.791 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T01:48:37.792 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.791+0000 7f5115bd6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:48:37.836 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:37.835+0000 7f5115bd6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:48:37.895 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.893+0000 7f15a0f30040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:48:37.930 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.928+0000 7fc9cf200640 1 Processor -- start 2024-09-16T01:48:37.931 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.928+0000 7fc9cf200640 1 -- start start 2024-09-16T01:48:37.932 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9cf200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc9c8154c90 0x7fc9c80c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:37.932 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9cf200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:37.932 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9cf200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9c815ac60 con 0x7fc9c80cbc00 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9cf200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc9c80c7520 con 0x7fc9c8154c90 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9cf200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c815af40 con 0x7fc9c815dfe0 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc9c8154c90 0x7fc9c80c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc9c80cbc00 0x7fc9c80cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34426/0 (socket says 172.21.15.146:34426) 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7400640 1 -- 172.21.15.146:0/2894840144 learned_addr learned my addr 172.21.15.146:0/2894840144 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9cd200640 1 --2- 172.21.15.146:0/2894840144 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c6a00640 1 -- 172.21.15.146:0/2894840144 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648521817 0 0) 0x7fc9c815ac60 con 0x7fc9c80cbc00 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c6a00640 1 -- 172.21.15.146:0/2894840144 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9ac003520 con 0x7fc9c80cbc00 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7e00640 1 -- 172.21.15.146:0/2894840144 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 msgr2=0x7fc9c80c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7e00640 1 --2- 172.21.15.146:0/2894840144 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7e00640 1 -- 172.21.15.146:0/2894840144 >> v1:172.21.15.161:6789/0 conn(0x7fc9c80cbc00 legacy=0x7fc9c80cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:37.933 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.930+0000 7fc9c7e00640 1 -- 172.21.15.146:0/2894840144 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c815b220 con 0x7fc9c8154c90 2024-09-16T01:48:37.934 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.932+0000 7fc9c7e00640 1 --2- 172.21.15.146:0/2894840144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c59c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80097a0 tx=0x7fc9b802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562160e27a3b47eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:37.934 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.932+0000 7fc9c6a00640 1 -- 172.21.15.146:0/2894840144 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8037070 con 0x7fc9c8154c90 2024-09-16T01:48:37.934 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.932+0000 7fc9c6a00640 1 -- 172.21.15.146:0/2894840144 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b8032440 con 0x7fc9c8154c90 2024-09-16T01:48:37.934 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.932+0000 7fc9c6a00640 1 -- 172.21.15.146:0/2894840144 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80327f0 con 0x7fc9c8154c90 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.932+0000 7fc9cf200640 1 -- 172.21.15.146:0/2894840144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8154c90 msgr2=0x7fc9c80c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.932+0000 7fc9cf200640 1 --2- 172.21.15.146:0/2894840144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c59c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80097a0 tx=0x7fc9b802a180 comp rx=0 tx=0).stop 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.933+0000 7fc9cf200640 1 -- 172.21.15.146:0/2894840144 shutdown_connections 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.933+0000 7fc9cf200640 1 --2- 172.21.15.146:0/2894840144 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.933+0000 7fc9cf200640 1 --2- 172.21.15.146:0/2894840144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c59c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.933+0000 7fc9cf200640 1 -- 172.21.15.146:0/2894840144 >> 172.21.15.146:0/2894840144 conn(0x7fc9c806e4d0 msgr2=0x7fc9c8067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:37.935 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.933+0000 7fc9cf200640 1 -- 172.21.15.146:0/2894840144 shutdown_connections 2024-09-16T01:48:37.936 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.933+0000 7fc9cf200640 1 -- 172.21.15.146:0/2894840144 wait complete. 2024-09-16T01:48:37.936 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.934+0000 7fc9cf200640 1 Processor -- start 2024-09-16T01:48:37.936 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.934+0000 7fc9cf200640 1 -- start start 2024-09-16T01:48:37.937 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.935+0000 7fc9cf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 0x7fc9c81495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.935+0000 7fc9cf200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.935+0000 7fc9cf200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.935+0000 7fc9cf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c8155070 con 0x7fc9c80cbc00 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.935+0000 7fc9cf200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c815b7e0 con 0x7fc9c8154c90 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.935+0000 7fc9cf200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c81551f0 con 0x7fc9c815dfe0 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 0x7fc9c81495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9cd200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 0x7fc9c81495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40270/0 (socket says 172.21.15.146:40270) 2024-09-16T01:48:37.938 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 -- 172.21.15.146:0/3624555318 learned_addr learned my addr 172.21.15.146:0/3624555318 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 -- 172.21.15.146:0/3624555318 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 msgr2=0x7fc9c80c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 -- 172.21.15.146:0/3624555318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c8154c90 msgr2=0x7fc9c80c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 -- 172.21.15.146:0/3624555318 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c815da90 con 0x7fc9c80cbc00 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.936+0000 7fc9c7e00640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 0x7fc9c81495d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9b802a770 tx=0x7fc9b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8037040 con 0x7fc9c80cbc00 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b80044c0 con 0x7fc9c80cbc00 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c815dd50 con 0x7fc9c80cbc00 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8003e90 con 0x7fc9c80cbc00 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c80ca380 con 0x7fc9c80cbc00 2024-09-16T01:48:37.940 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc9b8044020 con 0x7fc9c80cbc00 2024-09-16T01:48:37.941 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.937+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc9b803d760 con 0x7fc9c80cbc00 2024-09-16T01:48:37.941 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.938+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c80c59c0 con 0x7fc9c80cbc00 2024-09-16T01:48:37.942 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:37.941+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fc9b803da80 con 0x7fc9c80cbc00 2024-09-16T01:48:37.962 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:37.960+0000 7f15a0f30040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:48:38.005 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.003+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9c80ca580 con 0x7fc9c80cbc00 2024-09-16T01:48:38.005 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.003+0000 7fc9bfe00640 1 -- 172.21.15.146:0/3624555318 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc9b803dc60 con 0x7fc9c80cbc00 2024-09-16T01:48:38.005 INFO:teuthology.misc.health.smithi146.stdout: 2024-09-16T01:48:38.006 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":10,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:36.639012+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:48:38.006 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 msgr2=0x7fc9c81495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 0x7fc9c81495d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9b802a770 tx=0x7fc9b80046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 shutdown_connections 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815dfe0 0x7fc9c80c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c8154c90 0x7fc9c80c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 --2- 172.21.15.146:0/3624555318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c80cbc00 0x7fc9c81495d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 >> 172.21.15.146:0/3624555318 conn(0x7fc9c806e4d0 msgr2=0x7fc9c80cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 shutdown_connections 2024-09-16T01:48:38.007 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:38.004+0000 7fc9cf200640 1 -- 172.21.15.146:0/3624555318 wait complete. 2024-09-16T01:48:38.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T01:48:38.019 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.016+0000 7f15a0f30040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:48:38.133 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:48:38.132+0000 7f0c77727640 -1 Falling back to public interface 2024-09-16T01:48:38.144 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.142+0000 7f15a0f30040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:48:38.168 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:48:38.167+0000 7f3e2f218640 -1 Falling back to public interface 2024-09-16T01:48:38.202 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.200+0000 7f15a0f30040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:48:38.236 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:48:38.234+0000 7f2654a0f640 -1 Falling back to public interface 2024-09-16T01:48:38.244 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:48:38.241+0000 7fedd65dd640 -1 Falling back to public interface 2024-09-16T01:48:38.260 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:48:38.259+0000 7fcaf500f640 -1 Falling back to public interface 2024-09-16T01:48:38.265 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.263+0000 7f15a0f30040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:48:38.344 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.342+0000 7f15a0f30040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:48:38.384 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:48:38.383+0000 7f5e8063e640 -1 Falling back to public interface 2024-09-16T01:48:38.395 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.393+0000 7f15a0f30040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:48:38.448 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:38.446+0000 7f15a0f30040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:48:38.490 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:48:38.489+0000 7fa893f89640 -1 Falling back to public interface 2024-09-16T01:48:38.505 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:38.504+0000 7f5115bd6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:48:39.200 INFO:tasks.ceph.mgr.y.smithi146.stderr:2024-09-16T01:48:39.199+0000 7f15a0f30040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:48:39.525 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:39.524+0000 7f50b649b640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T01:48:39.525 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-09-16T01:48:39.524+0000 7f50b649b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T01:48:41.232 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:48:41.231+0000 7ff6185db640 -1 osd.4 0 log_to_monitors true 2024-09-16T01:48:42.017 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:48:42.016+0000 7fcaf500f640 -1 osd.3 0 log_to_monitors true 2024-09-16T01:48:42.121 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:48:42.120+0000 7fa893f89640 -1 osd.0 0 log_to_monitors true 2024-09-16T01:48:42.129 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:48:42.128+0000 7f5e8063e640 -1 osd.5 0 log_to_monitors true 2024-09-16T01:48:42.196 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:48:42.195+0000 7f3e2f218640 -1 osd.7 0 log_to_monitors true 2024-09-16T01:48:42.246 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:48:42.245+0000 7fedd65dd640 -1 osd.1 0 log_to_monitors true 2024-09-16T01:48:42.410 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:48:42.409+0000 7f0c77727640 -1 osd.6 0 log_to_monitors true 2024-09-16T01:48:42.700 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:48:42.699+0000 7f2654a0f640 -1 osd.2 0 log_to_monitors true 2024-09-16T01:48:43.089 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:48:43.088+0000 7ff61456a640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T01:48:43.104 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:48:43.103+0000 7ff60c38a640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.390 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:48:44.389+0000 7f3e2abe1640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T01:48:44.404 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:48:44.403+0000 7f3e22a01640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.447 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:48:44.446+0000 7f0c736b6640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T01:48:44.447 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:48:44.446+0000 7f5e7c5cd640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T01:48:44.457 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:48:44.456+0000 7f5e743ed640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.462 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:48:44.461+0000 7f0c6b4d6640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.759 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:48:44.758+0000 7fcaf0200640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T01:48:44.766 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:48:44.765+0000 7fcae5800640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.786 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:48:44.786+0000 7fedd1e00640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T01:48:44.794 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:48:44.793+0000 7fedc7400640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.806 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:48:44.805+0000 7fa88ee00640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T01:48:44.814 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:48:44.813+0000 7fa884e00640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:44.824 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T01:48:44.985 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.984+0000 7f0b6de00640 1 Processor -- start 2024-09-16T01:48:44.986 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.985+0000 7f0b6de00640 1 -- start start 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b6de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b680cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b6de00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0b680cbc00 0x7f0b680cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b6de00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6815ac60 con 0x7f0b68154c90 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b6de00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0b680c7520 con 0x7f0b680cbc00 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b6de00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6815af40 con 0x7f0b680c60d0 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b66a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0b68154c90 0x7f0b680c59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:36128/0 (socket says 172.21.15.146:36128) 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b66a00640 1 -- 172.21.15.146:0/1620160060 learned_addr learned my addr 172.21.15.146:0/1620160060 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b67400640 1 --2- 172.21.15.146:0/1620160060 >> v2:172.21.15.161:3300/0 conn(0x7f0b680cbc00 0x7f0b680cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:44.988 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b66000640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b680cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.986+0000 7f0b65600640 1 -- 172.21.15.146:0/1620160060 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2113691558 0 0) 0x7f0b6815ac60 con 0x7f0b68154c90 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.987+0000 7f0b65600640 1 -- 172.21.15.146:0/1620160060 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b4c003520 con 0x7f0b68154c90 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.987+0000 7f0b67400640 1 -- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 msgr2=0x7f0b680cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.987+0000 7f0b67400640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b680cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.987+0000 7f0b67400640 1 -- 172.21.15.146:0/1620160060 >> v1:172.21.15.146:6789/0 conn(0x7f0b68154c90 legacy=0x7f0b680c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.987+0000 7f0b67400640 1 -- 172.21.15.146:0/1620160060 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6815b220 con 0x7f0b680cbc00 2024-09-16T01:48:44.989 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.987+0000 7f0b66000640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b680cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:44.990 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.988+0000 7f0b67400640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b680cbc00 0x7f0b680cb260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009d90 tx=0x7f0b6002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e732a057be3512 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:44.990 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.988+0000 7f0b65600640 1 -- 172.21.15.146:0/1620160060 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60007740 con 0x7f0b680cbc00 2024-09-16T01:48:44.990 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.988+0000 7f0b65600640 1 -- 172.21.15.146:0/1620160060 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b600078e0 con 0x7f0b680cbc00 2024-09-16T01:48:44.990 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.988+0000 7f0b65600640 1 -- 172.21.15.146:0/1620160060 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60007cb0 con 0x7f0b680cbc00 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 -- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b680cbc00 msgr2=0x7f0b680cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b680cbc00 0x7f0b680cb260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009d90 tx=0x7f0b6002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 -- 172.21.15.146:0/1620160060 shutdown_connections 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b680cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 --2- 172.21.15.146:0/1620160060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b680cbc00 0x7f0b680cb260 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 -- 172.21.15.146:0/1620160060 >> 172.21.15.146:0/1620160060 conn(0x7f0b6806e4d0 msgr2=0x7f0b68067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 -- 172.21.15.146:0/1620160060 shutdown_connections 2024-09-16T01:48:44.991 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.989+0000 7f0b6de00640 1 -- 172.21.15.146:0/1620160060 wait complete. 2024-09-16T01:48:44.992 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.991+0000 7f0b6de00640 1 Processor -- start 2024-09-16T01:48:44.992 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.991+0000 7f0b6de00640 1 -- start start 2024-09-16T01:48:44.994 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b6de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b681f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b6de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 0x7f0b680b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b6de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b68154c90 0x7f0b68143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b6de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68155070 con 0x7f0b680cbc00 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b6de00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6815b7e0 con 0x7f0b68154c90 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b6de00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0b681551f0 con 0x7f0b680c60d0 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.992+0000 7f0b66000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 0x7f0b680b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 0x7f0b680b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46766/0 (socket says 172.21.15.146:46766) 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b681f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 -- 172.21.15.146:0/8722984 learned_addr learned my addr 172.21.15.146:0/8722984 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b67400640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b68154c90 0x7f0b68143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 -- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 msgr2=0x7f0b681f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:44.995 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b681f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 -- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b68154c90 msgr2=0x7f0b68143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b68154c90 0x7f0b68143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 -- 172.21.15.146:0/8722984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b681f4780 con 0x7f0b680cbc00 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66a00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b681f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b66000640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 0x7f0b680b4e30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b5400e990 tx=0x7f0b5400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.993+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b54017070 con 0x7f0b680cbc00 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.994+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5400c9e0 con 0x7f0b680cbc00 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.994+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5400cdb0 con 0x7f0b680cbc00 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.994+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b68153eb0 con 0x7f0b680cbc00 2024-09-16T01:48:44.996 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.994+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b681f3a40 con 0x7f0b680cbc00 2024-09-16T01:48:44.997 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.995+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b6804ffb0 con 0x7f0b680cbc00 2024-09-16T01:48:44.997 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.996+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 4) ==== 90790+0+0 (secure 0 0 0) 0x7f0b5401a070 con 0x7f0b680cbc00 2024-09-16T01:48:44.998 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.997+0000 7f0b5f400640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0b4806c220 0x7f0b4806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:44.998 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.997+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3351+0+0 (secure 0 0 0) 0x7f0b5408a7b0 con 0x7f0b680cbc00 2024-09-16T01:48:44.998 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.997+0000 7f0b66a00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0b4806c220 0x7f0b4806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:44.998 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:44.997+0000 7f0b66a00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0b4806c220 0x7f0b4806e6e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005880 tx=0x7f0b58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:45.003 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.002+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0b54059370 con 0x7f0b680cbc00 2024-09-16T01:48:45.226 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.225+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b680ca7d0 con 0x7f0b680cbc00 2024-09-16T01:48:45.226 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.226+0000 7f0b5f400640 1 -- 172.21.15.146:0/8722984 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9024 (secure 0 0 0) 0x7f0b5405d670 con 0x7f0b680cbc00 2024-09-16T01:48:45.227 INFO:teuthology.misc.health.smithi146.stdout: 2024-09-16T01:48:45.227 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":14,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:44.531015+0000","last_up_change":"2024-09-16T01:48:44.531015+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:48:45.230 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.228+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0b4806c220 msgr2=0x7f0b4806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:45.230 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0b4806c220 0x7f0b4806e6e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005880 tx=0x7f0b58009040 comp rx=0 tx=0).stop 2024-09-16T01:48:45.231 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 msgr2=0x7f0b680b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:45.231 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 0x7f0b680b4e30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b5400e990 tx=0x7f0b5400ee50 comp rx=0 tx=0).stop 2024-09-16T01:48:45.231 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 shutdown_connections 2024-09-16T01:48:45.231 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0b4806c220 0x7f0b4806e6e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:45.231 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0b68154c90 0x7f0b68143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:45.232 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b680cbc00 0x7f0b680b4e30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:45.232 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 --2- 172.21.15.146:0/8722984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b680c60d0 0x7f0b681f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:45.232 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 >> 172.21.15.146:0/8722984 conn(0x7f0b6806e4d0 msgr2=0x7f0b680d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:45.232 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 shutdown_connections 2024-09-16T01:48:45.232 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:45.229+0000 7f0b6de00640 1 -- 172.21.15.146:0/8722984 wait complete. 2024-09-16T01:48:45.243 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-09-16T01:48:45.487 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:48:45.486+0000 7f264f200640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T01:48:45.497 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:48:45.496+0000 7f2645200640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:48:52.050 DEBUG:teuthology.orchestra.run.smithi146:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T01:48:52.194 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.193+0000 7fd8bfe00640 1 Processor -- start 2024-09-16T01:48:52.194 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.193+0000 7fd8bfe00640 1 -- start start 2024-09-16T01:48:52.195 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00cccf0 0x7fd8c00cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.195 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bfe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c015e0e0 con 0x7fd8c006c1d0 2024-09-16T01:48:52.195 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bfe00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c015e3c0 con 0x7fd8c00caca0 2024-09-16T01:48:52.195 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bfe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c00cb640 con 0x7fd8c00cccf0 2024-09-16T01:48:52.195 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bf400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd8c006c1d0 0x7fd8c006c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:36144/0 (socket says 172.21.15.146:36144) 2024-09-16T01:48:52.195 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bf400640 1 -- 172.21.15.146:0/2006322870 learned_addr learned my addr 172.21.15.146:0/2006322870 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:52.196 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.194+0000 7fd8bea00640 1 --2- 172.21.15.146:0/2006322870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00cccf0 0x7fd8c00cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.196 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778357138 0 0) 0x7fd8c015e0e0 con 0x7fd8c006c1d0 2024-09-16T01:48:52.196 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8a4003520 con 0x7fd8c006c1d0 2024-09-16T01:48:52.197 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 683149848 0 0) 0x7fd8c015e3c0 con 0x7fd8c00caca0 2024-09-16T01:48:52.197 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8c015e0e0 con 0x7fd8c00caca0 2024-09-16T01:48:52.197 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8bea00640 1 -- 172.21.15.146:0/2006322870 >> v1:172.21.15.161:6789/0 conn(0x7fd8c00caca0 legacy=0x7fd8c00cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.197 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8bea00640 1 -- 172.21.15.146:0/2006322870 >> v1:172.21.15.146:6789/0 conn(0x7fd8c006c1d0 legacy=0x7fd8c006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.197 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.195+0000 7fd8bea00640 1 -- 172.21.15.146:0/2006322870 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c015e6a0 con 0x7fd8c00cccf0 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.196+0000 7fd8bea00640 1 --2- 172.21.15.146:0/2006322870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00cccf0 0x7fd8c00cd0d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00097d0 tx=0x7fd8b002a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35fc09eb13eed99a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.196+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0037070 con 0x7fd8c00cccf0 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.196+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b0032440 con 0x7fd8c00cccf0 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.196+0000 7fd8be000640 1 -- 172.21.15.146:0/2006322870 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00327f0 con 0x7fd8c00cccf0 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 -- 172.21.15.146:0/2006322870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00cccf0 msgr2=0x7fd8c00cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/2006322870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00cccf0 0x7fd8c00cd0d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00097d0 tx=0x7fd8b002a2a0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 -- 172.21.15.146:0/2006322870 shutdown_connections 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/2006322870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00cccf0 0x7fd8c00cd0d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 -- 172.21.15.146:0/2006322870 >> 172.21.15.146:0/2006322870 conn(0x7fd8c0067aa0 msgr2=0x7fd8c0067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:52.198 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 -- 172.21.15.146:0/2006322870 shutdown_connections 2024-09-16T01:48:52.199 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.197+0000 7fd8bfe00640 1 -- 172.21.15.146:0/2006322870 wait complete. 2024-09-16T01:48:52.199 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.199+0000 7fd8bfe00640 1 Processor -- start 2024-09-16T01:48:52.199 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.199+0000 7fd8bfe00640 1 -- start start 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 0x7fd8c01e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00caca0 0x7fd8c01e3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c00cccf0 0x7fd8c01f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c006bcb0 con 0x7fd8c006c1d0 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bfe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c015ec60 con 0x7fd8c00cccf0 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bfe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c015dd80 con 0x7fd8c00caca0 2024-09-16T01:48:52.204 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 0x7fd8c01e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 0x7fd8c01e46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46782/0 (socket says 172.21.15.146:46782) 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 -- 172.21.15.146:0/3176305166 learned_addr learned my addr 172.21.15.146:0/3176305166 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bf400640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c00cccf0 0x7fd8c01f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8b6000640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00caca0 0x7fd8c01e3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 -- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00caca0 msgr2=0x7fd8c01e3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00caca0 0x7fd8c01e3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 -- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c00cccf0 msgr2=0x7fd8c01f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c00cccf0 0x7fd8c01f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bea00640 1 -- 172.21.15.146:0/3176305166 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c00d0570 con 0x7fd8c006c1d0 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8b6000640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00caca0 0x7fd8c01e3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.200+0000 7fd8bf400640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c00cccf0 0x7fd8c01f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.201+0000 7fd8bea00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 0x7fd8c01e46b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0035040 tx=0x7fd8b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.201+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0037040 con 0x7fd8c006c1d0 2024-09-16T01:48:52.205 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.201+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b0042070 con 0x7fd8c006c1d0 2024-09-16T01:48:52.206 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.201+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c00d0850 con 0x7fd8c006c1d0 2024-09-16T01:48:52.206 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.201+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b003d860 con 0x7fd8c006c1d0 2024-09-16T01:48:52.206 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.201+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c00d0e10 con 0x7fd8c006c1d0 2024-09-16T01:48:52.207 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.202+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c015e3c0 con 0x7fd8c006c1d0 2024-09-16T01:48:52.207 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.203+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd8b003da00 con 0x7fd8c006c1d0 2024-09-16T01:48:52.207 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.204+0000 7fd8b7e00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89c06c2e0 0x7fd89c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.207 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.204+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fd8b00ab180 con 0x7fd8c006c1d0 2024-09-16T01:48:52.207 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.204+0000 7fd8b6000640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89c06c2e0 0x7fd89c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.207 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.204+0000 7fd8b6000640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89c06c2e0 0x7fd89c06e7a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd8c01e5350 tx=0x7fd8a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:52.209 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.209+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd8b004b020 con 0x7fd8c006c1d0 2024-09-16T01:48:52.385 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.383+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8c00cd550 con 0x7fd8c006c1d0 2024-09-16T01:48:52.385 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.384+0000 7fd8b7e00640 1 -- 172.21.15.146:0/3176305166 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12920 (secure 0 0 0) 0x7fd8b0030e20 con 0x7fd8c006c1d0 2024-09-16T01:48:52.386 INFO:teuthology.misc.health.smithi146.stdout: 2024-09-16T01:48:52.386 INFO:teuthology.misc.health.smithi146.stdout:{"epoch":18,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:48.761939+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:48:52.388 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89c06c2e0 msgr2=0x7fd89c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.388 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89c06c2e0 0x7fd89c06e7a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd8c01e5350 tx=0x7fd8a8009040 comp rx=0 tx=0).stop 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 msgr2=0x7fd8c01e46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 0x7fd8c01e46b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0035040 tx=0x7fd8b00046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 shutdown_connections 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89c06c2e0 0x7fd89c06e7a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8c00cccf0 0x7fd8c01f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8c00caca0 0x7fd8c01e3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 --2- 172.21.15.146:0/3176305166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8c006c1d0 0x7fd8c01e46b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 >> 172.21.15.146:0/3176305166 conn(0x7fd8c0067aa0 msgr2=0x7fd8c0153c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 shutdown_connections 2024-09-16T01:48:52.389 INFO:teuthology.misc.health.smithi146.stderr:2024-09-16T01:48:52.387+0000 7fd8bfe00640 1 -- 172.21.15.146:0/3176305166 wait complete. 2024-09-16T01:48:52.401 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-16T01:48:52.401 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T01:48:52.402 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:48:52.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:48:52.406 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:48:52.406 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:48:52.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:48:52.409 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:48:52.410 INFO:tasks.ceph.mds.e:Restarting daemon 2024-09-16T01:48:52.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-09-16T01:48:52.414 INFO:tasks.ceph.mds.e:Started 2024-09-16T01:48:52.414 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:48:52.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:48:52.417 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:48:52.417 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:48:52.417 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:48:52.419 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:48:52.419 INFO:tasks.ceph.mds.f:Restarting daemon 2024-09-16T01:48:52.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-09-16T01:48:52.422 INFO:tasks.ceph.mds.f:Started 2024-09-16T01:48:52.422 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-16T01:48:52.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:48:52.504 INFO:tasks.ceph.mds.b.smithi161.stdout:starting mds.b at 2024-09-16T01:48:52.504 INFO:tasks.ceph.mds.d.smithi161.stdout:starting mds.d at 2024-09-16T01:48:52.505 INFO:tasks.ceph.mds.f.smithi161.stdout:starting mds.f at 2024-09-16T01:48:52.544 INFO:tasks.ceph.mds.a.smithi146.stdout:starting mds.a at 2024-09-16T01:48:52.549 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-09-16T01:48:52.549 INFO:tasks.ceph.mds.e.smithi146.stdout:starting mds.e at 2024-09-16T01:48:52.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.593+0000 7ff51f200640 1 Processor -- start 2024-09-16T01:48:52.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.593+0000 7ff51f200640 1 -- start start 2024-09-16T01:48:52.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.594+0000 7ff51f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff52015b340 0x7ff520166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.594+0000 7ff51f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff52006d140 con 0x7ff5200c75a0 2024-09-16T01:48:52.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.594+0000 7ff51f200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff52006d420 con 0x7ff5200c6b70 2024-09-16T01:48:52.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.594+0000 7ff51f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff5200cab20 con 0x7ff52015b340 2024-09-16T01:48:52.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.594+0000 7ff51de00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff5200c6b70 0x7ff5200cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34096/0 (socket says 172.21.15.146:34096) 2024-09-16T01:48:52.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.595+0000 7ff51d400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff5200c75a0 0x7ff5200cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:36174/0 (socket says 172.21.15.146:36174) 2024-09-16T01:48:52.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.595+0000 7ff51d400640 1 -- 172.21.15.146:0/3307036255 learned_addr learned my addr 172.21.15.146:0/3307036255 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:52.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.595+0000 7ff51e800640 1 --2- 172.21.15.146:0/3307036255 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff52015b340 0x7ff520166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.595+0000 7ff51e800640 1 -- 172.21.15.146:0/3307036255 >> v1:172.21.15.161:6789/0 conn(0x7ff5200c6b70 legacy=0x7ff5200cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:52.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.595+0000 7ff51e800640 1 -- 172.21.15.146:0/3307036255 >> v1:172.21.15.146:6789/0 conn(0x7ff5200c75a0 legacy=0x7ff5200cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.595+0000 7ff51e800640 1 -- 172.21.15.146:0/3307036255 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52006d700 con 0x7ff52015b340 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.596+0000 7ff51e800640 1 --2- 172.21.15.146:0/3307036255 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff52015b340 0x7ff520166170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff518009dc0 tx=0x7ff51802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4baf3e130d6cfeb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.596+0000 7ff51ca00640 1 -- 172.21.15.146:0/3307036255 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518007740 con 0x7ff52015b340 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.596+0000 7ff51ca00640 1 -- 172.21.15.146:0/3307036255 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5180078e0 con 0x7ff52015b340 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.596+0000 7ff51ca00640 1 -- 172.21.15.146:0/3307036255 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518007cb0 con 0x7ff52015b340 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 -- 172.21.15.146:0/3307036255 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff52015b340 msgr2=0x7ff520166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 --2- 172.21.15.146:0/3307036255 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff52015b340 0x7ff520166170 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff518009dc0 tx=0x7ff51802a5f0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 -- 172.21.15.146:0/3307036255 shutdown_connections 2024-09-16T01:48:52.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 --2- 172.21.15.146:0/3307036255 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff52015b340 0x7ff520166170 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 -- 172.21.15.146:0/3307036255 >> 172.21.15.146:0/3307036255 conn(0x7ff5200ccc20 msgr2=0x7ff5200cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:52.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 -- 172.21.15.146:0/3307036255 shutdown_connections 2024-09-16T01:48:52.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.597+0000 7ff51f200640 1 -- 172.21.15.146:0/3307036255 wait complete. 2024-09-16T01:48:52.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.598+0000 7ff51f200640 1 Processor -- start 2024-09-16T01:48:52.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 -- start start 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5200c6b70 0x7ff52006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 0x7ff5201f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff52015b340 0x7ff5201f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff52015b7c0 con 0x7ff52015b340 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff5200cb050 con 0x7ff5200c75a0 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.599+0000 7ff51f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff5200cbc80 con 0x7ff5200c6b70 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff52015b340 0x7ff5201f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 0x7ff5201f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5200c6b70 0x7ff52006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff52015b340 0x7ff5201f7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46826/0 (socket says 172.21.15.146:46826) 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 0x7ff5201f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34802/0 (socket says 172.21.15.146:34802) 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51e800640 1 -- 172.21.15.146:0/1413629484 learned_addr learned my addr 172.21.15.146:0/1413629484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 -- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5200c6b70 msgr2=0x7ff52006f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5200c6b70 0x7ff52006f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 -- 172.21.15.146:0/1413629484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff52015b340 msgr2=0x7ff5201f7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff52015b340 0x7ff5201f7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 -- 172.21.15.146:0/1413629484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5201f8d90 con 0x7ff5200c75a0 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51de00640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5200c6b70 0x7ff52006f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.600+0000 7ff51d400640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 0x7ff5201f2b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff50c000f60 tx=0x7ff50c00c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:52.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.601+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c017070 con 0x7ff5200c75a0 2024-09-16T01:48:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.601+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff50c005ce0 con 0x7ff5200c75a0 2024-09-16T01:48:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.601+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c007310 con 0x7ff5200c75a0 2024-09-16T01:48:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.601+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5201f6ca0 con 0x7ff5200c75a0 2024-09-16T01:48:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.601+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201f7260 con 0x7ff5200c75a0 2024-09-16T01:48:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.602+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52006cd20 con 0x7ff5200c75a0 2024-09-16T01:48:52.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.603+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff50c01a070 con 0x7ff5200c75a0 2024-09-16T01:48:52.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.604+0000 7ff516000640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff50006c2e0 0x7ff50006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:52.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.604+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7ff50c08a410 con 0x7ff5200c75a0 2024-09-16T01:48:52.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.604+0000 7ff51de00640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff50006c2e0 0x7ff50006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:52.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.605+0000 7ff51de00640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff50006c2e0 0x7ff50006e7a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff510006fd0 tx=0x7ff510009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:52.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.608+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff50c058750 con 0x7ff5200c75a0 2024-09-16T01:48:52.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.819+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff52006da90 con 0x7ff5200c75a0 2024-09-16T01:48:52.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.819+0000 7ff516000640 1 -- 172.21.15.146:0/1413629484 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7ff50c05ca50 con 0x7ff5200c75a0 2024-09-16T01:48:52.821 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:48:52.821 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"btime":"2024-09-16T01:48:52:508848+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T01:48:52.821 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 2 2024-09-16T01:48:52.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff50006c2e0 msgr2=0x7ff50006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff50006c2e0 0x7ff50006e7a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff510006fd0 tx=0x7ff510009060 comp rx=0 tx=0).stop 2024-09-16T01:48:52.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 msgr2=0x7ff5201f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:52.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 0x7ff5201f2b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff50c000f60 tx=0x7ff50c00c760 comp rx=0 tx=0).stop 2024-09-16T01:48:52.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 shutdown_connections 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff50006c2e0 0x7ff50006e7a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff52015b340 0x7ff5201f7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5200c75a0 0x7ff5201f2b80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 --2- 172.21.15.146:0/1413629484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5200c6b70 0x7ff52006f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 >> 172.21.15.146:0/1413629484 conn(0x7ff5200ccc20 msgr2=0x7ff52015a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 shutdown_connections 2024-09-16T01:48:52.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:52.822+0000 7ff51f200640 1 -- 172.21.15.146:0/1413629484 wait complete. 2024-09-16T01:48:54.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:48:55.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.003+0000 7fa3da800640 1 Processor -- start 2024-09-16T01:48:55.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.003+0000 7fa3da800640 1 -- start start 2024-09-16T01:48:55.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.004+0000 7fa3da800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c6b70 0x7fa3d415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.004+0000 7fa3da800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3d40cc040 0x7fa3d40cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3da800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3d406d140 con 0x7fa3d40c75a0 2024-09-16T01:48:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3da800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa3d415b880 con 0x7fa3d40cc040 2024-09-16T01:48:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3da800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d40cab20 con 0x7fa3d40c6b70 2024-09-16T01:48:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d8a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa3d40c75a0 0x7fa3d40cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56664/0 (socket says 172.21.15.146:56664) 2024-09-16T01:48:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d8a00640 1 -- 172.21.15.146:0/2679552250 learned_addr learned my addr 172.21.15.146:0/2679552250 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9400640 1 --2- 172.21.15.146:0/2679552250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c6b70 0x7fa3d415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9e00640 1 --2- 172.21.15.146:0/2679552250 >> v2:172.21.15.161:3300/0 conn(0x7fa3d40cc040 0x7fa3d40cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d3e00640 1 -- 172.21.15.146:0/2679552250 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3923745952 0 0) 0x7fa3d406d140 con 0x7fa3d40c75a0 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d3e00640 1 -- 172.21.15.146:0/2679552250 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3bc003520 con 0x7fa3d40c75a0 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9400640 1 -- 172.21.15.146:0/2679552250 >> v2:172.21.15.161:3300/0 conn(0x7fa3d40cc040 msgr2=0x7fa3d40cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9400640 1 --2- 172.21.15.146:0/2679552250 >> v2:172.21.15.161:3300/0 conn(0x7fa3d40cc040 0x7fa3d40cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9400640 1 -- 172.21.15.146:0/2679552250 >> v1:172.21.15.146:6789/0 conn(0x7fa3d40c75a0 legacy=0x7fa3d40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9400640 1 -- 172.21.15.146:0/2679552250 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d406d420 con 0x7fa3d40c6b70 2024-09-16T01:48:55.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.005+0000 7fa3d9e00640 1 --2- 172.21.15.146:0/2679552250 >> v2:172.21.15.161:3300/0 conn(0x7fa3d40cc040 0x7fa3d40cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.006+0000 7fa3d9400640 1 --2- 172.21.15.146:0/2679552250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c6b70 0x7fa3d415b340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097a0 tx=0x7fa3c802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f82f83aa0e25408 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3d3e00640 1 -- 172.21.15.146:0/2679552250 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8037070 con 0x7fa3d40c6b70 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3d3e00640 1 -- 172.21.15.146:0/2679552250 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c802ae50 con 0x7fa3d40c6b70 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3d3e00640 1 -- 172.21.15.146:0/2679552250 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8032640 con 0x7fa3d40c6b70 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 -- 172.21.15.146:0/2679552250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c6b70 msgr2=0x7fa3d415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 --2- 172.21.15.146:0/2679552250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c6b70 0x7fa3d415b340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097a0 tx=0x7fa3c802a100 comp rx=0 tx=0).stop 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 -- 172.21.15.146:0/2679552250 shutdown_connections 2024-09-16T01:48:55.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 --2- 172.21.15.146:0/2679552250 >> v2:172.21.15.161:3300/0 conn(0x7fa3d40cc040 0x7fa3d40cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 --2- 172.21.15.146:0/2679552250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c6b70 0x7fa3d415b340 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 -- 172.21.15.146:0/2679552250 >> 172.21.15.146:0/2679552250 conn(0x7fa3d40ccc20 msgr2=0x7fa3d40cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:55.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 -- 172.21.15.146:0/2679552250 shutdown_connections 2024-09-16T01:48:55.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.007+0000 7fa3da800640 1 -- 172.21.15.146:0/2679552250 wait complete. 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.009+0000 7fa3da800640 1 Processor -- start 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.009+0000 7fa3da800640 1 -- start start 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3da800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3d40c6b70 0x7fa3d415d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3da800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c75a0 0x7fa3d415e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3da800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 0x7fa3d4156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3da800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d40cb050 con 0x7fa3d40cc040 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3da800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d40cad70 con 0x7fa3d40c6b70 2024-09-16T01:48:55.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3da800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d40cbc80 con 0x7fa3d40c75a0 2024-09-16T01:48:55.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 0x7fa3d4156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c75a0 0x7fa3d415e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 0x7fa3d4156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45796/0 (socket says 172.21.15.146:45796) 2024-09-16T01:48:55.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 -- 172.21.15.146:0/380652082 learned_addr learned my addr 172.21.15.146:0/380652082 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:55.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3d40c6b70 0x7fa3d415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 -- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c75a0 msgr2=0x7fa3d415e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c75a0 0x7fa3d415e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 -- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3d40c6b70 msgr2=0x7fa3d415d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3d40c6b70 0x7fa3d415d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.010+0000 7fa3d9e00640 1 -- 172.21.15.146:0/380652082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d40cd730 con 0x7fa3d40cc040 2024-09-16T01:48:55.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.011+0000 7fa3d9e00640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 0x7fa3d4156f00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00efc0 tx=0x7fa3cc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.011+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc018020 con 0x7fa3d40cc040 2024-09-16T01:48:55.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.011+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3cc014480 con 0x7fa3d40cc040 2024-09-16T01:48:55.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.011+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d415bbc0 con 0x7fa3d40cc040 2024-09-16T01:48:55.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.011+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc014870 con 0x7fa3d40cc040 2024-09-16T01:48:55.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.011+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d415c180 con 0x7fa3d40cc040 2024-09-16T01:48:55.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.012+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d406cd20 con 0x7fa3d40cc040 2024-09-16T01:48:55.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.013+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa3cc005d10 con 0x7fa3d40cc040 2024-09-16T01:48:55.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.014+0000 7fa3d1600640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa3b406c2b0 0x7fa3b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.014+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fa3cc08bf20 con 0x7fa3d40cc040 2024-09-16T01:48:55.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.014+0000 7fa3d9400640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa3b406c2b0 0x7fa3b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.015+0000 7fa3d9400640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa3b406c2b0 0x7fa3b406e770 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8034000 tx=0x7fa3c802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.019+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa3cc022030 con 0x7fa3d40cc040 2024-09-16T01:48:55.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.230+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3d40ca280 con 0x7fa3d40cc040 2024-09-16T01:48:55.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.231+0000 7fa3d1600640 1 -- 172.21.15.146:0/380652082 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5566 (secure 0 0 0) 0x7fa3cc010040 con 0x7fa3d40cc040 2024-09-16T01:48:55.232 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:48:55.232 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"btime":"2024-09-16T01:48:53:510653+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4277,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4288,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-16T01:48:55.232 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 3 2024-09-16T01:48:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa3b406c2b0 msgr2=0x7fa3b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa3b406c2b0 0x7fa3b406e770 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8034000 tx=0x7fa3c802e000 comp rx=0 tx=0).stop 2024-09-16T01:48:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 msgr2=0x7fa3d4156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 0x7fa3d4156f00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00efc0 tx=0x7fa3cc009f90 comp rx=0 tx=0).stop 2024-09-16T01:48:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 shutdown_connections 2024-09-16T01:48:55.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa3b406c2b0 0x7fa3b406e770 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3d40cc040 0x7fa3d4156f00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3d40c75a0 0x7fa3d415e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 --2- 172.21.15.146:0/380652082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3d40c6b70 0x7fa3d415d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 >> 172.21.15.146:0/380652082 conn(0x7fa3d40ccc20 msgr2=0x7fa3d40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 shutdown_connections 2024-09-16T01:48:55.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.233+0000 7fa3da800640 1 -- 172.21.15.146:0/380652082 wait complete. 2024-09-16T01:48:55.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:48:55.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:48:55.299 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:48:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.405+0000 7f8154c00640 1 Processor -- start 2024-09-16T01:48:55.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.405+0000 7f8154c00640 1 -- start start 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f8154c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8150154f10 0x7f81500c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f8154c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f815015a9b0 con 0x7f815006f610 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f8154c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f815015ac70 con 0x7f81500c6cd0 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f8154c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f81500c7f20 con 0x7f8150154f10 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8150154f10 0x7f81500c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814e800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f815006f610 0x7f8150165330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56678/0 (socket says 172.21.15.146:56678) 2024-09-16T01:48:55.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814e800640 1 -- 172.21.15.146:0/1464818315 learned_addr learned my addr 172.21.15.146:0/1464818315 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:55.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814de00640 1 -- 172.21.15.146:0/1464818315 >> v1:172.21.15.161:6789/0 conn(0x7f81500c6cd0 legacy=0x7f815006ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814de00640 1 -- 172.21.15.146:0/1464818315 >> v1:172.21.15.146:6789/0 conn(0x7f815006f610 legacy=0x7f8150165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814de00640 1 -- 172.21.15.146:0/1464818315 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815015af50 con 0x7f8150154f10 2024-09-16T01:48:55.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.407+0000 7f814d400640 1 -- 172.21.15.146:0/1464818315 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205860308 0 0) 0x7f815015a9b0 con 0x7f815006f610 2024-09-16T01:48:55.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.408+0000 7f814de00640 1 --2- 172.21.15.146:0/1464818315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8150154f10 0x7f81500c8c40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81400097a0 tx=0x7f814002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b64f4f714af02e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f814d400640 1 -- 172.21.15.146:0/1464818315 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140037070 con 0x7f8150154f10 2024-09-16T01:48:55.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f814d400640 1 -- 172.21.15.146:0/1464818315 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8140032640 con 0x7f8150154f10 2024-09-16T01:48:55.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f814d400640 1 -- 172.21.15.146:0/1464818315 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140032a10 con 0x7f8150154f10 2024-09-16T01:48:55.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 -- 172.21.15.146:0/1464818315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8150154f10 msgr2=0x7f81500c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 --2- 172.21.15.146:0/1464818315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8150154f10 0x7f81500c8c40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f81400097a0 tx=0x7f814002a2c0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 -- 172.21.15.146:0/1464818315 shutdown_connections 2024-09-16T01:48:55.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 --2- 172.21.15.146:0/1464818315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8150154f10 0x7f81500c8c40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 -- 172.21.15.146:0/1464818315 >> 172.21.15.146:0/1464818315 conn(0x7f815006bc10 msgr2=0x7f815006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:55.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 -- 172.21.15.146:0/1464818315 shutdown_connections 2024-09-16T01:48:55.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.409+0000 7f8154c00640 1 -- 172.21.15.146:0/1464818315 wait complete. 2024-09-16T01:48:55.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.411+0000 7f8154c00640 1 Processor -- start 2024-09-16T01:48:55.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.411+0000 7f8154c00640 1 -- start start 2024-09-16T01:48:55.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8154c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 0x7f81501e6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8154c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81500c6cd0 0x7f81500ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8154c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8150154f10 0x7f81500c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8154c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f815015b510 con 0x7f815006f610 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8154c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f815015b7f0 con 0x7f8150154f10 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8154c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f81500c8130 con 0x7f81500c6cd0 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 0x7f81501e6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8150154f10 0x7f81500c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f8145400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81500c6cd0 0x7f81500ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 0x7f81501e6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45814/0 (socket says 172.21.15.146:45814) 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 -- 172.21.15.146:0/2995733187 learned_addr learned my addr 172.21.15.146:0/2995733187 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 -- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81500c6cd0 msgr2=0x7f81500ca720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81500c6cd0 0x7f81500ca720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 -- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8150154f10 msgr2=0x7f81500c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8150154f10 0x7f81500c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.412+0000 7f814de00640 1 -- 172.21.15.146:0/2995733187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81501f4b70 con 0x7f815006f610 2024-09-16T01:48:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.413+0000 7f814de00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 0x7f81501e6470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81400097a0 tx=0x7f81400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.413+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140037050 con 0x7f815006f610 2024-09-16T01:48:55.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.413+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8140038040 con 0x7f815006f610 2024-09-16T01:48:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.413+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8140004490 con 0x7f815006f610 2024-09-16T01:48:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.413+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81500cb660 con 0x7f815006f610 2024-09-16T01:48:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.413+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81501e7e70 con 0x7f815006f610 2024-09-16T01:48:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.414+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81500c8530 con 0x7f815006f610 2024-09-16T01:48:55.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.415+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8140030900 con 0x7f815006f610 2024-09-16T01:48:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.416+0000 7f8146a00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f812c06c2b0 0x7f812c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.416+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f81400abd50 con 0x7f815006f610 2024-09-16T01:48:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.416+0000 7f8145400640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f812c06c2b0 0x7f812c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.417+0000 7f8145400640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f812c06c2b0 0x7f812c06e770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81500bf0f0 tx=0x7f8138009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.421+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f814007a240 con 0x7f815006f610 2024-09-16T01:48:55.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.600+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f81500c7130 con 0x7f815006f610 2024-09-16T01:48:55.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.600+0000 7f8146a00640 1 -- 172.21.15.146:0/2995733187 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f814007e540 con 0x7f815006f610 2024-09-16T01:48:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:48:55.602 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2024-09-16T01:48:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:48:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "poolnum": 1, 2024-09-16T01:48:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "poolname": ".mgr" 2024-09-16T01:48:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:48:55.602 INFO:teuthology.orchestra.run.smithi146.stdout:] 2024-09-16T01:48:55.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f812c06c2b0 msgr2=0x7f812c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f812c06c2b0 0x7f812c06e770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81500bf0f0 tx=0x7f8138009040 comp rx=0 tx=0).stop 2024-09-16T01:48:55.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 msgr2=0x7f81501e6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 0x7f81501e6470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81400097a0 tx=0x7f81400046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 shutdown_connections 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f812c06c2b0 0x7f812c06e770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8150154f10 0x7f81500c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81500c6cd0 0x7f81500ca720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 --2- 172.21.15.146:0/2995733187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815006f610 0x7f81501e6470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 >> 172.21.15.146:0/2995733187 conn(0x7f815006bc10 msgr2=0x7f8150071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 shutdown_connections 2024-09-16T01:48:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.602+0000 7f8154c00640 1 -- 172.21.15.146:0/2995733187 wait complete. 2024-09-16T01:48:55.619 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:48:55.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:48:55.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.776+0000 7f578ac00640 1 Processor -- start 2024-09-16T01:48:55.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.776+0000 7f578ac00640 1 -- start start 2024-09-16T01:48:55.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f578ac00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57840c64f0 0x7f57840c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f578ac00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840cbfe0 0x7f578415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f578ac00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578415ab30 con 0x7f57840c8ca0 2024-09-16T01:48:55.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f578ac00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f57840c82d0 con 0x7f57840c64f0 2024-09-16T01:48:55.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f578ac00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f578415ae10 con 0x7f57840cbfe0 2024-09-16T01:48:55.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f5788e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f57840c8ca0 0x7f578415dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56694/0 (socket says 172.21.15.146:56694) 2024-09-16T01:48:55.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f5788e00640 1 -- 172.21.15.146:0/2251250031 learned_addr learned my addr 172.21.15.146:0/2251250031 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:55.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f578a200640 1 --2- 172.21.15.146:0/2251250031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840cbfe0 0x7f578415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.777+0000 7f5789800640 1 --2- 172.21.15.146:0/2251250031 >> v2:172.21.15.161:3300/0 conn(0x7f57840c64f0 0x7f57840c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f5783e00640 1 -- 172.21.15.146:0/2251250031 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325646812 0 0) 0x7f578415ab30 con 0x7f57840c8ca0 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f5783e00640 1 -- 172.21.15.146:0/2251250031 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f576c003520 con 0x7f57840c8ca0 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f578a200640 1 -- 172.21.15.146:0/2251250031 >> v2:172.21.15.161:3300/0 conn(0x7f57840c64f0 msgr2=0x7f57840c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f578a200640 1 --2- 172.21.15.146:0/2251250031 >> v2:172.21.15.161:3300/0 conn(0x7f57840c64f0 0x7f57840c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f578a200640 1 -- 172.21.15.146:0/2251250031 >> v1:172.21.15.146:6789/0 conn(0x7f57840c8ca0 legacy=0x7f578415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f578a200640 1 -- 172.21.15.146:0/2251250031 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578415b0f0 con 0x7f57840cbfe0 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.778+0000 7f5789800640 1 --2- 172.21.15.146:0/2251250031 >> v2:172.21.15.161:3300/0 conn(0x7f57840c64f0 0x7f57840c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.779+0000 7f578a200640 1 --2- 172.21.15.146:0/2251250031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840cbfe0 0x7f578415e4a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f577c0097a0 tx=0x7f577c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae405ca54e19f74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f5783e00640 1 -- 172.21.15.146:0/2251250031 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c037070 con 0x7f57840cbfe0 2024-09-16T01:48:55.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f5783e00640 1 -- 172.21.15.146:0/2251250031 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f577c032440 con 0x7f57840cbfe0 2024-09-16T01:48:55.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f5783e00640 1 -- 172.21.15.146:0/2251250031 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c0327f0 con 0x7f57840cbfe0 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f578ac00640 1 -- 172.21.15.146:0/2251250031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840cbfe0 msgr2=0x7f578415e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f578ac00640 1 --2- 172.21.15.146:0/2251250031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840cbfe0 0x7f578415e4a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f577c0097a0 tx=0x7f577c02a210 comp rx=0 tx=0).stop 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f578ac00640 1 -- 172.21.15.146:0/2251250031 shutdown_connections 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f578ac00640 1 --2- 172.21.15.146:0/2251250031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840cbfe0 0x7f578415e4a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f578ac00640 1 --2- 172.21.15.146:0/2251250031 >> v2:172.21.15.161:3300/0 conn(0x7f57840c64f0 0x7f57840c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.782+0000 7f578ac00640 1 -- 172.21.15.146:0/2251250031 >> 172.21.15.146:0/2251250031 conn(0x7f57840d0170 msgr2=0x7f57840d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.783+0000 7f578ac00640 1 -- 172.21.15.146:0/2251250031 shutdown_connections 2024-09-16T01:48:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.783+0000 7f578ac00640 1 -- 172.21.15.146:0/2251250031 wait complete. 2024-09-16T01:48:55.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.784+0000 7f578ac00640 1 Processor -- start 2024-09-16T01:48:55.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.784+0000 7f578ac00640 1 -- start start 2024-09-16T01:48:55.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578ac00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 0x7f578415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57840c8ca0 0x7f578414e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578ac00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 0x7f578415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f57840c90a0 con 0x7f57840c8ca0 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578ac00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f578415b6b0 con 0x7f57840cbfe0 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578ac00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f578415a820 con 0x7f57840c64f0 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f5788e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57840c8ca0 0x7f578414e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f5789800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 0x7f578415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578a200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 0x7f578415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578a200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 0x7f578415cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:42470/0 (socket says 172.21.15.146:42470) 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f578a200640 1 -- 172.21.15.146:0/4183447772 learned_addr learned my addr 172.21.15.146:0/4183447772 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.785+0000 7f5789800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 0x7f578415d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:34846/0 (socket says 172.21.15.146:34846) 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578a200640 1 -- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 msgr2=0x7f578415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578a200640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 0x7f578415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578a200640 1 -- 172.21.15.146:0/4183447772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57840c8ca0 msgr2=0x7f578414e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578a200640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57840c8ca0 0x7f578414e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578a200640 1 -- 172.21.15.146:0/4183447772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578414a750 con 0x7f57840cbfe0 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f5789800640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 0x7f578415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578a200640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 0x7f578415cee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f577c0097a0 tx=0x7f577c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c041020 con 0x7f57840cbfe0 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f577c03d6e0 con 0x7f57840cbfe0 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f578415e430 con 0x7f57840cbfe0 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.786+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c037040 con 0x7f57840cbfe0 2024-09-16T01:48:55.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.787+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57841f02b0 con 0x7f57840cbfe0 2024-09-16T01:48:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.787+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57840cc470 con 0x7f57840cbfe0 2024-09-16T01:48:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.788+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f577c044020 con 0x7f57840cbfe0 2024-09-16T01:48:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.789+0000 7f5781600640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f576406c2b0 0x7f576406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.789+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f577c0ab760 con 0x7f57840cbfe0 2024-09-16T01:48:55.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.789+0000 7f5789800640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f576406c2b0 0x7f576406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:55.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.790+0000 7f5789800640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f576406c2b0 0x7f576406e770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5778009fd0 tx=0x7f577800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:55.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.794+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f577c04b030 con 0x7f57840cbfe0 2024-09-16T01:48:55.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:55.968+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f57840c8760 con 0x7f57840cbfe0 2024-09-16T01:48:56.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.790+0000 7f5781600640 1 -- 172.21.15.146:0/4183447772 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7f577c079ab0 con 0x7f57840cbfe0 2024-09-16T01:48:56.792 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' created 2024-09-16T01:48:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.795+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f576406c2b0 msgr2=0x7f576406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.795+0000 7f578ac00640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f576406c2b0 0x7f576406e770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5778009fd0 tx=0x7f577800b040 comp rx=0 tx=0).stop 2024-09-16T01:48:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.795+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 msgr2=0x7f578415cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.795+0000 7f578ac00640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 0x7f578415cee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f577c0097a0 tx=0x7f577c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 shutdown_connections 2024-09-16T01:48:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f576406c2b0 0x7f576406e770 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57840cbfe0 0x7f578415cee0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57840c8ca0 0x7f578414e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 --2- 172.21.15.146:0/4183447772 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57840c64f0 0x7f578415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 >> 172.21.15.146:0/4183447772 conn(0x7f57840d0170 msgr2=0x7f57840ccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.796+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 shutdown_connections 2024-09-16T01:48:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.797+0000 7f578ac00640 1 -- 172.21.15.146:0/4183447772 wait complete. 2024-09-16T01:48:56.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:48:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.976+0000 7fc5bde00640 1 Processor -- start 2024-09-16T01:48:56.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.976+0000 7fc5bde00640 1 -- start start 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5bde00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b80c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5bde00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5b8073100 0x7fc5b8151550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5bde00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b80c5780 con 0x7fc5b8072d20 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5bde00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc5b80c5a60 con 0x7fc5b8073100 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5bde00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b80c5d40 con 0x7fc5b80c8d50 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5ae000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b80c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5b7400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5b8073100 0x7fc5b8151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5ae000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b80c37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:34848/0 (socket says 172.21.15.146:34848) 2024-09-16T01:48:56.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5ae000640 1 -- 172.21.15.146:0/458555360 learned_addr learned my addr 172.21.15.146:0/458555360 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.978+0000 7fc5b7400640 1 -- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 msgr2=0x7fc5b80c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.979+0000 7fc5b7400640 1 --2- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b80c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.979+0000 7fc5b7400640 1 -- 172.21.15.146:0/458555360 >> v1:172.21.15.161:6789/0 conn(0x7fc5b8072d20 legacy=0x7fc5b80c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.979+0000 7fc5b7400640 1 -- 172.21.15.146:0/458555360 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b80c75f0 con 0x7fc5b8073100 2024-09-16T01:48:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.979+0000 7fc5b6000640 1 -- 172.21.15.146:0/458555360 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 708736910 0 0) 0x7fc5b80c5780 con 0x7fc5b8072d20 2024-09-16T01:48:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.979+0000 7fc5ae000640 1 --2- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b80c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.979+0000 7fc5b7400640 1 --2- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b8151550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009d90 tx=0x7fc5b002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c4ad78779ee08ef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.980+0000 7fc5b6000640 1 -- 172.21.15.146:0/458555360 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0007740 con 0x7fc5b8073100 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.980+0000 7fc5b6000640 1 -- 172.21.15.146:0/458555360 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b00078e0 con 0x7fc5b8073100 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.980+0000 7fc5b6000640 1 -- 172.21.15.146:0/458555360 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0007cb0 con 0x7fc5b8073100 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.980+0000 7fc5bde00640 1 -- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 msgr2=0x7fc5b8151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.980+0000 7fc5bde00640 1 --2- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b8151550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009d90 tx=0x7fc5b002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.981+0000 7fc5bde00640 1 -- 172.21.15.146:0/458555360 shutdown_connections 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.981+0000 7fc5bde00640 1 --2- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b8151550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.981+0000 7fc5bde00640 1 --2- 172.21.15.146:0/458555360 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b80c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.981+0000 7fc5bde00640 1 -- 172.21.15.146:0/458555360 >> 172.21.15.146:0/458555360 conn(0x7fc5b806e670 msgr2=0x7fc5b806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.981+0000 7fc5bde00640 1 -- 172.21.15.146:0/458555360 shutdown_connections 2024-09-16T01:48:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.981+0000 7fc5bde00640 1 -- 172.21.15.146:0/458555360 wait complete. 2024-09-16T01:48:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.982+0000 7fc5bde00640 1 Processor -- start 2024-09-16T01:48:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.982+0000 7fc5bde00640 1 -- start start 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5bde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 0x7fc5b81f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5bde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b81f4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5bde00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b814b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5bde00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b80c3d00 con 0x7fc5b8072d20 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5bde00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b80c72c0 con 0x7fc5b8073100 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5bde00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b80c65e0 con 0x7fc5b80c8d50 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5b6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 0x7fc5b81f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5b6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 0x7fc5b81f4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45832/0 (socket says 172.21.15.146:45832) 2024-09-16T01:48:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.983+0000 7fc5b6a00640 1 -- 172.21.15.146:0/1533711925 learned_addr learned my addr 172.21.15.146:0/1533711925 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b7400640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b814b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5ae000640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b81f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b6a00640 1 -- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 msgr2=0x7fc5b814b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b6a00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b814b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b6a00640 1 -- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 msgr2=0x7fc5b81f4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b6a00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b81f4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b6a00640 1 -- 172.21.15.146:0/1533711925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b80ccfb0 con 0x7fc5b8072d20 2024-09-16T01:48:56.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5b6a00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 0x7fc5b81f4000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c4b0 tx=0x7fc5a800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8017070 con 0x7fc5b8072d20 2024-09-16T01:48:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a8005480 con 0x7fc5b8072d20 2024-09-16T01:48:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8003830 con 0x7fc5b8072d20 2024-09-16T01:48:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.984+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b814db60 con 0x7fc5b8072d20 2024-09-16T01:48:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.985+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b814d880 con 0x7fc5b8072d20 2024-09-16T01:48:56.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.985+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b80c6cc0 con 0x7fc5b8072d20 2024-09-16T01:48:56.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.987+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc5a801a070 con 0x7fc5b8072d20 2024-09-16T01:48:56.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.987+0000 7fc5afe00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c2b0 0x7fc59406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:56.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.987+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7fc5a808a520 con 0x7fc5b8072d20 2024-09-16T01:48:56.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.988+0000 7fc5ae000640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c2b0 0x7fc59406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:56.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.988+0000 7fc5ae000640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c2b0 0x7fc59406e770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5b81ee450 tx=0x7fc5a0009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:56.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:56.992+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc5a8058820 con 0x7fc5b8072d20 2024-09-16T01:48:57.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.168+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc5b80c4f80 con 0x7fc5b8072d20 2024-09-16T01:48:57.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.789+0000 7fc5afe00640 1 -- 172.21.15.146:0/1533711925 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) ==== 170+0+0 (secure 0 0 0) 0x7fc5a805cb20 con 0x7fc5b8072d20 2024-09-16T01:48:57.790 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' created 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.791+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c2b0 msgr2=0x7fc59406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.791+0000 7fc5bde00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c2b0 0x7fc59406e770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5b81ee450 tx=0x7fc5a0009240 comp rx=0 tx=0).stop 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.791+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 msgr2=0x7fc5b81f4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.791+0000 7fc5bde00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 0x7fc5b81f4000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c4b0 tx=0x7fc5a800c970 comp rx=0 tx=0).stop 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 shutdown_connections 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c2b0 0x7fc59406e770 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b80c8d50 0x7fc5b814b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b8073100 0x7fc5b81f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 --2- 172.21.15.146:0/1533711925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b8072d20 0x7fc5b81f4000 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 >> 172.21.15.146:0/1533711925 conn(0x7fc5b806e670 msgr2=0x7fc5b80c33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:57.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 shutdown_connections 2024-09-16T01:48:57.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.792+0000 7fc5bde00640 1 -- 172.21.15.146:0/1533711925 wait complete. 2024-09-16T01:48:57.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:48:57.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.984+0000 7fac34e00640 1 Processor -- start 2024-09-16T01:48:57.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.984+0000 7fac34e00640 1 -- start start 2024-09-16T01:48:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac34e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30166440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac34e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3015ba20 con 0x7fac300cc480 2024-09-16T01:48:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac34e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3015aea0 con 0x7fac300cc0a0 2024-09-16T01:48:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac34e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fac3015e1d0 con 0x7fac3015ed70 2024-09-16T01:48:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac27e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fac300cc480 0x7fac3015e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56710/0 (socket says 172.21.15.146:56710) 2024-09-16T01:48:57.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac27e00640 1 -- 172.21.15.146:0/543321224 learned_addr learned my addr 172.21.15.146:0/543321224 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:57.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.985+0000 7fac2e800640 1 --2- 172.21.15.146:0/543321224 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30166440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3441616849 0 0) 0x7fac3015ba20 con 0x7fac300cc480 2024-09-16T01:48:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac14003520 con 0x7fac300cc480 2024-09-16T01:48:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1691768972 0 0) 0x7fac3015aea0 con 0x7fac300cc0a0 2024-09-16T01:48:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac3015ba20 con 0x7fac300cc0a0 2024-09-16T01:48:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2e800640 1 -- 172.21.15.146:0/543321224 >> v1:172.21.15.161:6789/0 conn(0x7fac300cc0a0 legacy=0x7fac300d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2e800640 1 -- 172.21.15.146:0/543321224 >> v1:172.21.15.146:6789/0 conn(0x7fac300cc480 legacy=0x7fac3015e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.986+0000 7fac2e800640 1 -- 172.21.15.146:0/543321224 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3015b180 con 0x7fac3015ed70 2024-09-16T01:48:57.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.987+0000 7fac2e800640 1 --2- 172.21.15.146:0/543321224 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30166440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac28009dc0 tx=0x7fac2802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d778032cb2545f84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:57.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.987+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28007740 con 0x7fac3015ed70 2024-09-16T01:48:57.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.987+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac280078e0 con 0x7fac3015ed70 2024-09-16T01:48:57.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.987+0000 7fac2d400640 1 -- 172.21.15.146:0/543321224 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28007cb0 con 0x7fac3015ed70 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 -- 172.21.15.146:0/543321224 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 msgr2=0x7fac30166440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 --2- 172.21.15.146:0/543321224 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30166440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac28009dc0 tx=0x7fac2802a760 comp rx=0 tx=0).stop 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 -- 172.21.15.146:0/543321224 shutdown_connections 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 --2- 172.21.15.146:0/543321224 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30166440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 -- 172.21.15.146:0/543321224 >> 172.21.15.146:0/543321224 conn(0x7fac3006d560 msgr2=0x7fac30073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 -- 172.21.15.146:0/543321224 shutdown_connections 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.988+0000 7fac34e00640 1 -- 172.21.15.146:0/543321224 wait complete. 2024-09-16T01:48:57.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.989+0000 7fac34e00640 1 Processor -- start 2024-09-16T01:48:57.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.989+0000 7fac34e00640 1 -- start start 2024-09-16T01:48:57.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac34e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 0x7fac3014b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac34e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac300cc480 0x7fac30150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac34e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac34e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac300cc8a0 con 0x7fac300cc480 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac34e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac3015b740 con 0x7fac300cc0a0 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac34e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fac300cbde0 con 0x7fac3015ed70 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac2de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 0x7fac3014b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac27e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac300cc480 0x7fac30150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac2de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 0x7fac3014b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:42500/0 (socket says 172.21.15.146:42500) 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac2de00640 1 -- 172.21.15.146:0/3635798550 learned_addr learned my addr 172.21.15.146:0/3635798550 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:57.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.990+0000 7fac2e800640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:57.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac2de00640 1 -- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 msgr2=0x7fac30157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac2de00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac2de00640 1 -- 172.21.15.146:0/3635798550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac300cc480 msgr2=0x7fac30150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac2de00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac300cc480 0x7fac30150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac2de00640 1 -- 172.21.15.146:0/3635798550 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3015c4f0 con 0x7fac300cc0a0 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac2de00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 0x7fac3014b380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac2000c430 tx=0x7fac2000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20017070 con 0x7fac300cc0a0 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac200052f0 con 0x7fac300cc0a0 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac301f0cd0 con 0x7fac300cc0a0 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac20003830 con 0x7fac300cc0a0 2024-09-16T01:48:57.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.991+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac301f1290 con 0x7fac300cc0a0 2024-09-16T01:48:57.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.993+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac300b3530 con 0x7fac300cc0a0 2024-09-16T01:48:57.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.993+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fac2001a070 con 0x7fac300cc0a0 2024-09-16T01:48:57.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.994+0000 7fac26000640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fac0c06c2e0 0x7fac0c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:57.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.994+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5609+0+0 (secure 0 0 0) 0x7fac2008a6b0 con 0x7fac300cc0a0 2024-09-16T01:48:57.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.994+0000 7fac27e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fac0c06c2e0 0x7fac0c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:57.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.995+0000 7fac27e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fac0c06c2e0 0x7fac0c06e7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac30156b40 tx=0x7fac18007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:57.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:57.999+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fac300b3530 con 0x7fac300cc0a0 2024-09-16T01:48:58.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.205+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fac300c56d0 con 0x7fac300cc0a0 2024-09-16T01:48:58.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.825+0000 7fac26000640 1 -- 172.21.15.146:0/3635798550 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fac300c56d0 con 0x7fac300cc0a0 2024-09-16T01:48:58.826 INFO:teuthology.orchestra.run.smithi146.stderr:new fs with metadata pool 2 and data pool 3 2024-09-16T01:48:58.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.831+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fac0c06c2e0 msgr2=0x7fac0c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:58.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.831+0000 7fac34e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fac0c06c2e0 0x7fac0c06e7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac30156b40 tx=0x7fac18007d30 comp rx=0 tx=0).stop 2024-09-16T01:48:58.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.831+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 msgr2=0x7fac3014b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:58.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.831+0000 7fac34e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 0x7fac3014b380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fac2000c430 tx=0x7fac2000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:48:58.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.832+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 shutdown_connections 2024-09-16T01:48:58.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.832+0000 7fac34e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fac0c06c2e0 0x7fac0c06e7a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:58.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.832+0000 7fac34e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac3015ed70 0x7fac30157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:58.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.832+0000 7fac34e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac300cc480 0x7fac30150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:58.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.832+0000 7fac34e00640 1 --2- 172.21.15.146:0/3635798550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac300cc0a0 0x7fac3014b380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:58.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.832+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 >> 172.21.15.146:0/3635798550 conn(0x7fac3006d560 msgr2=0x7fac300d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:58.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.833+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 shutdown_connections 2024-09-16T01:48:58.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:58.834+0000 7fac34e00640 1 -- 172.21.15.146:0/3635798550 wait complete. 2024-09-16T01:48:58.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:48:59.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.009+0000 7fa555e00640 1 Processor -- start 2024-09-16T01:48:59.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.009+0000 7fa555e00640 1 -- start start 2024-09-16T01:48:59.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa555e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cb2f0 0x7fa55015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa555e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55015b1d0 con 0x7fa5500cc4a0 2024-09-16T01:48:59.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa555e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa55015b4b0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa555e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa5500cbd00 con 0x7fa5500cb2f0 2024-09-16T01:48:59.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa54e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cb2f0 0x7fa55015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa54f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa5500cc4a0 0x7fa550165430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56720/0 (socket says 172.21.15.146:56720) 2024-09-16T01:48:59.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa54ea00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa5500cc0c0 0x7fa55015e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:42064/0 (socket says 172.21.15.146:42064) 2024-09-16T01:48:59.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.010+0000 7fa54f400640 1 -- 172.21.15.146:0/2425650962 learned_addr learned my addr 172.21.15.146:0/2425650962 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:59.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.011+0000 7fa54d600640 1 -- 172.21.15.146:0/2425650962 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878288289 0 0) 0x7fa55015b4b0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.011+0000 7fa54e000640 1 -- 172.21.15.146:0/2425650962 >> v1:172.21.15.161:6789/0 conn(0x7fa5500cc0c0 legacy=0x7fa55015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.011+0000 7fa54e000640 1 -- 172.21.15.146:0/2425650962 >> v1:172.21.15.146:6789/0 conn(0x7fa5500cc4a0 legacy=0x7fa550165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.011+0000 7fa54e000640 1 -- 172.21.15.146:0/2425650962 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55015b790 con 0x7fa5500cb2f0 2024-09-16T01:48:59.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.012+0000 7fa54e000640 1 --2- 172.21.15.146:0/2425650962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cb2f0 0x7fa55015aab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0097d0 tx=0x7fa53c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcadfffbd44d65b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.012+0000 7fa54d600640 1 -- 172.21.15.146:0/2425650962 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c037070 con 0x7fa5500cb2f0 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.012+0000 7fa54d600640 1 -- 172.21.15.146:0/2425650962 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53c032440 con 0x7fa5500cb2f0 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.012+0000 7fa54d600640 1 -- 172.21.15.146:0/2425650962 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c032810 con 0x7fa5500cb2f0 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.012+0000 7fa555e00640 1 -- 172.21.15.146:0/2425650962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cb2f0 msgr2=0x7fa55015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.012+0000 7fa555e00640 1 --2- 172.21.15.146:0/2425650962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cb2f0 0x7fa55015aab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0097d0 tx=0x7fa53c02a210 comp rx=0 tx=0).stop 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.013+0000 7fa555e00640 1 -- 172.21.15.146:0/2425650962 shutdown_connections 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.013+0000 7fa555e00640 1 --2- 172.21.15.146:0/2425650962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cb2f0 0x7fa55015aab0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.013+0000 7fa555e00640 1 -- 172.21.15.146:0/2425650962 >> 172.21.15.146:0/2425650962 conn(0x7fa55006d560 msgr2=0x7fa55006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.013+0000 7fa555e00640 1 -- 172.21.15.146:0/2425650962 shutdown_connections 2024-09-16T01:48:59.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.013+0000 7fa555e00640 1 -- 172.21.15.146:0/2425650962 wait complete. 2024-09-16T01:48:59.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.014+0000 7fa555e00640 1 Processor -- start 2024-09-16T01:48:59.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.014+0000 7fa555e00640 1 -- start start 2024-09-16T01:48:59.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.015+0000 7fa555e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5500cb2f0 0x7fa5500ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.015+0000 7fa555e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 0x7fa5501f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.015+0000 7fa555e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5500cc4a0 0x7fa55015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.015+0000 7fa555e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa55015e540 con 0x7fa5500cb2f0 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.015+0000 7fa555e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa55015ba70 con 0x7fa5500cc4a0 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.015+0000 7fa555e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa5500cc8b0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5500cb2f0 0x7fa5500ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 0x7fa5501f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5500cc4a0 0x7fa55015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5500cb2f0 0x7fa5500ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45854/0 (socket says 172.21.15.146:45854) 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54ea00640 1 -- 172.21.15.146:0/3023412880 learned_addr learned my addr 172.21.15.146:0/3023412880 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 0x7fa5501f1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:34918/0 (socket says 172.21.15.146:34918) 2024-09-16T01:48:59.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 -- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5500cc4a0 msgr2=0x7fa55015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5500cc4a0 0x7fa55015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 -- 172.21.15.146:0/3023412880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5500cb2f0 msgr2=0x7fa5500ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5500cb2f0 0x7fa5500ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 -- 172.21.15.146:0/3023412880 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa550157a90 con 0x7fa5500cc0c0 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.016+0000 7fa54e000640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 0x7fa5501f1360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa53c031000 tx=0x7fa53c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.017+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c032e60 con 0x7fa5500cc0c0 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.017+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53c0307b0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.017+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c037040 con 0x7fa5500cc0c0 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.017+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa550157d70 con 0x7fa5500cc0c0 2024-09-16T01:48:59.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.017+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5501f18a0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.018+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5500c5560 con 0x7fa5500cc0c0 2024-09-16T01:48:59.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.019+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa53c044020 con 0x7fa5500cc0c0 2024-09-16T01:48:59.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.019+0000 7fa547400640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa53006c2b0 0x7fa53006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.020+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa53c0ab7e0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.020+0000 7fa54ea00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa53006c2b0 0x7fa53006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.020+0000 7fa54ea00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa53006c2b0 0x7fa53006e770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa540007b10 tx=0x7fa540007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.024+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa53c0797d0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.200+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5500cbea0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.201+0000 7fa547400640 1 -- 172.21.15.146:0/3023412880 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16706 (secure 0 0 0) 0x7fa53c07dad0 con 0x7fa5500cc0c0 2024-09-16T01:48:59.203 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:48:59.203 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":21,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:58.207400+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:48:59.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa53006c2b0 msgr2=0x7fa53006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa53006c2b0 0x7fa53006e770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa540007b10 tx=0x7fa540007a60 comp rx=0 tx=0).stop 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 msgr2=0x7fa5501f1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 0x7fa5501f1360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa53c031000 tx=0x7fa53c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 shutdown_connections 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa53006c2b0 0x7fa53006e770 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5500cc4a0 0x7fa55015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5500cc0c0 0x7fa5501f1360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 --2- 172.21.15.146:0/3023412880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5500cb2f0 0x7fa5500ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 >> 172.21.15.146:0/3023412880 conn(0x7fa55006d560 msgr2=0x7fa5500b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 shutdown_connections 2024-09-16T01:48:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.204+0000 7fa555e00640 1 -- 172.21.15.146:0/3023412880 wait complete. 2024-09-16T01:48:59.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:48:59.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.374+0000 7f481d600640 1 Processor -- start 2024-09-16T01:48:59.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.375+0000 7f481d600640 1 -- start start 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f481d600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f48180c75a0 0x7f481815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f481d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481806d140 con 0x7f48180c6b70 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f481d600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481806d420 con 0x7f48180cc040 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f481d600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f481815b880 con 0x7f48180c75a0 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4817e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f48180c6b70 0x7f48180cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56724/0 (socket says 172.21.15.146:56724) 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4817e00640 1 -- 172.21.15.146:0/2638075946 learned_addr learned my addr 172.21.15.146:0/2638075946 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:48:59.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4817400640 1 --2- 172.21.15.146:0/2638075946 >> v2:172.21.15.146:3300/0 conn(0x7f48180c75a0 0x7f481815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1930971710 0 0) 0x7f481806d140 con 0x7f48180c6b70 2024-09-16T01:48:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47fc003520 con 0x7f48180c6b70 2024-09-16T01:48:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 498647942 0 0) 0x7f481806d420 con 0x7f48180cc040 2024-09-16T01:48:59.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f481806d140 con 0x7f48180cc040 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.376+0000 7f4817400640 1 -- 172.21.15.146:0/2638075946 >> v1:172.21.15.161:6789/0 conn(0x7f48180cc040 legacy=0x7f4818165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.377+0000 7f4817400640 1 -- 172.21.15.146:0/2638075946 >> v1:172.21.15.146:6789/0 conn(0x7f48180c6b70 legacy=0x7f48180cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.377+0000 7f4817400640 1 -- 172.21.15.146:0/2638075946 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481806d700 con 0x7f48180c75a0 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.377+0000 7f4817400640 1 --2- 172.21.15.146:0/2638075946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c75a0 0x7f481815b340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48040097a0 tx=0x7f480402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f0329e555b7a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804037070 con 0x7f48180c75a0 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4804032440 con 0x7f48180c75a0 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f4816a00640 1 -- 172.21.15.146:0/2638075946 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804032810 con 0x7f48180c75a0 2024-09-16T01:48:59.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 -- 172.21.15.146:0/2638075946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c75a0 msgr2=0x7f481815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 --2- 172.21.15.146:0/2638075946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c75a0 0x7f481815b340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48040097a0 tx=0x7f480402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 -- 172.21.15.146:0/2638075946 shutdown_connections 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 --2- 172.21.15.146:0/2638075946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c75a0 0x7f481815b340 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 -- 172.21.15.146:0/2638075946 >> 172.21.15.146:0/2638075946 conn(0x7f48180ccc20 msgr2=0x7f48180cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 -- 172.21.15.146:0/2638075946 shutdown_connections 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.378+0000 7f481d600640 1 -- 172.21.15.146:0/2638075946 wait complete. 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.380+0000 7f481d600640 1 Processor -- start 2024-09-16T01:48:59.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.380+0000 7f481d600640 1 -- start start 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 0x7f481815df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48180c75a0 0x7f481815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48180cc040 0x7f4818153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f48180cb050 con 0x7f48180c6b70 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481d600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f48180cab20 con 0x7f48180c75a0 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f48180cbc80 con 0x7f48180cc040 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 0x7f481815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 0x7f481815df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45876/0 (socket says 172.21.15.146:45876) 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 -- 172.21.15.146:0/3234631219 learned_addr learned my addr 172.21.15.146:0/3234631219 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f481cc00640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48180cc040 0x7f4818153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817400640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48180c75a0 0x7f481815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 -- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48180cc040 msgr2=0x7f4818153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48180cc040 0x7f4818153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 -- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48180c75a0 msgr2=0x7f481815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.381+0000 7f4817e00640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48180c75a0 0x7f481815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f4817e00640 1 -- 172.21.15.146:0/3234631219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48181f4ed0 con 0x7f48180c6b70 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f4817e00640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 0x7f481815df50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f480800cb50 tx=0x7f480800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808017070 con 0x7f48180c6b70 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480800ad40 con 0x7f48180c6b70 2024-09-16T01:48:59.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808012620 con 0x7f48180c6b70 2024-09-16T01:48:59.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f481815e7f0 con 0x7f48180c6b70 2024-09-16T01:48:59.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.382+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48181f0c50 con 0x7f48180c6b70 2024-09-16T01:48:59.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.383+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48181f0a40 con 0x7f48180c6b70 2024-09-16T01:48:59.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.384+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f480801a070 con 0x7f48180c6b70 2024-09-16T01:48:59.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.385+0000 7f480fe00640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47f806c2e0 0x7f47f806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.385+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f480808b1b0 con 0x7f48180c6b70 2024-09-16T01:48:59.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.385+0000 7f4817400640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47f806c2e0 0x7f47f806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.386+0000 7f4817400640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47f806c2e0 0x7f47f806e7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4804034000 tx=0x7f480402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.390+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f48080592c0 con 0x7f48180c6b70 2024-09-16T01:48:59.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.566+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f481806da90 con 0x7f48180c6b70 2024-09-16T01:48:59.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.567+0000 7f480fe00640 1 -- 172.21.15.146:0/3234631219 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16706 (secure 0 0 0) 0x7f480805d5c0 con 0x7f48180c6b70 2024-09-16T01:48:59.569 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:48:59.569 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":21,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:58.207400+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:48:59.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47f806c2e0 msgr2=0x7f47f806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47f806c2e0 0x7f47f806e7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4804034000 tx=0x7f480402e000 comp rx=0 tx=0).stop 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 msgr2=0x7f481815df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 0x7f481815df50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f480800cb50 tx=0x7f480800a430 comp rx=0 tx=0).stop 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 shutdown_connections 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47f806c2e0 0x7f47f806e7a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48180cc040 0x7f4818153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48180c75a0 0x7f481815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 --2- 172.21.15.146:0/3234631219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48180c6b70 0x7f481815df50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 >> 172.21.15.146:0/3234631219 conn(0x7f48180ccc20 msgr2=0x7f48180c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 shutdown_connections 2024-09-16T01:48:59.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.569+0000 7f481d600640 1 -- 172.21.15.146:0/3234631219 wait complete. 2024-09-16T01:48:59.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:48:59.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.742+0000 7f6cc1200640 1 Processor -- start 2024-09-16T01:48:59.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.742+0000 7f6cc1200640 1 -- start start 2024-09-16T01:48:59.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.743+0000 7f6cc1200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6cbc072d20 0x7f6cbc0c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cc1200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc073100 0x7f6cbc151550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cc1200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cbc0c8050 con 0x7f6cbc0c8d50 2024-09-16T01:48:59.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cc1200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6cbc0c5780 con 0x7f6cbc072d20 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cc1200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc0c5a60 con 0x7f6cbc073100 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6cbc072d20 0x7f6cbc0c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6cbc072d20 0x7f6cbc0c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45882/0 (socket says 172.21.15.146:45882) 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 -- 172.21.15.146:0/2221576957 learned_addr learned my addr 172.21.15.146:0/2221576957 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cba800640 1 --2- 172.21.15.146:0/2221576957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc073100 0x7f6cbc151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 -- 172.21.15.146:0/2221576957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc073100 msgr2=0x7f6cbc151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 --2- 172.21.15.146:0/2221576957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc073100 0x7f6cbc151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 -- 172.21.15.146:0/2221576957 >> v1:172.21.15.161:6789/0 conn(0x7f6cbc0c8d50 legacy=0x7f6cbc0c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.744+0000 7f6cb9e00640 1 -- 172.21.15.146:0/2221576957 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc0c5d40 con 0x7f6cbc072d20 2024-09-16T01:48:59.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.745+0000 7f6cb9e00640 1 --2- 172.21.15.146:0/2221576957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc072d20 0x7f6cbc0c7b10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00bb10 tx=0x7f6cac02c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661a391d6ed04c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.745+0000 7f6cb9400640 1 -- 172.21.15.146:0/2221576957 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac03a070 con 0x7f6cbc072d20 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cb9400640 1 -- 172.21.15.146:0/2221576957 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cac0326f0 con 0x7f6cbc072d20 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cb9400640 1 -- 172.21.15.146:0/2221576957 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac032ac0 con 0x7f6cbc072d20 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 -- 172.21.15.146:0/2221576957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc072d20 msgr2=0x7f6cbc0c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 --2- 172.21.15.146:0/2221576957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc072d20 0x7f6cbc0c7b10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00bb10 tx=0x7f6cac02c8b0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 -- 172.21.15.146:0/2221576957 shutdown_connections 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 --2- 172.21.15.146:0/2221576957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc073100 0x7f6cbc151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 --2- 172.21.15.146:0/2221576957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc072d20 0x7f6cbc0c7b10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 -- 172.21.15.146:0/2221576957 >> 172.21.15.146:0/2221576957 conn(0x7f6cbc06e670 msgr2=0x7f6cbc06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 -- 172.21.15.146:0/2221576957 shutdown_connections 2024-09-16T01:48:59.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.746+0000 7f6cc1200640 1 -- 172.21.15.146:0/2221576957 wait complete. 2024-09-16T01:48:59.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.748+0000 7f6cc1200640 1 Processor -- start 2024-09-16T01:48:59.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.748+0000 7f6cc1200640 1 -- start start 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cc1200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 0x7f6cbc1546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cc1200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc073100 0x7f6cbc14d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cc1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc0c8d50 0x7f6cbc1437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cc1200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc0c72c0 con 0x7f6cbc0c8d50 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cc1200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc0c5480 con 0x7f6cbc073100 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cc1200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc0c65e0 con 0x7f6cbc072d20 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 0x7f6cbc1546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb1400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc073100 0x7f6cbc14d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 0x7f6cbc1546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:34940/0 (socket says 172.21.15.146:34940) 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 -- 172.21.15.146:0/1855073331 learned_addr learned my addr 172.21.15.146:0/1855073331 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cba800640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc0c8d50 0x7f6cbc1437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 -- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc073100 msgr2=0x7f6cbc14d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc073100 0x7f6cbc14d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 -- 172.21.15.146:0/1855073331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc0c8d50 msgr2=0x7f6cbc1437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc0c8d50 0x7f6cbc1437d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.749+0000 7f6cb9e00640 1 -- 172.21.15.146:0/1855073331 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc149120 con 0x7f6cbc072d20 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cba800640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc0c8d50 0x7f6cbc1437d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cb1400640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc073100 0x7f6cbc14d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cb9e00640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 0x7f6cbc1546f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cac02c8f0 tx=0x7f6cac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac045070 con 0x7f6cbc072d20 2024-09-16T01:48:59.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cac004030 con 0x7f6cbc072d20 2024-09-16T01:48:59.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc149780 con 0x7f6cbc072d20 2024-09-16T01:48:59.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac03a070 con 0x7f6cbc072d20 2024-09-16T01:48:59.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.750+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc149a40 con 0x7f6cbc072d20 2024-09-16T01:48:59.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.751+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc070360 con 0x7f6cbc072d20 2024-09-16T01:48:59.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.752+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6cac048070 con 0x7f6cbc072d20 2024-09-16T01:48:59.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.753+0000 7f6cb2a00640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c9806c2b0 0x7f6c9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:48:59.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.753+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6cac0ab760 con 0x7f6cbc072d20 2024-09-16T01:48:59.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.753+0000 7f6cb1400640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c9806c2b0 0x7f6c9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:48:59.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.754+0000 7f6cb1400640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c9806c2b0 0x7f6c9806e770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc14e1d0 tx=0x7f6ca4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:48:59.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.757+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6cbc070360 con 0x7f6cbc072d20 2024-09-16T01:48:59.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:48:59.968+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6cbc0c4400 con 0x7f6cbc072d20 2024-09-16T01:49:00.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.831+0000 7f6cb2a00640 1 -- 172.21.15.146:0/1855073331 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f6cbc0c4400 con 0x7f6cbc072d20 2024-09-16T01:49:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.833+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c9806c2b0 msgr2=0x7f6c9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.833+0000 7f6cc1200640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c9806c2b0 0x7f6c9806e770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc14e1d0 tx=0x7f6ca4009040 comp rx=0 tx=0).stop 2024-09-16T01:49:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.833+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 msgr2=0x7f6cbc1546f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.833+0000 7f6cc1200640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 0x7f6cbc1546f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cac02c8f0 tx=0x7f6cac004730 comp rx=0 tx=0).stop 2024-09-16T01:49:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.833+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 shutdown_connections 2024-09-16T01:49:00.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c9806c2b0 0x7f6c9806e770 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6cbc0c8d50 0x7f6cbc1437d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cbc073100 0x7f6cbc14d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 --2- 172.21.15.146:0/1855073331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6cbc072d20 0x7f6cbc1546f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 >> 172.21.15.146:0/1855073331 conn(0x7f6cbc06e670 msgr2=0x7f6cbc0c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 shutdown_connections 2024-09-16T01:49:00.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:00.834+0000 7f6cc1200640 1 -- 172.21.15.146:0/1855073331 wait complete. 2024-09-16T01:49:00.852 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-09-16T01:49:00.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-09-16T01:49:01.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.013+0000 7f7b15a00640 1 Processor -- start 2024-09-16T01:49:01.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.013+0000 7f7b15a00640 1 -- start start 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.014+0000 7f7b15a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7b1015c480 0x7f7b10164110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.014+0000 7f7b15a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b10158890 con 0x7f7b10152760 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.014+0000 7f7b15a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b10158b00 con 0x7f7b100c6cc0 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.014+0000 7f7b15a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7b100c8530 con 0x7f7b1015c480 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0ea00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7b10152760 0x7f7b100c59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56730/0 (socket says 172.21.15.146:56730) 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0ea00640 1 -- 172.21.15.146:0/1742485885 learned_addr learned my addr 172.21.15.146:0/1742485885 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0f400640 1 --2- 172.21.15.146:0/1742485885 >> v2:172.21.15.161:3300/0 conn(0x7f7b1015c480 0x7f7b10164110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0d600640 1 -- 172.21.15.146:0/1742485885 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890836279 0 0) 0x7f7b10158890 con 0x7f7b10152760 2024-09-16T01:49:01.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0d600640 1 -- 172.21.15.146:0/1742485885 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7af4003520 con 0x7f7b10152760 2024-09-16T01:49:01.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0f400640 1 -- 172.21.15.146:0/1742485885 >> v1:172.21.15.161:6789/0 conn(0x7f7b100c6cc0 legacy=0x7f7b1015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0f400640 1 -- 172.21.15.146:0/1742485885 >> v1:172.21.15.146:6789/0 conn(0x7f7b10152760 legacy=0x7f7b100c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.015+0000 7f7b0f400640 1 -- 172.21.15.146:0/1742485885 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b10158de0 con 0x7f7b1015c480 2024-09-16T01:49:01.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.016+0000 7f7b0f400640 1 --2- 172.21.15.146:0/1742485885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b1015c480 0x7f7b10164110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b0802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57e65829c7b45ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:01.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.016+0000 7f7b0d600640 1 -- 172.21.15.146:0/1742485885 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b08037070 con 0x7f7b1015c480 2024-09-16T01:49:01.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.016+0000 7f7b0d600640 1 -- 172.21.15.146:0/1742485885 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b08007740 con 0x7f7b1015c480 2024-09-16T01:49:01.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.016+0000 7f7b0d600640 1 -- 172.21.15.146:0/1742485885 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b08007b10 con 0x7f7b1015c480 2024-09-16T01:49:01.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 -- 172.21.15.146:0/1742485885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b1015c480 msgr2=0x7f7b10164110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 --2- 172.21.15.146:0/1742485885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b1015c480 0x7f7b10164110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b0802a780 comp rx=0 tx=0).stop 2024-09-16T01:49:01.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 -- 172.21.15.146:0/1742485885 shutdown_connections 2024-09-16T01:49:01.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 --2- 172.21.15.146:0/1742485885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b1015c480 0x7f7b10164110 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 -- 172.21.15.146:0/1742485885 >> 172.21.15.146:0/1742485885 conn(0x7f7b1006e670 msgr2=0x7f7b1006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:01.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 -- 172.21.15.146:0/1742485885 shutdown_connections 2024-09-16T01:49:01.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.017+0000 7f7b15a00640 1 -- 172.21.15.146:0/1742485885 wait complete. 2024-09-16T01:49:01.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.019+0000 7f7b15a00640 1 Processor -- start 2024-09-16T01:49:01.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.019+0000 7f7b15a00640 1 -- start start 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.019+0000 7f7b15a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b100c6cc0 0x7f7b101f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.019+0000 7f7b15a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7b10152760 0x7f7b101f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b15a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 0x7f7b101512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b15a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7b10152b40 con 0x7f7b1015c480 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b15a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7b101593a0 con 0x7f7b100c6cc0 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b15a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7b10152cc0 con 0x7f7b10152760 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 0x7f7b101512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 0x7f7b101512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45906/0 (socket says 172.21.15.146:45906) 2024-09-16T01:49:01.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 -- 172.21.15.146:0/4294257325 learned_addr learned my addr 172.21.15.146:0/4294257325 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0ea00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b100c6cc0 0x7f7b101f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0e000640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7b10152760 0x7f7b101f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 -- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7b10152760 msgr2=0x7f7b101f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7b10152760 0x7f7b101f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 -- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b100c6cc0 msgr2=0x7f7b101f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b100c6cc0 0x7f7b101f0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.020+0000 7f7b0f400640 1 -- 172.21.15.146:0/4294257325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b10151b70 con 0x7f7b1015c480 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.021+0000 7f7b0f400640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 0x7f7b101512e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.021+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b08042070 con 0x7f7b1015c480 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.021+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b08003990 con 0x7f7b1015c480 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.021+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b08037040 con 0x7f7b1015c480 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.021+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b101581c0 con 0x7f7b1015c480 2024-09-16T01:49:01.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.021+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b101f3f40 con 0x7f7b1015c480 2024-09-16T01:49:01.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.022+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b1006f450 con 0x7f7b1015c480 2024-09-16T01:49:01.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.023+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7b08032870 con 0x7f7b1015c480 2024-09-16T01:49:01.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.024+0000 7f7b06a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7af006c2b0 0x7f7af006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:01.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.024+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7b080ac240 con 0x7f7b1015c480 2024-09-16T01:49:01.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.025+0000 7f7b0ea00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7af006c2b0 0x7f7af006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:01.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.025+0000 7f7b0ea00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7af006c2b0 0x7f7af006e770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b000011e0 tx=0x7f7b00007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:01.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.029+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7b1006f450 con 0x7f7b1015c480 2024-09-16T01:49:01.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.243+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f7b100c7350 con 0x7f7b1015c480 2024-09-16T01:49:01.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.833+0000 7f7b06a00640 1 -- 172.21.15.146:0/4294257325 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) ==== 105+0+0 (secure 0 0 0) 0x7f7b0804c020 con 0x7f7b1015c480 2024-09-16T01:49:01.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7af006c2b0 msgr2=0x7f7af006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7af006c2b0 0x7f7af006e770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7b000011e0 tx=0x7f7b00007d30 comp rx=0 tx=0).stop 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 msgr2=0x7f7b101512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 0x7f7b101512e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d90 tx=0x7f7b08004a40 comp rx=0 tx=0).stop 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 shutdown_connections 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7af006c2b0 0x7f7af006e770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b1015c480 0x7f7b101512e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7b10152760 0x7f7b101f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 --2- 172.21.15.146:0/4294257325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b100c6cc0 0x7f7b101f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 >> 172.21.15.146:0/4294257325 conn(0x7f7b1006e670 msgr2=0x7f7b100c2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 shutdown_connections 2024-09-16T01:49:01.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:01.837+0000 7f7b15a00640 1 -- 172.21.15.146:0/4294257325 wait complete. 2024-09-16T01:49:01.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:49:02.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.017+0000 7fc604c00640 1 Processor -- start 2024-09-16T01:49:02.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.017+0000 7fc604c00640 1 -- start start 2024-09-16T01:49:02.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc604c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc60015b190 0x7fc6000d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:02.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc604c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6000d0d00 0x7fc60015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:02.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc604c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc600154ea0 con 0x7fc600154ac0 2024-09-16T01:49:02.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc604c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc6000c5340 con 0x7fc60015b190 2024-09-16T01:49:02.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc604c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc6000c5620 con 0x7fc6000d0d00 2024-09-16T01:49:02.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc5ff400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc600154ac0 0x7fc60006d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56746/0 (socket says 172.21.15.146:56746) 2024-09-16T01:49:02.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc5ff400640 1 -- 172.21.15.146:0/243204274 learned_addr learned my addr 172.21.15.146:0/243204274 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:02.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc5fea00640 1 --2- 172.21.15.146:0/243204274 >> v2:172.21.15.146:3300/0 conn(0x7fc60015b190 0x7fc6000d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:02.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.018+0000 7fc5ffe00640 1 --2- 172.21.15.146:0/243204274 >> v2:172.21.15.161:3300/0 conn(0x7fc6000d0d00 0x7fc60015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.019+0000 7fc5fea00640 1 -- 172.21.15.146:0/243204274 >> v2:172.21.15.161:3300/0 conn(0x7fc6000d0d00 msgr2=0x7fc60015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.019+0000 7fc5fea00640 1 --2- 172.21.15.146:0/243204274 >> v2:172.21.15.161:3300/0 conn(0x7fc6000d0d00 0x7fc60015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.019+0000 7fc5fea00640 1 -- 172.21.15.146:0/243204274 >> v1:172.21.15.146:6789/0 conn(0x7fc600154ac0 legacy=0x7fc60006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.019+0000 7fc5fea00640 1 -- 172.21.15.146:0/243204274 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6000c5900 con 0x7fc60015b190 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.019+0000 7fc5fe000640 1 -- 172.21.15.146:0/243204274 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 180523520 0 0) 0x7fc600154ea0 con 0x7fc600154ac0 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.020+0000 7fc5fea00640 1 --2- 172.21.15.146:0/243204274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60015b190 0x7fc6000d07c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc60006bcc0 tx=0x7fc5ec02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2450e919e6ac166b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.020+0000 7fc5fe000640 1 -- 172.21.15.146:0/243204274 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec037070 con 0x7fc60015b190 2024-09-16T01:49:02.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.020+0000 7fc5fe000640 1 -- 172.21.15.146:0/243204274 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec032440 con 0x7fc60015b190 2024-09-16T01:49:02.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.020+0000 7fc5fe000640 1 -- 172.21.15.146:0/243204274 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec032810 con 0x7fc60015b190 2024-09-16T01:49:02.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 -- 172.21.15.146:0/243204274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60015b190 msgr2=0x7fc6000d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:02.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 --2- 172.21.15.146:0/243204274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60015b190 0x7fc6000d07c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc60006bcc0 tx=0x7fc5ec02a230 comp rx=0 tx=0).stop 2024-09-16T01:49:02.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 -- 172.21.15.146:0/243204274 shutdown_connections 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 --2- 172.21.15.146:0/243204274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc60015b190 0x7fc6000d07c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 --2- 172.21.15.146:0/243204274 >> v2:172.21.15.161:3300/0 conn(0x7fc6000d0d00 0x7fc60015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 -- 172.21.15.146:0/243204274 >> 172.21.15.146:0/243204274 conn(0x7fc60006e400 msgr2=0x7fc60006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 -- 172.21.15.146:0/243204274 shutdown_connections 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.021+0000 7fc604c00640 1 -- 172.21.15.146:0/243204274 wait complete. 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.022+0000 7fc604c00640 1 Processor -- start 2024-09-16T01:49:02.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.022+0000 7fc604c00640 1 -- start start 2024-09-16T01:49:02.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc604c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6000d0d00 0x7fc6000cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc604c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 0x7fc60014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc604c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc60015b190 0x7fc60014e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc604c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc6000c61a0 con 0x7fc600154ac0 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc604c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc6000c5ec0 con 0x7fc60015b190 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc604c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc6000cad60 con 0x7fc6000d0d00 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.023+0000 7fc5fea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 0x7fc60014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 0x7fc60014e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45916/0 (socket says 172.21.15.146:45916) 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 -- 172.21.15.146:0/2178689268 learned_addr learned my addr 172.21.15.146:0/2178689268 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5ffe00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc60015b190 0x7fc60014e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5ff400640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6000d0d00 0x7fc6000cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 -- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6000d0d00 msgr2=0x7fc6000cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6000d0d00 0x7fc6000cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 -- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc60015b190 msgr2=0x7fc60014e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc60015b190 0x7fc60014e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 -- 172.21.15.146:0/2178689268 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6000c82e0 con 0x7fc600154ac0 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5fea00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 0x7fc60014e1b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec000c00 tx=0x7fc5ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec037040 con 0x7fc600154ac0 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ec042070 con 0x7fc600154ac0 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec030900 con 0x7fc600154ac0 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.024+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6000ca0d0 con 0x7fc600154ac0 2024-09-16T01:49:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.025+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6000c8ac0 con 0x7fc600154ac0 2024-09-16T01:49:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.026+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60015b5e0 con 0x7fc600154ac0 2024-09-16T01:49:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.027+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc5ec030aa0 con 0x7fc600154ac0 2024-09-16T01:49:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.027+0000 7fc5f7400640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc5e006c2b0 0x7fc5e006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.028+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc5ec0abd10 con 0x7fc600154ac0 2024-09-16T01:49:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.028+0000 7fc5ff400640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc5e006c2b0 0x7fc5e006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.028+0000 7fc5ff400640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc5e006c2b0 0x7fc5e006e770 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000f60 tx=0x7fc5f0007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:02.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.031+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc5ec07a510 con 0x7fc600154ac0 2024-09-16T01:49:02.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.240+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fc6000ce990 con 0x7fc600154ac0 2024-09-16T01:49:02.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.840+0000 7fc5f7400640 1 -- 172.21.15.146:0/2178689268 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7fc5ec079e10 con 0x7fc600154ac0 2024-09-16T01:49:02.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc5e006c2b0 msgr2=0x7fc5e006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:02.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc5e006c2b0 0x7fc5e006e770 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000f60 tx=0x7fc5f0007b50 comp rx=0 tx=0).stop 2024-09-16T01:49:02.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 msgr2=0x7fc60014e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:02.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 0x7fc60014e1b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec000c00 tx=0x7fc5ec0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 shutdown_connections 2024-09-16T01:49:02.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc5e006c2b0 0x7fc5e006e770 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc60015b190 0x7fc60014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc600154ac0 0x7fc60014e1b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 --2- 172.21.15.146:0/2178689268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6000d0d00 0x7fc6000cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:02.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 >> 172.21.15.146:0/2178689268 conn(0x7fc60006e400 msgr2=0x7fc600150790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:02.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 shutdown_connections 2024-09-16T01:49:02.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:02.843+0000 7fc604c00640 1 -- 172.21.15.146:0/2178689268 wait complete. 2024-09-16T01:49:02.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-09-16T01:49:03.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.017+0000 7fa0c9600640 1 Processor -- start 2024-09-16T01:49:03.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.017+0000 7fa0c9600640 1 -- start start 2024-09-16T01:49:03.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0c9600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0c415b190 0x7fa0c40d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:03.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0c9600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c4154ea0 con 0x7fa0c4154ac0 2024-09-16T01:49:03.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0c9600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0c40c5340 con 0x7fa0c40d0d00 2024-09-16T01:49:03.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0c9600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa0c40c5620 con 0x7fa0c415b190 2024-09-16T01:49:03.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0c3e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa0c4154ac0 0x7fa0c406d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53572/0 (socket says 172.21.15.146:53572) 2024-09-16T01:49:03.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0c3e00640 1 -- 172.21.15.146:0/3917733866 learned_addr learned my addr 172.21.15.146:0/3917733866 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:03.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.018+0000 7fa0bbe00640 1 --2- 172.21.15.146:0/3917733866 >> v2:172.21.15.161:3300/0 conn(0x7fa0c415b190 0x7fa0c40d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 467152505 0 0) 0x7fa0c4154ea0 con 0x7fa0c4154ac0 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0a8003520 con 0x7fa0c4154ac0 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757808225 0 0) 0x7fa0c40c5340 con 0x7fa0c40d0d00 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0c4154ea0 con 0x7fa0c40d0d00 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0bbe00640 1 -- 172.21.15.146:0/3917733866 >> v1:172.21.15.161:6789/0 conn(0x7fa0c40d0d00 legacy=0x7fa0c415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0bbe00640 1 -- 172.21.15.146:0/3917733866 >> v1:172.21.15.146:6789/0 conn(0x7fa0c4154ac0 legacy=0x7fa0c406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.019+0000 7fa0bbe00640 1 -- 172.21.15.146:0/3917733866 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c40c5900 con 0x7fa0c415b190 2024-09-16T01:49:03.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.020+0000 7fa0bbe00640 1 --2- 172.21.15.146:0/3917733866 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c415b190 0x7fa0c40d07c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0097a0 tx=0x7fa0ac02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff693dfcaf6899 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:03.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.020+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac037070 con 0x7fa0c415b190 2024-09-16T01:49:03.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.020+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0ac032640 con 0x7fa0c415b190 2024-09-16T01:49:03.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.020+0000 7fa0c3400640 1 -- 172.21.15.146:0/3917733866 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0329f0 con 0x7fa0c415b190 2024-09-16T01:49:03.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 -- 172.21.15.146:0/3917733866 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c415b190 msgr2=0x7fa0c40d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 --2- 172.21.15.146:0/3917733866 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c415b190 0x7fa0c40d07c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0097a0 tx=0x7fa0ac02a2c0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 -- 172.21.15.146:0/3917733866 shutdown_connections 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 --2- 172.21.15.146:0/3917733866 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c415b190 0x7fa0c40d07c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 -- 172.21.15.146:0/3917733866 >> 172.21.15.146:0/3917733866 conn(0x7fa0c406e400 msgr2=0x7fa0c406e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 -- 172.21.15.146:0/3917733866 shutdown_connections 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.021+0000 7fa0c9600640 1 -- 172.21.15.146:0/3917733866 wait complete. 2024-09-16T01:49:03.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.022+0000 7fa0c9600640 1 Processor -- start 2024-09-16T01:49:03.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 -- start start 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c40d0d00 0x7fa0c415bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154ac0 0x7fa0c41f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 0x7fa0c414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c40c61a0 con 0x7fa0c415b190 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c40c5ec0 con 0x7fa0c4154ac0 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.023+0000 7fa0c9600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c40cad60 con 0x7fa0c40d0d00 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 0x7fa0c414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 0x7fa0c414ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58872/0 (socket says 172.21.15.146:58872) 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 -- 172.21.15.146:0/2155296148 learned_addr learned my addr 172.21.15.146:0/2155296148 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c3e00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c40d0d00 0x7fa0c415bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0bbe00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154ac0 0x7fa0c41f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 -- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c40d0d00 msgr2=0x7fa0c415bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c40d0d00 0x7fa0c415bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 -- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154ac0 msgr2=0x7fa0c41f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154ac0 0x7fa0c41f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 -- 172.21.15.146:0/2155296148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c41f4620 con 0x7fa0c415b190 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.024+0000 7fa0c8c00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 0x7fa0c414ab80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c7d0 tx=0x7fa0bc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.025+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc003b70 con 0x7fa0c415b190 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.025+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0bc003d30 con 0x7fa0c415b190 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.025+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00aa90 con 0x7fa0c415b190 2024-09-16T01:49:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.025+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c40ca1a0 con 0x7fa0c415b190 2024-09-16T01:49:03.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.025+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c414cff0 con 0x7fa0c415b190 2024-09-16T01:49:03.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.026+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c40ca6c0 con 0x7fa0c415b190 2024-09-16T01:49:03.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.027+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa0bc00ac30 con 0x7fa0c415b190 2024-09-16T01:49:03.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.028+0000 7fa0c0c00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0a006c2b0 0x7fa0a006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:03.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.028+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa0bc08b2c0 con 0x7fa0c415b190 2024-09-16T01:49:03.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.028+0000 7fa0c3e00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0a006c2b0 0x7fa0a006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:03.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.029+0000 7fa0c3e00640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0a006c2b0 0x7fa0a006e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40011e0 tx=0x7fa0b4007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:03.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.033+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa0bc0593e0 con 0x7fa0c415b190 2024-09-16T01:49:03.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.241+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa0c415b5e0 con 0x7fa0c415b190 2024-09-16T01:49:03.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.842+0000 7fa0c0c00640 1 -- 172.21.15.146:0/2155296148 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) ==== 115+0+0 (secure 0 0 0) 0x7fa0bc05d6e0 con 0x7fa0c415b190 2024-09-16T01:49:03.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0a006c2b0 msgr2=0x7fa0a006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0a006c2b0 0x7fa0a006e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40011e0 tx=0x7fa0b4007d30 comp rx=0 tx=0).stop 2024-09-16T01:49:03.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 msgr2=0x7fa0c414ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 0x7fa0c414ab80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c7d0 tx=0x7fa0bc00cc90 comp rx=0 tx=0).stop 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 shutdown_connections 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0a006c2b0 0x7fa0a006e770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0c415b190 0x7fa0c414ab80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0c4154ac0 0x7fa0c41f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 --2- 172.21.15.146:0/2155296148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0c40d0d00 0x7fa0c415bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 >> 172.21.15.146:0/2155296148 conn(0x7fa0c406e400 msgr2=0x7fa0c4150f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 shutdown_connections 2024-09-16T01:49:03.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:03.845+0000 7fa0c9600640 1 -- 172.21.15.146:0/2155296148 wait complete. 2024-09-16T01:49:03.865 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:49:03.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:04.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.022+0000 7fd8fea00640 1 Processor -- start 2024-09-16T01:49:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.022+0000 7fd8fea00640 1 -- start start 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fea00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd8f80c9700 0x7fd8f8164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f8072f50 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fea00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f815cb50 con 0x7fd8f8152850 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fea00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f8073210 con 0x7fd8f80c9700 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8ef400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd8f80c6bb0 0x7fd8f80c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53586/0 (socket says 172.21.15.146:53586) 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8ef400640 1 -- 172.21.15.146:0/770076988 learned_addr learned my addr 172.21.15.146:0/770076988 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fca00640 1 --2- 172.21.15.146:0/770076988 >> v2:172.21.15.161:3300/0 conn(0x7fd8f80c9700 0x7fd8f8164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876445600 0 0) 0x7fd8f8072f50 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8dc003520 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4016555768 0 0) 0x7fd8f815cb50 con 0x7fd8f8152850 2024-09-16T01:49:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8f8072f50 con 0x7fd8f8152850 2024-09-16T01:49:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fca00640 1 -- 172.21.15.146:0/770076988 >> v1:172.21.15.161:6789/0 conn(0x7fd8f8152850 legacy=0x7fd8f8158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.024+0000 7fd8fca00640 1 -- 172.21.15.146:0/770076988 >> v1:172.21.15.146:6789/0 conn(0x7fd8f80c6bb0 legacy=0x7fd8f80c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.025+0000 7fd8fca00640 1 -- 172.21.15.146:0/770076988 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f815bd90 con 0x7fd8f80c9700 2024-09-16T01:49:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.025+0000 7fd8fca00640 1 --2- 172.21.15.146:0/770076988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f8164000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00097a0 tx=0x7fd8f002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e0deb25af5b8000 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0037070 con 0x7fd8f80c9700 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f0032440 con 0x7fd8f80c9700 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8f7400640 1 -- 172.21.15.146:0/770076988 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00327f0 con 0x7fd8f80c9700 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8fea00640 1 -- 172.21.15.146:0/770076988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 msgr2=0x7fd8f8164000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8fea00640 1 --2- 172.21.15.146:0/770076988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f8164000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00097a0 tx=0x7fd8f002a230 comp rx=0 tx=0).stop 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8fea00640 1 -- 172.21.15.146:0/770076988 shutdown_connections 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8fea00640 1 --2- 172.21.15.146:0/770076988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f8164000 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.026+0000 7fd8fea00640 1 -- 172.21.15.146:0/770076988 >> 172.21.15.146:0/770076988 conn(0x7fd8f806f3c0 msgr2=0x7fd8f806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.027+0000 7fd8fea00640 1 -- 172.21.15.146:0/770076988 shutdown_connections 2024-09-16T01:49:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.027+0000 7fd8fea00640 1 -- 172.21.15.146:0/770076988 wait complete. 2024-09-16T01:49:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.028+0000 7fd8fea00640 1 Processor -- start 2024-09-16T01:49:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.028+0000 7fd8fea00640 1 -- start start 2024-09-16T01:49:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 0x7fd8f81427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f8152850 0x7fd8f80cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f815c630 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fea00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f815c350 con 0x7fd8f80c9700 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fea00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f8070200 con 0x7fd8f8152850 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8f7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 0x7fd8f81427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8f7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 0x7fd8f81427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58888/0 (socket says 172.21.15.146:58888) 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8f7e00640 1 -- 172.21.15.146:0/1739142969 learned_addr learned my addr 172.21.15.146:0/1739142969 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8ef400640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.029+0000 7fd8fca00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f8152850 0x7fd8f80cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f7e00640 1 -- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f8152850 msgr2=0x7fd8f80cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f7e00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f8152850 0x7fd8f80cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f7e00640 1 -- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 msgr2=0x7fd8f80cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f7e00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f80cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f7e00640 1 -- 172.21.15.146:0/1739142969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f81ee430 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f7e00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 0x7fd8f81427a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c7d0 tx=0x7fd8e800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8003b70 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8003d10 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e800aa90 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f81ee710 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.030+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f81eecd0 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.031+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f80718a0 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.032+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd8e800ac30 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.033+0000 7fd8f4c00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd8d406c2e0 0x7fd8d406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.033+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd8e808b2d0 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.033+0000 7fd8ef400640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd8d406c2e0 0x7fd8d406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.034+0000 7fd8ef400640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd8d406c2e0 0x7fd8d406e7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8148ed0 tx=0x7fd8e0007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.037+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd8e8059a30 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.247+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8f806eb50 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.248+0000 7fd8f4c00640 1 -- 172.21.15.146:0/1739142969 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6894 (secure 0 0 0) 0x7fd8e808b0d0 con 0x7fd8f80c6bb0 2024-09-16T01:49:04.249 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:04.249 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":9,"btime":"2024-09-16T01:49:03:845183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4259,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4277,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:03.845171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4291,"mds_1":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4291]},"id":1}]} 2024-09-16T01:49:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 9 2024-09-16T01:49:04.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd8d406c2e0 msgr2=0x7fd8d406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd8d406c2e0 0x7fd8d406e7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8148ed0 tx=0x7fd8e0007d30 comp rx=0 tx=0).stop 2024-09-16T01:49:04.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 msgr2=0x7fd8f81427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 0x7fd8f81427a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c7d0 tx=0x7fd8e800cc90 comp rx=0 tx=0).stop 2024-09-16T01:49:04.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 shutdown_connections 2024-09-16T01:49:04.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd8d406c2e0 0x7fd8d406e7a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f8152850 0x7fd8f80cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f80c9700 0x7fd8f80cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 --2- 172.21.15.146:0/1739142969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8f80c6bb0 0x7fd8f81427a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 >> 172.21.15.146:0/1739142969 conn(0x7fd8f806f3c0 msgr2=0x7fd8f8150da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 shutdown_connections 2024-09-16T01:49:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.250+0000 7fd8fea00640 1 -- 172.21.15.146:0/1739142969 wait complete. 2024-09-16T01:49:04.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:49:04.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.419+0000 7f4abca00640 1 Processor -- start 2024-09-16T01:49:04.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.419+0000 7f4abca00640 1 -- start start 2024-09-16T01:49:04.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4abca00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4ab80c75a0 0x7f4ab815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4abca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80cc040 0x7f4ab80cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4abca00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ab806d140 con 0x7f4ab80c6b70 2024-09-16T01:49:04.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4abca00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4ab815b880 con 0x7f4ab80c75a0 2024-09-16T01:49:04.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4abca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab80cab20 con 0x7f4ab80cc040 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4ab6800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4ab80c6b70 0x7f4ab80cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52630/0 (socket says 172.21.15.146:52630) 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4ab6800640 1 -- 172.21.15.146:0/3874427885 learned_addr learned my addr 172.21.15.146:0/3874427885 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4ab5e00640 1 --2- 172.21.15.146:0/3874427885 >> v2:172.21.15.161:3300/0 conn(0x7f4ab80c75a0 0x7f4ab815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.420+0000 7f4ab7200640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80cc040 0x7f4ab80cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab5400640 1 -- 172.21.15.146:0/3874427885 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859032798 0 0) 0x7f4ab806d140 con 0x7f4ab80c6b70 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab5400640 1 -- 172.21.15.146:0/3874427885 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a9c003520 con 0x7f4ab80c6b70 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab5e00640 1 -- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80cc040 msgr2=0x7f4ab80cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab5e00640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80cc040 0x7f4ab80cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab5e00640 1 -- 172.21.15.146:0/3874427885 >> v1:172.21.15.161:6789/0 conn(0x7f4ab80c6b70 legacy=0x7f4ab80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab5e00640 1 -- 172.21.15.146:0/3874427885 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab806d420 con 0x7f4ab80c75a0 2024-09-16T01:49:04.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.421+0000 7f4ab7200640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80cc040 0x7f4ab80cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:04.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.422+0000 7f4ab5e00640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab815b340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40097a0 tx=0x7f4aa402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83b88cef03795cbe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.422+0000 7f4ab5400640 1 -- 172.21.15.146:0/3874427885 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4037070 con 0x7f4ab80c75a0 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.422+0000 7f4ab5400640 1 -- 172.21.15.146:0/3874427885 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa402ae50 con 0x7f4ab80c75a0 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.422+0000 7f4ab5400640 1 -- 172.21.15.146:0/3874427885 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4032640 con 0x7f4ab80c75a0 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 -- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 msgr2=0x7f4ab815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab815b340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40097a0 tx=0x7f4aa402a100 comp rx=0 tx=0).stop 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 -- 172.21.15.146:0/3874427885 shutdown_connections 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80cc040 0x7f4ab80cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 --2- 172.21.15.146:0/3874427885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab815b340 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 -- 172.21.15.146:0/3874427885 >> 172.21.15.146:0/3874427885 conn(0x7f4ab80ccc20 msgr2=0x7f4ab80cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 -- 172.21.15.146:0/3874427885 shutdown_connections 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.423+0000 7f4abca00640 1 -- 172.21.15.146:0/3874427885 wait complete. 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.424+0000 7f4abca00640 1 Processor -- start 2024-09-16T01:49:04.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 -- start start 2024-09-16T01:49:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80c6b70 0x7f4ab815ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab8156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 0x7f4ab81f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab80cb050 con 0x7f4ab80cc040 2024-09-16T01:49:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab80cad70 con 0x7f4ab80c75a0 2024-09-16T01:49:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.425+0000 7f4abca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab80cbc80 con 0x7f4ab80c6b70 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 0x7f4ab81f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 0x7f4ab81f0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58898/0 (socket says 172.21.15.146:58898) 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 -- 172.21.15.146:0/691497399 learned_addr learned my addr 172.21.15.146:0/691497399 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab5e00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab8156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab6800640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80c6b70 0x7f4ab815ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 -- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80c6b70 msgr2=0x7f4ab815ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80c6b70 0x7f4ab815ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 -- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 msgr2=0x7f4ab8156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab8156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 -- 172.21.15.146:0/691497399 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab81f1410 con 0x7f4ab80cc040 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab5e00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab8156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.426+0000 7f4ab7200640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 0x7f4ab81f0b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000ef90 tx=0x7f4ab0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.427+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0018020 con 0x7f4ab80cc040 2024-09-16T01:49:04.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.427+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab0014480 con 0x7f4ab80cc040 2024-09-16T01:49:04.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.427+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab81f16f0 con 0x7f4ab80cc040 2024-09-16T01:49:04.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.427+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0014870 con 0x7f4ab80cc040 2024-09-16T01:49:04.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.427+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab81564d0 con 0x7f4ab80cc040 2024-09-16T01:49:04.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.428+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab80c9760 con 0x7f4ab80cc040 2024-09-16T01:49:04.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.429+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4ab0005d10 con 0x7f4ab80cc040 2024-09-16T01:49:04.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.430+0000 7f4aaea00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a9806c2b0 0x7f4a9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.430+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4ab008c1d0 con 0x7f4ab80cc040 2024-09-16T01:49:04.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.430+0000 7f4ab6800640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a9806c2b0 0x7f4a9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.431+0000 7f4ab6800640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a9806c2b0 0x7f4a9806e770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005880 tx=0x7f4aa8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.434+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4ab80c9760 con 0x7f4ab80cc040 2024-09-16T01:49:04.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.614+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ab806cd20 con 0x7f4ab80cc040 2024-09-16T01:49:04.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.616+0000 7f4aaea00640 1 -- 172.21.15.146:0/691497399 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16693 (secure 0 0 0) 0x7f4ab0022030 con 0x7f4ab80cc040 2024-09-16T01:49:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:04.617 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":22,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:59.816171+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a9806c2b0 msgr2=0x7f4a9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a9806c2b0 0x7f4a9806e770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005880 tx=0x7f4aa8009040 comp rx=0 tx=0).stop 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 msgr2=0x7f4ab81f0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 0x7f4ab81f0b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000ef90 tx=0x7f4ab0009f90 comp rx=0 tx=0).stop 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 shutdown_connections 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a9806c2b0 0x7f4a9806e770 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ab80cc040 0x7f4ab81f0b80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ab80c75a0 0x7f4ab8156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 --2- 172.21.15.146:0/691497399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ab80c6b70 0x7f4ab815ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.618+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 >> 172.21.15.146:0/691497399 conn(0x7f4ab80ccc20 msgr2=0x7f4ab80c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.619+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 shutdown_connections 2024-09-16T01:49:04.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.619+0000 7f4abca00640 1 -- 172.21.15.146:0/691497399 wait complete. 2024-09-16T01:49:04.635 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:49:04.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:04.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.786+0000 7f4d65000640 1 Processor -- start 2024-09-16T01:49:04.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.786+0000 7f4d65000640 1 -- start start 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d65000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4d6015b340 0x7f4d60166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d65000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d6006d140 con 0x7f4d600c75a0 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d65000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d6006d420 con 0x7f4d600c6b70 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d65000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4d600cab20 con 0x7f4d6015b340 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d5ea00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4d600c75a0 0x7f4d600cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53588/0 (socket says 172.21.15.146:53588) 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d5ea00640 1 -- 172.21.15.146:0/281547719 learned_addr learned my addr 172.21.15.146:0/281547719 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d5f400640 1 --1- 172.21.15.146:0/281547719 >> v1:172.21.15.161:6789/0 conn(0x7f4d600c6b70 0x7f4d600cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52638/0 (socket says 172.21.15.146:52638) 2024-09-16T01:49:04.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.787+0000 7f4d5fe00640 1 --2- 172.21.15.146:0/281547719 >> v2:172.21.15.146:3300/0 conn(0x7f4d6015b340 0x7f4d60166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054810189 0 0) 0x7f4d6006d140 con 0x7f4d600c75a0 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d44003520 con 0x7f4d600c75a0 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791704784 0 0) 0x7f4d6006d420 con 0x7f4d600c6b70 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d6006d140 con 0x7f4d600c6b70 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5fe00640 1 -- 172.21.15.146:0/281547719 >> v1:172.21.15.161:6789/0 conn(0x7f4d600c6b70 legacy=0x7f4d600cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5fe00640 1 -- 172.21.15.146:0/281547719 >> v1:172.21.15.146:6789/0 conn(0x7f4d600c75a0 legacy=0x7f4d600cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.788+0000 7f4d5fe00640 1 -- 172.21.15.146:0/281547719 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6006d700 con 0x7f4d6015b340 2024-09-16T01:49:04.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.789+0000 7f4d5fe00640 1 --2- 172.21.15.146:0/281547719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6015b340 0x7f4d60166170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009d90 tx=0x7f4d5802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b65de98cc5b6374 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.789+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58007740 con 0x7f4d6015b340 2024-09-16T01:49:04.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.789+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d580078e0 con 0x7f4d6015b340 2024-09-16T01:49:04.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.789+0000 7f4d5e000640 1 -- 172.21.15.146:0/281547719 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58007c90 con 0x7f4d6015b340 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 -- 172.21.15.146:0/281547719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6015b340 msgr2=0x7f4d60166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 --2- 172.21.15.146:0/281547719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6015b340 0x7f4d60166170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009d90 tx=0x7f4d5802a660 comp rx=0 tx=0).stop 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 -- 172.21.15.146:0/281547719 shutdown_connections 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 --2- 172.21.15.146:0/281547719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d6015b340 0x7f4d60166170 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 -- 172.21.15.146:0/281547719 >> 172.21.15.146:0/281547719 conn(0x7f4d600ccc20 msgr2=0x7f4d600cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 -- 172.21.15.146:0/281547719 shutdown_connections 2024-09-16T01:49:04.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.790+0000 7f4d65000640 1 -- 172.21.15.146:0/281547719 wait complete. 2024-09-16T01:49:04.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.791+0000 7f4d65000640 1 Processor -- start 2024-09-16T01:49:04.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.791+0000 7f4d65000640 1 -- start start 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.792+0000 7f4d65000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d600c6b70 0x7f4d601f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.792+0000 7f4d65000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 0x7f4d601e6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.792+0000 7f4d65000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4d6015b340 0x7f4d6015e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.792+0000 7f4d65000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6015b7c0 con 0x7f4d600c75a0 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.792+0000 7f4d65000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4d600cb050 con 0x7f4d600c6b70 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.792+0000 7f4d65000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4d600cbc80 con 0x7f4d6015b340 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 0x7f4d601e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d600c6b70 0x7f4d601f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4d6015b340 0x7f4d6015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 0x7f4d601e6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58924/0 (socket says 172.21.15.146:58924) 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d600c6b70 0x7f4d601f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44990/0 (socket says 172.21.15.146:44990) 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4d6015b340 0x7f4d6015e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37430/0 (socket says 172.21.15.146:37430) 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 -- 172.21.15.146:0/2946816707 learned_addr learned my addr 172.21.15.146:0/2946816707 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 -- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4d6015b340 msgr2=0x7f4d6015e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4d6015b340 0x7f4d6015e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 -- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d600c6b70 msgr2=0x7f4d601f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d600c6b70 0x7f4d601f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 -- 172.21.15.146:0/2946816707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d600c2fb0 con 0x7f4d600c75a0 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d5ea00640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 0x7f4d601e6240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c500 tx=0x7f4d4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.793+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c017070 con 0x7f4d600c75a0 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.794+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d4c005d00 con 0x7f4d600c75a0 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.794+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c007330 con 0x7f4d600c75a0 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.794+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d60156c40 con 0x7f4d600c75a0 2024-09-16T01:49:04.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.794+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d601574a0 con 0x7f4d600c75a0 2024-09-16T01:49:04.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.795+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d6006cd20 con 0x7f4d600c75a0 2024-09-16T01:49:04.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.796+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4d4c01a070 con 0x7f4d600c75a0 2024-09-16T01:49:04.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.796+0000 7f4d57400640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4d4006c2b0 0x7f4d4006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:04.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.797+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4d4c08a820 con 0x7f4d600c75a0 2024-09-16T01:49:04.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.797+0000 7f4d5f400640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4d4006c2b0 0x7f4d4006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:04.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.797+0000 7f4d5f400640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4d4006c2b0 0x7f4d4006e770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d500011e0 tx=0x7f4d50007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:04.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.801+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4d4c0589f0 con 0x7f4d600c75a0 2024-09-16T01:49:04.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.976+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d6006da90 con 0x7f4d4006c2b0 2024-09-16T01:49:04.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.984+0000 7f4d57400640 1 -- 172.21.15.146:0/2946816707 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251903 (secure 0 0 0) 0x7f4d6006da90 con 0x7f4d4006c2b0 2024-09-16T01:49:04.988 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:04.989 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.987+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4d4006c2b0 msgr2=0x7f4d4006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.987+0000 7f4d65000640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4d4006c2b0 0x7f4d4006e770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d500011e0 tx=0x7f4d50007d30 comp rx=0 tx=0).stop 2024-09-16T01:49:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.987+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 msgr2=0x7f4d601e6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.987+0000 7f4d65000640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 0x7f4d601e6240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c500 tx=0x7f4d4c00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.987+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 shutdown_connections 2024-09-16T01:49:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.987+0000 7f4d65000640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4d4006c2b0 0x7f4d4006e770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.988+0000 7f4d65000640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4d6015b340 0x7f4d6015e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.988+0000 7f4d65000640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d600c75a0 0x7f4d601e6240 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.988+0000 7f4d65000640 1 --2- 172.21.15.146:0/2946816707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4d600c6b70 0x7f4d601f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.988+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 >> 172.21.15.146:0/2946816707 conn(0x7f4d600ccc20 msgr2=0x7f4d600c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.988+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 shutdown_connections 2024-09-16T01:49:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:04.988+0000 7f4d65000640 1 -- 172.21.15.146:0/2946816707 wait complete. 2024-09-16T01:49:05.005 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T01:49:04.519183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218056,"kb_used_data":1808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756664,"statfs":{"total":773094113280,"available":772870823936,"internally_reserved":0,"allocated":1851392,"data_stored":1781744,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.726208"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829290+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:48:58.829290+0000","last_peered":"2024-09-16T01:48:58.829290+0000","last_clean":"2024-09-16T01:48:58.829290+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:48:58.829290+0000","last_undegraded":"2024-09-16T01:48:58.829290+0000","last_fullsized":"2024-09-16T01:48:58.829290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.830931+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:48:58.830931+0000","last_peered":"2024-09-16T01:48:58.830931+0000","last_clean":"2024-09-16T01:48:58.830931+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:48:58.830931+0000","last_undegraded":"2024-09-16T01:48:58.830931+0000","last_fullsized":"2024-09-16T01:48:58.830931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829354+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:48:58.829354+0000","last_peered":"2024-09-16T01:48:58.829354+0000","last_clean":"2024-09-16T01:48:58.829354+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:48:58.829354+0000","last_undegraded":"2024-09-16T01:48:58.829354+0000","last_fullsized":"2024-09-16T01:48:58.829354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.833773+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:48:58.833773+0000","last_peered":"2024-09-16T01:48:58.833773+0000","last_clean":"2024-09-16T01:48:58.833773+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:48:58.833773+0000","last_undegraded":"2024-09-16T01:48:58.833773+0000","last_fullsized":"2024-09-16T01:48:58.833773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839135+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:48:58.839135+0000","last_peered":"2024-09-16T01:48:58.839135+0000","last_clean":"2024-09-16T01:48:58.839135+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:48:58.839135+0000","last_undegraded":"2024-09-16T01:48:58.839135+0000","last_fullsized":"2024-09-16T01:48:58.839135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:43.838815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839412+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:58.839412+0000","last_peered":"2024-09-16T01:48:58.839412+0000","last_clean":"2024-09-16T01:48:58.839412+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:58.839412+0000","last_undegraded":"2024-09-16T01:48:58.839412+0000","last_fullsized":"2024-09-16T01:48:58.839412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:42.684463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839409+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:48:58.839409+0000","last_peered":"2024-09-16T01:48:58.839409+0000","last_clean":"2024-09-16T01:48:58.839409+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:48:58.839409+0000","last_undegraded":"2024-09-16T01:48:58.839409+0000","last_fullsized":"2024-09-16T01:48:58.839409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829168+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:48:58.829168+0000","last_peered":"2024-09-16T01:48:58.829168+0000","last_clean":"2024-09-16T01:48:58.829168+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:48:58.829168+0000","last_undegraded":"2024-09-16T01:48:58.829168+0000","last_fullsized":"2024-09-16T01:48:58.829168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.833752+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:48:58.833752+0000","last_peered":"2024-09-16T01:48:58.833752+0000","last_clean":"2024-09-16T01:48:58.833752+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:48:58.833752+0000","last_undegraded":"2024-09-16T01:48:58.833752+0000","last_fullsized":"2024-09-16T01:48:58.833752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.830946+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:48:58.830946+0000","last_peered":"2024-09-16T01:48:58.830946+0000","last_clean":"2024-09-16T01:48:58.830946+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:48:58.830946+0000","last_undegraded":"2024-09-16T01:48:58.830946+0000","last_fullsized":"2024-09-16T01:48:58.830946+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:48.348594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.814813+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:48:58.814813+0000","last_peered":"2024-09-16T01:48:58.814813+0000","last_clean":"2024-09-16T01:48:58.814813+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:48:58.814813+0000","last_undegraded":"2024-09-16T01:48:58.814813+0000","last_fullsized":"2024-09-16T01:48:58.814813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.841991+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:58.841991+0000","last_peered":"2024-09-16T01:48:58.841991+0000","last_clean":"2024-09-16T01:48:58.841991+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:58.841991+0000","last_undegraded":"2024-09-16T01:48:58.841991+0000","last_fullsized":"2024-09-16T01:48:58.841991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:03:51.311804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039329+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:01.039329+0000","last_peered":"2024-09-16T01:49:01.039329+0000","last_clean":"2024-09-16T01:49:01.039329+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:01.039329+0000","last_undegraded":"2024-09-16T01:49:01.039329+0000","last_fullsized":"2024-09-16T01:49:01.039329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825347+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:48:59.825347+0000","last_peered":"2024-09-16T01:48:59.825347+0000","last_clean":"2024-09-16T01:48:59.825347+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:48:59.825347+0000","last_undegraded":"2024-09-16T01:48:59.825347+0000","last_fullsized":"2024-09-16T01:48:59.825347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829333+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:48:58.829333+0000","last_peered":"2024-09-16T01:48:58.829333+0000","last_clean":"2024-09-16T01:48:58.829333+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:48:58.829333+0000","last_undegraded":"2024-09-16T01:48:58.829333+0000","last_fullsized":"2024-09-16T01:48:58.829333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:43.838815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038521+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:01.038521+0000","last_peered":"2024-09-16T01:49:01.038521+0000","last_clean":"2024-09-16T01:49:01.038521+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:01.038521+0000","last_undegraded":"2024-09-16T01:49:01.038521+0000","last_fullsized":"2024-09-16T01:49:01.038521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.831337+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:58.831337+0000","last_peered":"2024-09-16T01:48:58.831337+0000","last_clean":"2024-09-16T01:48:58.831337+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:58.831337+0000","last_undegraded":"2024-09-16T01:48:58.831337+0000","last_fullsized":"2024-09-16T01:48:58.831337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:15.774154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040713+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:01.040713+0000","last_peered":"2024-09-16T01:49:01.040713+0000","last_clean":"2024-09-16T01:49:01.040713+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:01.040713+0000","last_undegraded":"2024-09-16T01:49:01.040713+0000","last_fullsized":"2024-09-16T01:49:01.040713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039489+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:01.039489+0000","last_peered":"2024-09-16T01:49:01.039489+0000","last_clean":"2024-09-16T01:49:01.039489+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:01.039489+0000","last_undegraded":"2024-09-16T01:49:01.039489+0000","last_fullsized":"2024-09-16T01:49:01.039489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.842051+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:48:58.842051+0000","last_peered":"2024-09-16T01:48:58.842051+0000","last_clean":"2024-09-16T01:48:58.842051+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:48:58.842051+0000","last_undegraded":"2024-09-16T01:48:58.842051+0000","last_fullsized":"2024-09-16T01:48:58.842051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:27.080013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.816248+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:48:58.816248+0000","last_peered":"2024-09-16T01:48:58.816248+0000","last_clean":"2024-09-16T01:48:58.816248+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:48:58.816248+0000","last_undegraded":"2024-09-16T01:48:58.816248+0000","last_fullsized":"2024-09-16T01:48:58.816248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824172+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:48:59.824172+0000","last_peered":"2024-09-16T01:48:59.824172+0000","last_clean":"2024-09-16T01:48:59.824172+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:48:59.824172+0000","last_undegraded":"2024-09-16T01:48:59.824172+0000","last_fullsized":"2024-09-16T01:48:59.824172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.816181+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:48:58.816181+0000","last_peered":"2024-09-16T01:48:58.816181+0000","last_clean":"2024-09-16T01:48:58.816181+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:48:58.816181+0000","last_undegraded":"2024-09-16T01:48:58.816181+0000","last_fullsized":"2024-09-16T01:48:58.816181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039468+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:01.039468+0000","last_peered":"2024-09-16T01:49:01.039468+0000","last_clean":"2024-09-16T01:49:01.039468+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:01.039468+0000","last_undegraded":"2024-09-16T01:49:01.039468+0000","last_fullsized":"2024-09-16T01:49:01.039468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038796+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:01.038796+0000","last_peered":"2024-09-16T01:49:01.038796+0000","last_clean":"2024-09-16T01:49:01.038796+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:01.038796+0000","last_undegraded":"2024-09-16T01:49:01.038796+0000","last_fullsized":"2024-09-16T01:49:01.038796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.818853+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:48:58.818853+0000","last_peered":"2024-09-16T01:48:58.818853+0000","last_clean":"2024-09-16T01:48:58.818853+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:48:58.818853+0000","last_undegraded":"2024-09-16T01:48:58.818853+0000","last_fullsized":"2024-09-16T01:48:58.818853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039958+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:01.039958+0000","last_peered":"2024-09-16T01:49:01.039958+0000","last_clean":"2024-09-16T01:49:01.039958+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:01.039958+0000","last_undegraded":"2024-09-16T01:49:01.039958+0000","last_fullsized":"2024-09-16T01:49:01.039958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820575+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:48:58.820575+0000","last_peered":"2024-09-16T01:48:58.820575+0000","last_clean":"2024-09-16T01:48:58.820575+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:48:58.820575+0000","last_undegraded":"2024-09-16T01:48:58.820575+0000","last_fullsized":"2024-09-16T01:48:58.820575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.813221+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:48:58.813221+0000","last_peered":"2024-09-16T01:48:58.813221+0000","last_clean":"2024-09-16T01:48:58.813221+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:48:58.813221+0000","last_undegraded":"2024-09-16T01:48:58.813221+0000","last_fullsized":"2024-09-16T01:48:58.813221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825768+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:48:59.825768+0000","last_peered":"2024-09-16T01:48:59.825768+0000","last_clean":"2024-09-16T01:48:59.825768+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:48:59.825768+0000","last_undegraded":"2024-09-16T01:48:59.825768+0000","last_fullsized":"2024-09-16T01:48:59.825768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039827+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:01.039827+0000","last_peered":"2024-09-16T01:49:01.039827+0000","last_clean":"2024-09-16T01:49:01.039827+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:01.039827+0000","last_undegraded":"2024-09-16T01:49:01.039827+0000","last_fullsized":"2024-09-16T01:49:01.039827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839403+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:58.839403+0000","last_peered":"2024-09-16T01:48:58.839403+0000","last_clean":"2024-09-16T01:48:58.839403+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:58.839403+0000","last_undegraded":"2024-09-16T01:48:58.839403+0000","last_fullsized":"2024-09-16T01:48:58.839403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.059699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038630+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:01.038630+0000","last_peered":"2024-09-16T01:49:01.038630+0000","last_clean":"2024-09-16T01:49:01.038630+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:01.038630+0000","last_undegraded":"2024-09-16T01:49:01.038630+0000","last_fullsized":"2024-09-16T01:49:01.038630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.816050+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:48:58.816050+0000","last_peered":"2024-09-16T01:48:58.816050+0000","last_clean":"2024-09-16T01:48:58.816050+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:48:58.816050+0000","last_undegraded":"2024-09-16T01:48:58.816050+0000","last_fullsized":"2024-09-16T01:48:58.816050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.833799+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:48:58.833799+0000","last_peered":"2024-09-16T01:48:58.833799+0000","last_clean":"2024-09-16T01:48:58.833799+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:48:58.833799+0000","last_undegraded":"2024-09-16T01:48:58.833799+0000","last_fullsized":"2024-09-16T01:48:58.833799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039485+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:01.039485+0000","last_peered":"2024-09-16T01:49:01.039485+0000","last_clean":"2024-09-16T01:49:01.039485+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:01.039485+0000","last_undegraded":"2024-09-16T01:49:01.039485+0000","last_fullsized":"2024-09-16T01:49:01.039485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.813197+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:48:58.813197+0000","last_peered":"2024-09-16T01:48:58.813197+0000","last_clean":"2024-09-16T01:48:58.813197+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:48:58.813197+0000","last_undegraded":"2024-09-16T01:48:58.813197+0000","last_fullsized":"2024-09-16T01:48:58.813197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040340+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:01.040340+0000","last_peered":"2024-09-16T01:49:01.040340+0000","last_clean":"2024-09-16T01:49:01.040340+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:01.040340+0000","last_undegraded":"2024-09-16T01:49:01.040340+0000","last_fullsized":"2024-09-16T01:49:01.040340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820596+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:48:58.820596+0000","last_peered":"2024-09-16T01:48:58.820596+0000","last_clean":"2024-09-16T01:48:58.820596+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:48:58.820596+0000","last_undegraded":"2024-09-16T01:48:58.820596+0000","last_fullsized":"2024-09-16T01:48:58.820596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.834524+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:58.834524+0000","last_peered":"2024-09-16T01:48:58.834524+0000","last_clean":"2024-09-16T01:48:58.834524+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:58.834524+0000","last_undegraded":"2024-09-16T01:48:58.834524+0000","last_fullsized":"2024-09-16T01:48:58.834524+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:36.174533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040110+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:01.040110+0000","last_peered":"2024-09-16T01:49:01.040110+0000","last_clean":"2024-09-16T01:49:01.040110+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:01.040110+0000","last_undegraded":"2024-09-16T01:49:01.040110+0000","last_fullsized":"2024-09-16T01:49:01.040110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.831234+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:58.831234+0000","last_peered":"2024-09-16T01:48:58.831234+0000","last_clean":"2024-09-16T01:48:58.831234+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:58.831234+0000","last_undegraded":"2024-09-16T01:48:58.831234+0000","last_fullsized":"2024-09-16T01:48:58.831234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:14.706024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038599+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:01.038599+0000","last_peered":"2024-09-16T01:49:01.038599+0000","last_clean":"2024-09-16T01:49:01.038599+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:01.038599+0000","last_undegraded":"2024-09-16T01:49:01.038599+0000","last_fullsized":"2024-09-16T01:49:01.038599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820685+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:48:58.820685+0000","last_peered":"2024-09-16T01:48:58.820685+0000","last_clean":"2024-09-16T01:48:58.820685+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:48:58.820685+0000","last_undegraded":"2024-09-16T01:48:58.820685+0000","last_fullsized":"2024-09-16T01:48:58.820685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.831469+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:58.831469+0000","last_peered":"2024-09-16T01:48:58.831469+0000","last_clean":"2024-09-16T01:48:58.831469+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:58.831469+0000","last_undegraded":"2024-09-16T01:48:58.831469+0000","last_fullsized":"2024-09-16T01:48:58.831469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:03.685779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825690+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:48:59.825690+0000","last_peered":"2024-09-16T01:48:59.825690+0000","last_clean":"2024-09-16T01:48:59.825690+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:48:59.825690+0000","last_undegraded":"2024-09-16T01:48:59.825690+0000","last_fullsized":"2024-09-16T01:48:59.825690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.830979+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:48:58.830979+0000","last_peered":"2024-09-16T01:48:58.830979+0000","last_clean":"2024-09-16T01:48:58.830979+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:48:58.830979+0000","last_undegraded":"2024-09-16T01:48:58.830979+0000","last_fullsized":"2024-09-16T01:48:58.830979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:38.762210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039491+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:01.039491+0000","last_peered":"2024-09-16T01:49:01.039491+0000","last_clean":"2024-09-16T01:49:01.039491+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:01.039491+0000","last_undegraded":"2024-09-16T01:49:01.039491+0000","last_fullsized":"2024-09-16T01:49:01.039491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039109+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:01.039109+0000","last_peered":"2024-09-16T01:49:01.039109+0000","last_clean":"2024-09-16T01:49:01.039109+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:01.039109+0000","last_undegraded":"2024-09-16T01:49:01.039109+0000","last_fullsized":"2024-09-16T01:49:01.039109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039336+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:01.039336+0000","last_peered":"2024-09-16T01:49:01.039336+0000","last_clean":"2024-09-16T01:49:01.039336+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:01.039336+0000","last_undegraded":"2024-09-16T01:49:01.039336+0000","last_fullsized":"2024-09-16T01:49:01.039336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.828789+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:48:58.828789+0000","last_peered":"2024-09-16T01:48:58.828789+0000","last_clean":"2024-09-16T01:48:58.828789+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:48:58.828789+0000","last_undegraded":"2024-09-16T01:48:58.828789+0000","last_fullsized":"2024-09-16T01:48:58.828789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.814991+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:48:58.814991+0000","last_peered":"2024-09-16T01:48:58.814991+0000","last_clean":"2024-09-16T01:48:58.814991+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:48:58.814991+0000","last_undegraded":"2024-09-16T01:48:58.814991+0000","last_fullsized":"2024-09-16T01:48:58.814991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839438+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:48:58.839438+0000","last_peered":"2024-09-16T01:48:58.839438+0000","last_clean":"2024-09-16T01:48:58.839438+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:48:58.839438+0000","last_undegraded":"2024-09-16T01:48:58.839438+0000","last_fullsized":"2024-09-16T01:48:58.839438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820633+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:48:58.820633+0000","last_peered":"2024-09-16T01:48:58.820633+0000","last_clean":"2024-09-16T01:48:58.820633+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:48:58.820633+0000","last_undegraded":"2024-09-16T01:48:58.820633+0000","last_fullsized":"2024-09-16T01:48:58.820633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":668644,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":662344,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:05.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.197+0000 7fa851200640 1 Processor -- start 2024-09-16T01:49:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.197+0000 7fa851200640 1 -- start start 2024-09-16T01:49:05.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.198+0000 7fa851200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa84c0c75a0 0x7fa84c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.198+0000 7fa851200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.198+0000 7fa851200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa84c06d140 con 0x7fa84c0c6b70 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.198+0000 7fa851200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa84c15b880 con 0x7fa84c0c75a0 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.198+0000 7fa851200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa84c0cab20 con 0x7fa84c0cc040 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa84c0c6b70 0x7fa84c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53598/0 (socket says 172.21.15.146:53598) 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84b400640 1 -- 172.21.15.146:0/152351923 learned_addr learned my addr 172.21.15.146:0/152351923 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84aa00640 1 --2- 172.21.15.146:0/152351923 >> v2:172.21.15.161:3300/0 conn(0x7fa84c0c75a0 0x7fa84c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84be00640 1 --2- 172.21.15.146:0/152351923 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84a000640 1 -- 172.21.15.146:0/152351923 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992820928 0 0) 0x7fa84c06d140 con 0x7fa84c0c6b70 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84a000640 1 -- 172.21.15.146:0/152351923 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa830003520 con 0x7fa84c0c6b70 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84be00640 1 -- 172.21.15.146:0/152351923 >> v2:172.21.15.161:3300/0 conn(0x7fa84c0c75a0 msgr2=0x7fa84c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84be00640 1 --2- 172.21.15.146:0/152351923 >> v2:172.21.15.161:3300/0 conn(0x7fa84c0c75a0 0x7fa84c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84be00640 1 -- 172.21.15.146:0/152351923 >> v1:172.21.15.146:6789/0 conn(0x7fa84c0c6b70 legacy=0x7fa84c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84be00640 1 -- 172.21.15.146:0/152351923 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84c06d420 con 0x7fa84c0cc040 2024-09-16T01:49:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.199+0000 7fa84aa00640 1 --2- 172.21.15.146:0/152351923 >> v2:172.21.15.161:3300/0 conn(0x7fa84c0c75a0 0x7fa84c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.200+0000 7fa84be00640 1 --2- 172.21.15.146:0/152351923 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c0cc4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa844009d90 tx=0x7fa84402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1295e5b414089ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.200+0000 7fa84a000640 1 -- 172.21.15.146:0/152351923 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844031600 con 0x7fa84c0cc040 2024-09-16T01:49:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.200+0000 7fa84a000640 1 -- 172.21.15.146:0/152351923 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8440317a0 con 0x7fa84c0cc040 2024-09-16T01:49:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.200+0000 7fa84a000640 1 -- 172.21.15.146:0/152351923 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844031b70 con 0x7fa84c0cc040 2024-09-16T01:49:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 -- 172.21.15.146:0/152351923 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 msgr2=0x7fa84c0cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 --2- 172.21.15.146:0/152351923 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c0cc4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa844009d90 tx=0x7fa84402a5d0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 -- 172.21.15.146:0/152351923 shutdown_connections 2024-09-16T01:49:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 --2- 172.21.15.146:0/152351923 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c0cc4c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 --2- 172.21.15.146:0/152351923 >> v2:172.21.15.161:3300/0 conn(0x7fa84c0c75a0 0x7fa84c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 -- 172.21.15.146:0/152351923 >> 172.21.15.146:0/152351923 conn(0x7fa84c0ccc20 msgr2=0x7fa84c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 -- 172.21.15.146:0/152351923 shutdown_connections 2024-09-16T01:49:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.201+0000 7fa851200640 1 -- 172.21.15.146:0/152351923 wait complete. 2024-09-16T01:49:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.203+0000 7fa851200640 1 Processor -- start 2024-09-16T01:49:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.203+0000 7fa851200640 1 -- start start 2024-09-16T01:49:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa851200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 0x7fa84c1f1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa851200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c0c75a0 0x7fa84c157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa851200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c15c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa851200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c0cb050 con 0x7fa84c0c75a0 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa851200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c0cad70 con 0x7fa84c0c6b70 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa851200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa84c0cbc80 con 0x7fa84c0cc040 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 0x7fa84c1f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c0c75a0 0x7fa84c157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c15c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 0x7fa84c1f1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45014/0 (socket says 172.21.15.146:45014) 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c15c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37444/0 (socket says 172.21.15.146:37444) 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 -- 172.21.15.146:0/140890068 learned_addr learned my addr 172.21.15.146:0/140890068 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 -- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 msgr2=0x7fa84c15c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c15c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 -- 172.21.15.146:0/140890068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c0c75a0 msgr2=0x7fa84c157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.204+0000 7fa84b400640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c0c75a0 0x7fa84c157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa84b400640 1 -- 172.21.15.146:0/140890068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84c15e1c0 con 0x7fa84c0c6b70 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa84be00640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c15c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa84aa00640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c0c75a0 0x7fa84c157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa84b400640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 0x7fa84c1f1140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00c3f0 tx=0x7fa83c00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c017070 con 0x7fa84c0c6b70 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83c005ce0 con 0x7fa84c0c6b70 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0072f0 con 0x7fa84c0c6b70 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa84c15e430 con 0x7fa84c0c6b70 2024-09-16T01:49:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.205+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa84c15e9f0 con 0x7fa84c0c6b70 2024-09-16T01:49:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.206+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa84c06cd20 con 0x7fa84c0c6b70 2024-09-16T01:49:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.207+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa83c01a070 con 0x7fa84c0c6b70 2024-09-16T01:49:05.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.208+0000 7fa843400640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa82c06c2b0 0x7fa82c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.208+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa83c08a790 con 0x7fa84c0c6b70 2024-09-16T01:49:05.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.208+0000 7fa84aa00640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa82c06c2b0 0x7fa82c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.209+0000 7fa84aa00640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa82c06c2b0 0x7fa82c06e770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa838009560 tx=0x7fa838009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:05.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.213+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa83c0588a0 con 0x7fa84c0c6b70 2024-09-16T01:49:05.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.386+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa84c06da90 con 0x7fa82c06c2b0 2024-09-16T01:49:05.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.392+0000 7fa843400640 1 -- 172.21.15.146:0/140890068 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251903 (secure 0 0 0) 0x7fa84c06da90 con 0x7fa82c06c2b0 2024-09-16T01:49:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:05.397 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa82c06c2b0 msgr2=0x7fa82c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa82c06c2b0 0x7fa82c06e770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa838009560 tx=0x7fa838009190 comp rx=0 tx=0).stop 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 msgr2=0x7fa84c1f1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 0x7fa84c1f1140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00c3f0 tx=0x7fa83c00c8b0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 shutdown_connections 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa82c06c2b0 0x7fa82c06e770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa84c0cc040 0x7fa84c15c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c0c75a0 0x7fa84c157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 --2- 172.21.15.146:0/140890068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa84c0c6b70 0x7fa84c1f1140 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 >> 172.21.15.146:0/140890068 conn(0x7fa84c0ccc20 msgr2=0x7fa84c0c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 shutdown_connections 2024-09-16T01:49:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.396+0000 7fa851200640 1 -- 172.21.15.146:0/140890068 wait complete. 2024-09-16T01:49:05.413 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T01:49:04.519183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218056,"kb_used_data":1808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756664,"statfs":{"total":773094113280,"available":772870823936,"internally_reserved":0,"allocated":1851392,"data_stored":1781744,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.726208"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829290+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:48:58.829290+0000","last_peered":"2024-09-16T01:48:58.829290+0000","last_clean":"2024-09-16T01:48:58.829290+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:48:58.829290+0000","last_undegraded":"2024-09-16T01:48:58.829290+0000","last_fullsized":"2024-09-16T01:48:58.829290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.830931+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:48:58.830931+0000","last_peered":"2024-09-16T01:48:58.830931+0000","last_clean":"2024-09-16T01:48:58.830931+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:48:58.830931+0000","last_undegraded":"2024-09-16T01:48:58.830931+0000","last_fullsized":"2024-09-16T01:48:58.830931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829354+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:48:58.829354+0000","last_peered":"2024-09-16T01:48:58.829354+0000","last_clean":"2024-09-16T01:48:58.829354+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:48:58.829354+0000","last_undegraded":"2024-09-16T01:48:58.829354+0000","last_fullsized":"2024-09-16T01:48:58.829354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.833773+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:48:58.833773+0000","last_peered":"2024-09-16T01:48:58.833773+0000","last_clean":"2024-09-16T01:48:58.833773+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:48:58.833773+0000","last_undegraded":"2024-09-16T01:48:58.833773+0000","last_fullsized":"2024-09-16T01:48:58.833773+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839135+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:48:58.839135+0000","last_peered":"2024-09-16T01:48:58.839135+0000","last_clean":"2024-09-16T01:48:58.839135+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:48:58.839135+0000","last_undegraded":"2024-09-16T01:48:58.839135+0000","last_fullsized":"2024-09-16T01:48:58.839135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:43.838815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839412+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:58.839412+0000","last_peered":"2024-09-16T01:48:58.839412+0000","last_clean":"2024-09-16T01:48:58.839412+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:58.839412+0000","last_undegraded":"2024-09-16T01:48:58.839412+0000","last_fullsized":"2024-09-16T01:48:58.839412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:10:42.684463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839409+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:48:58.839409+0000","last_peered":"2024-09-16T01:48:58.839409+0000","last_clean":"2024-09-16T01:48:58.839409+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:48:58.839409+0000","last_undegraded":"2024-09-16T01:48:58.839409+0000","last_fullsized":"2024-09-16T01:48:58.839409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829168+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:48:58.829168+0000","last_peered":"2024-09-16T01:48:58.829168+0000","last_clean":"2024-09-16T01:48:58.829168+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:48:58.829168+0000","last_undegraded":"2024-09-16T01:48:58.829168+0000","last_fullsized":"2024-09-16T01:48:58.829168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.833752+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:48:58.833752+0000","last_peered":"2024-09-16T01:48:58.833752+0000","last_clean":"2024-09-16T01:48:58.833752+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:48:58.833752+0000","last_undegraded":"2024-09-16T01:48:58.833752+0000","last_fullsized":"2024-09-16T01:48:58.833752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.830946+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:48:58.830946+0000","last_peered":"2024-09-16T01:48:58.830946+0000","last_clean":"2024-09-16T01:48:58.830946+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:48:58.830946+0000","last_undegraded":"2024-09-16T01:48:58.830946+0000","last_fullsized":"2024-09-16T01:48:58.830946+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:48.348594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.814813+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:48:58.814813+0000","last_peered":"2024-09-16T01:48:58.814813+0000","last_clean":"2024-09-16T01:48:58.814813+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:48:58.814813+0000","last_undegraded":"2024-09-16T01:48:58.814813+0000","last_fullsized":"2024-09-16T01:48:58.814813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.841991+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:58.841991+0000","last_peered":"2024-09-16T01:48:58.841991+0000","last_clean":"2024-09-16T01:48:58.841991+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:58.841991+0000","last_undegraded":"2024-09-16T01:48:58.841991+0000","last_fullsized":"2024-09-16T01:48:58.841991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:03:51.311804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039329+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:01.039329+0000","last_peered":"2024-09-16T01:49:01.039329+0000","last_clean":"2024-09-16T01:49:01.039329+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:01.039329+0000","last_undegraded":"2024-09-16T01:49:01.039329+0000","last_fullsized":"2024-09-16T01:49:01.039329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825347+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:48:59.825347+0000","last_peered":"2024-09-16T01:48:59.825347+0000","last_clean":"2024-09-16T01:48:59.825347+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:48:59.825347+0000","last_undegraded":"2024-09-16T01:48:59.825347+0000","last_fullsized":"2024-09-16T01:48:59.825347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.829333+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:48:58.829333+0000","last_peered":"2024-09-16T01:48:58.829333+0000","last_clean":"2024-09-16T01:48:58.829333+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:48:58.829333+0000","last_undegraded":"2024-09-16T01:48:58.829333+0000","last_fullsized":"2024-09-16T01:48:58.829333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:43.838815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038521+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:01.038521+0000","last_peered":"2024-09-16T01:49:01.038521+0000","last_clean":"2024-09-16T01:49:01.038521+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:01.038521+0000","last_undegraded":"2024-09-16T01:49:01.038521+0000","last_fullsized":"2024-09-16T01:49:01.038521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.831337+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:58.831337+0000","last_peered":"2024-09-16T01:48:58.831337+0000","last_clean":"2024-09-16T01:48:58.831337+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:58.831337+0000","last_undegraded":"2024-09-16T01:48:58.831337+0000","last_fullsized":"2024-09-16T01:48:58.831337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:19:15.774154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040713+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:01.040713+0000","last_peered":"2024-09-16T01:49:01.040713+0000","last_clean":"2024-09-16T01:49:01.040713+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:01.040713+0000","last_undegraded":"2024-09-16T01:49:01.040713+0000","last_fullsized":"2024-09-16T01:49:01.040713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039489+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:01.039489+0000","last_peered":"2024-09-16T01:49:01.039489+0000","last_clean":"2024-09-16T01:49:01.039489+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:01.039489+0000","last_undegraded":"2024-09-16T01:49:01.039489+0000","last_fullsized":"2024-09-16T01:49:01.039489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.842051+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:48:58.842051+0000","last_peered":"2024-09-16T01:48:58.842051+0000","last_clean":"2024-09-16T01:48:58.842051+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:48:58.842051+0000","last_undegraded":"2024-09-16T01:48:58.842051+0000","last_fullsized":"2024-09-16T01:48:58.842051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:27.080013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.816248+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:48:58.816248+0000","last_peered":"2024-09-16T01:48:58.816248+0000","last_clean":"2024-09-16T01:48:58.816248+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:48:58.816248+0000","last_undegraded":"2024-09-16T01:48:58.816248+0000","last_fullsized":"2024-09-16T01:48:58.816248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824172+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:48:59.824172+0000","last_peered":"2024-09-16T01:48:59.824172+0000","last_clean":"2024-09-16T01:48:59.824172+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:48:59.824172+0000","last_undegraded":"2024-09-16T01:48:59.824172+0000","last_fullsized":"2024-09-16T01:48:59.824172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.816181+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:48:58.816181+0000","last_peered":"2024-09-16T01:48:58.816181+0000","last_clean":"2024-09-16T01:48:58.816181+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:48:58.816181+0000","last_undegraded":"2024-09-16T01:48:58.816181+0000","last_fullsized":"2024-09-16T01:48:58.816181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039468+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:01.039468+0000","last_peered":"2024-09-16T01:49:01.039468+0000","last_clean":"2024-09-16T01:49:01.039468+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:01.039468+0000","last_undegraded":"2024-09-16T01:49:01.039468+0000","last_fullsized":"2024-09-16T01:49:01.039468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038796+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:01.038796+0000","last_peered":"2024-09-16T01:49:01.038796+0000","last_clean":"2024-09-16T01:49:01.038796+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:01.038796+0000","last_undegraded":"2024-09-16T01:49:01.038796+0000","last_fullsized":"2024-09-16T01:49:01.038796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.818853+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:48:58.818853+0000","last_peered":"2024-09-16T01:48:58.818853+0000","last_clean":"2024-09-16T01:48:58.818853+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:48:58.818853+0000","last_undegraded":"2024-09-16T01:48:58.818853+0000","last_fullsized":"2024-09-16T01:48:58.818853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039958+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:01.039958+0000","last_peered":"2024-09-16T01:49:01.039958+0000","last_clean":"2024-09-16T01:49:01.039958+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:01.039958+0000","last_undegraded":"2024-09-16T01:49:01.039958+0000","last_fullsized":"2024-09-16T01:49:01.039958+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820575+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:48:58.820575+0000","last_peered":"2024-09-16T01:48:58.820575+0000","last_clean":"2024-09-16T01:48:58.820575+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:48:58.820575+0000","last_undegraded":"2024-09-16T01:48:58.820575+0000","last_fullsized":"2024-09-16T01:48:58.820575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.813221+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:48:58.813221+0000","last_peered":"2024-09-16T01:48:58.813221+0000","last_clean":"2024-09-16T01:48:58.813221+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:48:58.813221+0000","last_undegraded":"2024-09-16T01:48:58.813221+0000","last_fullsized":"2024-09-16T01:48:58.813221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825768+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:48:59.825768+0000","last_peered":"2024-09-16T01:48:59.825768+0000","last_clean":"2024-09-16T01:48:59.825768+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:48:59.825768+0000","last_undegraded":"2024-09-16T01:48:59.825768+0000","last_fullsized":"2024-09-16T01:48:59.825768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039827+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:01.039827+0000","last_peered":"2024-09-16T01:49:01.039827+0000","last_clean":"2024-09-16T01:49:01.039827+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:01.039827+0000","last_undegraded":"2024-09-16T01:49:01.039827+0000","last_fullsized":"2024-09-16T01:49:01.039827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839403+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:58.839403+0000","last_peered":"2024-09-16T01:48:58.839403+0000","last_clean":"2024-09-16T01:48:58.839403+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:58.839403+0000","last_undegraded":"2024-09-16T01:48:58.839403+0000","last_fullsized":"2024-09-16T01:48:58.839403+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:31.059699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038630+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:01.038630+0000","last_peered":"2024-09-16T01:49:01.038630+0000","last_clean":"2024-09-16T01:49:01.038630+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:01.038630+0000","last_undegraded":"2024-09-16T01:49:01.038630+0000","last_fullsized":"2024-09-16T01:49:01.038630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.816050+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:48:58.816050+0000","last_peered":"2024-09-16T01:48:58.816050+0000","last_clean":"2024-09-16T01:48:58.816050+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:48:58.816050+0000","last_undegraded":"2024-09-16T01:48:58.816050+0000","last_fullsized":"2024-09-16T01:48:58.816050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.833799+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:48:58.833799+0000","last_peered":"2024-09-16T01:48:58.833799+0000","last_clean":"2024-09-16T01:48:58.833799+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:48:58.833799+0000","last_undegraded":"2024-09-16T01:48:58.833799+0000","last_fullsized":"2024-09-16T01:48:58.833799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039485+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:01.039485+0000","last_peered":"2024-09-16T01:49:01.039485+0000","last_clean":"2024-09-16T01:49:01.039485+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:01.039485+0000","last_undegraded":"2024-09-16T01:49:01.039485+0000","last_fullsized":"2024-09-16T01:49:01.039485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.813197+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:48:58.813197+0000","last_peered":"2024-09-16T01:48:58.813197+0000","last_clean":"2024-09-16T01:48:58.813197+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:48:58.813197+0000","last_undegraded":"2024-09-16T01:48:58.813197+0000","last_fullsized":"2024-09-16T01:48:58.813197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040340+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:01.040340+0000","last_peered":"2024-09-16T01:49:01.040340+0000","last_clean":"2024-09-16T01:49:01.040340+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:01.040340+0000","last_undegraded":"2024-09-16T01:49:01.040340+0000","last_fullsized":"2024-09-16T01:49:01.040340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820596+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:48:58.820596+0000","last_peered":"2024-09-16T01:48:58.820596+0000","last_clean":"2024-09-16T01:48:58.820596+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:48:58.820596+0000","last_undegraded":"2024-09-16T01:48:58.820596+0000","last_fullsized":"2024-09-16T01:48:58.820596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.834524+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:58.834524+0000","last_peered":"2024-09-16T01:48:58.834524+0000","last_clean":"2024-09-16T01:48:58.834524+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:58.834524+0000","last_undegraded":"2024-09-16T01:48:58.834524+0000","last_fullsized":"2024-09-16T01:48:58.834524+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:36.174533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040110+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:01.040110+0000","last_peered":"2024-09-16T01:49:01.040110+0000","last_clean":"2024-09-16T01:49:01.040110+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:01.040110+0000","last_undegraded":"2024-09-16T01:49:01.040110+0000","last_fullsized":"2024-09-16T01:49:01.040110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.831234+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:58.831234+0000","last_peered":"2024-09-16T01:48:58.831234+0000","last_clean":"2024-09-16T01:48:58.831234+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:58.831234+0000","last_undegraded":"2024-09-16T01:48:58.831234+0000","last_fullsized":"2024-09-16T01:48:58.831234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:14.706024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038599+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:01.038599+0000","last_peered":"2024-09-16T01:49:01.038599+0000","last_clean":"2024-09-16T01:49:01.038599+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:01.038599+0000","last_undegraded":"2024-09-16T01:49:01.038599+0000","last_fullsized":"2024-09-16T01:49:01.038599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820685+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:48:58.820685+0000","last_peered":"2024-09-16T01:48:58.820685+0000","last_clean":"2024-09-16T01:48:58.820685+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:48:58.820685+0000","last_undegraded":"2024-09-16T01:48:58.820685+0000","last_fullsized":"2024-09-16T01:48:58.820685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.831469+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:58.831469+0000","last_peered":"2024-09-16T01:48:58.831469+0000","last_clean":"2024-09-16T01:48:58.831469+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:58.831469+0000","last_undegraded":"2024-09-16T01:48:58.831469+0000","last_fullsized":"2024-09-16T01:48:58.831469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:03.685779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825690+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:48:59.825690+0000","last_peered":"2024-09-16T01:48:59.825690+0000","last_clean":"2024-09-16T01:48:59.825690+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:48:59.825690+0000","last_undegraded":"2024-09-16T01:48:59.825690+0000","last_fullsized":"2024-09-16T01:48:59.825690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.830979+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:48:58.830979+0000","last_peered":"2024-09-16T01:48:58.830979+0000","last_clean":"2024-09-16T01:48:58.830979+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:48:58.830979+0000","last_undegraded":"2024-09-16T01:48:58.830979+0000","last_fullsized":"2024-09-16T01:48:58.830979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:38.762210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039491+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:01.039491+0000","last_peered":"2024-09-16T01:49:01.039491+0000","last_clean":"2024-09-16T01:49:01.039491+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:01.039491+0000","last_undegraded":"2024-09-16T01:49:01.039491+0000","last_fullsized":"2024-09-16T01:49:01.039491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039109+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:01.039109+0000","last_peered":"2024-09-16T01:49:01.039109+0000","last_clean":"2024-09-16T01:49:01.039109+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:01.039109+0000","last_undegraded":"2024-09-16T01:49:01.039109+0000","last_fullsized":"2024-09-16T01:49:01.039109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039336+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:01.039336+0000","last_peered":"2024-09-16T01:49:01.039336+0000","last_clean":"2024-09-16T01:49:01.039336+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:01.039336+0000","last_undegraded":"2024-09-16T01:49:01.039336+0000","last_fullsized":"2024-09-16T01:49:01.039336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.828789+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:48:58.828789+0000","last_peered":"2024-09-16T01:48:58.828789+0000","last_clean":"2024-09-16T01:48:58.828789+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:48:58.828789+0000","last_undegraded":"2024-09-16T01:48:58.828789+0000","last_fullsized":"2024-09-16T01:48:58.828789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.814991+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:48:58.814991+0000","last_peered":"2024-09-16T01:48:58.814991+0000","last_clean":"2024-09-16T01:48:58.814991+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:48:58.814991+0000","last_undegraded":"2024-09-16T01:48:58.814991+0000","last_fullsized":"2024-09-16T01:48:58.814991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.839438+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:48:58.839438+0000","last_peered":"2024-09-16T01:48:58.839438+0000","last_clean":"2024-09-16T01:48:58.839438+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:48:58.839438+0000","last_undegraded":"2024-09-16T01:48:58.839438+0000","last_fullsized":"2024-09-16T01:48:58.839438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:48:58.820633+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:48:58.820633+0000","last_peered":"2024-09-16T01:48:58.820633+0000","last_clean":"2024-09-16T01:48:58.820633+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:48:58.820633+0000","last_undegraded":"2024-09-16T01:48:58.820633+0000","last_fullsized":"2024-09-16T01:48:58.820633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":668644,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":71976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":662344,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:05.417 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:49:05.417 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T01:49:05.417 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T01:49:05.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T01:49:05.418 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T01:49:05.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.606+0000 7f9435800640 1 Processor -- start 2024-09-16T01:49:05.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.606+0000 7f9435800640 1 -- start start 2024-09-16T01:49:05.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9435800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f94300cacb0 0x7f94300c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9435800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cb090 0x7f943015c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9435800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94300c98b0 con 0x7f943015e1d0 2024-09-16T01:49:05.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9435800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f94300c9b70 con 0x7f94300cacb0 2024-09-16T01:49:05.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9435800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f94300c9e50 con 0x7f94300cb090 2024-09-16T01:49:05.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f942fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f94300cacb0 0x7f94300c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9434e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f943015e1d0 0x7f943015b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53610/0 (socket says 172.21.15.146:53610) 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9434e00640 1 -- 172.21.15.146:0/3947563070 learned_addr learned my addr 172.21.15.146:0/3947563070 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.607+0000 7f9427400640 1 --2- 172.21.15.146:0/3947563070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cb090 0x7f943015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.608+0000 7f942fe00640 1 -- 172.21.15.146:0/3947563070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cb090 msgr2=0x7f943015c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.608+0000 7f942fe00640 1 --2- 172.21.15.146:0/3947563070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cb090 0x7f943015c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.608+0000 7f942fe00640 1 -- 172.21.15.146:0/3947563070 >> v1:172.21.15.146:6789/0 conn(0x7f943015e1d0 legacy=0x7f943015b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.608+0000 7f942fe00640 1 -- 172.21.15.146:0/3947563070 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94300ca130 con 0x7f94300cacb0 2024-09-16T01:49:05.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.608+0000 7f942f400640 1 -- 172.21.15.146:0/3947563070 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3137806111 0 0) 0x7f94300c98b0 con 0x7f943015e1d0 2024-09-16T01:49:05.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f942fe00640 1 --2- 172.21.15.146:0/3947563070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cacb0 0x7f94300c8fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94200097a0 tx=0x7f942002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64882a4abd020866 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f942f400640 1 -- 172.21.15.146:0/3947563070 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420037070 con 0x7f94300cacb0 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f942f400640 1 -- 172.21.15.146:0/3947563070 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9420032440 con 0x7f94300cacb0 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f942f400640 1 -- 172.21.15.146:0/3947563070 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420032810 con 0x7f94300cacb0 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f9435800640 1 -- 172.21.15.146:0/3947563070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cacb0 msgr2=0x7f94300c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f9435800640 1 --2- 172.21.15.146:0/3947563070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cacb0 0x7f94300c8fb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94200097a0 tx=0x7f942002a210 comp rx=0 tx=0).stop 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f9435800640 1 -- 172.21.15.146:0/3947563070 shutdown_connections 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f9435800640 1 --2- 172.21.15.146:0/3947563070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cb090 0x7f943015c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f9435800640 1 --2- 172.21.15.146:0/3947563070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cacb0 0x7f94300c8fb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.609+0000 7f9435800640 1 -- 172.21.15.146:0/3947563070 >> 172.21.15.146:0/3947563070 conn(0x7f9430067950 msgr2=0x7f9430067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.610+0000 7f9435800640 1 -- 172.21.15.146:0/3947563070 shutdown_connections 2024-09-16T01:49:05.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.610+0000 7f9435800640 1 -- 172.21.15.146:0/3947563070 wait complete. 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.611+0000 7f9435800640 1 Processor -- start 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.611+0000 7f9435800640 1 -- start start 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9435800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 0x7f943006f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9435800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cb090 0x7f94301f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9435800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f943015e1d0 0x7f943006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9435800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f94300c94f0 con 0x7f94300cb090 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9435800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f94300ca6f0 con 0x7f943015e1d0 2024-09-16T01:49:05.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9435800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f94300c9670 con 0x7f94300cacb0 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9427400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cb090 0x7f94301f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f942fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 0x7f943006f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f942fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 0x7f943006f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37466/0 (socket says 172.21.15.146:37466) 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f942fe00640 1 -- 172.21.15.146:0/2894210803 learned_addr learned my addr 172.21.15.146:0/2894210803 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.612+0000 7f9434e00640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f943015e1d0 0x7f943006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942fe00640 1 -- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f943015e1d0 msgr2=0x7f943006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942fe00640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f943015e1d0 0x7f943006ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942fe00640 1 -- 172.21.15.146:0/2894210803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cb090 msgr2=0x7f94301f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942fe00640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cb090 0x7f94301f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942fe00640 1 -- 172.21.15.146:0/2894210803 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943006f170 con 0x7f94300cacb0 2024-09-16T01:49:05.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f9427400640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cb090 0x7f94301f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:05.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942fe00640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 0x7f943006f620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f942002a800 tx=0x7f94200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:05.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420037040 con 0x7f94300cacb0 2024-09-16T01:49:05.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9420030770 con 0x7f94300cacb0 2024-09-16T01:49:05.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94300be710 con 0x7f94300cacb0 2024-09-16T01:49:05.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.613+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420030b40 con 0x7f94300cacb0 2024-09-16T01:49:05.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.614+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94301f4000 con 0x7f94300cacb0 2024-09-16T01:49:05.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.614+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94300c80a0 con 0x7f94300cacb0 2024-09-16T01:49:05.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.616+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9420044020 con 0x7f94300cacb0 2024-09-16T01:49:05.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.616+0000 7f942cc00640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f940c06c2b0 0x7f940c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:05.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.617+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f94200ab840 con 0x7f94300cacb0 2024-09-16T01:49:05.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.617+0000 7f9427400640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f940c06c2b0 0x7f940c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:05.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.617+0000 7f9427400640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f940c06c2b0 0x7f940c06e770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f943006e260 tx=0x7f9418008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:05.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.622+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9420032e20 con 0x7f94300cacb0 2024-09-16T01:49:05.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.847+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f94300b3020 con 0x7f94300cacb0 2024-09-16T01:49:05.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.850+0000 7f942cc00640 1 -- 172.21.15.146:0/2894210803 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173512 (secure 0 0 0) 0x7f94200798c0 con 0x7f94300cacb0 2024-09-16T01:49:05.853 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:05.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f940c06c2b0 msgr2=0x7f940c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f940c06c2b0 0x7f940c06e770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f943006e260 tx=0x7f9418008040 comp rx=0 tx=0).stop 2024-09-16T01:49:05.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 msgr2=0x7f943006f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:05.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 0x7f943006f620 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f942002a800 tx=0x7f94200046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 shutdown_connections 2024-09-16T01:49:05.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f940c06c2b0 0x7f940c06e770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f943015e1d0 0x7f943006ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94300cb090 0x7f94301f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 --2- 172.21.15.146:0/2894210803 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94300cacb0 0x7f943006f620 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:05.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 >> 172.21.15.146:0/2894210803 conn(0x7f9430067950 msgr2=0x7f94300c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:05.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 shutdown_connections 2024-09-16T01:49:05.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:05.854+0000 7f9435800640 1 -- 172.21.15.146:0/2894210803 wait complete. 2024-09-16T01:49:05.871 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":3835694012},{"type":"v1","addr":"172.21.15.161:6833","nonce":3835694012}]},"active_addr":"172.21.15.161:6833/3835694012","active_change":"2024-09-16T01:48:38.506943+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":1446893610}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":2749638142}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3462011202}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":1400226773}]}]} 2024-09-16T01:49:05.876 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T01:49:05.876 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T01:49:05.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:49:06.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.029+0000 7f8b47e00640 1 Processor -- start 2024-09-16T01:49:06.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.029+0000 7f8b47e00640 1 -- start start 2024-09-16T01:49:06.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.030+0000 7f8b47e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b400c5280 0x7f8b400c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.030+0000 7f8b47e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b40151e70 con 0x7f8b400c6410 2024-09-16T01:49:06.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.030+0000 7f8b47e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b400c8130 con 0x7f8b40151a90 2024-09-16T01:49:06.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.030+0000 7f8b47e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8b400c8410 con 0x7f8b400c5280 2024-09-16T01:49:06.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b44a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8b400c6410 0x7f8b40158040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53620/0 (socket says 172.21.15.146:53620) 2024-09-16T01:49:06.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b44a00640 1 -- 172.21.15.146:0/1111655075 learned_addr learned my addr 172.21.15.146:0/1111655075 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:06.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b45e00640 1 --2- 172.21.15.146:0/1111655075 >> v2:172.21.15.161:3300/0 conn(0x7f8b400c5280 0x7f8b400c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4146459477 0 0) 0x7f8b40151e70 con 0x7f8b400c6410 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b28003520 con 0x7f8b400c6410 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577067428 0 0) 0x7f8b400c8130 con 0x7f8b40151a90 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b40151e70 con 0x7f8b40151a90 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b45e00640 1 -- 172.21.15.146:0/1111655075 >> v1:172.21.15.161:6789/0 conn(0x7f8b40151a90 legacy=0x7f8b400c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b45e00640 1 -- 172.21.15.146:0/1111655075 >> v1:172.21.15.146:6789/0 conn(0x7f8b400c6410 legacy=0x7f8b40158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.031+0000 7f8b45e00640 1 -- 172.21.15.146:0/1111655075 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b400c86f0 con 0x7f8b400c5280 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.032+0000 7f8b45e00640 1 --2- 172.21.15.146:0/1111655075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c5280 0x7f8b400c56f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009d90 tx=0x7f8b3802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90ea9b96f1719ed7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.032+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b38007740 con 0x7f8b400c5280 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.032+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b380078e0 con 0x7f8b400c5280 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.032+0000 7f8b3fe00640 1 -- 172.21.15.146:0/1111655075 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b38007c90 con 0x7f8b400c5280 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 -- 172.21.15.146:0/1111655075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c5280 msgr2=0x7f8b400c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 --2- 172.21.15.146:0/1111655075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c5280 0x7f8b400c56f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009d90 tx=0x7f8b3802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 -- 172.21.15.146:0/1111655075 shutdown_connections 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 --2- 172.21.15.146:0/1111655075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c5280 0x7f8b400c56f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 -- 172.21.15.146:0/1111655075 >> 172.21.15.146:0/1111655075 conn(0x7f8b4006f3c0 msgr2=0x7f8b4006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 -- 172.21.15.146:0/1111655075 shutdown_connections 2024-09-16T01:49:06.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.033+0000 7f8b47e00640 1 -- 172.21.15.146:0/1111655075 wait complete. 2024-09-16T01:49:06.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.035+0000 7f8b47e00640 1 Processor -- start 2024-09-16T01:49:06.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.035+0000 7f8b47e00640 1 -- start start 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b47e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 0x7f8b4014b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b47e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c6410 0x7f8b401e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b47e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8b40151a90 0x7f8b401e53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b47e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b40157b20 con 0x7f8b400c5280 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b47e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b40157840 con 0x7f8b400c6410 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b47e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8b400c71f0 con 0x7f8b40151a90 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 0x7f8b4014b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 0x7f8b4014b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58962/0 (socket says 172.21.15.146:58962) 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 -- 172.21.15.146:0/3872901473 learned_addr learned my addr 172.21.15.146:0/3872901473 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b44a00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c6410 0x7f8b401e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8b40151a90 0x7f8b401e53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 -- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8b40151a90 msgr2=0x7f8b401e53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8b40151a90 0x7f8b401e53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 -- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c6410 msgr2=0x7f8b401e4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c6410 0x7f8b401e4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b45400640 1 -- 172.21.15.146:0/3872901473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4014d850 con 0x7f8b400c5280 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.036+0000 7f8b44a00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c6410 0x7f8b401e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.037+0000 7f8b45400640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 0x7f8b4014b200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c460 tx=0x7f8b3400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.037+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b340052d0 con 0x7f8b400c5280 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.037+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b34005490 con 0x7f8b400c5280 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.037+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34012870 con 0x7f8b400c5280 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.037+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4014db20 con 0x7f8b400c5280 2024-09-16T01:49:06.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.037+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b401f0510 con 0x7f8b400c5280 2024-09-16T01:49:06.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.038+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b400c8c40 con 0x7f8b400c5280 2024-09-16T01:49:06.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.039+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8b34012a10 con 0x7f8b400c5280 2024-09-16T01:49:06.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.040+0000 7f8b3d600640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8b2006c2b0 0x7f8b2006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.040+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f8b3408afe0 con 0x7f8b400c5280 2024-09-16T01:49:06.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.040+0000 7f8b44a00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8b2006c2b0 0x7f8b2006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.041+0000 7f8b44a00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8b2006c2b0 0x7f8b2006e770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b40158160 tx=0x7f8b2c007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.044+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8b34058ff0 con 0x7f8b400c5280 2024-09-16T01:49:06.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.220+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b400c5ee0 con 0x7f8b400c5280 2024-09-16T01:49:06.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.221+0000 7f8b3d600640 1 -- 172.21.15.146:0/3872901473 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16693 (secure 0 0 0) 0x7f8b3405d2f0 con 0x7f8b400c5280 2024-09-16T01:49:06.222 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:06.223 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":22,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:59.816171+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:49:06.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.223+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8b2006c2b0 msgr2=0x7f8b2006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.223+0000 7f8b47e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8b2006c2b0 0x7f8b2006e770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b40158160 tx=0x7f8b2c007d30 comp rx=0 tx=0).stop 2024-09-16T01:49:06.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.223+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 msgr2=0x7f8b4014b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.223+0000 7f8b47e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 0x7f8b4014b200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c460 tx=0x7f8b3400c920 comp rx=0 tx=0).stop 2024-09-16T01:49:06.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 shutdown_connections 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8b2006c2b0 0x7f8b2006e770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8b40151a90 0x7f8b401e53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b400c6410 0x7f8b401e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 --2- 172.21.15.146:0/3872901473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b400c5280 0x7f8b4014b200 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 >> 172.21.15.146:0/3872901473 conn(0x7f8b4006f3c0 msgr2=0x7f8b40074ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 shutdown_connections 2024-09-16T01:49:06.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.224+0000 7f8b47e00640 1 -- 172.21.15.146:0/3872901473 wait complete. 2024-09-16T01:49:06.241 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T01:49:06.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:49:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.394+0000 7f61bd200640 1 Processor -- start 2024-09-16T01:49:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.394+0000 7f61bd200640 1 -- start start 2024-09-16T01:49:06.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61bd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61b815b340 0x7f61b8166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61bd200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b806d140 con 0x7f61b80c75a0 2024-09-16T01:49:06.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61bd200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b806d420 con 0x7f61b80c6b70 2024-09-16T01:49:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61bd200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f61b80cab20 con 0x7f61b815b340 2024-09-16T01:49:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61b6a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f61b80c75a0 0x7f61b80cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53626/0 (socket says 172.21.15.146:53626) 2024-09-16T01:49:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61b815b340 0x7f61b8166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61b7400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f61b80c6b70 0x7f61b80cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52658/0 (socket says 172.21.15.146:52658) 2024-09-16T01:49:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.395+0000 7f61b6a00640 1 -- 172.21.15.146:0/637258960 learned_addr learned my addr 172.21.15.146:0/637258960 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.396+0000 7f61b6000640 1 -- 172.21.15.146:0/637258960 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2305411157 0 0) 0x7f61b806d420 con 0x7f61b80c6b70 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.396+0000 7f61b6000640 1 -- 172.21.15.146:0/637258960 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f619c003520 con 0x7f61b80c6b70 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.396+0000 7f61b7e00640 1 -- 172.21.15.146:0/637258960 >> v1:172.21.15.161:6789/0 conn(0x7f61b80c6b70 legacy=0x7f61b80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.396+0000 7f61b7e00640 1 -- 172.21.15.146:0/637258960 >> v1:172.21.15.146:6789/0 conn(0x7f61b80c75a0 legacy=0x7f61b80cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.396+0000 7f61b6000640 1 -- 172.21.15.146:0/637258960 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567377141 0 0) 0x7f61b806d140 con 0x7f61b80c75a0 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.396+0000 7f61b7e00640 1 -- 172.21.15.146:0/637258960 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b806d700 con 0x7f61b815b340 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.397+0000 7f61b7e00640 1 --2- 172.21.15.146:0/637258960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b8166170 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009d90 tx=0x7f61b002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b3b3d0aef9091a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.397+0000 7f61b6000640 1 -- 172.21.15.146:0/637258960 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0007740 con 0x7f61b815b340 2024-09-16T01:49:06.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.397+0000 7f61b6000640 1 -- 172.21.15.146:0/637258960 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b00078e0 con 0x7f61b815b340 2024-09-16T01:49:06.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.397+0000 7f61b6000640 1 -- 172.21.15.146:0/637258960 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0007cb0 con 0x7f61b815b340 2024-09-16T01:49:06.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 -- 172.21.15.146:0/637258960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 msgr2=0x7f61b8166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 --2- 172.21.15.146:0/637258960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b8166170 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009d90 tx=0x7f61b002a660 comp rx=0 tx=0).stop 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 -- 172.21.15.146:0/637258960 shutdown_connections 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 --2- 172.21.15.146:0/637258960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b8166170 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 -- 172.21.15.146:0/637258960 >> 172.21.15.146:0/637258960 conn(0x7f61b80ccc20 msgr2=0x7f61b80cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 -- 172.21.15.146:0/637258960 shutdown_connections 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.398+0000 7f61bd200640 1 -- 172.21.15.146:0/637258960 wait complete. 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.399+0000 7f61bd200640 1 Processor -- start 2024-09-16T01:49:06.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.399+0000 7f61bd200640 1 -- start start 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.400+0000 7f61bd200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 0x7f61b815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.400+0000 7f61bd200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b80c75a0 0x7f61b815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.400+0000 7f61bd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b815d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.400+0000 7f61bd200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61b815b7c0 con 0x7f61b815b340 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.400+0000 7f61bd200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61b80cb050 con 0x7f61b80c75a0 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.400+0000 7f61bd200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f61b80cbc80 con 0x7f61b80c6b70 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 0x7f61b815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b6a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b80c75a0 0x7f61b815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b815d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 0x7f61b815c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37476/0 (socket says 172.21.15.146:37476) 2024-09-16T01:49:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 -- 172.21.15.146:0/4229689397 learned_addr learned my addr 172.21.15.146:0/4229689397 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 -- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b80c75a0 msgr2=0x7f61b815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b80c75a0 0x7f61b815e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 -- 172.21.15.146:0/4229689397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 msgr2=0x7f61b815d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b815d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 -- 172.21.15.146:0/4229689397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b815eaa0 con 0x7f61b80c6b70 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61b7400640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 0x7f61b815c450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61a800c390 tx=0x7f61a800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.401+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a80074e0 con 0x7f61b80c6b70 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.402+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a80047e0 con 0x7f61b80c6b70 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.402+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b81f3aa0 con 0x7f61b80c6b70 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.402+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8004bd0 con 0x7f61b80c6b70 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.402+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b80c97d0 con 0x7f61b80c6b70 2024-09-16T01:49:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.403+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b806cd20 con 0x7f61b80c6b70 2024-09-16T01:49:06.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.404+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f61a801a020 con 0x7f61b80c6b70 2024-09-16T01:49:06.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.404+0000 7f61af400640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f619806c2b0 0x7f619806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.404+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f61a808aa50 con 0x7f61b80c6b70 2024-09-16T01:49:06.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.405+0000 7f61b6a00640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f619806c2b0 0x7f619806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.405+0000 7f61b6a00640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f619806c2b0 0x7f619806e770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61b81e6750 tx=0x7f61a4007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.409+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f61a8058cc0 con 0x7f61b80c6b70 2024-09-16T01:49:06.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.591+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61b80ca280 con 0x7f61b80c6b70 2024-09-16T01:49:06.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.593+0000 7f61af400640 1 -- 172.21.15.146:0/4229689397 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16693 (secure 0 0 0) 0x7f61a805cfc0 con 0x7f61b80c6b70 2024-09-16T01:49:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:06.594 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":22,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:59.816171+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:49:06.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f619806c2b0 msgr2=0x7f619806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f619806c2b0 0x7f619806e770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f61b81e6750 tx=0x7f61a4007d50 comp rx=0 tx=0).stop 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 msgr2=0x7f61b815c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 0x7f61b815c450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61a800c390 tx=0x7f61a800c850 comp rx=0 tx=0).stop 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 shutdown_connections 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f619806c2b0 0x7f619806e770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61b815b340 0x7f61b815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b80c75a0 0x7f61b815e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 --2- 172.21.15.146:0/4229689397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61b80c6b70 0x7f61b815c450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 >> 172.21.15.146:0/4229689397 conn(0x7f61b80ccc20 msgr2=0x7f61b80c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 shutdown_connections 2024-09-16T01:49:06.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.595+0000 7f61bd200640 1 -- 172.21.15.146:0/4229689397 wait complete. 2024-09-16T01:49:06.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T01:49:06.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T01:49:06.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T01:49:06.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T01:49:06.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T01:49:06.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T01:49:06.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T01:49:06.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T01:49:06.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.889+0000 7f67c9200640 1 Processor -- start 2024-09-16T01:49:06.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.889+0000 7f67c9200640 1 -- start start 2024-09-16T01:49:06.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.891+0000 7f5d48a00640 1 Processor -- start 2024-09-16T01:49:06.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c9200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 0x7f67c40cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67c415af00 con 0x7f67c40cc0c0 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c9200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67c415b1d0 con 0x7f67c40cb2f0 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c9200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f67c40cbd00 con 0x7f67c415aab0 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 0x7f67c40cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 0x7f67c40cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37492/0 (socket says 172.21.15.146:37492) 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.892+0000 7f67c2800640 1 -- 172.21.15.146:0/3769134878 learned_addr learned my addr 172.21.15.146:0/3769134878 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.891+0000 7f5d48a00640 1 -- start start 2024-09-16T01:49:06.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.893+0000 7f67c2800640 1 -- 172.21.15.146:0/3769134878 >> v1:172.21.15.161:6789/0 conn(0x7f67c40cb2f0 legacy=0x7f67c4163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.893+0000 7f67c2800640 1 -- 172.21.15.146:0/3769134878 >> v1:172.21.15.146:6789/0 conn(0x7f67c40cc0c0 legacy=0x7f67c415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f67c2800640 1 -- 172.21.15.146:0/3769134878 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c415b4b0 con 0x7f67c415aab0 2024-09-16T01:49:06.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.895+0000 7fc2b2200640 1 Processor -- start 2024-09-16T01:49:06.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f5d48a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5d44158f40 0x7f5d4415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.895+0000 7fc2b2200640 1 -- start start 2024-09-16T01:49:06.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f5d48a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d44159320 0x7f5d440c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f5d48a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4415f430 con 0x7f5d44162c00 2024-09-16T01:49:06.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f5d48a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5d44162770 con 0x7f5d44159320 2024-09-16T01:49:06.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f5d48a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5d4415f710 con 0x7f5d44158f40 2024-09-16T01:49:06.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.894+0000 7f5d42800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5d44162c00 0x7f5d440c7430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52668/0 (socket says 172.21.15.146:52668) 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.895+0000 7f5d42800640 1 -- 172.21.15.146:0/1963791973 learned_addr learned my addr 172.21.15.146:0/1963791973 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.895+0000 7f5d41400640 1 --2- 172.21.15.146:0/1963791973 >> v2:172.21.15.146:3300/0 conn(0x7f5d44159320 0x7f5d440c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.895+0000 7f5d41e00640 1 --2- 172.21.15.146:0/1963791973 >> v2:172.21.15.161:3300/0 conn(0x7f5d44158f40 0x7f5d4415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.895+0000 7f5d40a00640 1 -- 172.21.15.146:0/1963791973 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474913912 0 0) 0x7f5d4415f430 con 0x7f5d44162c00 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f5d40a00640 1 -- 172.21.15.146:0/1963791973 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d28003520 con 0x7f5d44162c00 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f5d41400640 1 -- 172.21.15.146:0/1963791973 >> v2:172.21.15.161:3300/0 conn(0x7f5d44158f40 msgr2=0x7f5d4415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f67c2800640 1 --2- 172.21.15.146:0/3769134878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 0x7f67c40cc4a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098a0 tx=0x7f67b002a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8688c696e12dd46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f5d41400640 1 --2- 172.21.15.146:0/1963791973 >> v2:172.21.15.161:3300/0 conn(0x7f5d44158f40 0x7f5d4415ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f5d41400640 1 -- 172.21.15.146:0/1963791973 >> v1:172.21.15.161:6789/0 conn(0x7f5d44162c00 legacy=0x7f5d440c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f5d41400640 1 -- 172.21.15.146:0/1963791973 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4415f9f0 con 0x7f5d44159320 2024-09-16T01:49:06.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f67c0a00640 1 -- 172.21.15.146:0/3769134878 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0037070 con 0x7f67c415aab0 2024-09-16T01:49:06.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f5d41e00640 1 --2- 172.21.15.146:0/1963791973 >> v2:172.21.15.161:3300/0 conn(0x7f5d44158f40 0x7f5d4415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:06.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f67c0a00640 1 -- 172.21.15.146:0/3769134878 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67b0032640 con 0x7f67c415aab0 2024-09-16T01:49:06.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.896+0000 7f67c0a00640 1 -- 172.21.15.146:0/3769134878 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b0032a10 con 0x7f67c415aab0 2024-09-16T01:49:06.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.897+0000 7f5d41400640 1 --2- 172.21.15.146:0/1963791973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44159320 0x7f5d440c8b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d440cd270 tx=0x7f5d3c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693ec8730cba7e71 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f67c9200640 1 -- 172.21.15.146:0/3769134878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 msgr2=0x7f67c40cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f67c9200640 1 --2- 172.21.15.146:0/3769134878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 0x7f67c40cc4a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098a0 tx=0x7f67b002a2e0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f5d40a00640 1 -- 172.21.15.146:0/1963791973 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c007740 con 0x7f5d44159320 2024-09-16T01:49:06.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f5d40a00640 1 -- 172.21.15.146:0/1963791973 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3c0078e0 con 0x7f5d44159320 2024-09-16T01:49:06.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f5d40a00640 1 -- 172.21.15.146:0/1963791973 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c007cb0 con 0x7f5d44159320 2024-09-16T01:49:06.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f67c9200640 1 -- 172.21.15.146:0/3769134878 shutdown_connections 2024-09-16T01:49:06.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f67c9200640 1 --2- 172.21.15.146:0/3769134878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c415aab0 0x7f67c40cc4a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f67c9200640 1 -- 172.21.15.146:0/3769134878 >> 172.21.15.146:0/3769134878 conn(0x7f67c406d560 msgr2=0x7f67c406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f0762000640 1 Processor -- start 2024-09-16T01:49:06.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.898+0000 7f67c9200640 1 -- 172.21.15.146:0/3769134878 shutdown_connections 2024-09-16T01:49:06.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.899+0000 7f0762000640 1 -- start start 2024-09-16T01:49:06.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.899+0000 7f5d48a00640 1 -- 172.21.15.146:0/1963791973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44159320 msgr2=0x7f5d440c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.899+0000 7f5d48a00640 1 --2- 172.21.15.146:0/1963791973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44159320 0x7f5d440c8b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d440cd270 tx=0x7f5d3c02a710 comp rx=0 tx=0).stop 2024-09-16T01:49:06.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.899+0000 7f67c9200640 1 -- 172.21.15.146:0/3769134878 wait complete. 2024-09-16T01:49:06.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.899+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a4005a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.900+0000 7f5d48a00640 1 -- 172.21.15.146:0/1963791973 shutdown_connections 2024-09-16T01:49:06.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.900+0000 7f5d48a00640 1 --2- 172.21.15.146:0/1963791973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44159320 0x7f5d440c8b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.900+0000 7f5d48a00640 1 --2- 172.21.15.146:0/1963791973 >> v2:172.21.15.161:3300/0 conn(0x7f5d44158f40 0x7f5d4415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.900+0000 7f5d48a00640 1 -- 172.21.15.146:0/1963791973 >> 172.21.15.146:0/1963791973 conn(0x7f5d4406f3c0 msgr2=0x7f5d4406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0762000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f075c0cc0c0 0x7f075c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0762000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 0x7f075c0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0762000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f075c15b1d0 con 0x7f075c15e9d0 2024-09-16T01:49:06.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0762000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f075c0cbd00 con 0x7f075c0cc0c0 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0762000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f075c15ba70 con 0x7f075c0cb2f0 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0752a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 0x7f075c0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0752a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 0x7f075c0cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37516/0 (socket says 172.21.15.146:37516) 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f0752a00640 1 -- 172.21.15.146:0/4209036929 learned_addr learned my addr 172.21.15.146:0/4209036929 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f075b400640 1 --2- 172.21.15.146:0/4209036929 >> v2:172.21.15.146:3300/0 conn(0x7f075c0cc0c0 0x7f075c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.902+0000 7f0752a00640 1 -- 172.21.15.146:0/4209036929 >> v2:172.21.15.146:3300/0 conn(0x7f075c0cc0c0 msgr2=0x7f075c15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.902+0000 7f67c9200640 1 Processor -- start 2024-09-16T01:49:06.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.902+0000 7f0752a00640 1 --2- 172.21.15.146:0/4209036929 >> v2:172.21.15.146:3300/0 conn(0x7f075c0cc0c0 0x7f075c15aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.902+0000 7f0752a00640 1 -- 172.21.15.146:0/4209036929 >> v1:172.21.15.161:6789/0 conn(0x7f075c15e9d0 legacy=0x7f075c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.902+0000 7f67c9200640 1 -- start start 2024-09-16T01:49:06.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.902+0000 7f0752a00640 1 -- 172.21.15.146:0/4209036929 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c15b4b0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.900+0000 7fc2b2200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2a404c090 0x7fc2a40de360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.900+0000 7fc2abe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc2a40450f0 0x7fc2a40ddc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53646/0 (socket says 172.21.15.146:53646) 2024-09-16T01:49:06.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7fc2b1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2a404c090 0x7fc2a40de360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7fc2b1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2a404c090 0x7fc2a40de360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58998/0 (socket says 172.21.15.146:58998) 2024-09-16T01:49:06.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7fc2b0e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a4005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7fc2b0e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a4005a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37504/0 (socket says 172.21.15.146:37504) 2024-09-16T01:49:06.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67c40cb2f0 0x7f67c415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c9200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c40cc0c0 0x7f67c415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c9200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67c415aab0 0x7f67c414b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f67c415e540 con 0x7f67c40cb2f0 2024-09-16T01:49:06.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c9200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67c415ba70 con 0x7f67c415aab0 2024-09-16T01:49:06.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c9200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f67c415e1d0 con 0x7f67c40cc0c0 2024-09-16T01:49:06.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.903+0000 7fc2b2200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a404aa40 con 0x7fc2a40450f0 2024-09-16T01:49:06.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.903+0000 7fc2b2200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a4044c60 con 0x7fc2a404c090 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.903+0000 7fc2b2200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a404ace0 con 0x7fc2a40d4940 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.903+0000 7fc2abe00640 1 -- 172.21.15.146:0/4135209273 learned_addr learned my addr 172.21.15.146:0/4135209273 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7fc2b0e00640 1 -- 172.21.15.146:0/4135209273 >> v2:172.21.15.146:3300/0 conn(0x7fc2a404c090 msgr2=0x7fc2a40de360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c40cc0c0 0x7f67c415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c40cc0c0 0x7f67c415d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37522/0 (socket says 172.21.15.146:37522) 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c1400640 1 -- 172.21.15.146:0/1810081063 learned_addr learned my addr 172.21.15.146:0/1810081063 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c2800640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67c415aab0 0x7f67c414b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7f67c1e00640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67c40cb2f0 0x7f67c415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7fc2b0e00640 1 --2- 172.21.15.146:0/4135209273 >> v2:172.21.15.146:3300/0 conn(0x7fc2a404c090 0x7fc2a40de360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.904+0000 7fc2b0e00640 1 -- 172.21.15.146:0/4135209273 >> v1:172.21.15.146:6789/0 conn(0x7fc2a40450f0 legacy=0x7fc2a40ddc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7fc2b0e00640 1 -- 172.21.15.146:0/4135209273 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a404afa0 con 0x7fc2a40d4940 2024-09-16T01:49:06.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f67c1400640 1 -- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67c415aab0 msgr2=0x7f67c414b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f67c1400640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67c415aab0 0x7f67c414b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f67c1400640 1 -- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67c40cb2f0 msgr2=0x7f67c415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f67c1400640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67c40cb2f0 0x7f67c415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f67c1400640 1 -- 172.21.15.146:0/1810081063 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c41554b0 con 0x7f67c40cc0c0 2024-09-16T01:49:06.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f5d48a00640 1 -- 172.21.15.146:0/1963791973 shutdown_connections 2024-09-16T01:49:06.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.901+0000 7f5d48a00640 1 -- 172.21.15.146:0/1963791973 wait complete. 2024-09-16T01:49:06.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f0752a00640 1 --2- 172.21.15.146:0/4209036929 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 0x7f075c0cc4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f074c009870 tx=0x7f074c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17a06b5d6b503b6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7fc0ebe00640 1 Processor -- start 2024-09-16T01:49:06.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.903+0000 7f5d48a00640 1 Processor -- start 2024-09-16T01:49:06.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f67c1400640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c40cc0c0 0x7f67c415d870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00c730 tx=0x7f67bc00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f075aa00640 1 -- 172.21.15.146:0/4209036929 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c037070 con 0x7f075c0cb2f0 2024-09-16T01:49:06.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f075aa00640 1 -- 172.21.15.146:0/4209036929 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c032440 con 0x7f075c0cb2f0 2024-09-16T01:49:06.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7f075aa00640 1 -- 172.21.15.146:0/4209036929 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c032810 con 0x7f075c0cb2f0 2024-09-16T01:49:06.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.905+0000 7fc0ebe00640 1 -- start start 2024-09-16T01:49:06.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.906+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc016020 con 0x7f67c40cc0c0 2024-09-16T01:49:06.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.906+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67bc006d50 con 0x7f67c40cc0c0 2024-09-16T01:49:06.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.906+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc003b90 con 0x7f67c40cc0c0 2024-09-16T01:49:06.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f0762000640 1 -- 172.21.15.146:0/4209036929 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 msgr2=0x7f075c0cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f0762000640 1 --2- 172.21.15.146:0/4209036929 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 0x7f075c0cc4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f074c009870 tx=0x7f074c02a210 comp rx=0 tx=0).stop 2024-09-16T01:49:06.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.906+0000 7f5d48a00640 1 -- start start 2024-09-16T01:49:06.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc2b0e00640 1 --2- 172.21.15.146:0/4135209273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a4005a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc298011fc0 tx=0x7fc298032940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c808c5bc4fbbba04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f0762000640 1 -- 172.21.15.146:0/4209036929 shutdown_connections 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f0762000640 1 --2- 172.21.15.146:0/4209036929 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cb2f0 0x7f075c0cc4a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f0762000640 1 --2- 172.21.15.146:0/4209036929 >> v2:172.21.15.146:3300/0 conn(0x7f075c0cc0c0 0x7f075c15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f0762000640 1 -- 172.21.15.146:0/4209036929 >> 172.21.15.146:0/4209036929 conn(0x7f075c06d560 msgr2=0x7f075c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f5d48a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d44158f40 0x7f5d441f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f5d48a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d44159320 0x7f5d441f8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f5d48a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44162c00 0x7f5d44151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d41400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d44159320 0x7f5d441f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d44158f40 0x7f5d441f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d44158f40 0x7f5d441f54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37534/0 (socket says 172.21.15.146:37534) 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d41400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d44159320 0x7f5d441f8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45062/0 (socket says 172.21.15.146:45062) 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d48a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44072070 con 0x7f5d44162c00 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d48a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d44162400 con 0x7f5d44159320 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d48a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5d44071930 con 0x7f5d44158f40 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c4155750 con 0x7f67c40cc0c0 2024-09-16T01:49:06.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c4155d10 con 0x7f67c40cc0c0 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc2ab400640 1 -- 172.21.15.146:0/4135209273 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2980112e0 con 0x7fc2a40d4940 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc2ab400640 1 -- 172.21.15.146:0/4135209273 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc29803d070 con 0x7fc2a40d4940 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc2ab400640 1 -- 172.21.15.146:0/4135209273 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2980116a0 con 0x7fc2a40d4940 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc0ebe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 0x7fc0ec15de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc0ebe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc0ec06d950 0x7fc0ec15e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc0ebe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0ec0cc930 con 0x7fc0ec154c90 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc0ebe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc0ec15ab60 con 0x7fc0ec06d950 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.907+0000 7fc0ebe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ec15ae40 con 0x7fc0ec0b3020 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 0x7fc0ec15de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 0x7fc0ec15de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37532/0 (socket says 172.21.15.146:37532) 2024-09-16T01:49:06.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 -- 172.21.15.146:0/53118426 learned_addr learned my addr 172.21.15.146:0/53118426 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0eb400640 1 --2- 172.21.15.146:0/53118426 >> v2:172.21.15.161:3300/0 conn(0x7fc0ec06d950 0x7fc0ec15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 -- 172.21.15.146:0/53118426 >> v2:172.21.15.161:3300/0 conn(0x7fc0ec06d950 msgr2=0x7fc0ec15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 --2- 172.21.15.146:0/53118426 >> v2:172.21.15.161:3300/0 conn(0x7fc0ec06d950 0x7fc0ec15e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d41e00640 1 -- 172.21.15.146:0/4089260884 learned_addr learned my addr 172.21.15.146:0/4089260884 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7f5d42800640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44162c00 0x7f5d44151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d41400640 1 -- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d44158f40 msgr2=0x7f5d441f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f67c40c5560 con 0x7f67c40cc0c0 2024-09-16T01:49:06.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 -- 172.21.15.146:0/53118426 >> v1:172.21.15.146:6789/0 conn(0x7fc0ec154c90 legacy=0x7fc0ec0ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0e2000640 1 -- 172.21.15.146:0/53118426 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec15b120 con 0x7fc0ec0b3020 2024-09-16T01:49:06.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.908+0000 7fc0eb400640 1 --2- 172.21.15.146:0/53118426 >> v2:172.21.15.161:3300/0 conn(0x7fc0ec06d950 0x7fc0ec15e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d41400640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d44158f40 0x7f5d441f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d41400640 1 -- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44162c00 msgr2=0x7f5d44151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d41400640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44162c00 0x7f5d44151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d41400640 1 -- 172.21.15.146:0/4089260884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d441f8eb0 con 0x7f5d44159320 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d42800640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d44162c00 0x7f5d44151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f5d41400640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d44159320 0x7f5d441f8970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d44073710 tx=0x7f5d3c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.909+0000 7f0762000640 1 -- 172.21.15.146:0/4209036929 shutdown_connections 2024-09-16T01:49:06.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c031b40 con 0x7f5d44159320 2024-09-16T01:49:06.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc2b2200640 1 -- 172.21.15.146:0/4135209273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 msgr2=0x7fc2a4005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc2b2200640 1 --2- 172.21.15.146:0/4135209273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a4005a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc298011fc0 tx=0x7fc298032940 comp rx=0 tx=0).stop 2024-09-16T01:49:06.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc0e2000640 1 --2- 172.21.15.146:0/53118426 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 0x7fc0ec15de20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009870 tx=0x7fc0d402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f84ebfef92feda91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7f0762000640 1 -- 172.21.15.146:0/4209036929 wait complete. 2024-09-16T01:49:06.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc0ea000640 1 -- 172.21.15.146:0/53118426 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4037070 con 0x7fc0ec0b3020 2024-09-16T01:49:06.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc0ea000640 1 -- 172.21.15.146:0/53118426 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4032440 con 0x7fc0ec0b3020 2024-09-16T01:49:06.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc0ea000640 1 -- 172.21.15.146:0/53118426 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4032810 con 0x7fc0ec0b3020 2024-09-16T01:49:06.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7fc2b2200640 1 -- 172.21.15.146:0/4135209273 shutdown_connections 2024-09-16T01:49:06.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f67bc00a850 con 0x7f67c40cc0c0 2024-09-16T01:49:06.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7fc2b2200640 1 --2- 172.21.15.146:0/4135209273 >> v2:172.21.15.146:3300/0 conn(0x7fc2a404c090 0x7fc2a40de360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7fc2b2200640 1 --2- 172.21.15.146:0/4135209273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a4005a00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3c031d00 con 0x7f5d44159320 2024-09-16T01:49:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c03c660 con 0x7f5d44159320 2024-09-16T01:49:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d440c43b0 con 0x7f5d44159320 2024-09-16T01:49:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.910+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d440c4970 con 0x7f5d44159320 2024-09-16T01:49:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d44152ef0 con 0x7f5d44159320 2024-09-16T01:49:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7fc2b2200640 1 -- 172.21.15.146:0/4135209273 >> 172.21.15.146:0/4135209273 conn(0x7fc2a404fc50 msgr2=0x7fc2a4050060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7fc0ebe00640 1 -- 172.21.15.146:0/53118426 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 msgr2=0x7fc0ec15de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.911+0000 7fc0ebe00640 1 --2- 172.21.15.146:0/53118426 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 0x7fc0ec15de20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009870 tx=0x7fc0d402a290 comp rx=0 tx=0).stop 2024-09-16T01:49:06.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7fc0ebe00640 1 -- 172.21.15.146:0/53118426 shutdown_connections 2024-09-16T01:49:06.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7fc0ebe00640 1 --2- 172.21.15.146:0/53118426 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec0b3020 0x7fc0ec15de20 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009870 tx=0x7fc0d402a290 comp rx=0 tx=0).stop 2024-09-16T01:49:06.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7fc0ebe00640 1 --2- 172.21.15.146:0/53118426 >> v2:172.21.15.161:3300/0 conn(0x7fc0ec06d950 0x7fc0ec15e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7fc0ebe00640 1 -- 172.21.15.146:0/53118426 >> 172.21.15.146:0/53118426 conn(0x7fc0ec067950 msgr2=0x7fc0ec067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7fc2b2200640 1 -- 172.21.15.146:0/4135209273 shutdown_connections 2024-09-16T01:49:06.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7fc2b2200640 1 -- 172.21.15.146:0/4135209273 wait complete. 2024-09-16T01:49:06.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7f0762000640 1 Processor -- start 2024-09-16T01:49:06.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.912+0000 7f67ba000640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67a406c300 0x7f67a406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f67bc08afb0 con 0x7f67c40cc0c0 2024-09-16T01:49:06.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f0762000640 1 -- start start 2024-09-16T01:49:06.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f67ba000640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67a4074580 0x7f67a40769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f3b02000640 1 Processor -- start 2024-09-16T01:49:06.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67bc010e50 con 0x7f67a4074580 2024-09-16T01:49:06.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f67c2800640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67a4074580 0x7f67a40769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f67bc08b490 con 0x7f67c40cc0c0 2024-09-16T01:49:06.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7fc0ebe00640 1 -- 172.21.15.146:0/53118426 shutdown_connections 2024-09-16T01:49:06.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f67c1e00640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67a406c300 0x7f67a406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f0762000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f075c0cb2f0 0x7f075c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f0762000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cc0c0 0x7f075c1577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f0762000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f075c15e9d0 0x7f075c156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f0762000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f075c15e1d0 con 0x7f075c15e9d0 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f0762000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f075c15e540 con 0x7f075c0cb2f0 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f0762000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f075c0c5560 con 0x7f075c0cc0c0 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f075b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f075c0cb2f0 0x7f075c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f075b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f075c0cb2f0 0x7f075c0ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45074/0 (socket says 172.21.15.146:45074) 2024-09-16T01:49:06.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f075b400640 1 -- 172.21.15.146:0/1815518993 learned_addr learned my addr 172.21.15.146:0/1815518993 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5d3c039040 con 0x7f5d44159320 2024-09-16T01:49:06.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f67c2800640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67a4074580 0x7f67a40769a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7fc2b2200640 1 Processor -- start 2024-09-16T01:49:06.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.913+0000 7f3b02000640 1 -- start start 2024-09-16T01:49:06.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f075b400640 1 -- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cc0c0 msgr2=0x7f075c1577c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:49:06.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f075b400640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f075c0cc0c0 0x7f075c1577c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f075b400640 1 -- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f075c15e9d0 msgr2=0x7f075c156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f075b400640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f075c15e9d0 0x7f075c156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f075b400640 1 -- 172.21.15.146:0/1815518993 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c0c7af0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f39f9600640 1 Processor -- start 2024-09-16T01:49:06.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7f3b02000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 0x7f3afc15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.915+0000 7fc2b2200640 1 -- start start 2024-09-16T01:49:06.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3b02000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3afc159320 0x7f3afc0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3b02000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3afc162c00 0x7f3afc0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3b02000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3afc162770 con 0x7f3afc162c00 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3b02000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3afc15f430 con 0x7f3afc159320 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3b02000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc15f710 con 0x7f3afc158f40 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 0x7f3afc15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 0x7f3afc15ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37550/0 (socket says 172.21.15.146:37550) 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 -- 172.21.15.146:0/2801343867 learned_addr learned my addr 172.21.15.146:0/2801343867 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f5d3a000640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5d2406c380 0x7f5d2406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5d3c0ac9e0 con 0x7f5d44159320 2024-09-16T01:49:06.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3af2a00640 1 --2- 172.21.15.146:0/2801343867 >> v2:172.21.15.161:3300/0 conn(0x7f3afc159320 0x7f3afc0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 -- 172.21.15.146:0/2801343867 >> v2:172.21.15.161:3300/0 conn(0x7f3afc159320 msgr2=0x7f3afc0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 --2- 172.21.15.146:0/2801343867 >> v2:172.21.15.161:3300/0 conn(0x7f3afc159320 0x7f3afc0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 -- 172.21.15.146:0/2801343867 >> v2:172.21.15.146:3300/0 conn(0x7f3afc162c00 msgr2=0x7f3afc0c7430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:49:06.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 --2- 172.21.15.146:0/2801343867 >> v2:172.21.15.146:3300/0 conn(0x7f3afc162c00 0x7f3afc0c7430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f3afb400640 1 -- 172.21.15.146:0/2801343867 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc15f9f0 con 0x7f3afc158f40 2024-09-16T01:49:06.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67bc010e50 con 0x7f67a4074580 2024-09-16T01:49:06.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f5d3a000640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5d24074520 0x7f5d24076940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.916+0000 7f39f9600640 1 -- start start 2024-09-16T01:49:06.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.914+0000 7fc0ebe00640 1 -- 172.21.15.146:0/53118426 wait complete. 2024-09-16T01:49:06.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d24076ff0 con 0x7f5d24074520 2024-09-16T01:49:06.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f5d41e00640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5d2406c380 0x7f5d2406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5d3c076d60 con 0x7f5d44159320 2024-09-16T01:49:06.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f724b200640 1 Processor -- start 2024-09-16T01:49:06.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f724b200640 1 -- start start 2024-09-16T01:49:06.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f075b400640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f075c0cb2f0 0x7f075c0ca4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f074400e9e0 tx=0x7f074400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7f5d42800640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5d24074520 0x7f5d24076940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f3afb400640 1 --2- 172.21.15.146:0/2801343867 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 0x7f3afc15ee10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3afc069a70 tx=0x7f3aec02a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c7e3ffdf73e2a5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40450f0 0x7fc2a40ddde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2a404c090 0x7fc2a4171390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2b2200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a40d71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2b2200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a4046770 con 0x7fc2a40450f0 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2b2200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a4046490 con 0x7fc2a404c090 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2b2200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a4005f40 con 0x7fc2a40d4940 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2abe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2a404c090 0x7fc2a4171390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2abe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2a404c090 0x7fc2a4171390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45094/0 (socket says 172.21.15.146:45094) 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc2abe00640 1 -- 172.21.15.146:0/2103178836 learned_addr learned my addr 172.21.15.146:0/2103178836 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.917+0000 7fc0ebe00640 1 Processor -- start 2024-09-16T01:49:06.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f5d41e00640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5d2406c380 0x7f5d2406e840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009560 tx=0x7f5d34009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f3afaa00640 1 -- 172.21.15.146:0/2801343867 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec036070 con 0x7f3afc158f40 2024-09-16T01:49:06.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7fc2abe00640 1 -- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 msgr2=0x7fc2a40d71c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:49:06.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7fc2abe00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc2a40d4940 0x7fc2a40d71c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7fc2abe00640 1 -- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40450f0 msgr2=0x7fc2a40ddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7fc2abe00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40450f0 0x7fc2a40ddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7fc2abe00640 1 -- 172.21.15.146:0/2103178836 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a40d8cf0 con 0x7fc2a404c090 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744017070 con 0x7f075c0cb2f0 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074400c9d0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074400cdc0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075c0c7dd0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075c1555a0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f3afaa00640 1 -- 172.21.15.146:0/2801343867 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aec031600 con 0x7f3afc158f40 2024-09-16T01:49:06.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f3afaa00640 1 -- 172.21.15.146:0/2801343867 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0319b0 con 0x7f3afc158f40 2024-09-16T01:49:06.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7f67c1e00640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67a406c300 0x7f67a406e7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67b4000f60 tx=0x7f67b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.919+0000 7f39f9600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39ec0dcd10 0x7f39ec0e6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.918+0000 7fc0ebe00640 1 -- start start 2024-09-16T01:49:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.919+0000 7f724b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7244152850 0x7f7244158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 -- 172.21.15.146:0/2801343867 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 msgr2=0x7f3afc15ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 --2- 172.21.15.146:0/2801343867 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 0x7f3afc15ee10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3afc069a70 tx=0x7f3aec02a760 comp rx=0 tx=0).stop 2024-09-16T01:49:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc2abe00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2a404c090 0x7fc2a4171390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac06b730 tx=0x7fc2ac069890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.919+0000 7f724b200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7244158bb0 0x7f72440c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.919+0000 7f724b200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7244072f50 con 0x7f72440c6bb0 2024-09-16T01:49:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.919+0000 7f724b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f7244073210 con 0x7f7244152850 2024-09-16T01:49:06.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.919+0000 7f724b200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f724415cb50 con 0x7f7244158bb0 2024-09-16T01:49:06.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 -- 172.21.15.146:0/2801343867 shutdown_connections 2024-09-16T01:49:06.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 --2- 172.21.15.146:0/2801343867 >> v2:172.21.15.146:3300/0 conn(0x7f3afc162c00 0x7f3afc0c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 --2- 172.21.15.146:0/2801343867 >> v2:172.21.15.161:3300/0 conn(0x7f3afc159320 0x7f3afc0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 --2- 172.21.15.146:0/2801343867 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc158f40 0x7f3afc15ee10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 -- 172.21.15.146:0/2801343867 >> 172.21.15.146:0/2801343867 conn(0x7f3afc06f3c0 msgr2=0x7f3afc06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f075c0ccc00 con 0x7f075c0cb2f0 2024-09-16T01:49:06.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f7243e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7244152850 0x7f7244158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f7243e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7244152850 0x7f7244158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59062/0 (socket says 172.21.15.146:59062) 2024-09-16T01:49:06.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f7243e00640 1 -- 172.21.15.146:0/2266505505 learned_addr learned my addr 172.21.15.146:0/2266505505 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f3b02000640 1 -- 172.21.15.146:0/2801343867 shutdown_connections 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39ec0dcd10 0x7f39ec0e6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39ec0dcd10 0x7f39ec0e6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59054/0 (socket says 172.21.15.146:59054) 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f3e00640 1 -- 172.21.15.146:0/1283642337 learned_addr learned my addr 172.21.15.146:0/1283642337 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ec0e39e0 con 0x7f39ec00f670 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ec050840 con 0x7f39ec00e3a0 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f39ec04fea0 con 0x7f39ec0dcd10 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f8c00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f39ec00e3a0 0x7f39ec0ed380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52682/0 (socket says 172.21.15.146:52682) 2024-09-16T01:49:06.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc0ebe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec06d950 0x7fc0ec0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f7243e00640 1 -- 172.21.15.146:0/2266505505 >> v2:172.21.15.161:3300/0 conn(0x7f7244158bb0 msgr2=0x7f72440c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f7243400640 1 --2- 172.21.15.146:0/2266505505 >> v2:172.21.15.161:3300/0 conn(0x7f7244158bb0 0x7f72440c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f3b02000640 1 -- 172.21.15.146:0/2801343867 wait complete. 2024-09-16T01:49:06.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7f39f3400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f39ec00f670 0x7f39ec04e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53666/0 (socket says 172.21.15.146:53666) 2024-09-16T01:49:06.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353290523 0 0) 0x7f39ec050840 con 0x7f39ec00e3a0 2024-09-16T01:49:06.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f7243e00640 1 --2- 172.21.15.146:0/2266505505 >> v2:172.21.15.161:3300/0 conn(0x7f7244158bb0 0x7f72440c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f7243e00640 1 -- 172.21.15.146:0/2266505505 >> v1:172.21.15.161:6789/0 conn(0x7f72440c6bb0 legacy=0x7f72440c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39dc003520 con 0x7f39ec00e3a0 2024-09-16T01:49:06.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc0ebe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ec154c90 0x7fc0ec1fcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc0ebe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ec156ed0 0x7fc0ec15cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f5d42800640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5d24074520 0x7f5d24076940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc0ebe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec0cd360 con 0x7fc0ec154c90 2024-09-16T01:49:06.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc0ebe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec15a830 con 0x7fc0ec156ed0 2024-09-16T01:49:06.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc0ebe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ec0cd4e0 con 0x7fc0ec06d950 2024-09-16T01:49:06.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7fc0e2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ec154c90 0x7fc0ec1fcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7fc0eaa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec06d950 0x7fc0ec0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f7242a00640 1 -- 172.21.15.146:0/2266505505 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264664844 0 0) 0x7f7244072f50 con 0x7f72440c6bb0 2024-09-16T01:49:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f7243e00640 1 -- 172.21.15.146:0/2266505505 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724415bd90 con 0x7f7244152850 2024-09-16T01:49:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7fc0e2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ec154c90 0x7fc0ec1fcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59072/0 (socket says 172.21.15.146:59072) 2024-09-16T01:49:06.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7fc0e2000640 1 -- 172.21.15.146:0/3370161717 learned_addr learned my addr 172.21.15.146:0/3370161717 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 603349643 0 0) 0x7f39dc003520 con 0x7f39ec00e3a0 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 >> v2:172.21.15.146:3300/0 conn(0x7f39ec0dcd10 msgr2=0x7f39ec0e6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 --2- 172.21.15.146:0/1283642337 >> v2:172.21.15.146:3300/0 conn(0x7f39ec0dcd10 0x7f39ec0e6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 >> v1:172.21.15.146:6789/0 conn(0x7f39ec00f670 legacy=0x7f39ec04e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec050580 con 0x7f39ec00e3a0 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f39e8001670 con 0x7f39ec00e3a0 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 >> v1:172.21.15.161:6789/0 conn(0x7f39ec00e3a0 legacy=0x7f39ec0ed380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc003a00 0x7f39dc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39dc004800 0x7f39dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.921+0000 7f39f2a00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39dc005230 0x7f39dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec04fea0 con 0x7f39dc003a00 2024-09-16T01:49:06.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc00c180 con 0x7f39dc005230 2024-09-16T01:49:06.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f39dc00c380 con 0x7f39dc004800 2024-09-16T01:49:06.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f7243400640 1 --2- 172.21.15.146:0/2266505505 >> v2:172.21.15.161:3300/0 conn(0x7f7244158bb0 0x7f72440c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39dc005230 0x7f39dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f3400640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39dc004800 0x7f39dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f8c00640 1 -- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39dc004800 msgr2=0x7f39dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39dc004800 0x7f39dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f8c00640 1 -- 172.21.15.146:0/1283642337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc003a00 msgr2=0x7f39dc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc003a00 0x7f39dc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f39f8c00640 1 -- 172.21.15.146:0/1283642337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39dc003520 con 0x7f39dc005230 2024-09-16T01:49:06.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7fc0eb400640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ec156ed0 0x7fc0ec15cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.920+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac091070 con 0x7fc2a404c090 2024-09-16T01:49:06.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ac071850 con 0x7fc2a404c090 2024-09-16T01:49:06.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac071c80 con 0x7fc2a404c090 2024-09-16T01:49:06.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7fc0e2000640 1 -- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec06d950 msgr2=0x7fc0ec0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.922+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f074401a070 con 0x7f075c0cb2f0 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5d24076ff0 con 0x7f5d24074520 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7fc0e2000640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec06d950 0x7fc0ec0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7fc0e2000640 1 -- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ec156ed0 msgr2=0x7fc0ec15cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7fc0e2000640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ec156ed0 0x7fc0ec15cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7fc0e2000640 1 -- 172.21.15.146:0/3370161717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec155e90 con 0x7fc0ec154c90 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7fc0eaa00640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ec06d950 0x7fc0ec0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:06.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f39f8c00640 1 -- 172.21.15.146:0/1283642337 reap_dead start 2024-09-16T01:49:06.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a41718d0 con 0x7fc2a404c090 2024-09-16T01:49:06.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f3b02000640 1 Processor -- start 2024-09-16T01:49:06.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.923+0000 7f7243e00640 1 --2- 172.21.15.146:0/2266505505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244152850 0x7f7244158670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72440c67f0 tx=0x7f723002a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ee19c25a03b8e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39dc005230 0x7f39dc00b9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f39e8007f80 tx=0x7f39dc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baaa431f15935245 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a4048430 con 0x7fc2a404c090 2024-09-16T01:49:06.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f0753e00640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f073806c300 0x7f073806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f07440575a0 con 0x7f075c0cb2f0 2024-09-16T01:49:06.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f0752a00640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f073806c300 0x7f073806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f0753e00640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0738075120 0x7f07380775c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0738077c90 con 0x7f0738075120 2024-09-16T01:49:06.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f3b02000640 1 -- start start 2024-09-16T01:49:06.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.924+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8007ba0 con 0x7f39dc005230 2024-09-16T01:49:06.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e8007070 con 0x7f39dc005230 2024-09-16T01:49:06.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7f39f2a00640 1 -- 172.21.15.146:0/1283642337 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e8007460 con 0x7f39dc005230 2024-09-16T01:49:06.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7f7242a00640 1 -- 172.21.15.146:0/2266505505 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230037070 con 0x7f7244152850 2024-09-16T01:49:06.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0744055f30 con 0x7f075c0cb2f0 2024-09-16T01:49:06.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc2a40de320 con 0x7fc2a404c090 2024-09-16T01:49:06.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7f7242a00640 1 -- 172.21.15.146:0/2266505505 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7230032440 con 0x7f7244152850 2024-09-16T01:49:06.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7f7242a00640 1 -- 172.21.15.146:0/2266505505 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230032810 con 0x7f7244152850 2024-09-16T01:49:06.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39dc005230 msgr2=0x7f39dc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f39f9600640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39dc005230 0x7f39dc00b9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f39e8007f80 tx=0x7f39dc005830 comp rx=0 tx=0).stop 2024-09-16T01:49:06.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.925+0000 7fc0e2000640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ec154c90 0x7fc0ec1fcfe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0d402a800 tx=0x7fc0d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4037040 con 0x7fc0ec154c90 2024-09-16T01:49:06.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f075be00640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0738075120 0x7f07380775c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4042070 con 0x7fc0ec154c90 2024-09-16T01:49:06.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4030900 con 0x7fc0ec154c90 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f3b02000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3afc158f40 0x7f3afc0c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f3b02000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3afc159320 0x7f3afc1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f3b02000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc162c00 0x7f3afc15ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f3b02000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc072070 con 0x7f3afc158f40 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f3b02000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc162400 con 0x7f3afc159320 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f3b02000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc071930 con 0x7f3afc162c00 2024-09-16T01:49:06.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7f0752a00640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f073806c300 0x7f073806e7c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f074c034000 tx=0x7f074c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.926+0000 7fc0ebe00640 1 -- 172.21.15.146:0/3370161717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ec0c8380 con 0x7fc0ec154c90 2024-09-16T01:49:06.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc162c00 0x7f3afc15ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc162c00 0x7f3afc15ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37584/0 (socket says 172.21.15.146:37584) 2024-09-16T01:49:06.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 -- 172.21.15.146:0/716706638 learned_addr learned my addr 172.21.15.146:0/716706638 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f724b200640 1 -- 172.21.15.146:0/2266505505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244152850 msgr2=0x7f7244158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f724b200640 1 --2- 172.21.15.146:0/2266505505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244152850 0x7f7244158670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72440c67f0 tx=0x7f723002a280 comp rx=0 tx=0).stop 2024-09-16T01:49:06.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3af2a00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3afc159320 0x7f3afc1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f724b200640 1 -- 172.21.15.146:0/2266505505 shutdown_connections 2024-09-16T01:49:06.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f724b200640 1 --2- 172.21.15.146:0/2266505505 >> v2:172.21.15.161:3300/0 conn(0x7f7244158bb0 0x7f72440c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f724b200640 1 --2- 172.21.15.146:0/2266505505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244152850 0x7f7244158670 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f72440c67f0 tx=0x7f723002a280 comp rx=0 tx=0).stop 2024-09-16T01:49:06.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f724b200640 1 -- 172.21.15.146:0/2266505505 >> 172.21.15.146:0/2266505505 conn(0x7f724406f3c0 msgr2=0x7f724406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 shutdown_connections 2024-09-16T01:49:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f39f9600640 1 --2- 172.21.15.146:0/1283642337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39dc005230 0x7f39dc00b9c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 >> 172.21.15.146:0/1283642337 conn(0x7f39ec054450 msgr2=0x7f39ec054860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 shutdown_connections 2024-09-16T01:49:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f075be00640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0738075120 0x7f07380775c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f39f9600640 1 -- 172.21.15.146:0/1283642337 wait complete. 2024-09-16T01:49:06.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.928+0000 7f724b200640 1 -- 172.21.15.146:0/2266505505 shutdown_connections 2024-09-16T01:49:06.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.929+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0738077c90 con 0x7f0738075120 2024-09-16T01:49:06.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.929+0000 7f724b200640 1 -- 172.21.15.146:0/2266505505 wait complete. 2024-09-16T01:49:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.928+0000 7fc0ebe00640 1 -- 172.21.15.146:0/3370161717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ec1553b0 con 0x7fc0ec154c90 2024-09-16T01:49:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.930+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc2ac094070 con 0x7fc2a404c090 2024-09-16T01:49:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.931+0000 7fc2a8c00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc28c06c380 0x7fc28c06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 -- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3afc159320 msgr2=0x7f3afc1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3afc159320 0x7f3afc1552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 -- 172.21.15.146:0/716706638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3afc158f40 msgr2=0x7f3afc0c53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3afc158f40 0x7f3afc0c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.927+0000 7f3afbe00640 1 -- 172.21.15.146:0/716706638 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc1550e0 con 0x7f3afc162c00 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.930+0000 7f3afbe00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc162c00 0x7f3afc15ffd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3af400b410 tx=0x7f3af400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.931+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4015020 con 0x7f3afc162c00 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.931+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af4011830 con 0x7f3afc162c00 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.931+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4011c20 con 0x7f3afc162c00 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.930+0000 7f39f9600640 1 Processor -- start 2024-09-16T01:49:06.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc2ac1076d0 con 0x7fc2a404c090 2024-09-16T01:49:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc0d4030aa0 con 0x7fc0ec154c90 2024-09-16T01:49:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7fc2a8c00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc28c0745a0 0x7fc28c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3afc1537a0 con 0x7f3afc162c00 2024-09-16T01:49:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3afc154320 con 0x7f3afc162c00 2024-09-16T01:49:06.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2ac099e70 con 0x7fc28c0745a0 2024-09-16T01:49:06.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc2ac107ac0 con 0x7fc2a404c090 2024-09-16T01:49:06.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7f724b200640 1 Processor -- start 2024-09-16T01:49:06.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.933+0000 7fc2b1800640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc28c0745a0 0x7fc28c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.932+0000 7fc2b0e00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc28c06c380 0x7fc28c06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.933+0000 7f724b200640 1 -- start start 2024-09-16T01:49:06.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.934+0000 7fc2b1800640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc28c0745a0 0x7fc28c0769c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.935+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc2ac099e70 con 0x7fc28c0745a0 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.933+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3afc1630b0 con 0x7f3afc162c00 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3af4018020 con 0x7f3afc162c00 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.933+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc0ec1551c0 con 0x7fc0ec154c90 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.933+0000 7fc0e3400640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc0c806c2e0 0x7fc0c806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.934+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc0d40abd90 con 0x7fc0ec154c90 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.934+0000 7fc0e3400640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc0c80744e0 0x7fc0c8076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.935+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0c8076fb0 con 0x7fc0c80744e0 2024-09-16T01:49:06.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7fc0eaa00640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc0c806c2e0 0x7fc0c806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.934+0000 7f39f9600640 1 -- start start 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.935+0000 7f39f9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc005230 0x7f39ec17c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.935+0000 7f39f9600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39ec17c5c0 0x7f39ec179f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f39f9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39ec17a4c0 0x7f39ec0099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f39f9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec0e31e0 con 0x7f39dc005230 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f39f9600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec0e2f00 con 0x7f39ec17c5c0 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f39f9600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec0e34c0 con 0x7f39ec17a4c0 2024-09-16T01:49:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f724b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72440c6bb0 0x7f724414b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7fc2b0e00640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc28c06c380 0x7fc28c06e840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc29803c000 tx=0x7fc298037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc0d4075f70 con 0x7fc0ec154c90 2024-09-16T01:49:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.936+0000 7f724b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244158bb0 0x7f724415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f7243e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72440c6bb0 0x7f724414b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f7243e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72440c6bb0 0x7f724414b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45128/0 (socket says 172.21.15.146:45128) 2024-09-16T01:49:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f7243e00640 1 -- 172.21.15.146:0/156990164 learned_addr learned my addr 172.21.15.146:0/156990164 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f724b200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72440cca70 0x7f72441fa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7fc0eb400640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc0c80744e0 0x7fc0c8076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f3af3e00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3ad806c2e0 0x7f3ad806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f39f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc005230 0x7f39ec17c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3af408b4d0 con 0x7f3afc162c00 2024-09-16T01:49:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f3af3e00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3ad8074500 0x7f3ad8076920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ad8076fd0 con 0x7f3ad8074500 2024-09-16T01:49:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3af408b8c0 con 0x7f3afc162c00 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f39f3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39ec17c5c0 0x7f39ec179f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f39f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc005230 0x7f39ec17c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59110/0 (socket says 172.21.15.146:59110) 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f39f3e00640 1 -- 172.21.15.146:0/1681085952 learned_addr learned my addr 172.21.15.146:0/1681085952 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39ec17a4c0 0x7f39ec0099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f724415c630 con 0x7f7244158bb0 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f724415c350 con 0x7f72440c6bb0 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.937+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7244070200 con 0x7f72440cca70 2024-09-16T01:49:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f7249200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72440cca70 0x7f72441fa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f3af2a00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3ad8074500 0x7f3ad8076920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f39f3e00640 1 -- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39ec17a4c0 msgr2=0x7f39ec0099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f39f3e00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39ec17a4c0 0x7f39ec0099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f7243e00640 1 -- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72440cca70 msgr2=0x7f72441fa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f7243e00640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72440cca70 0x7f72441fa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f39f3e00640 1 -- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39ec17c5c0 msgr2=0x7f39ec179f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f7243e00640 1 -- 172.21.15.146:0/156990164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244158bb0 msgr2=0x7f724415aad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:49:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f7243e00640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7244158bb0 0x7f724415aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f39f3e00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39ec17c5c0 0x7f39ec179f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f39f3e00640 1 -- 172.21.15.146:0/1681085952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec17a940 con 0x7f39dc005230 2024-09-16T01:49:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7fc0eb400640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc0c80744e0 0x7fc0c8076900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.938+0000 7f7243e00640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724414b880 con 0x7f72440c6bb0 2024-09-16T01:49:06.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f7249200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72440cca70 0x7f72441fa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:06.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f39f3400640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39ec17c5c0 0x7f39ec179f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:06.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f3af2a00640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3ad8074500 0x7f3ad8076920 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f3afb400640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3ad806c2e0 0x7f3ad806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39ec17a4c0 0x7f39ec0099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:06.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7fc0eaa00640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc0c806c2e0 0x7fc0c806e7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc007eb0 tx=0x7fc0dc0073c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f39f3e00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc005230 0x7f39ec17c080 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39f40504d0 tx=0x7f39f4069890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f7243e00640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72440c6bb0 0x7f724414b340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7230009960 tx=0x7f72300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f4099070 con 0x7f39dc005230 2024-09-16T01:49:06.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.940+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ad8076fd0 con 0x7f3ad8074500 2024-09-16T01:49:06.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.940+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f4071810 con 0x7f39dc005230 2024-09-16T01:49:06.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.940+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f4071be0 con 0x7f39dc005230 2024-09-16T01:49:06.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.940+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec054c90 con 0x7f39dc005230 2024-09-16T01:49:06.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.940+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec04bfc0 con 0x7f39dc005230 2024-09-16T01:49:06.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.939+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0c8076fb0 con 0x7fc0c80744e0 2024-09-16T01:49:06.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.943+0000 7f3afb400640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3ad806c2e0 0x7f3ad806e7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3aec033000 tx=0x7f3aec02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.942+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f39ec04bd70 con 0x7f39dc005230 2024-09-16T01:49:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.941+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230037040 con 0x7f72440c6bb0 2024-09-16T01:49:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.941+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7230042070 con 0x7f72440c6bb0 2024-09-16T01:49:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.941+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230030bb0 con 0x7f72440c6bb0 2024-09-16T01:49:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.944+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72441fd310 con 0x7f72440c6bb0 2024-09-16T01:49:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.944+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f67c41f7ae0 con 0x7f67a4074580 2024-09-16T01:49:06.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.945+0000 7f67ba000640 1 -- 172.21.15.146:0/1810081063 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f67c41f7ae0 con 0x7f67a4074580 2024-09-16T01:49:06.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.944+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f39f409c070 con 0x7f39dc005230 2024-09-16T01:49:06.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.944+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f724414a270 con 0x7f72440c6bb0 2024-09-16T01:49:06.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67a4074580 msgr2=0x7f67a40769a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67a4074580 0x7f67a40769a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67a406c300 msgr2=0x7f67a406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67a406c300 0x7f67a406e7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f67b4000f60 tx=0x7f67b4009040 comp rx=0 tx=0).stop 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c40cc0c0 msgr2=0x7f67c415d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 --2- 172.21.15.146:0/1810081063 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67c40cc0c0 0x7f67c415d870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00c730 tx=0x7f67bc00cbf0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c2800640 1 -- 172.21.15.146:0/1810081063 reap_dead start 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 shutdown_connections 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 >> 172.21.15.146:0/1810081063 conn(0x7f67c406d560 msgr2=0x7f67c40cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.945+0000 7f39e7e00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39d406c300 0x7f39d406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.945+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f39f4107fc0 con 0x7f39dc005230 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.946+0000 7f39e7e00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f39d4074580 0x7f39d40769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.946+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39f409fe80 con 0x7f39d4074580 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.946+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f39f4108370 con 0x7f39dc005230 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f39f3400640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39d406c300 0x7f39d406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.947+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f39d4074580 0x7f39d40769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.945+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72440718c0 con 0x7f72440c6bb0 2024-09-16T01:49:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.948+0000 7f39f8c00640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f39d4074580 0x7f39d40769a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.949+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39f409fe80 con 0x7f39d4074580 2024-09-16T01:49:06.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.948+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7230047460 con 0x7f72440c6bb0 2024-09-16T01:49:06.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.949+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 shutdown_connections 2024-09-16T01:49:06.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.949+0000 7f39f3400640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39d406c300 0x7f39d406e7c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0e6310 tx=0x7f39e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.950+0000 7f723be00640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f722406c3b0 0x7f722406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.950+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f72300ac020 con 0x7f72440c6bb0 2024-09-16T01:49:06.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.950+0000 7f7243400640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f722406c3b0 0x7f722406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.951+0000 7f67c9200640 1 -- 172.21.15.146:0/1810081063 wait complete. 2024-09-16T01:49:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.951+0000 7f7243400640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f722406c3b0 0x7f722406e870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72440c7d00 tx=0x7f7234008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.952+0000 7f723be00640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7224074630 0x7f7224076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.952+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7224077100 con 0x7f7224074630 2024-09-16T01:49:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.952+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f72300b3050 con 0x7f72440c6bb0 2024-09-16T01:49:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.952+0000 7f7249200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7224074630 0x7f7224076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.953+0000 7f7249200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7224074630 0x7f7224076a50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.954+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d440c4b70 con 0x7f5d24074520 2024-09-16T01:49:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.954+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7224077100 con 0x7f7224074630 2024-09-16T01:49:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.955+0000 7f5d3a000640 1 -- 172.21.15.146:0/4089260884 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5d440c4b70 con 0x7f5d24074520 2024-09-16T01:49:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.955+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5d24074520 msgr2=0x7f5d24076940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.955+0000 7f5d48a00640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5d24074520 0x7f5d24076940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5d2406c380 msgr2=0x7f5d2406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5d2406c380 0x7f5d2406e840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009560 tx=0x7f5d34009190 comp rx=0 tx=0).stop 2024-09-16T01:49:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d44159320 msgr2=0x7f5d441f8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 --2- 172.21.15.146:0/4089260884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d44159320 0x7f5d441f8970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d44073710 tx=0x7f5d3c004a40 comp rx=0 tx=0).stop 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d42800640 1 -- 172.21.15.146:0/4089260884 reap_dead start 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 shutdown_connections 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 >> 172.21.15.146:0/4089260884 conn(0x7f5d4406f3c0 msgr2=0x7f5d4415ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 shutdown_connections 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f5d48a00640 1 -- 172.21.15.146:0/4089260884 wait complete. 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.959+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f075c0ca110 con 0x7f0738075120 2024-09-16T01:49:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.960+0000 7f0753e00640 1 -- 172.21.15.146:0/1815518993 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f075c0ca110 con 0x7f0738075120 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.961+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0738075120 msgr2=0x7f07380775c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.961+0000 7f0762000640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0738075120 0x7f07380775c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.962+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f073806c300 msgr2=0x7f073806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.962+0000 7f0762000640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f073806c300 0x7f073806e7c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f074c034000 tx=0x7f074c02e000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.962+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f075c0cb2f0 msgr2=0x7f075c0ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.963+0000 7f0762000640 1 --2- 172.21.15.146:0/1815518993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f075c0cb2f0 0x7f075c0ca4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f074400e9e0 tx=0x7f074400eea0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.963+0000 7f075be00640 1 -- 172.21.15.146:0/1815518993 reap_dead start 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.963+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 shutdown_connections 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.963+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 >> 172.21.15.146:0/1815518993 conn(0x7f075c06d560 msgr2=0x7f075c0cee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.966+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 shutdown_connections 2024-09-16T01:49:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.966+0000 7f0762000640 1 -- 172.21.15.146:0/1815518993 wait complete. 2024-09-16T01:49:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.970+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3afc0c9b60 con 0x7f3ad8074500 2024-09-16T01:49:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.970+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc2a404b2e0 con 0x7fc28c0745a0 2024-09-16T01:49:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.971+0000 7fc2a8c00640 1 -- 172.21.15.146:0/2103178836 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc2a404b2e0 con 0x7fc28c0745a0 2024-09-16T01:49:07.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.972+0000 7f3af3e00640 1 -- 172.21.15.146:0/716706638 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3afc0c9b60 con 0x7f3ad8074500 2024-09-16T01:49:07.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.972+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc28c0745a0 msgr2=0x7fc28c0769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.972+0000 7fc2b2200640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc28c0745a0 0x7fc28c0769c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc28c06c380 msgr2=0x7fc28c06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc28c06c380 0x7fc28c06e840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc29803c000 tx=0x7fc298037000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2a404c090 msgr2=0x7fc2a4171390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 --2- 172.21.15.146:0/2103178836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2a404c090 0x7fc2a4171390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac06b730 tx=0x7fc2ac069890 comp rx=0 tx=0).stop 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b1800640 1 -- 172.21.15.146:0/2103178836 reap_dead start 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 shutdown_connections 2024-09-16T01:49:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 >> 172.21.15.146:0/2103178836 conn(0x7fc2a404fc50 msgr2=0x7fc2a4166740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 shutdown_connections 2024-09-16T01:49:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7fc2b2200640 1 -- 172.21.15.146:0/2103178836 wait complete. 2024-09-16T01:49:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3ad8074500 msgr2=0x7f3ad8076920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.973+0000 7f3b02000640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3ad8074500 0x7f3ad8076920 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3ad806c2e0 msgr2=0x7f3ad806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3ad806c2e0 0x7f3ad806e7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3aec033000 tx=0x7f3aec02e000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc162c00 msgr2=0x7f3afc15ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 --2- 172.21.15.146:0/716706638 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3afc162c00 0x7f3afc15ffd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3af400b410 tx=0x7f3af400b8d0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3afbe00640 1 -- 172.21.15.146:0/716706638 reap_dead start 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 shutdown_connections 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 >> 172.21.15.146:0/716706638 conn(0x7f3afc06f3c0 msgr2=0x7f3afc1575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 shutdown_connections 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f3b02000640 1 -- 172.21.15.146:0/716706638 wait complete. 2024-09-16T01:49:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.972+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f724406ea10 con 0x7f7224074630 2024-09-16T01:49:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f723be00640 1 -- 172.21.15.146:0/156990164 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f724406ea10 con 0x7f7224074630 2024-09-16T01:49:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.974+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f39ec037c30 con 0x7f39d4074580 2024-09-16T01:49:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.975+0000 7f39e7e00640 1 -- 172.21.15.146:0/1681085952 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f39ec037c30 con 0x7f39d4074580 2024-09-16T01:49:07.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.975+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7224074630 msgr2=0x7f7224076a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.975+0000 7f724b200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7224074630 0x7f7224076a50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.009 INFO:teuthology.orchestra.run.smithi146.stdout:64424509447 2024-09-16T01:49:07.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f39d4074580 msgr2=0x7f39d40769a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f39d4074580 0x7f39d40769a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39d406c300 msgr2=0x7f39d406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39d406c300 0x7f39d406e7c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0e6310 tx=0x7f39e0009040 comp rx=0 tx=0).stop 2024-09-16T01:49:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc005230 msgr2=0x7f39ec17c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 --2- 172.21.15.146:0/1681085952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc005230 0x7f39ec17c080 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39f40504d0 tx=0x7f39f4069890 comp rx=0 tx=0).stop 2024-09-16T01:49:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f8c00640 1 -- 172.21.15.146:0/1681085952 reap_dead start 2024-09-16T01:49:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc0ec0cc4f0 con 0x7fc0c80744e0 2024-09-16T01:49:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f722406c3b0 msgr2=0x7f722406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f722406c3b0 0x7f722406e870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72440c7d00 tx=0x7f7234008040 comp rx=0 tx=0).stop 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72440c6bb0 msgr2=0x7f724414b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 --2- 172.21.15.146:0/156990164 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72440c6bb0 0x7f724414b340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7230009960 tx=0x7f72300046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f7249200640 1 -- 172.21.15.146:0/156990164 reap_dead start 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 shutdown_connections 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 >> 172.21.15.146:0/156990164 conn(0x7f724406f3c0 msgr2=0x7f7244150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 shutdown_connections 2024-09-16T01:49:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f724b200640 1 -- 172.21.15.146:0/156990164 wait complete. 2024-09-16T01:49:07.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T01:49:07.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 shutdown_connections 2024-09-16T01:49:07.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.976+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 >> 172.21.15.146:0/1681085952 conn(0x7f39ec054450 msgr2=0x7f39ec04e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.977+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 shutdown_connections 2024-09-16T01:49:07.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.977+0000 7f39f9600640 1 -- 172.21.15.146:0/1681085952 wait complete. 2024-09-16T01:49:07.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.977+0000 7fc0e3400640 1 -- 172.21.15.146:0/3370161717 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc0ec0cc4f0 con 0x7fc0c80744e0 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.978+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc0c80744e0 msgr2=0x7fc0c8076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.978+0000 7fc0dbe00640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc0c80744e0 0x7fc0c8076900 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc0c806c2e0 msgr2=0x7fc0c806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0dbe00640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc0c806c2e0 0x7fc0c806e7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc007eb0 tx=0x7fc0dc0073c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ec154c90 msgr2=0x7fc0ec1fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0dbe00640 1 --2- 172.21.15.146:0/3370161717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ec154c90 0x7fc0ec1fcfe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0d402a800 tx=0x7fc0d40046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0eb400640 1 -- 172.21.15.146:0/3370161717 reap_dead start 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 shutdown_connections 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.979+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 >> 172.21.15.146:0/3370161717 conn(0x7fc0ec067950 msgr2=0x7fc0ec158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.982+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 shutdown_connections 2024-09-16T01:49:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:06.982+0000 7fc0dbe00640 1 -- 172.21.15.146:0/3370161717 wait complete. 2024-09-16T01:49:07.015 INFO:teuthology.orchestra.run.smithi146.stdout:64424509448 2024-09-16T01:49:07.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T01:49:07.016 INFO:teuthology.orchestra.run.smithi146.stdout:64424509448 2024-09-16T01:49:07.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T01:49:07.017 INFO:teuthology.orchestra.run.smithi146.stdout:60129542151 2024-09-16T01:49:07.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T01:49:07.018 INFO:teuthology.orchestra.run.smithi146.stdout:64424509448 2024-09-16T01:49:07.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T01:49:07.019 INFO:teuthology.orchestra.run.smithi146.stdout:55834574855 2024-09-16T01:49:07.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T01:49:07.020 INFO:teuthology.orchestra.run.smithi146.stdout:60129542151 2024-09-16T01:49:07.021 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T01:49:07.021 INFO:teuthology.orchestra.run.smithi146.stdout:60129542151 2024-09-16T01:49:07.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T01:49:07.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.326+0000 7fd5b7000640 1 Processor -- start 2024-09-16T01:49:07.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.326+0000 7fd5b7000640 1 -- start start 2024-09-16T01:49:07.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.326+0000 7fe25c000640 1 Processor -- start 2024-09-16T01:49:07.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.326+0000 7fe25c000640 1 -- start start 2024-09-16T01:49:07.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fd5b7000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd5a8007340 0x7fd5a804ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fd5b7000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fd5b7000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a8050520 con 0x7fd5a80d8bb0 2024-09-16T01:49:07.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fd5b7000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a8050800 con 0x7fd5a8007340 2024-09-16T01:49:07.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fd5b7000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a804f070 con 0x7fd5a804f870 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fe25c000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe2540caea0 0x7fe2540cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.328+0000 7fe25c000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 0x7fe254070570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 0x7fe254070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 0x7fe254070570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37630/0 (socket says 172.21.15.146:37630) 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 -- 172.21.15.146:0/3100689608 learned_addr learned my addr 172.21.15.146:0/3100689608 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2540cc040 con 0x7fe254070ab0 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe254162d90 con 0x7fe2540caea0 2024-09-16T01:49:07.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe2540cc6d0 con 0x7fe2540cb280 2024-09-16T01:49:07.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fd5b5200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd5a8007340 0x7fd5a804ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fd5b5200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd5a8007340 0x7fd5a804ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45146/0 (socket says 172.21.15.146:45146) 2024-09-16T01:49:07.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fd5b5200640 1 -- 172.21.15.146:0/4121778846 learned_addr learned my addr 172.21.15.146:0/4121778846 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 -- 172.21.15.146:0/3100689608 >> v2:172.21.15.146:3300/0 conn(0x7fe2540caea0 msgr2=0x7fe2540cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe259600640 1 --2- 172.21.15.146:0/3100689608 >> v2:172.21.15.146:3300/0 conn(0x7fe2540caea0 0x7fe2540cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 --2- 172.21.15.146:0/3100689608 >> v2:172.21.15.146:3300/0 conn(0x7fe2540caea0 0x7fe2540cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 -- 172.21.15.146:0/3100689608 >> v1:172.21.15.161:6789/0 conn(0x7fe254070ab0 legacy=0x7fe254072f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.329+0000 7fe258c00640 1 -- 172.21.15.146:0/3100689608 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe254163220 con 0x7fe2540cb280 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.330+0000 7fe259600640 1 --2- 172.21.15.146:0/3100689608 >> v2:172.21.15.146:3300/0 conn(0x7fe2540caea0 0x7fe2540cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.330+0000 7fd5b5200640 1 -- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 msgr2=0x7fd5a8005d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.330+0000 7fd5b5200640 1 --2- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8005d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.330+0000 7fd5b5200640 1 -- 172.21.15.146:0/4121778846 >> v1:172.21.15.161:6789/0 conn(0x7fd5a80d8bb0 legacy=0x7fd5a800e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.330+0000 7fd5b5200640 1 -- 172.21.15.146:0/4121778846 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a804f350 con 0x7fd5a8007340 2024-09-16T01:49:07.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.331+0000 7fe258c00640 1 --2- 172.21.15.146:0/3100689608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 0x7fe254070570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe248009e50 tx=0x7fe24802a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5c5c964c153bba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.331+0000 7fe253e00640 1 -- 172.21.15.146:0/3100689608 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248036070 con 0x7fe2540cb280 2024-09-16T01:49:07.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.331+0000 7fd5b5200640 1 --2- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a804ea00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4009870 tx=0x7fd5a402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b9633f2d602c861 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.331+0000 7fe253e00640 1 -- 172.21.15.146:0/3100689608 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe248031600 con 0x7fe2540cb280 2024-09-16T01:49:07.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.331+0000 7fe253e00640 1 -- 172.21.15.146:0/3100689608 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2480319d0 con 0x7fe2540cb280 2024-09-16T01:49:07.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.332+0000 7fa8c7800640 1 Processor -- start 2024-09-16T01:49:07.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.332+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 msgr2=0x7fe254070570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.332+0000 7fe25c000640 1 --2- 172.21.15.146:0/3100689608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 0x7fe254070570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe248009e50 tx=0x7fe24802a7a0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.332+0000 7fa8c7800640 1 -- start start 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 shutdown_connections 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fe25c000640 1 --2- 172.21.15.146:0/3100689608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2540cb280 0x7fe254070570 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe248009e50 tx=0x7fe24802a7a0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fe25c000640 1 --2- 172.21.15.146:0/3100689608 >> v2:172.21.15.146:3300/0 conn(0x7fe2540caea0 0x7fe2540cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 >> 172.21.15.146:0/3100689608 conn(0x7fe2540d1b80 msgr2=0x7fe2540d1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fd5afe00640 1 -- 172.21.15.146:0/4121778846 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4037070 con 0x7fd5a8007340 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fd5afe00640 1 -- 172.21.15.146:0/4121778846 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a4032440 con 0x7fd5a8007340 2024-09-16T01:49:07.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fd5afe00640 1 -- 172.21.15.146:0/4121778846 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4032850 con 0x7fd5a8007340 2024-09-16T01:49:07.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 shutdown_connections 2024-09-16T01:49:07.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7f5bb0000640 1 Processor -- start 2024-09-16T01:49:07.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.333+0000 7f5bb0000640 1 -- start start 2024-09-16T01:49:07.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fe25c000640 1 -- 172.21.15.146:0/3100689608 wait complete. 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fd5b7000640 1 -- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 msgr2=0x7fd5a804ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fd5b7000640 1 --2- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a804ea00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4009870 tx=0x7fd5a402a210 comp rx=0 tx=0).stop 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fd5b7000640 1 -- 172.21.15.146:0/4121778846 shutdown_connections 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fd5b7000640 1 --2- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8005d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fd5b7000640 1 --2- 172.21.15.146:0/4121778846 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a804ea00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.334+0000 7fd5b7000640 1 -- 172.21.15.146:0/4121778846 >> 172.21.15.146:0/4121778846 conn(0x7fd5a8054470 msgr2=0x7fd5a8054880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7fd5b7000640 1 -- 172.21.15.146:0/4121778846 shutdown_connections 2024-09-16T01:49:07.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7fd5b7000640 1 -- 172.21.15.146:0/4121778846 wait complete. 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7f5bb0000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ba8159080 0x7f5ba8159460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7f5bb0000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ba815a4d0 0x7f5ba80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7f5bb0000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ba80c72d0 con 0x7f5ba80cdfe0 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7f5bb0000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba80c75b0 con 0x7f5ba815a4d0 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.335+0000 7f5bb0000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5ba80c7890 con 0x7f5ba8159080 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.336+0000 7f5bad600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ba8159080 0x7f5ba8159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.336+0000 7f5bad600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ba8159080 0x7f5ba8159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45158/0 (socket says 172.21.15.146:45158) 2024-09-16T01:49:07.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.336+0000 7f5bad600640 1 -- 172.21.15.146:0/2215862605 learned_addr learned my addr 172.21.15.146:0/2215862605 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.336+0000 7fe25c000640 1 Processor -- start 2024-09-16T01:49:07.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.336+0000 7f5bae000640 1 --2- 172.21.15.146:0/2215862605 >> v2:172.21.15.146:3300/0 conn(0x7f5ba815a4d0 0x7f5ba80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.336+0000 7fe25c000640 1 -- start start 2024-09-16T01:49:07.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7f519be00640 1 Processor -- start 2024-09-16T01:49:07.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7fd5b7000640 1 Processor -- start 2024-09-16T01:49:07.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7fd5b7000640 1 -- start start 2024-09-16T01:49:07.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7f519be00640 1 -- start start 2024-09-16T01:49:07.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7f5bae000640 1 -- 172.21.15.146:0/2215862605 >> v2:172.21.15.161:3300/0 conn(0x7f5ba8159080 msgr2=0x7f5ba8159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7f5bae000640 1 --2- 172.21.15.146:0/2215862605 >> v2:172.21.15.161:3300/0 conn(0x7f5ba8159080 0x7f5ba8159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7f5bae000640 1 -- 172.21.15.146:0/2215862605 >> v1:172.21.15.146:6789/0 conn(0x7f5ba80cdfe0 legacy=0x7f5ba8159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.337+0000 7f5bae000640 1 -- 172.21.15.146:0/2215862605 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba80c7b70 con 0x7f5ba815a4d0 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.338+0000 7fa8c7800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c0158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.338+0000 7fa8c7800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa8c0158bb0 0x7fa8c00c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.338+0000 7fa8c7800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c0072f50 con 0x7fa8c00c6bb0 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.338+0000 7fa8c7800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c0073210 con 0x7fa8c0158bb0 2024-09-16T01:49:07.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.338+0000 7fa8c7800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c015cb50 con 0x7fa8c0152850 2024-09-16T01:49:07.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.338+0000 7f5bae000640 1 --2- 172.21.15.146:0/2215862605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba815a4d0 0x7f5ba80ce400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ba806f250 tx=0x7f5b9402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b837bca1c5b7a72 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f5ba7e00640 1 -- 172.21.15.146:0/2215862605 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94004030 con 0x7f5ba815a4d0 2024-09-16T01:49:07.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f5ba7e00640 1 -- 172.21.15.146:0/2215862605 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b940041d0 con 0x7f5ba815a4d0 2024-09-16T01:49:07.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f5ba7e00640 1 -- 172.21.15.146:0/2215862605 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94003610 con 0x7f5ba815a4d0 2024-09-16T01:49:07.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fe25c000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe254070ab0 0x7fe2541618d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fd5b7000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a80d5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fd5b7000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8053370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fd5b7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a80d8bb0 0x7fd5a8174f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fe25c000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 0x7fe2540c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f519be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f519c06c4f0 0x7f519c0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f519be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519c158630 con 0x7f519c14f1d0 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f519be00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519c073f00 con 0x7f519c0c8ef0 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7f519be00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f519c158f20 con 0x7f519c06c4f0 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fe259600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe254070ab0 0x7fe2541618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fe259600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe254070ab0 0x7fe2541618d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37642/0 (socket says 172.21.15.146:37642) 2024-09-16T01:49:07.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fe25c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe25415f9c0 0x7fe2540c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 -- 172.21.15.146:0/2215862605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba815a4d0 msgr2=0x7f5ba80ce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 --2- 172.21.15.146:0/2215862605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba815a4d0 0x7f5ba80ce400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ba806f250 tx=0x7f5b9402a270 comp rx=0 tx=0).stop 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fe25c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe254162740 con 0x7fe25415f9c0 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fe25c000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe254162460 con 0x7fe2540caea0 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fe25c000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe254162a20 con 0x7fe254070ab0 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fe258c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 0x7fe2540c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fe258c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 0x7fe2540c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45180/0 (socket says 172.21.15.146:45180) 2024-09-16T01:49:07.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fe258c00640 1 -- 172.21.15.146:0/2126854979 learned_addr learned my addr 172.21.15.146:0/2126854979 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fd5b7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8049340 con 0x7fd5a80d8bb0 2024-09-16T01:49:07.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 -- 172.21.15.146:0/2215862605 shutdown_connections 2024-09-16T01:49:07.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 --2- 172.21.15.146:0/2215862605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba815a4d0 0x7f5ba80ce400 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 --2- 172.21.15.146:0/2215862605 >> v2:172.21.15.161:3300/0 conn(0x7f5ba8159080 0x7f5ba8159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 -- 172.21.15.146:0/2215862605 >> 172.21.15.146:0/2215862605 conn(0x7f5ba8067950 msgr2=0x7f5ba8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f519b400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f519c0c8ef0 0x7f519c162ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52720/0 (socket says 172.21.15.146:52720) 2024-09-16T01:49:07.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f519b400640 1 -- 172.21.15.146:0/2002332420 learned_addr learned my addr 172.21.15.146:0/2002332420 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:07.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f519a000640 1 -- 172.21.15.146:0/2002332420 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779528624 0 0) 0x7f519c073f00 con 0x7f519c0c8ef0 2024-09-16T01:49:07.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f519a000640 1 -- 172.21.15.146:0/2002332420 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5180003520 con 0x7f519c0c8ef0 2024-09-16T01:49:07.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5192000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f519c06c4f0 0x7f519c0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fd5b7000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a80491c0 con 0x7fd5a8007340 2024-09-16T01:49:07.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7fd5b7000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a800eae0 con 0x7fd5a804f870 2024-09-16T01:49:07.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.340+0000 7f5bb0000640 1 -- 172.21.15.146:0/2215862605 shutdown_connections 2024-09-16T01:49:07.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a80d5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7f5bb0000640 1 -- 172.21.15.146:0/2215862605 wait complete. 2024-09-16T01:49:07.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fe258c00640 1 -- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe254070ab0 msgr2=0x7fe2541618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fe258c00640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe254070ab0 0x7fe2541618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8053370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fe258c00640 1 -- 172.21.15.146:0/2126854979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe25415f9c0 msgr2=0x7fe2540c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fe258c00640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe25415f9c0 0x7fe2540c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fe258c00640 1 -- 172.21.15.146:0/2126854979 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2541f5990 con 0x7fe2540caea0 2024-09-16T01:49:07.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8053370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37646/0 (socket says 172.21.15.146:37646) 2024-09-16T01:49:07.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5200640 1 -- 172.21.15.146:0/3515095082 learned_addr learned my addr 172.21.15.146:0/3515095082 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b6600640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a80d8bb0 0x7fd5a8174f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fa8bfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa8c0158bb0 0x7fa8c00c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fa8bfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa8c0158bb0 0x7fa8c00c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45162/0 (socket says 172.21.15.146:45162) 2024-09-16T01:49:07.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.339+0000 7fa8bfe00640 1 -- 172.21.15.146:0/3549108060 learned_addr learned my addr 172.21.15.146:0/3549108060 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fa8c4e00640 1 --2- 172.21.15.146:0/3549108060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c0158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a80d5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45164/0 (socket says 172.21.15.146:45164) 2024-09-16T01:49:07.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7f5192000640 1 -- 172.21.15.146:0/2002332420 >> v1:172.21.15.161:6789/0 conn(0x7f519c0c8ef0 legacy=0x7f519c162ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7f5192000640 1 -- 172.21.15.146:0/2002332420 >> v1:172.21.15.146:6789/0 conn(0x7f519c14f1d0 legacy=0x7f519c15bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5c00640 1 -- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 msgr2=0x7fd5a8053370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5c00640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8053370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fd5b5c00640 1 -- 172.21.15.146:0/3515095082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a80d8bb0 msgr2=0x7fd5a8174f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.342+0000 7fd5b5c00640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a80d8bb0 0x7fd5a8174f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.342+0000 7fd5b5c00640 1 -- 172.21.15.146:0/3515095082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a80d5f10 con 0x7fd5a8007340 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.342+0000 7fd5b6600640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a80d8bb0 0x7fd5a8174f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fa8c4e00640 1 -- 172.21.15.146:0/3549108060 >> v2:172.21.15.161:3300/0 conn(0x7fa8c0158bb0 msgr2=0x7fa8c00c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fa8c4e00640 1 --2- 172.21.15.146:0/3549108060 >> v2:172.21.15.161:3300/0 conn(0x7fa8c0158bb0 0x7fa8c00c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.342+0000 7fd5b5c00640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a80d5150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00504d0 tx=0x7fd5b0069890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fa8c4e00640 1 -- 172.21.15.146:0/3549108060 >> v1:172.21.15.161:6789/0 conn(0x7fa8c00c6bb0 legacy=0x7fa8c00c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7fa8c4e00640 1 -- 172.21.15.146:0/3549108060 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c015bd90 con 0x7fa8c0152850 2024-09-16T01:49:07.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7fe258c00640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 0x7fe2540c74d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe248000f80 tx=0x7fe248004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.341+0000 7f5192000640 1 -- 172.21.15.146:0/2002332420 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c0c2fd0 con 0x7f519c06c4f0 2024-09-16T01:49:07.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7f5bb0000640 1 Processor -- start 2024-09-16T01:49:07.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7fa8c4e00640 1 --2- 172.21.15.146:0/3549108060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c0158670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400dbc0 tx=0x7fa8b402e460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdccc0bbc3d30c49 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7f5192000640 1 --2- 172.21.15.146:0/2002332420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c0c89b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f518c0097a0 tx=0x7f518c02a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12a7cadec1511640 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7f59c1400640 1 Processor -- start 2024-09-16T01:49:07.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7f5bb0000640 1 -- start start 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fa8bf400640 1 -- 172.21.15.146:0/3549108060 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b403a070 con 0x7fa8c0152850 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fa8bf400640 1 -- 172.21.15.146:0/3549108060 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b400b650 con 0x7fa8c0152850 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fa8bf400640 1 -- 172.21.15.146:0/3549108060 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400ba60 con 0x7fa8c0152850 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0091070 con 0x7fd5a8007340 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b0071840 con 0x7fd5a8007340 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0071c90 con 0x7fd5a8007340 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a80d4e70 con 0x7fd5a8007340 2024-09-16T01:49:07.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a8006830 con 0x7fd5a8007340 2024-09-16T01:49:07.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248041020 con 0x7fe2540caea0 2024-09-16T01:49:07.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.343+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe248008710 con 0x7fe2540caea0 2024-09-16T01:49:07.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248036040 con 0x7fe2540caea0 2024-09-16T01:49:07.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2541f4e10 con 0x7fe2540caea0 2024-09-16T01:49:07.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2541f53d0 con 0x7fe2540caea0 2024-09-16T01:49:07.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 -- 172.21.15.146:0/3549108060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 msgr2=0x7fa8c0158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 --2- 172.21.15.146:0/3549108060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c0158670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400dbc0 tx=0x7fa8b402e460 comp rx=0 tx=0).stop 2024-09-16T01:49:07.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.344+0000 7f59c1400640 1 -- start start 2024-09-16T01:49:07.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7f519a000640 1 -- 172.21.15.146:0/2002332420 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c037070 con 0x7f519c06c4f0 2024-09-16T01:49:07.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 -- 172.21.15.146:0/3549108060 shutdown_connections 2024-09-16T01:49:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 --2- 172.21.15.146:0/3549108060 >> v2:172.21.15.161:3300/0 conn(0x7fa8c0158bb0 0x7fa8c00c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 --2- 172.21.15.146:0/3549108060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c0158670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 -- 172.21.15.146:0/3549108060 >> 172.21.15.146:0/3549108060 conn(0x7fa8c006f3c0 msgr2=0x7fa8c006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7f519a000640 1 -- 172.21.15.146:0/2002332420 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c032440 con 0x7f519c06c4f0 2024-09-16T01:49:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7f519a000640 1 -- 172.21.15.146:0/2002332420 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c032850 con 0x7f519c06c4f0 2024-09-16T01:49:07.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a8038300 con 0x7fd5a8007340 2024-09-16T01:49:07.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7f038b200640 1 Processor -- start 2024-09-16T01:49:07.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7f5bb0000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ba80cdfe0 0x7f5ba806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f5bb0000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 0x7f5ba806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2540cb920 con 0x7fe2540caea0 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f5bb0000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ba815a4d0 0x7f5ba8151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f5bacc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 0x7f5ba806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f5bacc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 0x7f5ba806d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59158/0 (socket says 172.21.15.146:59158) 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f5bacc00640 1 -- 172.21.15.146:0/123091009 learned_addr learned my addr 172.21.15.146:0/123091009 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f5bae000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ba815a4d0 0x7f5ba8151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba806f3e0 con 0x7f5ba8159080 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba80c8130 con 0x7f5ba80cdfe0 2024-09-16T01:49:07.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f519be00640 1 -- 172.21.15.146:0/2002332420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 msgr2=0x7f519c0c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f519be00640 1 --2- 172.21.15.146:0/2002332420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c0c89b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f518c0097a0 tx=0x7f518c02a250 comp rx=0 tx=0).stop 2024-09-16T01:49:07.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f519be00640 1 -- 172.21.15.146:0/2002332420 shutdown_connections 2024-09-16T01:49:07.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f519be00640 1 --2- 172.21.15.146:0/2002332420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c0c89b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f59c1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 0x7f59bc150b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f59c1400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f59bc0c70e0 0x7f59bc072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f519be00640 1 -- 172.21.15.146:0/2002332420 >> 172.21.15.146:0/2002332420 conn(0x7f519c067950 msgr2=0x7f519c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7f038b200640 1 -- start start 2024-09-16T01:49:07.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba806f560 con 0x7f5ba815a4d0 2024-09-16T01:49:07.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59c1400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59bc1569d0 con 0x7f59bc072dd0 2024-09-16T01:49:07.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f038b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0384158ea0 0x7f03840c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f038b200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f038415f670 0x7f03840c1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f519be00640 1 -- 172.21.15.146:0/2002332420 shutdown_connections 2024-09-16T01:49:07.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f519be00640 1 -- 172.21.15.146:0/2002332420 wait complete. 2024-09-16T01:49:07.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f038b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f038415fb60 con 0x7f0384073e90 2024-09-16T01:49:07.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f038b200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f03840c6b60 con 0x7f038415f670 2024-09-16T01:49:07.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f038b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f038406cda0 con 0x7f0384158ea0 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.345+0000 7fa8c7800640 1 -- 172.21.15.146:0/3549108060 shutdown_connections 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.346+0000 7fa8c7800640 1 -- 172.21.15.146:0/3549108060 wait complete. 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd5b0094070 con 0x7fd5a8007340 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59c1400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f59bc157580 con 0x7f59bc0c70e0 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59c1400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f59bc071ce0 con 0x7f59bc06fee0 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59ba800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 0x7f59bc150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59ba800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 0x7f59bc150b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37658/0 (socket says 172.21.15.146:37658) 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59ba800640 1 -- 172.21.15.146:0/172855371 learned_addr learned my addr 172.21.15.146:0/172855371 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59b9e00640 1 --2- 172.21.15.146:0/172855371 >> v2:172.21.15.146:3300/0 conn(0x7f59bc0c70e0 0x7f59bc072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.347+0000 7f59ba800640 1 -- 172.21.15.146:0/172855371 >> v2:172.21.15.146:3300/0 conn(0x7f59bc0c70e0 msgr2=0x7f59bc072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f0389400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f038415f670 0x7f03840c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f0389400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f038415f670 0x7f03840c1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45206/0 (socket says 172.21.15.146:45206) 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f0389400640 1 -- 172.21.15.146:0/4140817825 learned_addr learned my addr 172.21.15.146:0/4140817825 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f0389e00640 1 --2- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0384158ea0 0x7f03840c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f59ba800640 1 --2- 172.21.15.146:0/172855371 >> v2:172.21.15.146:3300/0 conn(0x7f59bc0c70e0 0x7f59bc072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f59ba800640 1 -- 172.21.15.146:0/172855371 >> v1:172.21.15.161:6789/0 conn(0x7f59bc072dd0 legacy=0x7f59bc06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f59ba800640 1 -- 172.21.15.146:0/172855371 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59bc156c50 con 0x7f59bc06fee0 2024-09-16T01:49:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7fe3a7200640 1 Processor -- start 2024-09-16T01:49:07.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7fa8c7800640 1 Processor -- start 2024-09-16T01:49:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7fd5ad600640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd59006c2b0 0x7fd59006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd5b0107760 con 0x7fd5a8007340 2024-09-16T01:49:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe248044020 con 0x7fe2540caea0 2024-09-16T01:49:07.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7fd5b5200640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd59006c2b0 0x7fd59006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f5bad600640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ba80cdfe0 0x7f5ba806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f5bae000640 1 -- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ba80cdfe0 msgr2=0x7f5ba806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f5bae000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ba80cdfe0 0x7f5ba806df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f5bae000640 1 -- 172.21.15.146:0/123091009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 msgr2=0x7f5ba806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f5bae000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 0x7f5ba806d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.348+0000 7f5bae000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba81f54c0 con 0x7f5ba815a4d0 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f5bad600640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ba80cdfe0 0x7f5ba806df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f5bacc00640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 0x7f5ba806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:07.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7fe3a7200640 1 -- start start 2024-09-16T01:49:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f519be00640 1 Processor -- start 2024-09-16T01:49:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f0389400640 1 -- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0384158ea0 msgr2=0x7f03840c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f0389400640 1 --2- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0384158ea0 0x7f03840c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f0389400640 1 -- 172.21.15.146:0/4140817825 >> v1:172.21.15.146:6789/0 conn(0x7f0384073e90 legacy=0x7f0384162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f0389400640 1 -- 172.21.15.146:0/4140817825 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038406c820 con 0x7f038415f670 2024-09-16T01:49:07.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f0388a00640 1 -- 172.21.15.146:0/4140817825 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311046264 0 0) 0x7f038415fb60 con 0x7f0384073e90 2024-09-16T01:49:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f519be00640 1 -- start start 2024-09-16T01:49:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7fa8c7800640 1 -- start start 2024-09-16T01:49:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe251600640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe23406c3b0 0x7fe23406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe2480ac640 con 0x7fe2540caea0 2024-09-16T01:49:07.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7f0389400640 1 --2- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f038415f670 0x7f03840c1580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0374009870 tx=0x7f037402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2622b02b462d97d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fd5b5200640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd59006c2b0 0x7fd59006e770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40090b0 tx=0x7fd5a402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7f5bae000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ba815a4d0 0x7f5ba8151c00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80cbbf0 tx=0x7f5b940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe259600640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe23406c3b0 0x7fe23406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.349+0000 7f59b9e00640 1 --2- 172.21.15.146:0/172855371 >> v2:172.21.15.146:3300/0 conn(0x7f59bc0c70e0 0x7f59bc072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f59ba800640 1 --2- 172.21.15.146:0/172855371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 0x7f59bc150b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009870 tx=0x7f59ac02a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a7194b8c3f10bd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f59b9400640 1 -- 172.21.15.146:0/172855371 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac037070 con 0x7f59bc06fee0 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f59b9400640 1 -- 172.21.15.146:0/172855371 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ac032640 con 0x7f59bc06fee0 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.350+0000 7f59b9400640 1 -- 172.21.15.146:0/172855371 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac032a50 con 0x7f59bc06fee0 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7f59c1400640 1 -- 172.21.15.146:0/172855371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 msgr2=0x7f59bc150b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7f59c1400640 1 --2- 172.21.15.146:0/172855371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 0x7f59bc150b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009870 tx=0x7f59ac02a330 comp rx=0 tx=0).stop 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe3a7200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a8150e30 0x7fe3a80c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe3a7200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 0x7fe3a80c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe3a7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3a80c79e0 con 0x7fe3a8150a50 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe3a7200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a80c7290 con 0x7fe3a80c6990 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fe3a7200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a80c1af0 con 0x7fe3a8150e30 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a6800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 0x7fe3a80c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a8150e30 0x7fe3a80c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a6800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 0x7fe3a80c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45212/0 (socket says 172.21.15.146:45212) 2024-09-16T01:49:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a6800640 1 -- 172.21.15.146:0/2058808377 learned_addr learned my addr 172.21.15.146:0/2058808377 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe259600640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe23406c3b0 0x7fe23406e870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe24000aac0 tx=0x7fe240008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f519be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c1501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f59c1400640 1 -- 172.21.15.146:0/172855371 shutdown_connections 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f59c1400640 1 --2- 172.21.15.146:0/172855371 >> v2:172.21.15.146:3300/0 conn(0x7f59bc0c70e0 0x7f59bc072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f59c1400640 1 --2- 172.21.15.146:0/172855371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc06fee0 0x7f59bc150b80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f59c1400640 1 -- 172.21.15.146:0/172855371 >> 172.21.15.146:0/172855371 conn(0x7f59bc06f3c0 msgr2=0x7f59bc06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a5400640 1 -- 172.21.15.146:0/2058808377 >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 msgr2=0x7fe3a80c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a5400640 1 --2- 172.21.15.146:0/2058808377 >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 0x7fe3a80c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fe3a5400640 1 -- 172.21.15.146:0/2058808377 >> v1:172.21.15.146:6789/0 conn(0x7fe3a8150a50 legacy=0x7fe3a81568b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:49:07.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f519be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 0x7f519c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd5a80d5f10 con 0x7fd5a8007340 2024-09-16T01:49:07.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fa8c7800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8c00c6bb0 0x7fa8c00c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.351+0000 7fa8c7800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c00c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f519be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f519c14f1d0 0x7f519c150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f59c1400640 1 -- 172.21.15.146:0/172855371 shutdown_connections 2024-09-16T01:49:07.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.352+0000 7f59c1400640 1 -- 172.21.15.146:0/172855371 wait complete. 2024-09-16T01:49:07.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5192000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 0x7f519c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5192000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 0x7f519c0cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37698/0 (socket says 172.21.15.146:37698) 2024-09-16T01:49:07.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5192000640 1 -- 172.21.15.146:0/190792390 learned_addr learned my addr 172.21.15.146:0/190792390 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1588d0 con 0x7f519c14f1d0 2024-09-16T01:49:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f519c0c9300 con 0x7f519c06c4f0 2024-09-16T01:49:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f519c158bb0 con 0x7f519c0c8ef0 2024-09-16T01:49:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f519aa00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c1501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94042030 con 0x7f5ba815a4d0 2024-09-16T01:49:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b9403e680 con 0x7f5ba815a4d0 2024-09-16T01:49:07.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94004030 con 0x7f5ba815a4d0 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fe3a5400640 1 -- 172.21.15.146:0/2058808377 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a80c2590 con 0x7fe3a8150e30 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fe3a6800640 1 --2- 172.21.15.146:0/2058808377 >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 0x7fe3a80c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8c7800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c0158bb0 0x7fa8c014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8c7800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c015c630 con 0x7fa8c00c6bb0 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8c7800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c015c350 con 0x7fa8c0158bb0 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8c7800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c0070200 con 0x7fa8c0152850 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8c4e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8c00c6bb0 0x7fa8c00c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8bfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c00c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8bfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c00c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37710/0 (socket says 172.21.15.146:37710) 2024-09-16T01:49:07.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba806ce50 con 0x7f5ba815a4d0 2024-09-16T01:49:07.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba806d0f0 con 0x7f5ba815a4d0 2024-09-16T01:49:07.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7fa8bfe00640 1 -- 172.21.15.146:0/4063150877 learned_addr learned my addr 172.21.15.146:0/4063150877 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7f519aa00640 1 -- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 msgr2=0x7f519c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7f519aa00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 0x7f519c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7f519aa00640 1 -- 172.21.15.146:0/190792390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f519c14f1d0 msgr2=0x7f519c150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7fa8bfe00640 1 -- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c0158bb0 msgr2=0x7fa8c014b7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7fa8bfe00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c0158bb0 0x7fa8c014b7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7fa8bfe00640 1 -- 172.21.15.146:0/4063150877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8c00c6bb0 msgr2=0x7fa8c00c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7fa8bfe00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8c00c6bb0 0x7fa8c00c4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7fa8bfe00640 1 -- 172.21.15.146:0/4063150877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c00ccbd0 con 0x7fa8c0152850 2024-09-16T01:49:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.353+0000 7f519b400640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f519c14f1d0 0x7f519c150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7f519aa00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f519c14f1d0 0x7f519c150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f519aa00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c150880 con 0x7f519c06c4f0 2024-09-16T01:49:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f5192000640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 0x7f519c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f519b400640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f519c14f1d0 0x7f519c150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:07.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f59c1400640 1 Processor -- start 2024-09-16T01:49:07.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.354+0000 7fa8c4e00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8c00c6bb0 0x7fa8c00c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:07.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7fe3a5400640 1 --2- 172.21.15.146:0/2058808377 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a8150e30 0x7fe3a80c8ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe39400ba40 tx=0x7fe39402c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2168e19a53c3aa15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba8159460 con 0x7f5ba815a4d0 2024-09-16T01:49:07.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f519aa00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c1501f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f518400c3a0 tx=0x7f518400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7fa8bfe00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c00c4c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00e870 tx=0x7fa8ac00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7fe3a4a00640 1 -- 172.21.15.146:0/2058808377 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39403a070 con 0x7fe3a8150e30 2024-09-16T01:49:07.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7fe3a4a00640 1 -- 172.21.15.146:0/2058808377 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe394032480 con 0x7fe3a8150e30 2024-09-16T01:49:07.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7fe3a4a00640 1 -- 172.21.15.146:0/2058808377 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394032850 con 0x7fe3a8150e30 2024-09-16T01:49:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.355+0000 7f59c1400640 1 -- start start 2024-09-16T01:49:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f0388a00640 1 -- 172.21.15.146:0/4140817825 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374037070 con 0x7f038415f670 2024-09-16T01:49:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f0388a00640 1 -- 172.21.15.146:0/4140817825 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0374032440 con 0x7f038415f670 2024-09-16T01:49:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f0388a00640 1 -- 172.21.15.146:0/4140817825 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374032810 con 0x7f038415f670 2024-09-16T01:49:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac017070 con 0x7fa8c0152850 2024-09-16T01:49:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8ac00c940 con 0x7fa8c0152850 2024-09-16T01:49:07.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00cd10 con 0x7fa8c0152850 2024-09-16T01:49:07.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51840052d0 con 0x7f519c06c4f0 2024-09-16T01:49:07.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5184005490 con 0x7f519c06c4f0 2024-09-16T01:49:07.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5184007260 con 0x7f519c06c4f0 2024-09-16T01:49:07.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c00cce90 con 0x7fa8c0152850 2024-09-16T01:49:07.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c01532b0 con 0x7fa8c0152850 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f59c1400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59bc06fee0 0x7f59bc140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f59c1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc072dd0 0x7f59bc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f59c1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59bc0c70e0 0x7f59bc1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f59c1400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc0c9a90 con 0x7f59bc0c70e0 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f59c1400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc157210 con 0x7f59bc06fee0 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.356+0000 7f59c1400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f59bc0718c0 con 0x7f59bc072dd0 2024-09-16T01:49:07.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe24807ae20 con 0x7fe2540caea0 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519c151f70 con 0x7f519c06c4f0 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59bb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59bc0c70e0 0x7f59bc1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59ba800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59bc06fee0 0x7f59bc140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59bb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59bc0c70e0 0x7f59bc1597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59184/0 (socket says 172.21.15.146:59184) 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59bb200640 1 -- 172.21.15.146:0/1067619575 learned_addr learned my addr 172.21.15.146:0/1067619575 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59ba800640 1 -- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc072dd0 msgr2=0x7f59bc0c49a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59ba800640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc072dd0 0x7f59bc0c49a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59ba800640 1 -- 172.21.15.146:0/1067619575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59bc0c70e0 msgr2=0x7f59bc1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59ba800640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59bc0c70e0 0x7f59bc1597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f59ba800640 1 -- 172.21.15.146:0/1067619575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59bc140b90 con 0x7f59bc06fee0 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7f59ba800640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59bc06fee0 0x7f59bc140310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac031000 tx=0x7f59ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 -- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f038415f670 msgr2=0x7f03840c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 --2- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f038415f670 0x7f03840c1580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0374009870 tx=0x7f037402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 -- 172.21.15.146:0/4140817825 shutdown_connections 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 --2- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f038415f670 0x7f03840c1580 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 --2- 172.21.15.146:0/4140817825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0384158ea0 0x7f03840c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 -- 172.21.15.146:0/4140817825 >> 172.21.15.146:0/4140817825 conn(0x7f0384067950 msgr2=0x7f0384067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 -- 172.21.15.146:0/4140817825 shutdown_connections 2024-09-16T01:49:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.357+0000 7f038b200640 1 -- 172.21.15.146:0/4140817825 wait complete. 2024-09-16T01:49:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7fe3a7200640 1 -- 172.21.15.146:0/2058808377 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a8150e30 msgr2=0x7fe3a80c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7fe3a7200640 1 --2- 172.21.15.146:0/2058808377 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a8150e30 0x7fe3a80c8ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe39400ba40 tx=0x7fe39402c550 comp rx=0 tx=0).stop 2024-09-16T01:49:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519c152530 con 0x7f519c06c4f0 2024-09-16T01:49:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5b94045070 con 0x7f5ba815a4d0 2024-09-16T01:49:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7fe3a7200640 1 -- 172.21.15.146:0/2058808377 shutdown_connections 2024-09-16T01:49:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7fe3a7200640 1 --2- 172.21.15.146:0/2058808377 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a8150e30 0x7fe3a80c8ee0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7fe3a7200640 1 --2- 172.21.15.146:0/2058808377 >> v2:172.21.15.161:3300/0 conn(0x7fe3a80c6990 0x7fe3a80c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7fe3a7200640 1 -- 172.21.15.146:0/2058808377 >> 172.21.15.146:0/2058808377 conn(0x7fe3a8067950 msgr2=0x7fe3a8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7fe3a7200640 1 -- 172.21.15.146:0/2058808377 shutdown_connections 2024-09-16T01:49:07.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7f5ba5600640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b8806c300 0x7f5b8806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac037040 con 0x7f59bc06fee0 2024-09-16T01:49:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ac032440 con 0x7f59bc06fee0 2024-09-16T01:49:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.358+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac030c90 con 0x7f59bc06fee0 2024-09-16T01:49:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.360+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59bc0cbee0 con 0x7f59bc06fee0 2024-09-16T01:49:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7f038b200640 1 Processor -- start 2024-09-16T01:49:07.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7f038b200640 1 -- start start 2024-09-16T01:49:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa8ac01a070 con 0x7fa8c0152850 2024-09-16T01:49:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa89c005290 con 0x7fa8c0152850 2024-09-16T01:49:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.360+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5b940ac1b0 con 0x7f5ba815a4d0 2024-09-16T01:49:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f5bad600640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b8806c300 0x7f5b8806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f5bad600640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b8806c300 0x7f5b8806e7c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009ac0 tx=0x7f5b9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.359+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f519c0741c0 con 0x7f519c06c4f0 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.360+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59bc0cae40 con 0x7f59bc06fee0 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59bc0cd2a0 con 0x7f59bc06fee0 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.360+0000 7f038b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 0x7f03841572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f038b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0384158ea0 0x7f03841559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f038b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f038415f670 0x7f0384153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f038b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03840c7360 con 0x7f0384073e90 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f038b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f03840c7080 con 0x7f0384158ea0 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.361+0000 7f038b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0384159280 con 0x7f038415f670 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0384158ea0 0x7f03841559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 0x7f03841572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 0x7f03841572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59200/0 (socket says 172.21.15.146:59200) 2024-09-16T01:49:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 -- 172.21.15.146:0/1118462018 learned_addr learned my addr 172.21.15.146:0/1118462018 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.360+0000 7fe3a7200640 1 -- 172.21.15.146:0/2058808377 wait complete. 2024-09-16T01:49:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7fe3a7200640 1 Processor -- start 2024-09-16T01:49:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7fe3a7200640 1 -- start start 2024-09-16T01:49:07.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5184007400 con 0x7f519c06c4f0 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f038a800640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f038415f670 0x7f0384153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 -- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f038415f670 msgr2=0x7f0384153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f038415f670 0x7f0384153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 -- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0384158ea0 msgr2=0x7f03841559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0384158ea0 0x7f03841559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389e00640 1 -- 172.21.15.146:0/1118462018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0384156360 con 0x7f0384073e90 2024-09-16T01:49:07.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.362+0000 7f0389400640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0384158ea0 0x7f03841559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f0389e00640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 0x7f03841572c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f037801e840 tx=0x7f037801ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037801f040 con 0x7f0384073e90 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037802a070 con 0x7f0384073e90 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037800b8e0 con 0x7f0384073e90 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03841f8540 con 0x7f0384073e90 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03841551e0 con 0x7f0384073e90 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.363+0000 7fa8bcc00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa8a006c300 0x7fa8a006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.364+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa8ac08b170 con 0x7fa8c0152850 2024-09-16T01:49:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.364+0000 7fe3a7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3a80c6990 0x7fe3a814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.364+0000 7f5193400640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f517806c380 0x7f517806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a7200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3a8150a50 0x7fe3a80c5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f518408a7d0 con 0x7f519c06c4f0 2024-09-16T01:49:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a7200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 0x7fe3a81ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7f5192000640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f517806c380 0x7f517806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a80c2070 con 0x7fe3a80c6990 2024-09-16T01:49:07.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03841f7570 con 0x7f0384073e90 2024-09-16T01:49:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a6800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 0x7fe3a81ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a6800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 0x7fe3a81ec1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37740/0 (socket says 172.21.15.146:37740) 2024-09-16T01:49:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a6800640 1 -- 172.21.15.146:0/1517924343 learned_addr learned my addr 172.21.15.146:0/1517924343 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a5400640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3a8150a50 0x7fe3a80c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a80c1d90 con 0x7fe3a8150a50 2024-09-16T01:49:07.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5b9407aa20 con 0x7f5ba815a4d0 2024-09-16T01:49:07.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a80c7bf0 con 0x7fe3a814b910 2024-09-16T01:49:07.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.365+0000 7fa8c4e00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa8a006c300 0x7fa8a006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.366+0000 7fa8c4e00640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa8a006c300 0x7fa8a006e7c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4037000 tx=0x7fa8b400d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.366+0000 7fe3a5400640 1 -- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 msgr2=0x7fe3a81ec1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.366+0000 7fe3a5400640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 0x7fe3a81ec1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.367+0000 7fe3a5400640 1 -- 172.21.15.146:0/1517924343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3a80c6990 msgr2=0x7fe3a814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.367+0000 7fe3a5400640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3a80c6990 0x7fe3a814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.367+0000 7fe3a5400640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a8156b90 con 0x7fe3a8150a50 2024-09-16T01:49:07.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.367+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5184058960 con 0x7f519c06c4f0 2024-09-16T01:49:07.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.367+0000 7fe3a6800640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 0x7fe3a81ec1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:07.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.366+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f037800ba80 con 0x7f0384073e90 2024-09-16T01:49:07.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.367+0000 7f037e000640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f036806c2b0 0x7f036806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.368+0000 7f5192000640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f517806c380 0x7f517806e840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f519c15c520 tx=0x7f518c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.368+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f037809c7e0 con 0x7f0384073e90 2024-09-16T01:49:07.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.368+0000 7fe3a5400640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3a8150a50 0x7fe3a80c5790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80c1c70 tx=0x7fe3940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.370+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39403a070 con 0x7fe3a8150a50 2024-09-16T01:49:07.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.370+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa8ac059300 con 0x7fa8c0152850 2024-09-16T01:49:07.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.370+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe39403b040 con 0x7fe3a8150a50 2024-09-16T01:49:07.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.370+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394035820 con 0x7fe3a8150a50 2024-09-16T01:49:07.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.370+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a81ef8d0 con 0x7fe3a8150a50 2024-09-16T01:49:07.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.370+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a81efe90 con 0x7fe3a8150a50 2024-09-16T01:49:07.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.368+0000 7f0389400640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f036806c2b0 0x7f036806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.369+0000 7f0389400640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f036806c2b0 0x7f036806e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0374034000 tx=0x7f037402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.373+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a80c7680 con 0x7fe3a8150a50 2024-09-16T01:49:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.375+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe394046070 con 0x7fe3a8150a50 2024-09-16T01:49:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.369+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f59ac044020 con 0x7f59bc06fee0 2024-09-16T01:49:07.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.371+0000 7f59b2a00640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f599c06c300 0x7f599c06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.376+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f59ac0325f0 con 0x7f59bc06fee0 2024-09-16T01:49:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.376+0000 7f59b9e00640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f599c06c300 0x7f599c06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.376+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f59ac04b020 con 0x7f59bc06fee0 2024-09-16T01:49:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.377+0000 7fe39e000640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3880706f0 0x7fe388072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:07.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.377+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f03841f7570 con 0x7f0384073e90 2024-09-16T01:49:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.377+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe3940ab280 con 0x7fe3a8150a50 2024-09-16T01:49:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.377+0000 7fe3a5e00640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3880706f0 0x7fe388072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.381+0000 7f59b9e00640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f599c06c300 0x7f599c06e7c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59bc149130 tx=0x7f59a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.381+0000 7fe3a5e00640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3880706f0 0x7fe388072bb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00043c0 tx=0x7fe3a0007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:07.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.382+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe3940799e0 con 0x7fe3a8150a50 2024-09-16T01:49:07.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.756+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd5a8038020 con 0x7fd5a8007340 2024-09-16T01:49:07.761 INFO:teuthology.orchestra.run.smithi146.stdout:64424509445 2024-09-16T01:49:07.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.757+0000 7fd5ad600640 1 -- 172.21.15.146:0/3515095082 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd5b00d58f0 con 0x7fd5a8007340 2024-09-16T01:49:07.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.761+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd59006c2b0 msgr2=0x7fd59006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.761+0000 7fd5b7000640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd59006c2b0 0x7fd59006e770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40090b0 tx=0x7fd5a402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.761+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 msgr2=0x7fd5a80d5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.761+0000 7fd5b7000640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a80d5150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00504d0 tx=0x7fd5b0069890 comp rx=0 tx=0).stop 2024-09-16T01:49:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.762+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 shutdown_connections 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.762+0000 7fd5b7000640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd59006c2b0 0x7fd59006e770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.762+0000 7fd5b7000640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5a80d8bb0 0x7fd5a8174f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.762+0000 7fd5b7000640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5a804f870 0x7fd5a8053370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.762+0000 7fd5b7000640 1 --2- 172.21.15.146:0/3515095082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5a8007340 0x7fd5a80d5150 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.762+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 >> 172.21.15.146:0/3515095082 conn(0x7fd5a8054470 msgr2=0x7fd5a8004a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.763+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 shutdown_connections 2024-09-16T01:49:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.763+0000 7fd5b7000640 1 -- 172.21.15.146:0/3515095082 wait complete. 2024-09-16T01:49:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.774+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe25406cb30 con 0x7fe2540caea0 2024-09-16T01:49:07.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.775+0000 7fe251600640 1 -- 172.21.15.146:0/2126854979 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe24807a720 con 0x7fe2540caea0 2024-09-16T01:49:07.776 INFO:teuthology.orchestra.run.smithi146.stdout:64424509446 2024-09-16T01:49:07.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.779+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe23406c3b0 msgr2=0x7fe23406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.779+0000 7fe25c000640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe23406c3b0 0x7fe23406e870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe24000aac0 tx=0x7fe240008040 comp rx=0 tx=0).stop 2024-09-16T01:49:07.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.779+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 msgr2=0x7fe2540c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.779+0000 7fe25c000640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 0x7fe2540c74d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe248000f80 tx=0x7fe248004a40 comp rx=0 tx=0).stop 2024-09-16T01:49:07.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.780+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 shutdown_connections 2024-09-16T01:49:07.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.780+0000 7fe25c000640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe23406c3b0 0x7fe23406e870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.780+0000 7fe25c000640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe25415f9c0 0x7fe2540c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.780+0000 7fe25c000640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2540caea0 0x7fe2540c74d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.780+0000 7fe25c000640 1 --2- 172.21.15.146:0/2126854979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe254070ab0 0x7fe2541618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.780+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 >> 172.21.15.146:0/2126854979 conn(0x7fe2540d1b80 msgr2=0x7fe2540d1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.781+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 shutdown_connections 2024-09-16T01:49:07.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.781+0000 7fe25c000640 1 -- 172.21.15.146:0/2126854979 wait complete. 2024-09-16T01:49:07.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.786+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f519c14daf0 con 0x7f519c06c4f0 2024-09-16T01:49:07.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.786+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe3a80c7680 con 0x7fe3a8150a50 2024-09-16T01:49:07.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.787+0000 7f5193400640 1 -- 172.21.15.146:0/190792390 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f518405cc60 con 0x7f519c06c4f0 2024-09-16T01:49:07.788 INFO:teuthology.orchestra.run.smithi146.stdout:64424509446 2024-09-16T01:49:07.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.787+0000 7fe39e000640 1 -- 172.21.15.146:0/1517924343 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe3940792e0 con 0x7fe3a8150a50 2024-09-16T01:49:07.789 INFO:teuthology.orchestra.run.smithi146.stdout:60129542150 2024-09-16T01:49:07.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.789+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa89c005580 con 0x7fa8c0152850 2024-09-16T01:49:07.791 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.1 2024-09-16T01:49:07.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.790+0000 7fa8bcc00640 1 -- 172.21.15.146:0/4063150877 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa8ac05d600 con 0x7fa8c0152850 2024-09-16T01:49:07.791 INFO:teuthology.orchestra.run.smithi146.stdout:64424509446 2024-09-16T01:49:07.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5ba8158710 con 0x7f5ba815a4d0 2024-09-16T01:49:07.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f517806c380 msgr2=0x7f517806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f517806c380 0x7f517806e840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f519c15c520 tx=0x7f518c02e000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 msgr2=0x7f519c1501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c1501f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f518400c3a0 tx=0x7f518400c860 comp rx=0 tx=0).stop 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3880706f0 msgr2=0x7fe388072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3880706f0 0x7fe388072bb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00043c0 tx=0x7fe3a0007d30 comp rx=0 tx=0).stop 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3a8150a50 msgr2=0x7fe3a80c5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3a8150a50 0x7fe3a80c5790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80c1c70 tx=0x7fe3940046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 shutdown_connections 2024-09-16T01:49:07.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3880706f0 0x7fe388072bb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3a814b910 0x7fe3a81ec1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3a8150a50 0x7fe3a80c5790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 --2- 172.21.15.146:0/1517924343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3a80c6990 0x7fe3a814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 >> 172.21.15.146:0/1517924343 conn(0x7fe3a8067950 msgr2=0x7fe3a806fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 shutdown_connections 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 shutdown_connections 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f517806c380 0x7f517806e840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f519c14f1d0 0x7f519c150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f519c0c8ef0 0x7f519c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 --2- 172.21.15.146:0/190792390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f519c06c4f0 0x7f519c1501f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 >> 172.21.15.146:0/190792390 conn(0x7f519c067950 msgr2=0x7f519c0c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.791+0000 7fe3a7200640 1 -- 172.21.15.146:0/1517924343 wait complete. 2024-09-16T01:49:07.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.792+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 shutdown_connections 2024-09-16T01:49:07.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.792+0000 7f5ba5600640 1 -- 172.21.15.146:0/123091009 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b9407a320 con 0x7f5ba815a4d0 2024-09-16T01:49:07.796 INFO:teuthology.orchestra.run.smithi146.stdout:60129542150 2024-09-16T01:49:07.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.792+0000 7f519be00640 1 -- 172.21.15.146:0/190792390 wait complete. 2024-09-16T01:49:07.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa8a006c300 msgr2=0x7fa8a006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa8a006c300 0x7fa8a006e7c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4037000 tx=0x7fa8b400d000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 msgr2=0x7fa8c00c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c00c4c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00e870 tx=0x7fa8ac00ed30 comp rx=0 tx=0).stop 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 shutdown_connections 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa8a006c300 0x7fa8a006e7c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8c0158bb0 0x7fa8c014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8c0152850 0x7fa8c00c4c80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 --2- 172.21.15.146:0/4063150877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8c00c6bb0 0x7fa8c00c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 >> 172.21.15.146:0/4063150877 conn(0x7fa8c006f3c0 msgr2=0x7fa8c0072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 shutdown_connections 2024-09-16T01:49:07.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.794+0000 7fa8c7800640 1 -- 172.21.15.146:0/4063150877 wait complete. 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b8806c300 msgr2=0x7f5b8806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b8806c300 0x7f5b8806e7c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009ac0 tx=0x7f5b9c008040 comp rx=0 tx=0).stop 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ba815a4d0 msgr2=0x7f5ba8151c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ba815a4d0 0x7f5ba8151c00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80cbbf0 tx=0x7f5b940046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 shutdown_connections 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b8806c300 0x7f5b8806e7c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ba815a4d0 0x7f5ba8151c00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba8159080 0x7f5ba806d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 --2- 172.21.15.146:0/123091009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ba80cdfe0 0x7f5ba806df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.796+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 >> 172.21.15.146:0/123091009 conn(0x7f5ba8067950 msgr2=0x7f5ba80cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.797+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 shutdown_connections 2024-09-16T01:49:07.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.797+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f59bc072070 con 0x7f59bc06fee0 2024-09-16T01:49:07.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.797+0000 7f5bb0000640 1 -- 172.21.15.146:0/123091009 wait complete. 2024-09-16T01:49:07.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.797+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0384068ef0 con 0x7f0384073e90 2024-09-16T01:49:07.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.797+0000 7f59b2a00640 1 -- 172.21.15.146:0/1067619575 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f59ac07a8b0 con 0x7f59bc06fee0 2024-09-16T01:49:07.801 INFO:teuthology.orchestra.run.smithi146.stdout:55834574854 2024-09-16T01:49:07.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.799+0000 7f037e000640 1 -- 172.21.15.146:0/1118462018 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f037806aeb0 con 0x7f0384073e90 2024-09-16T01:49:07.801 INFO:teuthology.orchestra.run.smithi146.stdout:60129542150 2024-09-16T01:49:07.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f599c06c300 msgr2=0x7f599c06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f599c06c300 0x7f599c06e7c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59bc149130 tx=0x7f59a8008040 comp rx=0 tx=0).stop 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59bc06fee0 msgr2=0x7f59bc140310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59bc06fee0 0x7f59bc140310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac031000 tx=0x7f59ac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 shutdown_connections 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f599c06c300 0x7f599c06e7c0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59bc149130 tx=0x7f59a8008040 comp rx=0 tx=0).stop 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59bc0c70e0 0x7f59bc1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59bc072dd0 0x7f59bc0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.801+0000 7f59c1400640 1 --2- 172.21.15.146:0/1067619575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59bc06fee0 0x7f59bc140310 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59ac031000 tx=0x7f59ac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.802+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 >> 172.21.15.146:0/1067619575 conn(0x7f59bc06f3c0 msgr2=0x7f59bc0c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.802+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 shutdown_connections 2024-09-16T01:49:07.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.802+0000 7f59c1400640 1 -- 172.21.15.146:0/1067619575 wait complete. 2024-09-16T01:49:07.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f036806c2b0 msgr2=0x7f036806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f036806c2b0 0x7f036806e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0374034000 tx=0x7f037402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 msgr2=0x7f03841572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 0x7f03841572c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f037801e840 tx=0x7f037801ed00 comp rx=0 tx=0).stop 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 shutdown_connections 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f036806c2b0 0x7f036806e770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f038415f670 0x7f0384153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0384158ea0 0x7f03841559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 --2- 172.21.15.146:0/1118462018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0384073e90 0x7f03841572c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 >> 172.21.15.146:0/1118462018 conn(0x7f0384067950 msgr2=0x7f038406f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 shutdown_connections 2024-09-16T01:49:07.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:07.803+0000 7f038b200640 1 -- 172.21.15.146:0/1118462018 wait complete. 2024-09-16T01:49:07.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509446 for osd.0 2024-09-16T01:49:07.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.6 2024-09-16T01:49:07.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509446 for osd.3 2024-09-16T01:49:07.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509446 for osd.2 2024-09-16T01:49:07.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2024-09-16T01:49:07.828 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.5 2024-09-16T01:49:07.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2024-09-16T01:49:08.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T01:49:08.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T01:49:08.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T01:49:08.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T01:49:08.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T01:49:08.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T01:49:08.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T01:49:08.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T01:49:09.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.042+0000 7f2849a00640 1 Processor -- start 2024-09-16T01:49:09.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.042+0000 7f2849a00640 1 -- start start 2024-09-16T01:49:09.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.046+0000 7f2849a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f283c007340 0x7f283c04ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.046+0000 7f2849a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 0x7f283c005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.046+0000 7f2849a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f283c050520 con 0x7f283c0d8bb0 2024-09-16T01:49:09.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.046+0000 7f2843400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 0x7f283c005d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.046+0000 7f2843400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 0x7f283c005d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45260/0 (socket says 172.21.15.146:45260) 2024-09-16T01:49:09.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.046+0000 7f2843400640 1 -- 172.21.15.146:0/191943477 learned_addr learned my addr 172.21.15.146:0/191943477 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.047+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f283c050800 con 0x7f283c007340 2024-09-16T01:49:09.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.047+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f283c04f070 con 0x7f283c04f870 2024-09-16T01:49:09.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.048+0000 7f2842000640 1 --2- 172.21.15.146:0/191943477 >> v2:172.21.15.146:3300/0 conn(0x7f283c007340 0x7f283c04ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.050+0000 7f2842000640 1 -- 172.21.15.146:0/191943477 >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 msgr2=0x7f283c005d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:09.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.050+0000 7f2842000640 1 --2- 172.21.15.146:0/191943477 >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 0x7f283c005d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.050+0000 7f2842000640 1 -- 172.21.15.146:0/191943477 >> v1:172.21.15.146:6789/0 conn(0x7f283c0d8bb0 legacy=0x7f283c00e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.050+0000 7f2842000640 1 -- 172.21.15.146:0/191943477 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c04f350 con 0x7f283c007340 2024-09-16T01:49:09.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.051+0000 7f2843400640 1 --2- 172.21.15.146:0/191943477 >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 0x7f283c005d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:09.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.052+0000 7f2842000640 1 --2- 172.21.15.146:0/191943477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c007340 0x7f283c04ea00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f28440518b0 tx=0x7f28440639f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180247091b027de1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.053+0000 7f2841600640 1 -- 172.21.15.146:0/191943477 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28440a8990 con 0x7f283c007340 2024-09-16T01:49:09.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.053+0000 7f2841600640 1 -- 172.21.15.146:0/191943477 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28440a8b30 con 0x7f283c007340 2024-09-16T01:49:09.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.053+0000 7f2841600640 1 -- 172.21.15.146:0/191943477 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28440b0420 con 0x7f283c007340 2024-09-16T01:49:09.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.055+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c007340 msgr2=0x7f283c04ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.055+0000 7f2849a00640 1 --2- 172.21.15.146:0/191943477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c007340 0x7f283c04ea00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f28440518b0 tx=0x7f28440639f0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.055+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 shutdown_connections 2024-09-16T01:49:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.055+0000 7f2849a00640 1 --2- 172.21.15.146:0/191943477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c007340 0x7f283c04ea00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.056+0000 7f2849a00640 1 --2- 172.21.15.146:0/191943477 >> v2:172.21.15.161:3300/0 conn(0x7f283c04f870 0x7f283c005d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.056+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 >> 172.21.15.146:0/191943477 conn(0x7f283c054470 msgr2=0x7f283c054880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.056+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 shutdown_connections 2024-09-16T01:49:09.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.057+0000 7f2849a00640 1 -- 172.21.15.146:0/191943477 wait complete. 2024-09-16T01:49:09.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.060+0000 7f2849a00640 1 Processor -- start 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.061+0000 7f2849a00640 1 -- start start 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.062+0000 7f2849a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f283c007340 0x7f283c005300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.062+0000 7f2849a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f283c04f870 0x7f283c04d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.062+0000 7f2849a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 0x7f283c0387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.062+0000 7f2849a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f283c049340 con 0x7f283c0d8bb0 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.062+0000 7f2849a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f283c0491c0 con 0x7f283c04f870 2024-09-16T01:49:09.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.062+0000 7f2849a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f283c00eae0 con 0x7f283c007340 2024-09-16T01:49:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.064+0000 7f2842000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f283c04f870 0x7f283c04d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.064+0000 7f2843400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 0x7f283c0387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.064+0000 7f2843400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 0x7f283c0387a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59232/0 (socket says 172.21.15.146:59232) 2024-09-16T01:49:09.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.064+0000 7f2843400640 1 -- 172.21.15.146:0/484830365 learned_addr learned my addr 172.21.15.146:0/484830365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.064+0000 7f2842a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f283c007340 0x7f283c005300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.066+0000 7f2843400640 1 -- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f283c007340 msgr2=0x7f283c005300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.066+0000 7f2843400640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f283c007340 0x7f283c005300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.066+0000 7f2843400640 1 -- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f283c04f870 msgr2=0x7f283c04d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.066+0000 7f2843400640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f283c04f870 0x7f283c04d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.066+0000 7f2843400640 1 -- 172.21.15.146:0/484830365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c039260 con 0x7f283c0d8bb0 2024-09-16T01:49:09.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.067+0000 7f2842000640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f283c04f870 0x7f283c04d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:09.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.068+0000 7f2843400640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 0x7f283c0387a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f283000ec10 tx=0x7f283000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.068+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830017070 con 0x7f283c0d8bb0 2024-09-16T01:49:09.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.069+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f283000edc0 con 0x7f283c0d8bb0 2024-09-16T01:49:09.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.069+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830017070 con 0x7f283c0d8bb0 2024-09-16T01:49:09.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.069+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f283c0d28d0 con 0x7f283c0d8bb0 2024-09-16T01:49:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.070+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f283c04ddc0 con 0x7f283c0d8bb0 2024-09-16T01:49:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.076+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f283000cc30 con 0x7f283c0d8bb0 2024-09-16T01:49:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.075+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283c039430 con 0x7f283c0d8bb0 2024-09-16T01:49:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.077+0000 7f283aa00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f282406c4b0 0x7f282406e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.078+0000 7f2842a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f282406c4b0 0x7f282406e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.086+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2830057340 con 0x7f283c0d8bb0 2024-09-16T01:49:09.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.087+0000 7f2842a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f282406c4b0 0x7f282406e970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28340016e0 tx=0x7f2834009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.089+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f28300592a0 con 0x7f283c0d8bb0 2024-09-16T01:49:09.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.122+0000 7f484fa00640 1 Processor -- start 2024-09-16T01:49:09.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.122+0000 7f20b7400640 1 Processor -- start 2024-09-16T01:49:09.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.122+0000 7f484fa00640 1 -- start start 2024-09-16T01:49:09.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.124+0000 7fd372a00640 1 Processor -- start 2024-09-16T01:49:09.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.122+0000 7f20b7400640 1 -- start start 2024-09-16T01:49:09.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.124+0000 7f20b7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20b00cc0c0 0x7f20b015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.124+0000 7f20b7400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20b00cc4a0 0x7f20b0165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.124+0000 7f20b7400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b015b1d0 con 0x7f20b00cb2f0 2024-09-16T01:49:09.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.125+0000 7fd372a00640 1 -- start start 2024-09-16T01:49:09.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.124+0000 7f20b7400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f20b00cbd00 con 0x7f20b00cc0c0 2024-09-16T01:49:09.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.124+0000 7f20b7400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f20b015ba70 con 0x7f20b00cc4a0 2024-09-16T01:49:09.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.125+0000 7f20b5400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20b00cc4a0 0x7f20b0165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.125+0000 7f20afe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f20b00cb2f0 0x7f20b015e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52726/0 (socket says 172.21.15.146:52726) 2024-09-16T01:49:09.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.125+0000 7f20afe00640 1 -- 172.21.15.146:0/2664356181 learned_addr learned my addr 172.21.15.146:0/2664356181 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:09.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.126+0000 7f20b5400640 1 -- 172.21.15.146:0/2664356181 >> v2:172.21.15.146:3300/0 conn(0x7f20b00cc0c0 msgr2=0x7f20b015aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:49:09.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.126+0000 7f484fa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cb420 0x7f484815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.126+0000 7f20af400640 1 -- 172.21.15.146:0/2664356181 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897136636 0 0) 0x7f20b015b1d0 con 0x7f20b00cb2f0 2024-09-16T01:49:09.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.126+0000 7fd372a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3640d8a50 0x7fd36404d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.126+0000 7fd372a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 0x7fd364048420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.126+0000 7f20b4a00640 1 --2- 172.21.15.146:0/2664356181 >> v2:172.21.15.146:3300/0 conn(0x7f20b00cc0c0 0x7f20b015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7f20b5400640 1 --2- 172.21.15.146:0/2664356181 >> v2:172.21.15.146:3300/0 conn(0x7f20b00cc0c0 0x7f20b015aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7f20b5400640 1 -- 172.21.15.146:0/2664356181 >> v1:172.21.15.161:6789/0 conn(0x7f20b00cb2f0 legacy=0x7f20b015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7f20b5400640 1 -- 172.21.15.146:0/2664356181 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b015b4b0 con 0x7f20b00cc4a0 2024-09-16T01:49:09.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7f484fa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f48480cb800 0x7f48480d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd372a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3640d8e30 con 0x7fd364007f30 2024-09-16T01:49:09.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd372a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd36404d810 con 0x7fd3640d8a50 2024-09-16T01:49:09.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd372a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd36404e0d0 con 0x7fd364047f70 2024-09-16T01:49:09.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7f20b4a00640 1 --2- 172.21.15.146:0/2664356181 >> v2:172.21.15.146:3300/0 conn(0x7f20b00cc0c0 0x7f20b015aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:09.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd371600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3640d8a50 0x7fd36404d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7f484d000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cb420 0x7f484815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.128+0000 7f484fa00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48480cc750 con 0x7f48480cd4d0 2024-09-16T01:49:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.128+0000 7f484fa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f484815bb30 con 0x7f48480cb800 2024-09-16T01:49:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.128+0000 7f484fa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f48480d0b90 con 0x7f48480cb420 2024-09-16T01:49:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.128+0000 7f484d000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cb420 0x7f484815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37758/0 (socket says 172.21.15.146:37758) 2024-09-16T01:49:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.128+0000 7f484d000640 1 -- 172.21.15.146:0/583685664 learned_addr learned my addr 172.21.15.146:0/583685664 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd372000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 0x7fd364048420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd372000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 0x7fd364048420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37772/0 (socket says 172.21.15.146:37772) 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.127+0000 7fd372000640 1 -- 172.21.15.146:0/3547669949 learned_addr learned my addr 172.21.15.146:0/3547669949 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.128+0000 7fd372000640 1 -- 172.21.15.146:0/3547669949 >> v2:172.21.15.146:3300/0 conn(0x7fd3640d8a50 msgr2=0x7fd36404d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.129+0000 7fd372000640 1 --2- 172.21.15.146:0/3547669949 >> v2:172.21.15.146:3300/0 conn(0x7fd3640d8a50 0x7fd36404d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.129+0000 7fd372000640 1 -- 172.21.15.146:0/3547669949 >> v1:172.21.15.146:6789/0 conn(0x7fd364007f30 legacy=0x7fd3640e2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.129+0000 7fd372000640 1 -- 172.21.15.146:0/3547669949 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd364007bb0 con 0x7fd364047f70 2024-09-16T01:49:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.129+0000 7fd371600640 1 --2- 172.21.15.146:0/3547669949 >> v2:172.21.15.146:3300/0 conn(0x7fd3640d8a50 0x7fd36404d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.129+0000 7f4847400640 1 -- 172.21.15.146:0/583685664 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410002406 0 0) 0x7f48480cc750 con 0x7f48480cd4d0 2024-09-16T01:49:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.129+0000 7f4847e00640 1 --2- 172.21.15.146:0/583685664 >> v2:172.21.15.146:3300/0 conn(0x7f48480cb800 0x7f48480d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7f20b5400640 1 --2- 172.21.15.146:0/2664356181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc4a0 0x7f20b0165430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f20b006e100 tx=0x7f20a802a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98318f175a8ecaf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7fd372000640 1 --2- 172.21.15.146:0/3547669949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 0x7fd364048420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd358009870 tx=0x7fd35802a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f22b3c01a037e779 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7f4847400640 1 -- 172.21.15.146:0/583685664 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4830003520 con 0x7f48480cd4d0 2024-09-16T01:49:09.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7f20af400640 1 -- 172.21.15.146:0/2664356181 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8007740 con 0x7f20b00cc4a0 2024-09-16T01:49:09.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7fd36be00640 1 -- 172.21.15.146:0/3547669949 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358037070 con 0x7fd364047f70 2024-09-16T01:49:09.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7fd36be00640 1 -- 172.21.15.146:0/3547669949 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd358032440 con 0x7fd364047f70 2024-09-16T01:49:09.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7fd36be00640 1 -- 172.21.15.146:0/3547669949 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd358032810 con 0x7fd364047f70 2024-09-16T01:49:09.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f4847400640 1 -- 172.21.15.146:0/583685664 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2493261023 0 0) 0x7f4830003520 con 0x7f48480cd4d0 2024-09-16T01:49:09.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7f20af400640 1 -- 172.21.15.146:0/2664356181 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a80078e0 con 0x7f20b00cc4a0 2024-09-16T01:49:09.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.130+0000 7f20af400640 1 -- 172.21.15.146:0/2664356181 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8007cf0 con 0x7f20b00cc4a0 2024-09-16T01:49:09.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f20b7400640 1 -- 172.21.15.146:0/2664356181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc4a0 msgr2=0x7f20b0165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f20b7400640 1 --2- 172.21.15.146:0/2664356181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc4a0 0x7f20b0165430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f20b006e100 tx=0x7f20a802a6a0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7fd372a00640 1 -- 172.21.15.146:0/3547669949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 msgr2=0x7fd364048420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7fd372a00640 1 --2- 172.21.15.146:0/3547669949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 0x7fd364048420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd358009870 tx=0x7fd35802a140 comp rx=0 tx=0).stop 2024-09-16T01:49:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f4847e00640 1 -- 172.21.15.146:0/583685664 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cb420 msgr2=0x7f484815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7f20b7400640 1 -- 172.21.15.146:0/2664356181 shutdown_connections 2024-09-16T01:49:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7f20b7400640 1 --2- 172.21.15.146:0/2664356181 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc4a0 0x7f20b0165430 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7f20b7400640 1 --2- 172.21.15.146:0/2664356181 >> v2:172.21.15.146:3300/0 conn(0x7f20b00cc0c0 0x7f20b015aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7f20b7400640 1 -- 172.21.15.146:0/2664356181 >> 172.21.15.146:0/2664356181 conn(0x7f20b006d560 msgr2=0x7f20b006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7fd372a00640 1 -- 172.21.15.146:0/3547669949 shutdown_connections 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7fd372a00640 1 --2- 172.21.15.146:0/3547669949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd364047f70 0x7fd364048420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7fd372a00640 1 --2- 172.21.15.146:0/3547669949 >> v2:172.21.15.146:3300/0 conn(0x7fd3640d8a50 0x7fd36404d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7fd372a00640 1 -- 172.21.15.146:0/3547669949 >> 172.21.15.146:0/3547669949 conn(0x7fd36404e8f0 msgr2=0x7fd36404ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f4847e00640 1 --2- 172.21.15.146:0/583685664 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cb420 0x7f484815b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f4847e00640 1 -- 172.21.15.146:0/583685664 >> v1:172.21.15.161:6789/0 conn(0x7f48480cd4d0 legacy=0x7f48481547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.131+0000 7f4847e00640 1 -- 172.21.15.146:0/583685664 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48480cc9f0 con 0x7f48480cb800 2024-09-16T01:49:09.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7fd372a00640 1 -- 172.21.15.146:0/3547669949 shutdown_connections 2024-09-16T01:49:09.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7fd372a00640 1 -- 172.21.15.146:0/3547669949 wait complete. 2024-09-16T01:49:09.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7f20b7400640 1 -- 172.21.15.146:0/2664356181 shutdown_connections 2024-09-16T01:49:09.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.132+0000 7f20b7400640 1 -- 172.21.15.146:0/2664356181 wait complete. 2024-09-16T01:49:09.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.133+0000 7f4847e00640 1 --2- 172.21.15.146:0/583685664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480d02e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f48340097a0 tx=0x7f483402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde8625a0f8431c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7f4847400640 1 -- 172.21.15.146:0/583685664 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834037070 con 0x7f48480cb800 2024-09-16T01:49:09.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7f4847400640 1 -- 172.21.15.146:0/583685664 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f483402ac30 con 0x7f48480cb800 2024-09-16T01:49:09.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7f4847400640 1 -- 172.21.15.146:0/583685664 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834037070 con 0x7f48480cb800 2024-09-16T01:49:09.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7fd372a00640 1 Processor -- start 2024-09-16T01:49:09.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7fd372a00640 1 -- start start 2024-09-16T01:49:09.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7fd05b000640 1 Processor -- start 2024-09-16T01:49:09.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.134+0000 7fd05b000640 1 -- start start 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.135+0000 7fd372a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364007f30 0x7fd36404bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.135+0000 7fd372a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 0x7fd364168800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.135+0000 7fd372a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3640d8a50 0x7fd3640db160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.135+0000 7fd372a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd36404c7f0 con 0x7fd364007f30 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.135+0000 7fd372a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd36404e600 con 0x7fd364047f70 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.135+0000 7fd372a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd36404cad0 con 0x7fd3640d8a50 2024-09-16T01:49:09.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 0x7fd364168800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 0x7fd364168800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45294/0 (socket says 172.21.15.146:45294) 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 -- 172.21.15.146:0/3412278783 learned_addr learned my addr 172.21.15.146:0/3412278783 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd372000640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3640d8a50 0x7fd3640db160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 -- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3640d8a50 msgr2=0x7fd3640db160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd371600640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364007f30 0x7fd36404bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3640d8a50 0x7fd3640db160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 -- 172.21.15.146:0/3412278783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364007f30 msgr2=0x7fd36404bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364007f30 0x7fd36404bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd370c00640 1 -- 172.21.15.146:0/3412278783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36417c570 con 0x7fd364047f70 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7fd371600640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364007f30 0x7fd36404bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd370c00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 0x7fd364168800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c3c0 tx=0x7fd36000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360017070 con 0x7fd364047f70 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3600052f0 con 0x7fd364047f70 2024-09-16T01:49:09.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360003830 con 0x7fd364047f70 2024-09-16T01:49:09.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36417c850 con 0x7fd364047f70 2024-09-16T01:49:09.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7f484fa00640 1 -- 172.21.15.146:0/583685664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 msgr2=0x7f48480d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7f484fa00640 1 --2- 172.21.15.146:0/583685664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480d02e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f48340097a0 tx=0x7f483402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7f20b7400640 1 Processor -- start 2024-09-16T01:49:09.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.136+0000 7f20b7400640 1 -- start start 2024-09-16T01:49:09.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd05b000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0540c70e0 0x7fd05406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd05b000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0541569d0 con 0x7fd054150c10 2024-09-16T01:49:09.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd05b000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd054071ce0 con 0x7fd05406fee0 2024-09-16T01:49:09.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.137+0000 7fd05b000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd054157580 con 0x7fd0540c70e0 2024-09-16T01:49:09.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.138+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd36417ce10 con 0x7fd364047f70 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.138+0000 7fd053e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd05406fee0 0x7fd054072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52752/0 (socket says 172.21.15.146:52752) 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.138+0000 7fd053e00640 1 -- 172.21.15.146:0/126296832 learned_addr learned my addr 172.21.15.146:0/126296832 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f484fa00640 1 -- 172.21.15.146:0/583685664 shutdown_connections 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f484fa00640 1 --2- 172.21.15.146:0/583685664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480d02e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f484fa00640 1 --2- 172.21.15.146:0/583685664 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cb420 0x7f484815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f484fa00640 1 -- 172.21.15.146:0/583685664 >> 172.21.15.146:0/583685664 conn(0x7f484806d7c0 msgr2=0x7f484806e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f20b7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 0x7f20b014a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f484fa00640 1 -- 172.21.15.146:0/583685664 shutdown_connections 2024-09-16T01:49:09.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f484fa00640 1 -- 172.21.15.146:0/583685664 wait complete. 2024-09-16T01:49:09.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f20b7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc0c0 0x7f20b0148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.140+0000 7fd053400640 1 --2- 172.21.15.146:0/126296832 >> v2:172.21.15.146:3300/0 conn(0x7fd0540c70e0 0x7fd05406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f20b4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 0x7f20b014a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f20b4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 0x7f20b014a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37778/0 (socket says 172.21.15.146:37778) 2024-09-16T01:49:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.140+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3045340778 0 0) 0x7fd054071ce0 con 0x7fd05406fee0 2024-09-16T01:49:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.140+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd038003520 con 0x7fd05406fee0 2024-09-16T01:49:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.140+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039048461 0 0) 0x7fd04c001490 con 0x7fd054150c10 2024-09-16T01:49:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.140+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd054071ce0 con 0x7fd054150c10 2024-09-16T01:49:09.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7f20b4a00640 1 -- 172.21.15.146:0/687231920 learned_addr learned my addr 172.21.15.146:0/687231920 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.140+0000 7fd094c00640 1 Processor -- start 2024-09-16T01:49:09.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.139+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3640077c0 con 0x7fd364047f70 2024-09-16T01:49:09.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f49c7400640 1 Processor -- start 2024-09-16T01:49:09.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd3dae00640 1 Processor -- start 2024-09-16T01:49:09.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f49c7400640 1 -- start start 2024-09-16T01:49:09.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f20afe00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc0c0 0x7f20b0148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b00cc4a0 0x7f20b0149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd3dae00640 1 -- start start 2024-09-16T01:49:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20b015e1d0 con 0x7f20b00cc4a0 2024-09-16T01:49:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20b015e540 con 0x7f20b00cc0c0 2024-09-16T01:49:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f20b00cc8b0 con 0x7f20b00cb2f0 2024-09-16T01:49:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd36001a070 con 0x7fd364047f70 2024-09-16T01:49:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd053400640 1 -- 172.21.15.146:0/126296832 >> v1:172.21.15.161:6789/0 conn(0x7fd05406fee0 legacy=0x7fd054072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd053400640 1 -- 172.21.15.146:0/126296832 >> v1:172.21.15.146:6789/0 conn(0x7fd054150c10 legacy=0x7fd05415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd053400640 1 -- 172.21.15.146:0/126296832 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd054156c50 con 0x7fd0540c70e0 2024-09-16T01:49:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b5400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b00cc4a0 0x7f20b0149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f484fa00640 1 Processor -- start 2024-09-16T01:49:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b4a00640 1 -- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc0c0 msgr2=0x7f20b0148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b4a00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc0c0 0x7f20b0148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b4a00640 1 -- 172.21.15.146:0/687231920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b00cc4a0 msgr2=0x7f20b0149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b4a00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b00cc4a0 0x7f20b0149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b4a00640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b0151c70 con 0x7f20b00cb2f0 2024-09-16T01:49:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7fd369600640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd34c06c2e0 0x7fd34c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.142+0000 7f20b5400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b00cc4a0 0x7f20b0149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:09.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f484fa00640 1 -- start start 2024-09-16T01:49:09.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd36008a720 con 0x7fd364047f70 2024-09-16T01:49:09.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7fd3dae00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0520 0x7fd3d40b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7fd3dae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3d40d0900 0x7fd3d41628d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f20b4a00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 0x7f20b014a3b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20a400c820 tx=0x7f20a400cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4015070 con 0x7f20b00cb2f0 2024-09-16T01:49:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a4007a30 con 0x7f20b00cb2f0 2024-09-16T01:49:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7fd371600640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd34c06c2e0 0x7fd34c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 0x7f49c00c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49c00c6410 0x7f49c0158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c7400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49c00c5280 0x7f49c00c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c7400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f49c0151e70 con 0x7f49c00c6410 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c7400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f49c00c8130 con 0x7f49c00c5280 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c7400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f49c00c8410 con 0x7f49c0151a90 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 0x7f49c00c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 0x7f49c00c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37794/0 (socket says 172.21.15.146:37794) 2024-09-16T01:49:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.143+0000 7f49c4a00640 1 -- 172.21.15.146:0/2258191843 learned_addr learned my addr 172.21.15.146:0/2258191843 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.141+0000 7fd094c00640 1 -- start start 2024-09-16T01:49:09.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3d8e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd3d40cc2a0 0x7fd3d4158be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:52758/0 (socket says 172.21.15.146:52758) 2024-09-16T01:49:09.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3d8e00640 1 -- 172.21.15.146:0/632479059 learned_addr learned my addr 172.21.15.146:0/632479059 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:09.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd053400640 1 --2- 172.21.15.146:0/126296832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0540c70e0 0x7fd05406da50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd044009a00 tx=0x7fd04402a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9979152252d32c71 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49c5400640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.161:3300/0 conn(0x7f49c00c5280 0x7f49c00c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd371600640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd34c06c2e0 0x7fd34c06e7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd36c06b460 tx=0x7fd36c08cf90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d415f140 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49bfe00640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.146:3300/0 conn(0x7f49c00c6410 0x7f49c0158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d415f420 con 0x7fd3d40d0900 2024-09-16T01:49:09.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d415f700 con 0x7fd3d40d0520 2024-09-16T01:49:09.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044036070 con 0x7fd0540c70e0 2024-09-16T01:49:09.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd044031600 con 0x7fd0540c70e0 2024-09-16T01:49:09.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd052a00640 1 -- 172.21.15.146:0/126296832 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0440319d0 con 0x7fd0540c70e0 2024-09-16T01:49:09.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3d3e00640 1 --2- 172.21.15.146:0/632479059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0520 0x7fd3d40b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b01f1070 con 0x7f20b00cb2f0 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b01521f0 con 0x7f20b00cb2f0 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4015070 con 0x7f20b00cb2f0 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7fd3d3400640 1 --2- 172.21.15.146:0/632479059 >> v2:172.21.15.146:3300/0 conn(0x7fd3d40d0900 0x7fd3d41628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49c4a00640 1 -- 172.21.15.146:0/2258191843 >> v2:172.21.15.161:3300/0 conn(0x7f49c00c5280 msgr2=0x7f49c00c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49c4a00640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.161:3300/0 conn(0x7f49c00c5280 0x7f49c00c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49c4a00640 1 -- 172.21.15.146:0/2258191843 >> v2:172.21.15.146:3300/0 conn(0x7f49c00c6410 msgr2=0x7f49c0158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49c4a00640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.146:3300/0 conn(0x7f49c00c6410 0x7f49c0158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.144+0000 7f49c4a00640 1 -- 172.21.15.146:0/2258191843 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c00c86f0 con 0x7f49c0151a90 2024-09-16T01:49:09.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd3d3e00640 1 -- 172.21.15.146:0/632479059 >> v2:172.21.15.146:3300/0 conn(0x7fd3d40d0900 msgr2=0x7fd3d41628d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:09.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd3d3e00640 1 --2- 172.21.15.146:0/632479059 >> v2:172.21.15.146:3300/0 conn(0x7fd3d40d0900 0x7fd3d41628d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd3d3e00640 1 -- 172.21.15.146:0/632479059 >> v1:172.21.15.161:6789/0 conn(0x7fd3d40cc2a0 legacy=0x7fd3d4158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd3d3e00640 1 -- 172.21.15.146:0/632479059 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d415f9e0 con 0x7fd3d40d0520 2024-09-16T01:49:09.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b014f050 con 0x7f20b00cb2f0 2024-09-16T01:49:09.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7f49c5400640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.161:3300/0 conn(0x7f49c00c5280 0x7f49c00c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:09.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd05b000640 1 -- 172.21.15.146:0/126296832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0540c70e0 msgr2=0x7fd05406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7f484fa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48480cb420 0x7f484814b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7f484fa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd05b000640 1 --2- 172.21.15.146:0/126296832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0540c70e0 0x7fd05406da50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd044009a00 tx=0x7fd04402a430 comp rx=0 tx=0).stop 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd05b000640 1 -- 172.21.15.146:0/126296832 shutdown_connections 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd05b000640 1 --2- 172.21.15.146:0/126296832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0540c70e0 0x7fd05406da50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.145+0000 7fd05b000640 1 -- 172.21.15.146:0/126296832 >> 172.21.15.146:0/126296832 conn(0x7fd05406f3c0 msgr2=0x7fd05406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7fd05b000640 1 -- 172.21.15.146:0/126296832 shutdown_connections 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7fd094c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0900c5eb0 0x7fd0900cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7fd05b000640 1 -- 172.21.15.146:0/126296832 wait complete. 2024-09-16T01:49:09.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f49c4a00640 1 --2- 172.21.15.146:0/2258191843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 0x7f49c00c36b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f49c015bd40 tx=0x7f49b402a1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba62ae365fc2475 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7fd094c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09015af00 con 0x7fd09015aa70 2024-09-16T01:49:09.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7fd3d3e00640 1 --2- 172.21.15.146:0/632479059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0520 0x7fd3d40b4240 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009d90 tx=0x7fd3cc02a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2710dd80ec86a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7fd094c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09015b1e0 con 0x7fd0900c6290 2024-09-16T01:49:09.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7fd094c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd09006dbd0 con 0x7fd0900c5eb0 2024-09-16T01:49:09.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7f484fa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cd4d0 0x7f48480b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f484fa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f48480d0e70 con 0x7f48480cb800 2024-09-16T01:49:09.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f484fa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f48480ccfb0 con 0x7f48480cb420 2024-09-16T01:49:09.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f484fa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f48480c5530 con 0x7f48480cd4d0 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.146+0000 7f484d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48480cb420 0x7f484814b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f484d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48480cb420 0x7f484814b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45324/0 (socket says 172.21.15.146:45324) 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f484d000640 1 -- 172.21.15.146:0/1923465385 learned_addr learned my addr 172.21.15.146:0/1923465385 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f4847e00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd3600587a0 con 0x7fd364047f70 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7fd08ea00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd09015aa70 0x7fd090154910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53730/0 (socket says 172.21.15.146:53730) 2024-09-16T01:49:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7fd08ea00640 1 -- 172.21.15.146:0/2973470107 learned_addr learned my addr 172.21.15.146:0/2973470107 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:09.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f484da00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cd4d0 0x7f48480b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7fd08f400640 1 --2- 172.21.15.146:0/2973470107 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0900c5eb0 0x7fd0900cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7fd08e000640 1 -- 172.21.15.146:0/2973470107 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588043283 0 0) 0x7fd09015b1e0 con 0x7fd0900c6290 2024-09-16T01:49:09.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7fd08e000640 1 -- 172.21.15.146:0/2973470107 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd074003520 con 0x7fd0900c6290 2024-09-16T01:49:09.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f20a4018070 con 0x7f20b00cb2f0 2024-09-16T01:49:09.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7fd08f400640 1 -- 172.21.15.146:0/2973470107 >> v1:172.21.15.161:6789/0 conn(0x7fd0900c6290 legacy=0x7fd09006c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7fd08f400640 1 -- 172.21.15.146:0/2973470107 >> v1:172.21.15.146:6789/0 conn(0x7fd09015aa70 legacy=0x7fd090154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7fd08f400640 1 -- 172.21.15.146:0/2973470107 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09015b4c0 con 0x7fd0900c5eb0 2024-09-16T01:49:09.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.147+0000 7f4847e00640 1 -- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cd4d0 msgr2=0x7f48480b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7f4847e00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cd4d0 0x7f48480b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7f4847e00640 1 -- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48480cb420 msgr2=0x7f484814b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7f4847e00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48480cb420 0x7f484814b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7f4847e00640 1 -- 172.21.15.146:0/1923465385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484814b570 con 0x7f48480cb800 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.148+0000 7f484da00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cd4d0 0x7f48480b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.149+0000 7f4847e00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480b4ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4834031000 tx=0x7f48340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.149+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834041020 con 0x7f48480cb800 2024-09-16T01:49:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.149+0000 7f20acc00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f209006c4e0 0x7f209006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.149+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f483403d620 con 0x7f48480cb800 2024-09-16T01:49:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.149+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48481f4a20 con 0x7f48480cb800 2024-09-16T01:49:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484814e410 con 0x7f48480cb800 2024-09-16T01:49:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834041020 con 0x7f48480cb800 2024-09-16T01:49:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7fd05b000640 1 Processor -- start 2024-09-16T01:49:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7fd08f400640 1 --2- 172.21.15.146:0/2973470107 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0900c5eb0 0x7fd0900cac30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0800097d0 tx=0x7fd08002a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6a0cc1a731c27c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7fd05b000640 1 -- start start 2024-09-16T01:49:09.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f20a408b0f0 con 0x7f20b00cb2f0 2024-09-16T01:49:09.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.150+0000 7f20afe00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f209006c4e0 0x7f209006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7f20afe00640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f209006c4e0 0x7f209006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20b00b3280 tx=0x7f209c007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4824005290 con 0x7f48480cb800 2024-09-16T01:49:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd3d2a00640 1 -- 172.21.15.146:0/632479059 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc007740 con 0x7fd3d40d0520 2024-09-16T01:49:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd3d2a00640 1 -- 172.21.15.146:0/632479059 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc0078e0 con 0x7fd3d40d0520 2024-09-16T01:49:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd3d2a00640 1 -- 172.21.15.146:0/632479059 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc007cb0 con 0x7fd3d40d0520 2024-09-16T01:49:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd05b000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 0x7fd0541f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7f49bf400640 1 -- 172.21.15.146:0/2258191843 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4037070 con 0x7f49c0151a90 2024-09-16T01:49:09.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7f49bf400640 1 -- 172.21.15.146:0/2258191843 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b4032440 con 0x7f49c0151a90 2024-09-16T01:49:09.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7f49bf400640 1 -- 172.21.15.146:0/2258191843 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4032810 con 0x7f49c0151a90 2024-09-16T01:49:09.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd08e000640 1 -- 172.21.15.146:0/2973470107 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080037070 con 0x7fd0900c5eb0 2024-09-16T01:49:09.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd08e000640 1 -- 172.21.15.146:0/2973470107 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd080032440 con 0x7fd0900c5eb0 2024-09-16T01:49:09.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7fd05b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 0x7fd0541f8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.151+0000 7fd08e000640 1 -- 172.21.15.146:0/2973470107 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080032810 con 0x7fd0900c5eb0 2024-09-16T01:49:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7fd053e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 0x7fd0541f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7fd053e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 0x7fd0541f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45330/0 (socket says 172.21.15.146:45330) 2024-09-16T01:49:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7fd053400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 0x7fd0541f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0520 msgr2=0x7fd3d40b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd3dae00640 1 --2- 172.21.15.146:0/632479059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0520 0x7fd3d40b4240 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009d90 tx=0x7fd3cc02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 -- 172.21.15.146:0/2973470107 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0900c5eb0 msgr2=0x7fd0900cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 --2- 172.21.15.146:0/2973470107 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0900c5eb0 0x7fd0900cac30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0800097d0 tx=0x7fd08002a2a0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 shutdown_connections 2024-09-16T01:49:09.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd05b000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd05415a790 0x7fd05414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7f49c7400640 1 -- 172.21.15.146:0/2258191843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 msgr2=0x7f49c00c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7f49c7400640 1 --2- 172.21.15.146:0/2258191843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 0x7f49c00c36b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f49c015bd40 tx=0x7f49b402a1f0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f20b014f050 con 0x7f20b00cb2f0 2024-09-16T01:49:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 -- 172.21.15.146:0/2973470107 shutdown_connections 2024-09-16T01:49:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 --2- 172.21.15.146:0/2973470107 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0900c5eb0 0x7fd0900cac30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 -- 172.21.15.146:0/2973470107 >> 172.21.15.146:0/2973470107 conn(0x7fd090067b30 msgr2=0x7fd090067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd053e00640 1 -- 172.21.15.146:0/4132534520 learned_addr learned my addr 172.21.15.146:0/4132534520 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0540c9a90 con 0x7fd054150c10 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd054157210 con 0x7fd05406fee0 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd0540718c0 con 0x7fd05415a790 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.152+0000 7fd053400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 0x7fd0541f8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59306/0 (socket says 172.21.15.146:59306) 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd059000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd05415a790 0x7fd05414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 -- 172.21.15.146:0/2973470107 shutdown_connections 2024-09-16T01:49:09.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7fd094c00640 1 -- 172.21.15.146:0/2973470107 wait complete. 2024-09-16T01:49:09.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7fd3dae00640 1 --2- 172.21.15.146:0/632479059 >> v2:172.21.15.146:3300/0 conn(0x7fd3d40d0900 0x7fd3d41628d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7fd3dae00640 1 --2- 172.21.15.146:0/632479059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0520 0x7fd3d40b4240 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 >> 172.21.15.146:0/632479059 conn(0x7fd3d406ca90 msgr2=0x7fd3d406e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7fd053e00640 1 -- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd05415a790 msgr2=0x7fd05414fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7f49c7400640 1 -- 172.21.15.146:0/2258191843 shutdown_connections 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7f49c7400640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.161:3300/0 conn(0x7f49c00c5280 0x7f49c00c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7f49c7400640 1 --2- 172.21.15.146:0/2258191843 >> v2:172.21.15.146:3300/0 conn(0x7f49c00c6410 0x7f49c0158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7f49c7400640 1 --2- 172.21.15.146:0/2258191843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c0151a90 0x7f49c00c36b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.153+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4834044020 con 0x7f48480cb800 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7f4844c00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f482806c4b0 0x7f482806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f48340ab210 con 0x7f48480cb800 2024-09-16T01:49:09.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7f49c7400640 1 -- 172.21.15.146:0/2258191843 >> 172.21.15.146:0/2258191843 conn(0x7f49c006f3c0 msgr2=0x7f49c006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 shutdown_connections 2024-09-16T01:49:09.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7f49c7400640 1 -- 172.21.15.146:0/2258191843 shutdown_connections 2024-09-16T01:49:09.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd094c00640 1 Processor -- start 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7f484d000640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f482806c4b0 0x7f482806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd3dae00640 1 -- 172.21.15.146:0/632479059 wait complete. 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.154+0000 7fd053e00640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd05415a790 0x7fd05414fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd053e00640 1 -- 172.21.15.146:0/4132534520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 msgr2=0x7fd0541f8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd053e00640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 0x7fd0541f8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd053e00640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd054144ed0 con 0x7fd05406fee0 2024-09-16T01:49:09.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7fd053400640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 0x7fd0541f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:09.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.155+0000 7f49c7400640 1 -- 172.21.15.146:0/2258191843 wait complete. 2024-09-16T01:49:09.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.156+0000 7fd094c00640 1 -- start start 2024-09-16T01:49:09.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.156+0000 7fd053e00640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 0x7fd0541f4970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c4d0 tx=0x7fd04000c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.156+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040018070 con 0x7fd05406fee0 2024-09-16T01:49:09.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.156+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0400074e0 con 0x7fd05406fee0 2024-09-16T01:49:09.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.156+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040004a20 con 0x7fd05406fee0 2024-09-16T01:49:09.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.157+0000 7f484d000640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f482806c4b0 0x7f482806e970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f483c0094a0 tx=0x7f483c009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.156+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd054145170 con 0x7fd05406fee0 2024-09-16T01:49:09.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.157+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0541f8130 con 0x7fd05406fee0 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.158+0000 7fd094c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0900c5eb0 0x7fd0901e5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.158+0000 7fd094c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0900c6290 0x7fd0900c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.158+0000 7fd094c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 0x7fd0900c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0900c5eb0 0x7fd0901e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd094c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd090155020 con 0x7fd0900c5eb0 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd094c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd09006c370 con 0x7fd0900c6290 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd094c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd0901551a0 con 0x7fd09015a640 2024-09-16T01:49:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0900c6290 0x7fd0900c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 0x7fd0900c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 0x7fd0900c7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37842/0 (socket says 172.21.15.146:37842) 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08fe00640 1 -- 172.21.15.146:0/3658030072 learned_addr learned my addr 172.21.15.146:0/3658030072 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08fe00640 1 -- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0900c6290 msgr2=0x7fd0900c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08fe00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0900c6290 0x7fd0900c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08fe00640 1 -- 172.21.15.146:0/3658030072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0900c5eb0 msgr2=0x7fd0901e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.159+0000 7fd08ea00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0900c6290 0x7fd0900c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.158+0000 7fd3dae00640 1 Processor -- start 2024-09-16T01:49:09.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.158+0000 7fd3dae00640 1 -- start start 2024-09-16T01:49:09.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd08fe00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0900c5eb0 0x7fd0901e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3dae00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 0x7fd3d41e8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3dae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40d0520 0x7fd3d4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3dae00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0900 0x7fd3d4161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7f49c7400640 1 Processor -- start 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3dae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40b4780 con 0x7fd3d40d0520 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3dae00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40cbde0 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3dae00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d41592f0 con 0x7fd3d40d0900 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 0x7fd3d41e8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 0x7fd3d41e8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45344/0 (socket says 172.21.15.146:45344) 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 -- 172.21.15.146:0/3438665090 learned_addr learned my addr 172.21.15.146:0/3438665090 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7f49c7400640 1 -- start start 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 -- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0900 msgr2=0x7fd3d4161230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0900 0x7fd3d4161230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 -- 172.21.15.146:0/3438665090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40d0520 msgr2=0x7fd3d4160cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40d0520 0x7fd3d4160cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd3d3e00640 1 -- 172.21.15.146:0/3438665090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d40d00e0 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.161+0000 7fd3d3e00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 0x7fd3d41e8940 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009d90 tx=0x7fd3cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.161+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc040070 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.161+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc031bc0 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.161+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc045460 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d40cfe00 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d41604f0 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.161+0000 7f49c7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c00c5280 0x7f49c0148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c00c6410 0x7f49c0148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49c0151a90 0x7f49c01f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c00c5280 0x7f49c0148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c7400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f49c0157b20 con 0x7f49c0151a90 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c7400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49c0157840 con 0x7f49c00c5280 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c00c5280 0x7f49c0148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45348/0 (socket says 172.21.15.146:45348) 2024-09-16T01:49:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7f49c4a00640 1 -- 172.21.15.146:0/3809917438 learned_addr learned my addr 172.21.15.146:0/3809917438 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.163+0000 7f49c5400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49c0151a90 0x7f49c01f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd08fe00640 1 -- 172.21.15.146:0/3658030072 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0900c9d10 con 0x7fd09015a640 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.160+0000 7fd08f400640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0900c5eb0 0x7fd0901e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.161+0000 7fd08fe00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 0x7fd0900c7c40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c6f0 tx=0x7fd08800cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088003970 con 0x7fd09015a640 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd088003b30 con 0x7fd09015a640 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800a850 con 0x7fd09015a640 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd094c00640 1 -- 172.21.15.146:0/3658030072 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0901f5480 con 0x7fd09015a640 2024-09-16T01:49:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.162+0000 7fd094c00640 1 -- 172.21.15.146:0/3658030072 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0900ca290 con 0x7fd09015a640 2024-09-16T01:49:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.164+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f483404b030 con 0x7f48480cb800 2024-09-16T01:49:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.164+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd04001b070 con 0x7fd05406fee0 2024-09-16T01:49:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.164+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f49c00c3bf0 con 0x7f49c00c6410 2024-09-16T01:49:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.164+0000 7f49bfe00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c00c6410 0x7f49c0148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.164+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd02c005290 con 0x7fd05406fee0 2024-09-16T01:49:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ac005290 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7f49bfe00640 1 -- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c00c5280 msgr2=0x7f49c0148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7f49bfe00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c00c5280 0x7f49c0148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7f49bfe00640 1 -- 172.21.15.146:0/3809917438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49c0151a90 msgr2=0x7f49c01f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7f49bfe00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49c0151a90 0x7f49c01f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7f49bfe00640 1 -- 172.21.15.146:0/3809917438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c0146490 con 0x7f49c00c6410 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.165+0000 7f49c5400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49c0151a90 0x7f49c01f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.166+0000 7f49bfe00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c00c6410 0x7f49c0148680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00d5a0 tx=0x7f49ac00da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.166+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac019070 con 0x7f49c00c6410 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.166+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ac004240 con 0x7f49c00c6410 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.166+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0900c56f0 con 0x7fd09015a640 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.166+0000 7fd04be00640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd03406c3b0 0x7fd03406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.167+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd04008a7b0 con 0x7fd05406fee0 2024-09-16T01:49:09.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.167+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c0146730 con 0x7f49c00c6410 2024-09-16T01:49:09.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.167+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac003610 con 0x7f49c00c6410 2024-09-16T01:49:09.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.167+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c0147850 con 0x7f49c00c6410 2024-09-16T01:49:09.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.167+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd3cc039040 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.168+0000 7fd053400640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd03406c3b0 0x7fd03406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.168+0000 7fd3cbe00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd3b406c2b0 0x7fd3b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.168+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c00c71f0 con 0x7f49c00c6410 2024-09-16T01:49:09.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.169+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd3cc0acce0 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.169+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd08800a9f0 con 0x7fd09015a640 2024-09-16T01:49:09.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.169+0000 7fd3d3400640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd3b406c2b0 0x7fd3b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.170+0000 7fd3d3400640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd3b406c2b0 0x7fd3b406e770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3d41e9450 tx=0x7fd3c4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.170+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f49ac01c070 con 0x7f49c00c6410 2024-09-16T01:49:09.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.170+0000 7fd053400640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd03406c3b0 0x7fd03406e870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd044033000 tx=0x7fd044008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.171+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd0400588b0 con 0x7fd05406fee0 2024-09-16T01:49:09.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.171+0000 7fd087400640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd07006c3b0 0x7fd07006e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.171+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd08808b220 con 0x7fd09015a640 2024-09-16T01:49:09.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.171+0000 7f49bcc00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49a006c300 0x7f49a006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.172+0000 7f49c4a00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49a006c300 0x7f49a006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.172+0000 7fd08f400640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd07006c3b0 0x7fd07006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.172+0000 7f49c4a00640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49a006c300 0x7f49a006e7c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009630 tx=0x7f49b402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.173+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f49ac0892d0 con 0x7f49c00c6410 2024-09-16T01:49:09.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.173+0000 7fd08f400640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd07006c3b0 0x7fd07006e870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd080034000 tx=0x7fd08002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.173+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd088059970 con 0x7fd09015a640 2024-09-16T01:49:09.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.175+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd3cc07b440 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.178+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f49ac058db0 con 0x7f49c00c6410 2024-09-16T01:49:09.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.484+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f283c04e4b0 con 0x7f283c0d8bb0 2024-09-16T01:49:09.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.485+0000 7f283aa00640 1 -- 172.21.15.146:0/484830365 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f283005d5a0 con 0x7f283c0d8bb0 2024-09-16T01:49:09.486 INFO:teuthology.orchestra.run.smithi146.stdout:64424509447 2024-09-16T01:49:09.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.489+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f282406c4b0 msgr2=0x7f282406e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.489+0000 7f2849a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f282406c4b0 0x7f282406e970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28340016e0 tx=0x7f2834009040 comp rx=0 tx=0).stop 2024-09-16T01:49:09.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.489+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 msgr2=0x7f283c0387a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.489+0000 7f2849a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 0x7f283c0387a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f283000ec10 tx=0x7f283000c520 comp rx=0 tx=0).stop 2024-09-16T01:49:09.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.490+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 shutdown_connections 2024-09-16T01:49:09.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.490+0000 7f2849a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f282406c4b0 0x7f282406e970 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.490+0000 7f2849a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0d8bb0 0x7f283c0387a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.490+0000 7f2849a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f283c04f870 0x7f283c04d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.490+0000 7f2849a00640 1 --2- 172.21.15.146:0/484830365 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f283c007340 0x7f283c005300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.490+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 >> 172.21.15.146:0/484830365 conn(0x7f283c054470 msgr2=0x7f283c054850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.491+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 shutdown_connections 2024-09-16T01:49:09.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.491+0000 7f2849a00640 1 -- 172.21.15.146:0/484830365 wait complete. 2024-09-16T01:49:09.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2024-09-16T01:49:09.518 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.546+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd3ac005580 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.548 INFO:teuthology.orchestra.run.smithi146.stdout:64424509448 2024-09-16T01:49:09.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.547+0000 7fd3cbe00640 1 -- 172.21.15.146:0/3438665090 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd3cc07ad40 con 0x7fd3d40cc2a0 2024-09-16T01:49:09.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd3b406c2b0 msgr2=0x7fd3b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd3b406c2b0 0x7fd3b406e770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3d41e9450 tx=0x7fd3c4009190 comp rx=0 tx=0).stop 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 msgr2=0x7fd3d41e8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 0x7fd3d41e8940 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009d90 tx=0x7fd3cc004a40 comp rx=0 tx=0).stop 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 shutdown_connections 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd3b406c2b0 0x7fd3b406e770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3d40d0900 0x7fd3d4161230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3d40d0520 0x7fd3d4160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 --2- 172.21.15.146:0/3438665090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3d40cc2a0 0x7fd3d41e8940 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 >> 172.21.15.146:0/3438665090 conn(0x7fd3d406ca90 msgr2=0x7fd3d40ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.549+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 shutdown_connections 2024-09-16T01:49:09.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.550+0000 7fd3dae00640 1 -- 172.21.15.146:0/3438665090 wait complete. 2024-09-16T01:49:09.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.2 2024-09-16T01:49:09.566 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.567+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd364034b40 con 0x7fd364047f70 2024-09-16T01:49:09.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.567+0000 7fd369600640 1 -- 172.21.15.146:0/3412278783 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd36005caa0 con 0x7fd364047f70 2024-09-16T01:49:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:64424509448 2024-09-16T01:49:09.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd34c06c2e0 msgr2=0x7fd34c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd34c06c2e0 0x7fd34c06e7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd36c06b460 tx=0x7fd36c08cf90 comp rx=0 tx=0).stop 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 msgr2=0x7fd364168800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 0x7fd364168800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c3c0 tx=0x7fd36000c880 comp rx=0 tx=0).stop 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 shutdown_connections 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd34c06c2e0 0x7fd34c06e7a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd3640d8a50 0x7fd3640db160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd364047f70 0x7fd364168800 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 --2- 172.21.15.146:0/3412278783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd364007f30 0x7fd36404bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.571+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 >> 172.21.15.146:0/3412278783 conn(0x7fd36404e8f0 msgr2=0x7fd36404d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.572+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 shutdown_connections 2024-09-16T01:49:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.572+0000 7fd372a00640 1 -- 172.21.15.146:0/3412278783 wait complete. 2024-09-16T01:49:09.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.573+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f20b00c5560 con 0x7f20b00cb2f0 2024-09-16T01:49:09.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.573+0000 7f20acc00640 1 -- 172.21.15.146:0/687231920 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f20a40592a0 con 0x7f20b00cb2f0 2024-09-16T01:49:09.574 INFO:teuthology.orchestra.run.smithi146.stdout:64424509448 2024-09-16T01:49:09.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.577+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f209006c4e0 msgr2=0x7f209006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.577+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f209006c4e0 0x7f209006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20b00b3280 tx=0x7f209c007e40 comp rx=0 tx=0).stop 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.577+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 msgr2=0x7f20b014a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.577+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 0x7f20b014a3b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20a400c820 tx=0x7f20a400cc20 comp rx=0 tx=0).stop 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 shutdown_connections 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f209006c4e0 0x7f209006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20b00cc4a0 0x7f20b0149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20b00cc0c0 0x7f20b0148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 --2- 172.21.15.146:0/687231920 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20b00cb2f0 0x7f20b014a3b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 >> 172.21.15.146:0/687231920 conn(0x7f20b006d560 msgr2=0x7f20b01538c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 shutdown_connections 2024-09-16T01:49:09.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f20b7400640 1 -- 172.21.15.146:0/687231920 wait complete. 2024-09-16T01:49:09.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.578+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4824005580 con 0x7f48480cb800 2024-09-16T01:49:09.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.579+0000 7f4844c00640 1 -- 172.21.15.146:0/1923465385 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f484814b570 con 0x7f48480cb800 2024-09-16T01:49:09.580 INFO:teuthology.orchestra.run.smithi146.stdout:60129542151 2024-09-16T01:49:09.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f482806c4b0 msgr2=0x7f482806e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f482806c4b0 0x7f482806e970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f483c0094a0 tx=0x7f483c009240 comp rx=0 tx=0).stop 2024-09-16T01:49:09.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 msgr2=0x7f48480b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480b4ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4834031000 tx=0x7f48340046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 shutdown_connections 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f482806c4b0 0x7f482806e970 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f48480cd4d0 0x7f48480b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48480cb800 0x7f48480b4ba0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 --2- 172.21.15.146:0/1923465385 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48480cb420 0x7f484814b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 >> 172.21.15.146:0/1923465385 conn(0x7f484806d7c0 msgr2=0x7f48481594f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 shutdown_connections 2024-09-16T01:49:09.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.583+0000 7f484fa00640 1 -- 172.21.15.146:0/1923465385 wait complete. 2024-09-16T01:49:09.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.593+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f49c0155f80 con 0x7f49c00c6410 2024-09-16T01:49:09.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.593+0000 7f49bcc00640 1 -- 172.21.15.146:0/3809917438 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f49ac05d0b0 con 0x7f49c00c6410 2024-09-16T01:49:09.594 INFO:teuthology.orchestra.run.smithi146.stdout:60129542151 2024-09-16T01:49:09.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.3 2024-09-16T01:49:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.596+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd02c005580 con 0x7fd05406fee0 2024-09-16T01:49:09.597 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.596+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd0900c68d0 con 0x7fd09015a640 2024-09-16T01:49:09.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49a006c300 msgr2=0x7f49a006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49a006c300 0x7f49a006e7c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009630 tx=0x7f49b402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:09.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c00c6410 msgr2=0x7f49c0148680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c00c6410 0x7f49c0148680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00d5a0 tx=0x7f49ac00da60 comp rx=0 tx=0).stop 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7fd04be00640 1 -- 172.21.15.146:0/4132534520 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd04005f0d0 con 0x7fd05406fee0 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 shutdown_connections 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49a006c300 0x7f49a006e7c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49c0151a90 0x7f49c01f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49c00c6410 0x7f49c0148680 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 --2- 172.21.15.146:0/3809917438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c00c5280 0x7f49c0148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.600 INFO:teuthology.orchestra.run.smithi146.stdout:55834574855 2024-09-16T01:49:09.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 >> 172.21.15.146:0/3809917438 conn(0x7f49c006f3c0 msgr2=0x7f49c00c7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 shutdown_connections 2024-09-16T01:49:09.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7f49c7400640 1 -- 172.21.15.146:0/3809917438 wait complete. 2024-09-16T01:49:09.601 INFO:teuthology.orchestra.run.smithi146.stdout:60129542151 2024-09-16T01:49:09.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.597+0000 7fd087400640 1 -- 172.21.15.146:0/3658030072 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd088059270 con 0x7fd09015a640 2024-09-16T01:49:09.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd03406c3b0 msgr2=0x7fd03406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd03406c3b0 0x7fd03406e870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd044033000 tx=0x7fd044008030 comp rx=0 tx=0).stop 2024-09-16T01:49:09.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 msgr2=0x7fd0541f4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 0x7fd0541f4970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c4d0 tx=0x7fd04000c990 comp rx=0 tx=0).stop 2024-09-16T01:49:09.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 shutdown_connections 2024-09-16T01:49:09.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd03406c3b0 0x7fd03406e870 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd05415a790 0x7fd05414fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054150c10 0x7fd0541f8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 --2- 172.21.15.146:0/4132534520 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd05406fee0 0x7fd0541f4970 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 >> 172.21.15.146:0/4132534520 conn(0x7fd05406f3c0 msgr2=0x7fd054146800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.0 2024-09-16T01:49:09.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 shutdown_connections 2024-09-16T01:49:09.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd05b000640 1 -- 172.21.15.146:0/4132534520 wait complete. 2024-09-16T01:49:09.606 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd07006c3b0 msgr2=0x7fd07006e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd07006c3b0 0x7fd07006e870 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd080034000 tx=0x7fd08002e000 comp rx=0 tx=0).stop 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 msgr2=0x7fd0900c7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 0x7fd0900c7c40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd08800c6f0 tx=0x7fd08800cbb0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 shutdown_connections 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd07006c3b0 0x7fd07006e870 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd09015a640 0x7fd0900c7c40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0900c6290 0x7fd0900c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 --2- 172.21.15.146:0/3658030072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0900c5eb0 0x7fd0901e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.601+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 >> 172.21.15.146:0/3658030072 conn(0x7fd090067b30 msgr2=0x7fd0901546e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.602+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 shutdown_connections 2024-09-16T01:49:09.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.602+0000 7fd084c00640 1 -- 172.21.15.146:0/3658030072 wait complete. 2024-09-16T01:49:09.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.6 2024-09-16T01:49:09.608 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-09-16T01:49:09.619 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2024-09-16T01:49:09.622 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-09-16T01:49:09.624 DEBUG:teuthology.parallel:result is None 2024-09-16T01:49:09.624 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:49:09.624 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.817+0000 7f95e6e00640 1 Processor -- start 2024-09-16T01:49:09.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.817+0000 7f95e6e00640 1 -- start start 2024-09-16T01:49:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e6e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f95e015b340 0x7f95e0166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e6e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e006d140 con 0x7f95e00c6b70 2024-09-16T01:49:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e6e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e006d420 con 0x7f95e00c75a0 2024-09-16T01:49:09.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e6e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f95e00cab20 con 0x7f95e015b340 2024-09-16T01:49:09.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e5a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f95e00c6b70 0x7f95e00cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53734/0 (socket says 172.21.15.146:53734) 2024-09-16T01:49:09.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e5a00640 1 -- 172.21.15.146:0/294883365 learned_addr learned my addr 172.21.15.146:0/294883365 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:09.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.818+0000 7f95e6400640 1 --2- 172.21.15.146:0/294883365 >> v2:172.21.15.146:3300/0 conn(0x7f95e015b340 0x7f95e0166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.819+0000 7f95dfe00640 1 -- 172.21.15.146:0/294883365 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 332174058 0 0) 0x7f95e006d140 con 0x7f95e00c6b70 2024-09-16T01:49:09.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.819+0000 7f95dfe00640 1 -- 172.21.15.146:0/294883365 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95c8003520 con 0x7f95e00c6b70 2024-09-16T01:49:09.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.819+0000 7f95e6400640 1 -- 172.21.15.146:0/294883365 >> v1:172.21.15.161:6789/0 conn(0x7f95e00c75a0 legacy=0x7f95e00cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.819+0000 7f95dfe00640 1 -- 172.21.15.146:0/294883365 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1695362530 0 0) 0x7f95e006d420 con 0x7f95e00c75a0 2024-09-16T01:49:09.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.819+0000 7f95e6400640 1 -- 172.21.15.146:0/294883365 >> v1:172.21.15.146:6789/0 conn(0x7f95e00c6b70 legacy=0x7f95e00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.819+0000 7f95e6400640 1 -- 172.21.15.146:0/294883365 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e006d700 con 0x7f95e015b340 2024-09-16T01:49:09.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.820+0000 7f95e6400640 1 --2- 172.21.15.146:0/294883365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e0166170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009d90 tx=0x7f95d802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2914691651ad4fcb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.820+0000 7f95dfe00640 1 -- 172.21.15.146:0/294883365 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8007740 con 0x7f95e015b340 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.820+0000 7f95dfe00640 1 -- 172.21.15.146:0/294883365 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d80078e0 con 0x7f95e015b340 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.820+0000 7f95dfe00640 1 -- 172.21.15.146:0/294883365 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8007c90 con 0x7f95e015b340 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 -- 172.21.15.146:0/294883365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 msgr2=0x7f95e0166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 --2- 172.21.15.146:0/294883365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e0166170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f95d8009d90 tx=0x7f95d802a660 comp rx=0 tx=0).stop 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 -- 172.21.15.146:0/294883365 shutdown_connections 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 --2- 172.21.15.146:0/294883365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e0166170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 -- 172.21.15.146:0/294883365 >> 172.21.15.146:0/294883365 conn(0x7f95e00ccc20 msgr2=0x7f95e00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 -- 172.21.15.146:0/294883365 shutdown_connections 2024-09-16T01:49:09.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.821+0000 7f95e6e00640 1 -- 172.21.15.146:0/294883365 wait complete. 2024-09-16T01:49:09.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.822+0000 7f95e6e00640 1 Processor -- start 2024-09-16T01:49:09.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.822+0000 7f95e6e00640 1 -- start start 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.823+0000 7f95e6e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95e00c6b70 0x7f95e0159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.823+0000 7f95e6e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 0x7f95e015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.823+0000 7f95e6e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e015d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.823+0000 7f95e6e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f95e015b7c0 con 0x7f95e015b340 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.823+0000 7f95e6e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f95e00cb050 con 0x7f95e00c75a0 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.823+0000 7f95e6e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f95e00cbc80 con 0x7f95e00c6b70 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 0x7f95e015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95e00c6b70 0x7f95e0159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 0x7f95e015d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45362/0 (socket says 172.21.15.146:45362) 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 -- 172.21.15.146:0/2716943192 learned_addr learned my addr 172.21.15.146:0/2716943192 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e6400640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e015d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 -- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95e00c6b70 msgr2=0x7f95e0159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95e00c6b70 0x7f95e0159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 -- 172.21.15.146:0/2716943192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 msgr2=0x7f95e015d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e015d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 -- 172.21.15.146:0/2716943192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e00ce170 con 0x7f95e00c75a0 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5a00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95e00c6b70 0x7f95e0159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e6400640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e015d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.824+0000 7f95e5000640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 0x7f95e015d020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00c520 tx=0x7f95cc00c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.825+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc017070 con 0x7f95e00c75a0 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.825+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc0035f0 con 0x7f95e00c75a0 2024-09-16T01:49:09.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.825+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0039e0 con 0x7f95e00c75a0 2024-09-16T01:49:09.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.825+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e00c8800 con 0x7f95e00c75a0 2024-09-16T01:49:09.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.825+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e00ce770 con 0x7f95e00c75a0 2024-09-16T01:49:09.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.826+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e006cd20 con 0x7f95e00c75a0 2024-09-16T01:49:09.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.827+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f95cc01a070 con 0x7f95e00c75a0 2024-09-16T01:49:09.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.828+0000 7f95dd600640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95c006c2b0 0x7f95c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:09.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.828+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f95cc08a820 con 0x7f95e00c75a0 2024-09-16T01:49:09.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.828+0000 7f95e5a00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95c006c2b0 0x7f95c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:09.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.828+0000 7f95e5a00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95c006c2b0 0x7f95c006e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95d4005880 tx=0x7f95d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:09.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:09.832+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f95cc058930 con 0x7f95e00c75a0 2024-09-16T01:49:10.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.005+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95e00c9020 con 0x7f95c006c2b0 2024-09-16T01:49:10.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.012+0000 7f95dd600640 1 -- 172.21.15.146:0/2716943192 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251908 (secure 0 0 0) 0x7f95d4001800 con 0x7f95c006c2b0 2024-09-16T01:49:10.016 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95c006c2b0 msgr2=0x7f95c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95c006c2b0 0x7f95c006e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95d4005880 tx=0x7f95d4009040 comp rx=0 tx=0).stop 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 msgr2=0x7f95e015d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 0x7f95e015d020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00c520 tx=0x7f95cc00c9e0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 shutdown_connections 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95c006c2b0 0x7f95c006e770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95e015b340 0x7f95e015d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95e00c75a0 0x7f95e015d020 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 --2- 172.21.15.146:0/2716943192 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95e00c6b70 0x7f95e0159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.015+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 >> 172.21.15.146:0/2716943192 conn(0x7f95e00ccc20 msgr2=0x7f95e0154600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.016+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 shutdown_connections 2024-09-16T01:49:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.016+0000 7f95e6e00640 1 -- 172.21.15.146:0/2716943192 wait complete. 2024-09-16T01:49:10.032 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-16T01:49:08.520100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218116,"kb_used_data":1920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756604,"statfs":{"total":773094113280,"available":772870762496,"internally_reserved":0,"allocated":1966080,"data_stored":1811764,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.710939"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039277+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:49:01.039277+0000","last_peered":"2024-09-16T01:49:01.039277+0000","last_clean":"2024-09-16T01:49:01.039277+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:49:01.039277+0000","last_undegraded":"2024-09-16T01:49:01.039277+0000","last_fullsized":"2024-09-16T01:49:01.039277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039170+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:49:01.039170+0000","last_peered":"2024-09-16T01:49:01.039170+0000","last_clean":"2024-09-16T01:49:01.039170+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:49:01.039170+0000","last_undegraded":"2024-09-16T01:49:01.039170+0000","last_fullsized":"2024-09-16T01:49:01.039170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038749+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:49:01.038749+0000","last_peered":"2024-09-16T01:49:01.038749+0000","last_clean":"2024-09-16T01:49:01.038749+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:49:01.038749+0000","last_undegraded":"2024-09-16T01:49:01.038749+0000","last_fullsized":"2024-09-16T01:49:01.038749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038386+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:49:01.038386+0000","last_peered":"2024-09-16T01:49:01.038386+0000","last_clean":"2024-09-16T01:49:01.038386+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:49:01.038386+0000","last_undegraded":"2024-09-16T01:49:01.038386+0000","last_fullsized":"2024-09-16T01:49:01.038386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:09.260093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038850+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:49:01.038850+0000","last_peered":"2024-09-16T01:49:01.038850+0000","last_clean":"2024-09-16T01:49:01.038850+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:49:01.038850+0000","last_undegraded":"2024-09-16T01:49:01.038850+0000","last_fullsized":"2024-09-16T01:49:01.038850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824333+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:59.824333+0000","last_peered":"2024-09-16T01:48:59.824333+0000","last_clean":"2024-09-16T01:48:59.824333+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:59.824333+0000","last_undegraded":"2024-09-16T01:48:59.824333+0000","last_fullsized":"2024-09-16T01:48:59.824333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:54.394499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038902+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:49:01.038902+0000","last_peered":"2024-09-16T01:49:01.038902+0000","last_clean":"2024-09-16T01:49:01.038902+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:49:01.038902+0000","last_undegraded":"2024-09-16T01:49:01.038902+0000","last_fullsized":"2024-09-16T01:49:01.038902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039312+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:49:01.039312+0000","last_peered":"2024-09-16T01:49:01.039312+0000","last_clean":"2024-09-16T01:49:01.039312+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:49:01.039312+0000","last_undegraded":"2024-09-16T01:49:01.039312+0000","last_fullsized":"2024-09-16T01:49:01.039312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038909+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:49:01.038909+0000","last_peered":"2024-09-16T01:49:01.038909+0000","last_clean":"2024-09-16T01:49:01.038909+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:49:01.038909+0000","last_undegraded":"2024-09-16T01:49:01.038909+0000","last_fullsized":"2024-09-16T01:49:01.038909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038603+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:49:01.038603+0000","last_peered":"2024-09-16T01:49:01.038603+0000","last_clean":"2024-09-16T01:49:01.038603+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:49:01.038603+0000","last_undegraded":"2024-09-16T01:49:01.038603+0000","last_fullsized":"2024-09-16T01:49:01.038603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039081+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:49:01.039081+0000","last_peered":"2024-09-16T01:49:01.039081+0000","last_clean":"2024-09-16T01:49:01.039081+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:49:01.039081+0000","last_undegraded":"2024-09-16T01:49:01.039081+0000","last_fullsized":"2024-09-16T01:49:01.039081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824444+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:59.824444+0000","last_peered":"2024-09-16T01:48:59.824444+0000","last_clean":"2024-09-16T01:48:59.824444+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:59.824444+0000","last_undegraded":"2024-09-16T01:48:59.824444+0000","last_fullsized":"2024-09-16T01:48:59.824444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:21.869634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.543617+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:01.543617+0000","last_peered":"2024-09-16T01:49:01.543617+0000","last_clean":"2024-09-16T01:49:01.543617+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:01.543617+0000","last_undegraded":"2024-09-16T01:49:01.543617+0000","last_fullsized":"2024-09-16T01:49:01.543617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825347+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:48:59.825347+0000","last_peered":"2024-09-16T01:48:59.825347+0000","last_clean":"2024-09-16T01:48:59.825347+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:48:59.825347+0000","last_undegraded":"2024-09-16T01:48:59.825347+0000","last_fullsized":"2024-09-16T01:48:59.825347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038968+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:49:01.038968+0000","last_peered":"2024-09-16T01:49:01.038968+0000","last_clean":"2024-09-16T01:49:01.038968+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:49:01.038968+0000","last_undegraded":"2024-09-16T01:49:01.038968+0000","last_fullsized":"2024-09-16T01:49:01.038968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038521+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:01.038521+0000","last_peered":"2024-09-16T01:49:01.038521+0000","last_clean":"2024-09-16T01:49:01.038521+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:01.038521+0000","last_undegraded":"2024-09-16T01:49:01.038521+0000","last_fullsized":"2024-09-16T01:49:01.038521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824266+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:59.824266+0000","last_peered":"2024-09-16T01:48:59.824266+0000","last_clean":"2024-09-16T01:48:59.824266+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:59.824266+0000","last_undegraded":"2024-09-16T01:48:59.824266+0000","last_fullsized":"2024-09-16T01:48:59.824266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:31.622539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040713+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:01.040713+0000","last_peered":"2024-09-16T01:49:01.040713+0000","last_clean":"2024-09-16T01:49:01.040713+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:01.040713+0000","last_undegraded":"2024-09-16T01:49:01.040713+0000","last_fullsized":"2024-09-16T01:49:01.040713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039489+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:01.039489+0000","last_peered":"2024-09-16T01:49:01.039489+0000","last_clean":"2024-09-16T01:49:01.039489+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:01.039489+0000","last_undegraded":"2024-09-16T01:49:01.039489+0000","last_fullsized":"2024-09-16T01:49:01.039489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039431+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:49:01.039431+0000","last_peered":"2024-09-16T01:49:01.039431+0000","last_clean":"2024-09-16T01:49:01.039431+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:49:01.039431+0000","last_undegraded":"2024-09-16T01:49:01.039431+0000","last_fullsized":"2024-09-16T01:49:01.039431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.830468+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:49:03.830468+0000","last_peered":"2024-09-16T01:49:03.830468+0000","last_clean":"2024-09-16T01:49:03.830468+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:49:03.830468+0000","last_undegraded":"2024-09-16T01:49:03.830468+0000","last_fullsized":"2024-09-16T01:49:03.830468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.834087+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:49:03.834087+0000","last_peered":"2024-09-16T01:49:03.834087+0000","last_clean":"2024-09-16T01:49:03.834087+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:49:03.834087+0000","last_undegraded":"2024-09-16T01:49:03.834087+0000","last_fullsized":"2024-09-16T01:49:03.834087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.832229+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:49:03.832229+0000","last_peered":"2024-09-16T01:49:03.832229+0000","last_clean":"2024-09-16T01:49:03.832229+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:49:03.832229+0000","last_undegraded":"2024-09-16T01:49:03.832229+0000","last_fullsized":"2024-09-16T01:49:03.832229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039468+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:01.039468+0000","last_peered":"2024-09-16T01:49:01.039468+0000","last_clean":"2024-09-16T01:49:01.039468+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:01.039468+0000","last_undegraded":"2024-09-16T01:49:01.039468+0000","last_fullsized":"2024-09-16T01:49:01.039468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038796+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:01.038796+0000","last_peered":"2024-09-16T01:49:01.038796+0000","last_clean":"2024-09-16T01:49:01.038796+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:01.038796+0000","last_undegraded":"2024-09-16T01:49:01.038796+0000","last_fullsized":"2024-09-16T01:49:01.038796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.831520+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:49:03.831520+0000","last_peered":"2024-09-16T01:49:03.831520+0000","last_clean":"2024-09-16T01:49:03.831520+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:49:03.831520+0000","last_undegraded":"2024-09-16T01:49:03.831520+0000","last_fullsized":"2024-09-16T01:49:03.831520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.542455+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:01.542455+0000","last_peered":"2024-09-16T01:49:01.542455+0000","last_clean":"2024-09-16T01:49:01.542455+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:01.542455+0000","last_undegraded":"2024-09-16T01:49:01.542455+0000","last_fullsized":"2024-09-16T01:49:01.542455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829853+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:49:03.829853+0000","last_peered":"2024-09-16T01:49:03.829853+0000","last_clean":"2024-09-16T01:49:03.829853+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:49:03.829853+0000","last_undegraded":"2024-09-16T01:49:03.829853+0000","last_fullsized":"2024-09-16T01:49:03.829853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039188+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:49:01.039188+0000","last_peered":"2024-09-16T01:49:01.039188+0000","last_clean":"2024-09-16T01:49:01.039188+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:49:01.039188+0000","last_undegraded":"2024-09-16T01:49:01.039188+0000","last_fullsized":"2024-09-16T01:49:01.039188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825768+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:48:59.825768+0000","last_peered":"2024-09-16T01:48:59.825768+0000","last_clean":"2024-09-16T01:48:59.825768+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:48:59.825768+0000","last_undegraded":"2024-09-16T01:48:59.825768+0000","last_fullsized":"2024-09-16T01:48:59.825768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039827+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:01.039827+0000","last_peered":"2024-09-16T01:49:01.039827+0000","last_clean":"2024-09-16T01:49:01.039827+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:01.039827+0000","last_undegraded":"2024-09-16T01:49:01.039827+0000","last_fullsized":"2024-09-16T01:49:01.039827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824354+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:59.824354+0000","last_peered":"2024-09-16T01:48:59.824354+0000","last_clean":"2024-09-16T01:48:59.824354+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:59.824354+0000","last_undegraded":"2024-09-16T01:48:59.824354+0000","last_fullsized":"2024-09-16T01:48:59.824354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:03.840176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038630+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:01.038630+0000","last_peered":"2024-09-16T01:49:01.038630+0000","last_clean":"2024-09-16T01:49:01.038630+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:01.038630+0000","last_undegraded":"2024-09-16T01:49:01.038630+0000","last_fullsized":"2024-09-16T01:49:01.038630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038252+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:49:01.038252+0000","last_peered":"2024-09-16T01:49:01.038252+0000","last_clean":"2024-09-16T01:49:01.038252+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:49:01.038252+0000","last_undegraded":"2024-09-16T01:49:01.038252+0000","last_fullsized":"2024-09-16T01:49:01.038252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038525+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:49:01.038525+0000","last_peered":"2024-09-16T01:49:01.038525+0000","last_clean":"2024-09-16T01:49:01.038525+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:49:01.038525+0000","last_undegraded":"2024-09-16T01:49:01.038525+0000","last_fullsized":"2024-09-16T01:49:01.038525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:20.864782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039485+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:01.039485+0000","last_peered":"2024-09-16T01:49:01.039485+0000","last_clean":"2024-09-16T01:49:01.039485+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:01.039485+0000","last_undegraded":"2024-09-16T01:49:01.039485+0000","last_fullsized":"2024-09-16T01:49:01.039485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039101+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:49:01.039101+0000","last_peered":"2024-09-16T01:49:01.039101+0000","last_clean":"2024-09-16T01:49:01.039101+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:49:01.039101+0000","last_undegraded":"2024-09-16T01:49:01.039101+0000","last_fullsized":"2024-09-16T01:49:01.039101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040340+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:01.040340+0000","last_peered":"2024-09-16T01:49:01.040340+0000","last_clean":"2024-09-16T01:49:01.040340+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:01.040340+0000","last_undegraded":"2024-09-16T01:49:01.040340+0000","last_fullsized":"2024-09-16T01:49:01.040340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824783+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:48:59.824783+0000","last_peered":"2024-09-16T01:48:59.824783+0000","last_clean":"2024-09-16T01:48:59.824783+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:48:59.824783+0000","last_undegraded":"2024-09-16T01:48:59.824783+0000","last_fullsized":"2024-09-16T01:48:59.824783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824807+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:59.824807+0000","last_peered":"2024-09-16T01:48:59.824807+0000","last_clean":"2024-09-16T01:48:59.824807+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:59.824807+0000","last_undegraded":"2024-09-16T01:48:59.824807+0000","last_fullsized":"2024-09-16T01:48:59.824807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:08:09.528402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829883+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:03.829883+0000","last_peered":"2024-09-16T01:49:03.829883+0000","last_clean":"2024-09-16T01:49:03.829883+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:03.829883+0000","last_undegraded":"2024-09-16T01:49:03.829883+0000","last_fullsized":"2024-09-16T01:49:03.829883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824748+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:59.824748+0000","last_peered":"2024-09-16T01:48:59.824748+0000","last_clean":"2024-09-16T01:48:59.824748+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:59.824748+0000","last_undegraded":"2024-09-16T01:48:59.824748+0000","last_fullsized":"2024-09-16T01:48:59.824748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:51:26.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038599+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:01.038599+0000","last_peered":"2024-09-16T01:49:01.038599+0000","last_clean":"2024-09-16T01:49:01.038599+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:01.038599+0000","last_undegraded":"2024-09-16T01:49:01.038599+0000","last_fullsized":"2024-09-16T01:49:01.038599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824219+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:48:59.824219+0000","last_peered":"2024-09-16T01:48:59.824219+0000","last_clean":"2024-09-16T01:48:59.824219+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:48:59.824219+0000","last_undegraded":"2024-09-16T01:48:59.824219+0000","last_fullsized":"2024-09-16T01:48:59.824219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824251+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:59.824251+0000","last_peered":"2024-09-16T01:48:59.824251+0000","last_clean":"2024-09-16T01:48:59.824251+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:59.824251+0000","last_undegraded":"2024-09-16T01:48:59.824251+0000","last_fullsized":"2024-09-16T01:48:59.824251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:26.577467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825690+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:48:59.825690+0000","last_peered":"2024-09-16T01:48:59.825690+0000","last_clean":"2024-09-16T01:48:59.825690+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:48:59.825690+0000","last_undegraded":"2024-09-16T01:48:59.825690+0000","last_fullsized":"2024-09-16T01:48:59.825690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038826+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:49:01.038826+0000","last_peered":"2024-09-16T01:49:01.038826+0000","last_clean":"2024-09-16T01:49:01.038826+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:49:01.038826+0000","last_undegraded":"2024-09-16T01:49:01.038826+0000","last_fullsized":"2024-09-16T01:49:01.038826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.834393+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:03.834393+0000","last_peered":"2024-09-16T01:49:03.834393+0000","last_clean":"2024-09-16T01:49:03.834393+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:03.834393+0000","last_undegraded":"2024-09-16T01:49:03.834393+0000","last_fullsized":"2024-09-16T01:49:03.834393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.832484+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:03.832484+0000","last_peered":"2024-09-16T01:49:03.832484+0000","last_clean":"2024-09-16T01:49:03.832484+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:03.832484+0000","last_undegraded":"2024-09-16T01:49:03.832484+0000","last_fullsized":"2024-09-16T01:49:03.832484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.832458+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:03.832458+0000","last_peered":"2024-09-16T01:49:03.832458+0000","last_clean":"2024-09-16T01:49:03.832458+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:03.832458+0000","last_undegraded":"2024-09-16T01:49:03.832458+0000","last_fullsized":"2024-09-16T01:49:03.832458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039335+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:49:01.039335+0000","last_peered":"2024-09-16T01:49:01.039335+0000","last_clean":"2024-09-16T01:49:01.039335+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:49:01.039335+0000","last_undegraded":"2024-09-16T01:49:01.039335+0000","last_fullsized":"2024-09-16T01:49:01.039335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039461+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:49:01.039461+0000","last_peered":"2024-09-16T01:49:01.039461+0000","last_clean":"2024-09-16T01:49:01.039461+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:49:01.039461+0000","last_undegraded":"2024-09-16T01:49:01.039461+0000","last_fullsized":"2024-09-16T01:49:01.039461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039274+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:49:01.039274+0000","last_peered":"2024-09-16T01:49:01.039274+0000","last_clean":"2024-09-16T01:49:01.039274+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:49:01.039274+0000","last_undegraded":"2024-09-16T01:49:01.039274+0000","last_fullsized":"2024-09-16T01:49:01.039274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.834294+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:49:03.834294+0000","last_peered":"2024-09-16T01:49:03.834294+0000","last_clean":"2024-09-16T01:49:03.834294+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:49:03.834294+0000","last_undegraded":"2024-09-16T01:49:03.834294+0000","last_fullsized":"2024-09-16T01:49:03.834294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":78858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":79888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":668714,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":668736,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":80560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:10.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:10.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.225+0000 7f2cea800640 1 Processor -- start 2024-09-16T01:49:10.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.225+0000 7f2cea800640 1 -- start start 2024-09-16T01:49:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2cea800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cb2f0 0x7f2ce415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2cea800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ce40cc4a0 0x7f2ce4165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2cea800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce415b1d0 con 0x7f2ce40cc0c0 2024-09-16T01:49:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2cea800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f2ce40cbd00 con 0x7f2ce40cc4a0 2024-09-16T01:49:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2cea800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce415ba70 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2ce3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ce40cc4a0 0x7f2ce4165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2ce3400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2ce40cc0c0 0x7f2ce415e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53742/0 (socket says 172.21.15.146:53742) 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.226+0000 7f2ce3400640 1 -- 172.21.15.146:0/3564390273 learned_addr learned my addr 172.21.15.146:0/3564390273 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce2a00640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cb2f0 0x7f2ce415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce3e00640 1 -- 172.21.15.146:0/3564390273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cb2f0 msgr2=0x7f2ce415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce3e00640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cb2f0 0x7f2ce415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce3e00640 1 -- 172.21.15.146:0/3564390273 >> v1:172.21.15.146:6789/0 conn(0x7f2ce40cc0c0 legacy=0x7f2ce415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce3e00640 1 -- 172.21.15.146:0/3564390273 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce415b4b0 con 0x7f2ce40cc4a0 2024-09-16T01:49:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce2000640 1 -- 172.21.15.146:0/3564390273 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714242935 0 0) 0x7f2ce415b1d0 con 0x7f2ce40cc0c0 2024-09-16T01:49:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.227+0000 7f2ce2a00640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cb2f0 0x7f2ce415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.228+0000 7f2ce3e00640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce4165430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009d90 tx=0x7f2cdc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5654a5a1c53b25fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.228+0000 7f2ce2000640 1 -- 172.21.15.146:0/3564390273 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc007740 con 0x7f2ce40cc4a0 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.228+0000 7f2ce2000640 1 -- 172.21.15.146:0/3564390273 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0078e0 con 0x7f2ce40cc4a0 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.228+0000 7f2ce2000640 1 -- 172.21.15.146:0/3564390273 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc007cb0 con 0x7f2ce40cc4a0 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.228+0000 7f2cea800640 1 -- 172.21.15.146:0/3564390273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cc4a0 msgr2=0x7f2ce4165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.228+0000 7f2cea800640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce4165430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009d90 tx=0x7f2cdc02a660 comp rx=0 tx=0).stop 2024-09-16T01:49:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.229+0000 7f2cea800640 1 -- 172.21.15.146:0/3564390273 shutdown_connections 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.229+0000 7f2cea800640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce4165430 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.229+0000 7f2cea800640 1 --2- 172.21.15.146:0/3564390273 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cb2f0 0x7f2ce415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.229+0000 7f2cea800640 1 -- 172.21.15.146:0/3564390273 >> 172.21.15.146:0/3564390273 conn(0x7f2ce406d560 msgr2=0x7f2ce406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.229+0000 7f2cea800640 1 -- 172.21.15.146:0/3564390273 shutdown_connections 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.229+0000 7f2cea800640 1 -- 172.21.15.146:0/3564390273 wait complete. 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.230+0000 7f2cea800640 1 Processor -- start 2024-09-16T01:49:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.230+0000 7f2cea800640 1 -- start start 2024-09-16T01:49:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2cea800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 0x7f2ce41f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2cea800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cc0c0 0x7f2ce41f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2cea800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce41f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2cea800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce415e1d0 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2cea800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce415e540 con 0x7f2ce40cc4a0 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2cea800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce40cc8b0 con 0x7f2ce40cc0c0 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2ce3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 0x7f2ce41f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2ce3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 0x7f2ce41f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59380/0 (socket says 172.21.15.146:59380) 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.231+0000 7f2ce3400640 1 -- 172.21.15.146:0/842224790 learned_addr learned my addr 172.21.15.146:0/842224790 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3e00640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce41f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce2a00640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cc0c0 0x7f2ce41f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3400640 1 -- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cc0c0 msgr2=0x7f2ce41f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3400640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cc0c0 0x7f2ce41f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3400640 1 -- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce40cc4a0 msgr2=0x7f2ce41f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3400640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce41f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3400640 1 -- 172.21.15.146:0/842224790 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce414d4e0 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3e00640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce41f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce2a00640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cc0c0 0x7f2ce41f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2ce3400640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 0x7f2ce41f17d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400c500 tx=0x7f2cd400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4018070 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd4007580 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce414d7a0 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.232+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4004a20 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.233+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce414dd60 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.234+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce40c5560 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.234+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2cd401b070 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.235+0000 7f2cdb400640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cc406c2b0 0x7f2cc406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.235+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2cd408a6e0 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.236+0000 7f2ce2a00640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cc406c2b0 0x7f2cc406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.236+0000 7f2ce2a00640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cc406c2b0 0x7f2cc406e770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ce41f47e0 tx=0x7f2cd0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:10.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.240+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2cd4058ee0 con 0x7f2ce40cb2f0 2024-09-16T01:49:10.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.409+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ce40cbea0 con 0x7f2cc406c2b0 2024-09-16T01:49:10.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.417+0000 7f2cdb400640 1 -- 172.21.15.146:0/842224790 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251908 (secure 0 0 0) 0x7f2ce40cbea0 con 0x7f2cc406c2b0 2024-09-16T01:49:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:10.422 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:10.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cc406c2b0 msgr2=0x7f2cc406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cc406c2b0 0x7f2cc406e770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ce41f47e0 tx=0x7f2cd0008040 comp rx=0 tx=0).stop 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 msgr2=0x7f2ce41f17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 0x7f2ce41f17d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400c500 tx=0x7f2cd400c9c0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 shutdown_connections 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cc406c2b0 0x7f2cc406e770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce40cc4a0 0x7f2ce41f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ce40cc0c0 0x7f2ce41f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 --2- 172.21.15.146:0/842224790 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ce40cb2f0 0x7f2ce41f17d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 >> 172.21.15.146:0/842224790 conn(0x7f2ce406d560 msgr2=0x7f2ce41530c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:10.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 shutdown_connections 2024-09-16T01:49:10.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.420+0000 7f2cea800640 1 -- 172.21.15.146:0/842224790 wait complete. 2024-09-16T01:49:10.437 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-16T01:49:08.520100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218116,"kb_used_data":1920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756604,"statfs":{"total":773094113280,"available":772870762496,"internally_reserved":0,"allocated":1966080,"data_stored":1811764,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.710939"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039277+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:49:01.039277+0000","last_peered":"2024-09-16T01:49:01.039277+0000","last_clean":"2024-09-16T01:49:01.039277+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:49:01.039277+0000","last_undegraded":"2024-09-16T01:49:01.039277+0000","last_fullsized":"2024-09-16T01:49:01.039277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039170+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:49:01.039170+0000","last_peered":"2024-09-16T01:49:01.039170+0000","last_clean":"2024-09-16T01:49:01.039170+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:49:01.039170+0000","last_undegraded":"2024-09-16T01:49:01.039170+0000","last_fullsized":"2024-09-16T01:49:01.039170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038749+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:49:01.038749+0000","last_peered":"2024-09-16T01:49:01.038749+0000","last_clean":"2024-09-16T01:49:01.038749+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:49:01.038749+0000","last_undegraded":"2024-09-16T01:49:01.038749+0000","last_fullsized":"2024-09-16T01:49:01.038749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038386+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:49:01.038386+0000","last_peered":"2024-09-16T01:49:01.038386+0000","last_clean":"2024-09-16T01:49:01.038386+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:49:01.038386+0000","last_undegraded":"2024-09-16T01:49:01.038386+0000","last_fullsized":"2024-09-16T01:49:01.038386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:09.260093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038850+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:49:01.038850+0000","last_peered":"2024-09-16T01:49:01.038850+0000","last_clean":"2024-09-16T01:49:01.038850+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:49:01.038850+0000","last_undegraded":"2024-09-16T01:49:01.038850+0000","last_fullsized":"2024-09-16T01:49:01.038850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824333+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:59.824333+0000","last_peered":"2024-09-16T01:48:59.824333+0000","last_clean":"2024-09-16T01:48:59.824333+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:59.824333+0000","last_undegraded":"2024-09-16T01:48:59.824333+0000","last_fullsized":"2024-09-16T01:48:59.824333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:54.394499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038902+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:49:01.038902+0000","last_peered":"2024-09-16T01:49:01.038902+0000","last_clean":"2024-09-16T01:49:01.038902+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:49:01.038902+0000","last_undegraded":"2024-09-16T01:49:01.038902+0000","last_fullsized":"2024-09-16T01:49:01.038902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039312+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:49:01.039312+0000","last_peered":"2024-09-16T01:49:01.039312+0000","last_clean":"2024-09-16T01:49:01.039312+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:49:01.039312+0000","last_undegraded":"2024-09-16T01:49:01.039312+0000","last_fullsized":"2024-09-16T01:49:01.039312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038909+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:49:01.038909+0000","last_peered":"2024-09-16T01:49:01.038909+0000","last_clean":"2024-09-16T01:49:01.038909+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:49:01.038909+0000","last_undegraded":"2024-09-16T01:49:01.038909+0000","last_fullsized":"2024-09-16T01:49:01.038909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038603+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:49:01.038603+0000","last_peered":"2024-09-16T01:49:01.038603+0000","last_clean":"2024-09-16T01:49:01.038603+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:49:01.038603+0000","last_undegraded":"2024-09-16T01:49:01.038603+0000","last_fullsized":"2024-09-16T01:49:01.038603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039081+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:49:01.039081+0000","last_peered":"2024-09-16T01:49:01.039081+0000","last_clean":"2024-09-16T01:49:01.039081+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:49:01.039081+0000","last_undegraded":"2024-09-16T01:49:01.039081+0000","last_fullsized":"2024-09-16T01:49:01.039081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824444+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:59.824444+0000","last_peered":"2024-09-16T01:48:59.824444+0000","last_clean":"2024-09-16T01:48:59.824444+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:59.824444+0000","last_undegraded":"2024-09-16T01:48:59.824444+0000","last_fullsized":"2024-09-16T01:48:59.824444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:21.869634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.543617+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:01.543617+0000","last_peered":"2024-09-16T01:49:01.543617+0000","last_clean":"2024-09-16T01:49:01.543617+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:01.543617+0000","last_undegraded":"2024-09-16T01:49:01.543617+0000","last_fullsized":"2024-09-16T01:49:01.543617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825347+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:48:59.825347+0000","last_peered":"2024-09-16T01:48:59.825347+0000","last_clean":"2024-09-16T01:48:59.825347+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:48:59.825347+0000","last_undegraded":"2024-09-16T01:48:59.825347+0000","last_fullsized":"2024-09-16T01:48:59.825347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038968+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:49:01.038968+0000","last_peered":"2024-09-16T01:49:01.038968+0000","last_clean":"2024-09-16T01:49:01.038968+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:49:01.038968+0000","last_undegraded":"2024-09-16T01:49:01.038968+0000","last_fullsized":"2024-09-16T01:49:01.038968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038521+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:01.038521+0000","last_peered":"2024-09-16T01:49:01.038521+0000","last_clean":"2024-09-16T01:49:01.038521+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:01.038521+0000","last_undegraded":"2024-09-16T01:49:01.038521+0000","last_fullsized":"2024-09-16T01:49:01.038521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824266+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:59.824266+0000","last_peered":"2024-09-16T01:48:59.824266+0000","last_clean":"2024-09-16T01:48:59.824266+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:59.824266+0000","last_undegraded":"2024-09-16T01:48:59.824266+0000","last_fullsized":"2024-09-16T01:48:59.824266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:31.622539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040713+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:01.040713+0000","last_peered":"2024-09-16T01:49:01.040713+0000","last_clean":"2024-09-16T01:49:01.040713+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:01.040713+0000","last_undegraded":"2024-09-16T01:49:01.040713+0000","last_fullsized":"2024-09-16T01:49:01.040713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039489+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:01.039489+0000","last_peered":"2024-09-16T01:49:01.039489+0000","last_clean":"2024-09-16T01:49:01.039489+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:01.039489+0000","last_undegraded":"2024-09-16T01:49:01.039489+0000","last_fullsized":"2024-09-16T01:49:01.039489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039431+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:49:01.039431+0000","last_peered":"2024-09-16T01:49:01.039431+0000","last_clean":"2024-09-16T01:49:01.039431+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:49:01.039431+0000","last_undegraded":"2024-09-16T01:49:01.039431+0000","last_fullsized":"2024-09-16T01:49:01.039431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.830468+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:49:03.830468+0000","last_peered":"2024-09-16T01:49:03.830468+0000","last_clean":"2024-09-16T01:49:03.830468+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:49:03.830468+0000","last_undegraded":"2024-09-16T01:49:03.830468+0000","last_fullsized":"2024-09-16T01:49:03.830468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.834087+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:49:03.834087+0000","last_peered":"2024-09-16T01:49:03.834087+0000","last_clean":"2024-09-16T01:49:03.834087+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:49:03.834087+0000","last_undegraded":"2024-09-16T01:49:03.834087+0000","last_fullsized":"2024-09-16T01:49:03.834087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.832229+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:49:03.832229+0000","last_peered":"2024-09-16T01:49:03.832229+0000","last_clean":"2024-09-16T01:49:03.832229+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:49:03.832229+0000","last_undegraded":"2024-09-16T01:49:03.832229+0000","last_fullsized":"2024-09-16T01:49:03.832229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039468+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:01.039468+0000","last_peered":"2024-09-16T01:49:01.039468+0000","last_clean":"2024-09-16T01:49:01.039468+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:01.039468+0000","last_undegraded":"2024-09-16T01:49:01.039468+0000","last_fullsized":"2024-09-16T01:49:01.039468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038796+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:01.038796+0000","last_peered":"2024-09-16T01:49:01.038796+0000","last_clean":"2024-09-16T01:49:01.038796+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:01.038796+0000","last_undegraded":"2024-09-16T01:49:01.038796+0000","last_fullsized":"2024-09-16T01:49:01.038796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.831520+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:49:03.831520+0000","last_peered":"2024-09-16T01:49:03.831520+0000","last_clean":"2024-09-16T01:49:03.831520+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:49:03.831520+0000","last_undegraded":"2024-09-16T01:49:03.831520+0000","last_fullsized":"2024-09-16T01:49:03.831520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.542455+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:01.542455+0000","last_peered":"2024-09-16T01:49:01.542455+0000","last_clean":"2024-09-16T01:49:01.542455+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:01.542455+0000","last_undegraded":"2024-09-16T01:49:01.542455+0000","last_fullsized":"2024-09-16T01:49:01.542455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829853+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:49:03.829853+0000","last_peered":"2024-09-16T01:49:03.829853+0000","last_clean":"2024-09-16T01:49:03.829853+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:49:03.829853+0000","last_undegraded":"2024-09-16T01:49:03.829853+0000","last_fullsized":"2024-09-16T01:49:03.829853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039188+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:49:01.039188+0000","last_peered":"2024-09-16T01:49:01.039188+0000","last_clean":"2024-09-16T01:49:01.039188+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:49:01.039188+0000","last_undegraded":"2024-09-16T01:49:01.039188+0000","last_fullsized":"2024-09-16T01:49:01.039188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825768+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:48:59.825768+0000","last_peered":"2024-09-16T01:48:59.825768+0000","last_clean":"2024-09-16T01:48:59.825768+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:48:59.825768+0000","last_undegraded":"2024-09-16T01:48:59.825768+0000","last_fullsized":"2024-09-16T01:48:59.825768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039827+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:01.039827+0000","last_peered":"2024-09-16T01:49:01.039827+0000","last_clean":"2024-09-16T01:49:01.039827+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:01.039827+0000","last_undegraded":"2024-09-16T01:49:01.039827+0000","last_fullsized":"2024-09-16T01:49:01.039827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824354+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:59.824354+0000","last_peered":"2024-09-16T01:48:59.824354+0000","last_clean":"2024-09-16T01:48:59.824354+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:59.824354+0000","last_undegraded":"2024-09-16T01:48:59.824354+0000","last_fullsized":"2024-09-16T01:48:59.824354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:03.840176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038630+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:01.038630+0000","last_peered":"2024-09-16T01:49:01.038630+0000","last_clean":"2024-09-16T01:49:01.038630+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:01.038630+0000","last_undegraded":"2024-09-16T01:49:01.038630+0000","last_fullsized":"2024-09-16T01:49:01.038630+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038252+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:49:01.038252+0000","last_peered":"2024-09-16T01:49:01.038252+0000","last_clean":"2024-09-16T01:49:01.038252+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:49:01.038252+0000","last_undegraded":"2024-09-16T01:49:01.038252+0000","last_fullsized":"2024-09-16T01:49:01.038252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038525+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:49:01.038525+0000","last_peered":"2024-09-16T01:49:01.038525+0000","last_clean":"2024-09-16T01:49:01.038525+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:49:01.038525+0000","last_undegraded":"2024-09-16T01:49:01.038525+0000","last_fullsized":"2024-09-16T01:49:01.038525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:20.864782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039485+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:01.039485+0000","last_peered":"2024-09-16T01:49:01.039485+0000","last_clean":"2024-09-16T01:49:01.039485+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:01.039485+0000","last_undegraded":"2024-09-16T01:49:01.039485+0000","last_fullsized":"2024-09-16T01:49:01.039485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039101+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:49:01.039101+0000","last_peered":"2024-09-16T01:49:01.039101+0000","last_clean":"2024-09-16T01:49:01.039101+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:49:01.039101+0000","last_undegraded":"2024-09-16T01:49:01.039101+0000","last_fullsized":"2024-09-16T01:49:01.039101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040340+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:01.040340+0000","last_peered":"2024-09-16T01:49:01.040340+0000","last_clean":"2024-09-16T01:49:01.040340+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:01.040340+0000","last_undegraded":"2024-09-16T01:49:01.040340+0000","last_fullsized":"2024-09-16T01:49:01.040340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824783+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:48:59.824783+0000","last_peered":"2024-09-16T01:48:59.824783+0000","last_clean":"2024-09-16T01:48:59.824783+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:48:59.824783+0000","last_undegraded":"2024-09-16T01:48:59.824783+0000","last_fullsized":"2024-09-16T01:48:59.824783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824807+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:59.824807+0000","last_peered":"2024-09-16T01:48:59.824807+0000","last_clean":"2024-09-16T01:48:59.824807+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:59.824807+0000","last_undegraded":"2024-09-16T01:48:59.824807+0000","last_fullsized":"2024-09-16T01:48:59.824807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:08:09.528402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829883+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:03.829883+0000","last_peered":"2024-09-16T01:49:03.829883+0000","last_clean":"2024-09-16T01:49:03.829883+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:03.829883+0000","last_undegraded":"2024-09-16T01:49:03.829883+0000","last_fullsized":"2024-09-16T01:49:03.829883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824748+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:59.824748+0000","last_peered":"2024-09-16T01:48:59.824748+0000","last_clean":"2024-09-16T01:48:59.824748+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:59.824748+0000","last_undegraded":"2024-09-16T01:48:59.824748+0000","last_fullsized":"2024-09-16T01:48:59.824748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:51:26.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038599+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:01.038599+0000","last_peered":"2024-09-16T01:49:01.038599+0000","last_clean":"2024-09-16T01:49:01.038599+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:01.038599+0000","last_undegraded":"2024-09-16T01:49:01.038599+0000","last_fullsized":"2024-09-16T01:49:01.038599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824219+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:48:59.824219+0000","last_peered":"2024-09-16T01:48:59.824219+0000","last_clean":"2024-09-16T01:48:59.824219+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:48:59.824219+0000","last_undegraded":"2024-09-16T01:48:59.824219+0000","last_fullsized":"2024-09-16T01:48:59.824219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824251+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:59.824251+0000","last_peered":"2024-09-16T01:48:59.824251+0000","last_clean":"2024-09-16T01:48:59.824251+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:59.824251+0000","last_undegraded":"2024-09-16T01:48:59.824251+0000","last_fullsized":"2024-09-16T01:48:59.824251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:26.577467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825690+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:48:59.825690+0000","last_peered":"2024-09-16T01:48:59.825690+0000","last_clean":"2024-09-16T01:48:59.825690+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:48:59.825690+0000","last_undegraded":"2024-09-16T01:48:59.825690+0000","last_fullsized":"2024-09-16T01:48:59.825690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038826+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:49:01.038826+0000","last_peered":"2024-09-16T01:49:01.038826+0000","last_clean":"2024-09-16T01:49:01.038826+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:49:01.038826+0000","last_undegraded":"2024-09-16T01:49:01.038826+0000","last_fullsized":"2024-09-16T01:49:01.038826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.834393+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:03.834393+0000","last_peered":"2024-09-16T01:49:03.834393+0000","last_clean":"2024-09-16T01:49:03.834393+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:03.834393+0000","last_undegraded":"2024-09-16T01:49:03.834393+0000","last_fullsized":"2024-09-16T01:49:03.834393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.832484+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:03.832484+0000","last_peered":"2024-09-16T01:49:03.832484+0000","last_clean":"2024-09-16T01:49:03.832484+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:03.832484+0000","last_undegraded":"2024-09-16T01:49:03.832484+0000","last_fullsized":"2024-09-16T01:49:03.832484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.832458+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:03.832458+0000","last_peered":"2024-09-16T01:49:03.832458+0000","last_clean":"2024-09-16T01:49:03.832458+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:03.832458+0000","last_undegraded":"2024-09-16T01:49:03.832458+0000","last_fullsized":"2024-09-16T01:49:03.832458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039335+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:49:01.039335+0000","last_peered":"2024-09-16T01:49:01.039335+0000","last_clean":"2024-09-16T01:49:01.039335+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:49:01.039335+0000","last_undegraded":"2024-09-16T01:49:01.039335+0000","last_fullsized":"2024-09-16T01:49:01.039335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039461+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:49:01.039461+0000","last_peered":"2024-09-16T01:49:01.039461+0000","last_clean":"2024-09-16T01:49:01.039461+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:49:01.039461+0000","last_undegraded":"2024-09-16T01:49:01.039461+0000","last_fullsized":"2024-09-16T01:49:01.039461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039274+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:49:01.039274+0000","last_peered":"2024-09-16T01:49:01.039274+0000","last_clean":"2024-09-16T01:49:01.039274+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:49:01.039274+0000","last_undegraded":"2024-09-16T01:49:01.039274+0000","last_fullsized":"2024-09-16T01:49:01.039274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.834294+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:49:03.834294+0000","last_peered":"2024-09-16T01:49:03.834294+0000","last_clean":"2024-09-16T01:49:03.834294+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:49:03.834294+0000","last_undegraded":"2024-09-16T01:49:03.834294+0000","last_fullsized":"2024-09-16T01:49:03.834294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":78858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":79888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":668714,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":78276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":294912,"data_stored":668736,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":80560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:10.441 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:49:10.441 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T01:49:10.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T01:49:10.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T01:49:10.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.630+0000 7f705e600640 1 Processor -- start 2024-09-16T01:49:10.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.631+0000 7f705e600640 1 -- start start 2024-09-16T01:49:10.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705e600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7058154b40 0x7f70580cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70580cbef0 0x7f705815e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705e600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f705815e7e0 0x7f70580ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f705815ec70 con 0x7f70580cbef0 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705e600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f705815aac0 con 0x7f705815e7e0 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705e600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f705815ada0 con 0x7f7058154b40 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f7057e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70580cbef0 0x7f705815e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f7057e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70580cbef0 0x7f705815e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59388/0 (socket says 172.21.15.146:59388) 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7058154b40 0x7f70580cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f7057e00640 1 -- 172.21.15.146:0/4216129880 learned_addr learned my addr 172.21.15.146:0/4216129880 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f705dc00640 1 --2- 172.21.15.146:0/4216129880 >> v2:172.21.15.161:3300/0 conn(0x7f705815e7e0 0x7f70580ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f7057e00640 1 -- 172.21.15.146:0/4216129880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7058154b40 msgr2=0x7f70580cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.632+0000 7f7057e00640 1 --2- 172.21.15.146:0/4216129880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7058154b40 0x7f70580cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.633+0000 7f7057e00640 1 -- 172.21.15.146:0/4216129880 >> v2:172.21.15.161:3300/0 conn(0x7f705815e7e0 msgr2=0x7f70580ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:10.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.633+0000 7f7057e00640 1 --2- 172.21.15.146:0/4216129880 >> v2:172.21.15.161:3300/0 conn(0x7f705815e7e0 0x7f70580ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.633+0000 7f7057e00640 1 -- 172.21.15.146:0/4216129880 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705815b080 con 0x7f70580cbef0 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.633+0000 7f7057e00640 1 --2- 172.21.15.146:0/4216129880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580cbef0 0x7f705815e2a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70580b3100 tx=0x7f704402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75c30962d86dedf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f7057400640 1 -- 172.21.15.146:0/4216129880 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044037070 con 0x7f70580cbef0 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f7057400640 1 -- 172.21.15.146:0/4216129880 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7044032440 con 0x7f70580cbef0 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f7057400640 1 -- 172.21.15.146:0/4216129880 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044032810 con 0x7f70580cbef0 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 -- 172.21.15.146:0/4216129880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580cbef0 msgr2=0x7f705815e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 --2- 172.21.15.146:0/4216129880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580cbef0 0x7f705815e2a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70580b3100 tx=0x7f704402a270 comp rx=0 tx=0).stop 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 -- 172.21.15.146:0/4216129880 shutdown_connections 2024-09-16T01:49:10.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 --2- 172.21.15.146:0/4216129880 >> v2:172.21.15.161:3300/0 conn(0x7f705815e7e0 0x7f70580ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 --2- 172.21.15.146:0/4216129880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580cbef0 0x7f705815e2a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 --2- 172.21.15.146:0/4216129880 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7058154b40 0x7f70580cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.634+0000 7f705e600640 1 -- 172.21.15.146:0/4216129880 >> 172.21.15.146:0/4216129880 conn(0x7f705806d2d0 msgr2=0x7f705806d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.635+0000 7f705e600640 1 -- 172.21.15.146:0/4216129880 shutdown_connections 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.635+0000 7f705e600640 1 -- 172.21.15.146:0/4216129880 wait complete. 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.636+0000 7f705e600640 1 Processor -- start 2024-09-16T01:49:10.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.636+0000 7f705e600640 1 -- start start 2024-09-16T01:49:10.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705e600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580cbef0 0x7f70580d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7058154b40 0x7f70581f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f705815e7e0 0x7f70581f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7058154f20 con 0x7f7058154b40 2024-09-16T01:49:10.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705e600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f705815b640 con 0x7f705815e7e0 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705e600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70580cc6b0 con 0x7f70580cbef0 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705dc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f705815e7e0 0x7f70581f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580cbef0 0x7f70580d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705dc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f705815e7e0 0x7f70581f40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:45382/0 (socket says 172.21.15.146:45382) 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f7057e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7058154b40 0x7f70581f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.637+0000 7f705dc00640 1 -- 172.21.15.146:0/223917093 learned_addr learned my addr 172.21.15.146:0/223917093 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7057e00640 1 -- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580cbef0 msgr2=0x7f70580d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7057e00640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580cbef0 0x7f70580d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7057e00640 1 -- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f705815e7e0 msgr2=0x7f70581f40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7057e00640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f705815e7e0 0x7f70581f40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7057e00640 1 -- 172.21.15.146:0/223917093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70580d0360 con 0x7f7058154b40 2024-09-16T01:49:10.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7057e00640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7058154b40 0x7f70581f1d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7044031000 tx=0x7f70440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:10.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044041020 con 0x7f7058154b40 2024-09-16T01:49:10.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f704403d660 con 0x7f7058154b40 2024-09-16T01:49:10.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70580d0080 con 0x7f7058154b40 2024-09-16T01:49:10.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044037040 con 0x7f7058154b40 2024-09-16T01:49:10.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.638+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70580ceda0 con 0x7f7058154b40 2024-09-16T01:49:10.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.639+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70580b3960 con 0x7f7058154b40 2024-09-16T01:49:10.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.640+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7044044020 con 0x7f7058154b40 2024-09-16T01:49:10.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.642+0000 7f7054c00640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:10.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.642+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f70440ab9e0 con 0x7f7058154b40 2024-09-16T01:49:10.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.642+0000 7f705d200640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:10.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.642+0000 7f705d200640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f704c009ac0 tx=0x7f704c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:10.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.646+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f70580b3960 con 0x7f7058154b40 2024-09-16T01:49:10.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.856+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f705815e2a0 con 0x7f7058154b40 2024-09-16T01:49:10.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.857+0000 7f7054c00640 1 -- 172.21.15.146:0/223917093 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f704404b030 con 0x7f7058154b40 2024-09-16T01:49:10.858 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:10.858 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-16T01:49:10.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 msgr2=0x7f703806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f704c009ac0 tx=0x7f704c009190 comp rx=0 tx=0).stop 2024-09-16T01:49:10.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7058154b40 msgr2=0x7f70581f1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:10.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7058154b40 0x7f70581f1d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7044031000 tx=0x7f70440046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 shutdown_connections 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f705815e7e0 0x7f70581f40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7058154b40 0x7f70581f1d40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 --2- 172.21.15.146:0/223917093 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580cbef0 0x7f70580d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 >> 172.21.15.146:0/223917093 conn(0x7f705806d2d0 msgr2=0x7f70580b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 shutdown_connections 2024-09-16T01:49:10.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:10.859+0000 7f705e600640 1 -- 172.21.15.146:0/223917093 wait complete. 2024-09-16T01:49:13.523 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-09-16T01:49:13.522+0000 7f36b7200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:49:13.523 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-09-16T01:49:13.522+0000 7f36b7200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-16T01:49:13.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T01:49:14.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.041+0000 7ffbd9600640 1 Processor -- start 2024-09-16T01:49:14.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.041+0000 7ffbd9600640 1 -- start start 2024-09-16T01:49:14.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.042+0000 7ffbd9600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffbd415b6f0 0x7ffbd40c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:14.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.042+0000 7ffbd9600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd406c050 con 0x7ffbd4073f30 2024-09-16T01:49:14.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.042+0000 7ffbd9600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd40731b0 con 0x7ffbd40c1e70 2024-09-16T01:49:14.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.042+0000 7ffbd9600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd4073490 con 0x7ffbd415b6f0 2024-09-16T01:49:14.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.042+0000 7ffbd3400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffbd4073f30 0x7ffbd4165330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54182/0 (socket says 172.21.15.146:54182) 2024-09-16T01:49:14.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.042+0000 7ffbd3400640 1 -- 172.21.15.146:0/2031582968 learned_addr learned my addr 172.21.15.146:0/2031582968 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:14.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbca000640 1 --2- 172.21.15.146:0/2031582968 >> v2:172.21.15.161:3300/0 conn(0x7ffbd415b6f0 0x7ffbd40c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:14.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1536599442 0 0) 0x7ffbd406c050 con 0x7ffbd4073f30 2024-09-16T01:49:14.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbb8003520 con 0x7ffbd4073f30 2024-09-16T01:49:14.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476589027 0 0) 0x7ffbd40731b0 con 0x7ffbd40c1e70 2024-09-16T01:49:14.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbd406c050 con 0x7ffbd40c1e70 2024-09-16T01:49:14.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbca000640 1 -- 172.21.15.146:0/2031582968 >> v1:172.21.15.161:6789/0 conn(0x7ffbd40c1e70 legacy=0x7ffbd415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbca000640 1 -- 172.21.15.146:0/2031582968 >> v1:172.21.15.146:6789/0 conn(0x7ffbd4073f30 legacy=0x7ffbd4165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.043+0000 7ffbca000640 1 -- 172.21.15.146:0/2031582968 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd4073770 con 0x7ffbd415b6f0 2024-09-16T01:49:14.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.044+0000 7ffbca000640 1 --2- 172.21.15.146:0/2031582968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd415b6f0 0x7ffbd40c2250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0097a0 tx=0x7ffbbc02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5c0ecc2230b679 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:14.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.044+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc037070 con 0x7ffbd415b6f0 2024-09-16T01:49:14.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.044+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbbc032640 con 0x7ffbd415b6f0 2024-09-16T01:49:14.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.044+0000 7ffbd2000640 1 -- 172.21.15.146:0/2031582968 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc032a10 con 0x7ffbd415b6f0 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 -- 172.21.15.146:0/2031582968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd415b6f0 msgr2=0x7ffbd40c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 --2- 172.21.15.146:0/2031582968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd415b6f0 0x7ffbd40c2250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0097a0 tx=0x7ffbbc02a2c0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 -- 172.21.15.146:0/2031582968 shutdown_connections 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 --2- 172.21.15.146:0/2031582968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd415b6f0 0x7ffbd40c2250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 -- 172.21.15.146:0/2031582968 >> 172.21.15.146:0/2031582968 conn(0x7ffbd40ccb10 msgr2=0x7ffbd40ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 -- 172.21.15.146:0/2031582968 shutdown_connections 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.045+0000 7ffbd9600640 1 -- 172.21.15.146:0/2031582968 wait complete. 2024-09-16T01:49:14.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.047+0000 7ffbd9600640 1 Processor -- start 2024-09-16T01:49:14.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.047+0000 7ffbd9600640 1 -- start start 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.047+0000 7ffbd9600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd4073f30 0x7ffbd40c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 0x7ffbd41f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffbd415b6f0 0x7ffbd4153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd40b3130 con 0x7ffbd40c1e70 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd9600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd415aa70 con 0x7ffbd4073f30 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd9600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd4155040 con 0x7ffbd415b6f0 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 0x7ffbd41f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 0x7ffbd41f0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59844/0 (socket says 172.21.15.146:59844) 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 -- 172.21.15.146:0/511899899 learned_addr learned my addr 172.21.15.146:0/511899899 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:14.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd2a00640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd4073f30 0x7ffbd40c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd3400640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffbd415b6f0 0x7ffbd4153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 -- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffbd415b6f0 msgr2=0x7ffbd4153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffbd415b6f0 0x7ffbd4153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 -- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd4073f30 msgr2=0x7ffbd40c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd4073f30 0x7ffbd40c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 -- 172.21.15.146:0/511899899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd41f4740 con 0x7ffbd40c1e70 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbd3400640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffbd415b6f0 0x7ffbd4153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.048+0000 7ffbca000640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 0x7ffbd41f0370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0098c0 tx=0x7ffbbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.049+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc037040 con 0x7ffbd40c1e70 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.049+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbbc032440 con 0x7ffbd40c1e70 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.049+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc030990 con 0x7ffbd40c1e70 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.049+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd40cc010 con 0x7ffbd40c1e70 2024-09-16T01:49:14.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.049+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd41f3c60 con 0x7ffbd40c1e70 2024-09-16T01:49:14.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.050+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd40731b0 con 0x7ffbd40c1e70 2024-09-16T01:49:14.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.051+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ffbbc044020 con 0x7ffbd40c1e70 2024-09-16T01:49:14.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.052+0000 7ffbcb400640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffbb006c2b0 0x7ffbb006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:14.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.052+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ffbbc0ab6f0 con 0x7ffbd40c1e70 2024-09-16T01:49:14.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.052+0000 7ffbd2a00640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffbb006c2b0 0x7ffbb006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:14.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.053+0000 7ffbd2a00640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffbb006c2b0 0x7ffbb006e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40011e0 tx=0x7ffbc4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:14.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.056+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ffbbc079dc0 con 0x7ffbd40c1e70 2024-09-16T01:49:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.265+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ffbd40cbb20 con 0x7ffbd40c1e70 2024-09-16T01:49:14.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.265+0000 7ffbcb400640 1 -- 172.21.15.146:0/511899899 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7ffbbc0796c0 con 0x7ffbd40c1e70 2024-09-16T01:49:14.266 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:14.267 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffbb006c2b0 msgr2=0x7ffbb006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffbb006c2b0 0x7ffbb006e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40011e0 tx=0x7ffbc4009040 comp rx=0 tx=0).stop 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 msgr2=0x7ffbd41f0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 0x7ffbd41f0370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0098c0 tx=0x7ffbbc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 shutdown_connections 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffbb006c2b0 0x7ffbb006e770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffbd415b6f0 0x7ffbd4153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd40c1e70 0x7ffbd41f0370 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 --2- 172.21.15.146:0/511899899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffbd4073f30 0x7ffbd40c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:14.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 >> 172.21.15.146:0/511899899 conn(0x7ffbd40ccb10 msgr2=0x7ffbd4074ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:14.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.267+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 shutdown_connections 2024-09-16T01:49:14.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:14.268+0000 7ffbd9600640 1 -- 172.21.15.146:0/511899899 wait complete. 2024-09-16T01:49:17.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T01:49:17.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.446+0000 7f2243a00640 1 Processor -- start 2024-09-16T01:49:17.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.446+0000 7f2243a00640 1 -- start start 2024-09-16T01:49:17.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.447+0000 7f2243a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f223c151820 0x7f223c0c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2243a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f223c0c7980 0x7f223c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2243a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 0x7f223c0c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2243a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f223c1580a0 con 0x7f223c0c7980 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2243a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f223c0c6680 con 0x7f223c151820 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2243a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f223c0c74f0 con 0x7f223c0c8070 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f223c151820 0x7f223c0c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 0x7f223c0c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f223c151820 0x7f223c0c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:32806/0 (socket says 172.21.15.146:32806) 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 0x7f223c0c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44744/0 (socket says 172.21.15.146:44744) 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 -- 172.21.15.146:0/1355414945 learned_addr learned my addr 172.21.15.146:0/1355414945 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2233e00640 1 --2- 172.21.15.146:0/1355414945 >> v2:172.21.15.146:3300/0 conn(0x7f223c0c7980 0x7f223c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 -- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 msgr2=0x7f223c0c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 --2- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 0x7f223c0c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 -- 172.21.15.146:0/1355414945 >> v2:172.21.15.146:3300/0 conn(0x7f223c0c7980 msgr2=0x7f223c071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 --2- 172.21.15.146:0/1355414945 >> v2:172.21.15.146:3300/0 conn(0x7f223c0c7980 0x7f223c071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241000640 1 -- 172.21.15.146:0/1355414945 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c1585b0 con 0x7f223c151820 2024-09-16T01:49:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2241a00640 1 --2- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 0x7f223c0c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:17.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.448+0000 7f2233e00640 1 --2- 172.21.15.146:0/1355414945 >> v2:172.21.15.146:3300/0 conn(0x7f223c0c7980 0x7f223c071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.449+0000 7f2241000640 1 --2- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c151820 0x7f223c0c36b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f223c071690 tx=0x7f222c02c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5024f5c314732c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.450+0000 7f223be00640 1 -- 172.21.15.146:0/1355414945 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c036020 con 0x7f223c151820 2024-09-16T01:49:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.450+0000 7f223be00640 1 -- 172.21.15.146:0/1355414945 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222c032820 con 0x7f223c151820 2024-09-16T01:49:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.450+0000 7f223be00640 1 -- 172.21.15.146:0/1355414945 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c032bf0 con 0x7f223c151820 2024-09-16T01:49:17.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.450+0000 7f2243a00640 1 -- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c151820 msgr2=0x7f223c0c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.450+0000 7f2243a00640 1 --2- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c151820 0x7f223c0c36b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f223c071690 tx=0x7f222c02c920 comp rx=0 tx=0).stop 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 -- 172.21.15.146:0/1355414945 shutdown_connections 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 --2- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c8070 0x7f223c0c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 --2- 172.21.15.146:0/1355414945 >> v2:172.21.15.146:3300/0 conn(0x7f223c0c7980 0x7f223c071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 --2- 172.21.15.146:0/1355414945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c151820 0x7f223c0c36b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 -- 172.21.15.146:0/1355414945 >> 172.21.15.146:0/1355414945 conn(0x7f223c06bc10 msgr2=0x7f223c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 -- 172.21.15.146:0/1355414945 shutdown_connections 2024-09-16T01:49:17.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.451+0000 7f2243a00640 1 -- 172.21.15.146:0/1355414945 wait complete. 2024-09-16T01:49:17.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.452+0000 7f2243a00640 1 Processor -- start 2024-09-16T01:49:17.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.452+0000 7f2243a00640 1 -- start start 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2243a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c7980 0x7f223c153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2243a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c0c8070 0x7f223c0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2243a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 0x7f223c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2243a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f223c0c6ea0 con 0x7f223c151820 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2243a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f223c0c6bc0 con 0x7f223c0c8070 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2243a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f223c0c7180 con 0x7f223c0c7980 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.453+0000 7f2241a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 0x7f223c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2233e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c0c8070 0x7f223c0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c7980 0x7f223c153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 0x7f223c0cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59850/0 (socket says 172.21.15.146:59850) 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 -- 172.21.15.146:0/422013992 learned_addr learned my addr 172.21.15.146:0/422013992 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 -- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c7980 msgr2=0x7f223c153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c7980 0x7f223c153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 -- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c0c8070 msgr2=0x7f223c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c0c8070 0x7f223c0cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 -- 172.21.15.146:0/422013992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c0cc7b0 con 0x7f223c151820 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2241a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 0x7f223c0cbfe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2234009f00 tx=0x7f223400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234018030 con 0x7f223c151820 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223400ce50 con 0x7f223c151820 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234003bb0 con 0x7f223c151820 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.454+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223c152840 con 0x7f223c151820 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.455+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223c0ccd00 con 0x7f223c151820 2024-09-16T01:49:17.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.456+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f223c0c8520 con 0x7f223c151820 2024-09-16T01:49:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.456+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f223401b070 con 0x7f223c151820 2024-09-16T01:49:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.457+0000 7f2239600640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f221806c2b0 0x7f221806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.457+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f223408bd10 con 0x7f223c151820 2024-09-16T01:49:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.458+0000 7f2241000640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f221806c2b0 0x7f221806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:17.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.458+0000 7f2241000640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f221806c2b0 0x7f221806e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f222c037000 tx=0x7f222c03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:17.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.462+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2234091020 con 0x7f223c151820 2024-09-16T01:49:17.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.674+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f223c0c7e30 con 0x7f223c151820 2024-09-16T01:49:17.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.675+0000 7f2239600640 1 -- 172.21.15.146:0/422013992 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f2234010040 con 0x7f223c151820 2024-09-16T01:49:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:17.677 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-09-16T01:49:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f221806c2b0 msgr2=0x7f221806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f221806c2b0 0x7f221806e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f222c037000 tx=0x7f222c03d040 comp rx=0 tx=0).stop 2024-09-16T01:49:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 msgr2=0x7f223c0cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 0x7f223c0cbfe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2234009f00 tx=0x7f223400c490 comp rx=0 tx=0).stop 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 shutdown_connections 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f221806c2b0 0x7f221806e770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f223c151820 0x7f223c0cbfe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f223c0c8070 0x7f223c0cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 --2- 172.21.15.146:0/422013992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f223c0c7980 0x7f223c153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 >> 172.21.15.146:0/422013992 conn(0x7f223c06bc10 msgr2=0x7f223c154a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 shutdown_connections 2024-09-16T01:49:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:17.677+0000 7f2243a00640 1 -- 172.21.15.146:0/422013992 wait complete. 2024-09-16T01:49:18.523 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-09-16T01:49:18.521+0000 7f36b7200640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-16T01:49:20.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T01:49:20.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.859+0000 7f56dbe00640 1 Processor -- start 2024-09-16T01:49:20.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.859+0000 7f56dbe00640 1 -- start start 2024-09-16T01:49:20.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56dbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc15de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:20.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56dbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56dc15e3b0 0x7f56dc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:20.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56dbe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56dc0cc860 con 0x7f56dc154bd0 2024-09-16T01:49:20.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56dbe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f56dc15b850 con 0x7f56dc15e3b0 2024-09-16T01:49:20.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56dbe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f56dc15b3c0 con 0x7f56dc0cacb0 2024-09-16T01:49:20.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56daa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f56dc154bd0 0x7f56dc0cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54198/0 (socket says 172.21.15.146:54198) 2024-09-16T01:49:20.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56daa00640 1 -- 172.21.15.146:0/4265915018 learned_addr learned my addr 172.21.15.146:0/4265915018 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:20.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56db400640 1 --2- 172.21.15.146:0/4265915018 >> v2:172.21.15.146:3300/0 conn(0x7f56dc15e3b0 0x7f56dc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:20.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.860+0000 7f56da000640 1 --2- 172.21.15.146:0/4265915018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:20.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.861+0000 7f56d9600640 1 -- 172.21.15.146:0/4265915018 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2242728590 0 0) 0x7f56dc0cc860 con 0x7f56dc154bd0 2024-09-16T01:49:20.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.861+0000 7f56d9600640 1 -- 172.21.15.146:0/4265915018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56c0003520 con 0x7f56dc154bd0 2024-09-16T01:49:20.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.861+0000 7f56db400640 1 -- 172.21.15.146:0/4265915018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 msgr2=0x7f56dc15de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:20.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.861+0000 7f56db400640 1 --2- 172.21.15.146:0/4265915018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc15de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:20.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.861+0000 7f56db400640 1 -- 172.21.15.146:0/4265915018 >> v1:172.21.15.146:6789/0 conn(0x7f56dc154bd0 legacy=0x7f56dc0cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:20.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.861+0000 7f56db400640 1 -- 172.21.15.146:0/4265915018 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc15aa90 con 0x7f56dc15e3b0 2024-09-16T01:49:20.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.862+0000 7f56db400640 1 --2- 172.21.15.146:0/4265915018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc15e3b0 0x7f56dc06e0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009d90 tx=0x7f56d402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23784a04cece9d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:20.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.862+0000 7f56d9600640 1 -- 172.21.15.146:0/4265915018 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4007740 con 0x7f56dc15e3b0 2024-09-16T01:49:20.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.862+0000 7f56d9600640 1 -- 172.21.15.146:0/4265915018 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d40078e0 con 0x7f56dc15e3b0 2024-09-16T01:49:20.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.862+0000 7f56d9600640 1 -- 172.21.15.146:0/4265915018 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4007c90 con 0x7f56dc15e3b0 2024-09-16T01:49:20.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 -- 172.21.15.146:0/4265915018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc15e3b0 msgr2=0x7f56dc06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:20.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 --2- 172.21.15.146:0/4265915018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc15e3b0 0x7f56dc06e0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009d90 tx=0x7f56d402a6f0 comp rx=0 tx=0).stop 2024-09-16T01:49:20.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 -- 172.21.15.146:0/4265915018 shutdown_connections 2024-09-16T01:49:20.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 --2- 172.21.15.146:0/4265915018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc15e3b0 0x7f56dc06e0b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:20.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 --2- 172.21.15.146:0/4265915018 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc15de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:20.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 -- 172.21.15.146:0/4265915018 >> 172.21.15.146:0/4265915018 conn(0x7f56dc06c9f0 msgr2=0x7f56dc06d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:20.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 -- 172.21.15.146:0/4265915018 shutdown_connections 2024-09-16T01:49:20.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.863+0000 7f56dbe00640 1 -- 172.21.15.146:0/4265915018 wait complete. 2024-09-16T01:49:20.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.864+0000 7f56dbe00640 1 Processor -- start 2024-09-16T01:49:20.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 -- start start 2024-09-16T01:49:20.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc14c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:20.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 0x7f56dc1584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:20.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56dc15e3b0 0x7f56dc149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:20.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc15b050 con 0x7f56dc154bd0 2024-09-16T01:49:20.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc154fb0 con 0x7f56dc15e3b0 2024-09-16T01:49:20.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.865+0000 7f56dbe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f56dc0cd970 con 0x7f56dc0cacb0 2024-09-16T01:49:20.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 0x7f56dc1584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:20.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 0x7f56dc1584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59872/0 (socket says 172.21.15.146:59872) 2024-09-16T01:49:20.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 -- 172.21.15.146:0/2241704890 learned_addr learned my addr 172.21.15.146:0/2241704890 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:20.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56daa00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc14c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:20.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56db400640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56dc15e3b0 0x7f56dc149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:20.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 -- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 msgr2=0x7f56dc14c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:20.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc14c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:20.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 -- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56dc15e3b0 msgr2=0x7f56dc149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:20.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56dc15e3b0 0x7f56dc149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:20.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 -- 172.21.15.146:0/2241704890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc14e2a0 con 0x7f56dc154bd0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56da000640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 0x7f56dc1584c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f56c800e9b0 tx=0x7f56c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.866+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c8017070 con 0x7f56dc154bd0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.867+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56c800c9a0 con 0x7f56dc154bd0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.867+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c800cd70 con 0x7f56dc154bd0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.867+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56dc1f39a0 con 0x7f56dc154bd0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.867+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56dc1f3f60 con 0x7f56dc154bd0 2024-09-16T01:49:20.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.868+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56dc15de70 con 0x7f56dc154bd0 2024-09-16T01:49:20.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.869+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f56c801a070 con 0x7f56dc154bd0 2024-09-16T01:49:20.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.869+0000 7f56d2a00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f56bc06c2b0 0x7f56bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:20.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.869+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f56c808b810 con 0x7f56dc154bd0 2024-09-16T01:49:20.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.870+0000 7f56daa00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f56bc06c2b0 0x7f56bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:20.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.870+0000 7f56daa00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f56bc06c2b0 0x7f56bc06e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56cc005880 tx=0x7f56cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:20.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:20.874+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f56c8059800 con 0x7f56dc154bd0 2024-09-16T01:49:21.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.095+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f56dc152f00 con 0x7f56dc154bd0 2024-09-16T01:49:21.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.095+0000 7f56d2a00640 1 -- 172.21.15.146:0/2241704890 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f56c805db00 con 0x7f56dc154bd0 2024-09-16T01:49:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:21.097 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f56bc06c2b0 msgr2=0x7f56bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f56bc06c2b0 0x7f56bc06e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56cc005880 tx=0x7f56cc009040 comp rx=0 tx=0).stop 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 msgr2=0x7f56dc1584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 0x7f56dc1584c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f56c800e9b0 tx=0x7f56c800ee70 comp rx=0 tx=0).stop 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 shutdown_connections 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f56bc06c2b0 0x7f56bc06e770 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56dc15e3b0 0x7f56dc149b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56dc154bd0 0x7f56dc1584c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 --2- 172.21.15.146:0/2241704890 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f56dc0cacb0 0x7f56dc14c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 >> 172.21.15.146:0/2241704890 conn(0x7f56dc06c9f0 msgr2=0x7f56dc0c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 shutdown_connections 2024-09-16T01:49:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.097+0000 7f56dbe00640 1 -- 172.21.15.146:0/2241704890 wait complete. 2024-09-16T01:49:21.115 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T01:49:21.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:21.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.268+0000 7fbe1cc00640 1 Processor -- start 2024-09-16T01:49:21.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.268+0000 7fbe1cc00640 1 -- start start 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.269+0000 7fbe1cc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbe18151a90 0x7fbe180c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe1cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe180c5280 0x7fbe180c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe1cc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe18151e70 con 0x7fbe180c6410 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe1cc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbe180c8130 con 0x7fbe18151a90 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe1cc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbe180c8410 con 0x7fbe180c5280 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe0d400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbe180c6410 0x7fbe18158040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54212/0 (socket says 172.21.15.146:54212) 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe0d400640 1 -- 172.21.15.146:0/3377234249 learned_addr learned my addr 172.21.15.146:0/3377234249 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe16800640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe180c5280 0x7fbe180c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15e00640 1 --2- 172.21.15.146:0/3377234249 >> v2:172.21.15.161:3300/0 conn(0x7fbe18151a90 0x7fbe180c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15400640 1 -- 172.21.15.146:0/3377234249 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2245898129 0 0) 0x7fbe18151e70 con 0x7fbe180c6410 2024-09-16T01:49:21.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15400640 1 -- 172.21.15.146:0/3377234249 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdfc003520 con 0x7fbe180c6410 2024-09-16T01:49:21.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15e00640 1 -- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe180c5280 msgr2=0x7fbe180c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:21.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15e00640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe180c5280 0x7fbe180c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15e00640 1 -- 172.21.15.146:0/3377234249 >> v1:172.21.15.146:6789/0 conn(0x7fbe180c6410 legacy=0x7fbe18158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe15e00640 1 -- 172.21.15.146:0/3377234249 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe180c86f0 con 0x7fbe18151a90 2024-09-16T01:49:21.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.270+0000 7fbe16800640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe180c5280 0x7fbe180c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:21.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.271+0000 7fbe15e00640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe18151a90 0x7fbe180c36b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe080097a0 tx=0x7fbe0802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5759ce6034cc9fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe15400640 1 -- 172.21.15.146:0/3377234249 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08037070 con 0x7fbe18151a90 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe15400640 1 -- 172.21.15.146:0/3377234249 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe08032440 con 0x7fbe18151a90 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe15400640 1 -- 172.21.15.146:0/3377234249 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080327f0 con 0x7fbe18151a90 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 -- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe18151a90 msgr2=0x7fbe180c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe18151a90 0x7fbe180c36b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe080097a0 tx=0x7fbe0802a180 comp rx=0 tx=0).stop 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 -- 172.21.15.146:0/3377234249 shutdown_connections 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe180c5280 0x7fbe180c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/3377234249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe18151a90 0x7fbe180c36b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 -- 172.21.15.146:0/3377234249 >> 172.21.15.146:0/3377234249 conn(0x7fbe1806f3c0 msgr2=0x7fbe1806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:21.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 -- 172.21.15.146:0/3377234249 shutdown_connections 2024-09-16T01:49:21.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.272+0000 7fbe1cc00640 1 -- 172.21.15.146:0/3377234249 wait complete. 2024-09-16T01:49:21.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.274+0000 7fbe1cc00640 1 Processor -- start 2024-09-16T01:49:21.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.274+0000 7fbe1cc00640 1 -- start start 2024-09-16T01:49:21.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe1cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe180c5280 0x7fbe181589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe1cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 0x7fbe181f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe1cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe18151a90 0x7fbe180c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe1cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbe18157b20 con 0x7fbe180c6410 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe1cc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbe18157840 con 0x7fbe180c5280 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe1cc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbe180c3bf0 con 0x7fbe18151a90 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe0d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 0x7fbe181f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe0d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 0x7fbe181f42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59888/0 (socket says 172.21.15.146:59888) 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe15e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe180c5280 0x7fbe181589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe0d400640 1 -- 172.21.15.146:0/2861598043 learned_addr learned my addr 172.21.15.146:0/2861598043 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.275+0000 7fbe16800640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe18151a90 0x7fbe180c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0d400640 1 -- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe18151a90 msgr2=0x7fbe180c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0d400640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe18151a90 0x7fbe180c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0d400640 1 -- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe180c5280 msgr2=0x7fbe181589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0d400640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe180c5280 0x7fbe181589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0d400640 1 -- 172.21.15.146:0/2861598043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe181f0440 con 0x7fbe180c6410 2024-09-16T01:49:21.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0d400640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 0x7fbe181f42b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000c400 tx=0x7fbe0000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:21.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00017070 con 0x7fbe180c6410 2024-09-16T01:49:21.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe00005d00 con 0x7fbe180c6410 2024-09-16T01:49:21.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe180cc200 con 0x7fbe180c6410 2024-09-16T01:49:21.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00007310 con 0x7fbe180c6410 2024-09-16T01:49:21.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.276+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe180cc7c0 con 0x7fbe180c6410 2024-09-16T01:49:21.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.277+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe18157370 con 0x7fbe180c6410 2024-09-16T01:49:21.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.278+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbe0001a070 con 0x7fbe180c6410 2024-09-16T01:49:21.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.280+0000 7fbe0ea00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbdf406c2b0 0x7fbdf406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.280+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fbe0008a8a0 con 0x7fbe180c6410 2024-09-16T01:49:21.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.280+0000 7fbe15e00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbdf406c2b0 0x7fbdf406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.280+0000 7fbe15e00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbdf406c2b0 0x7fbdf406e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe08034000 tx=0x7fbe0802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:21.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.284+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbe18157370 con 0x7fbe180c6410 2024-09-16T01:49:21.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.493+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe180c71f0 con 0x7fbe180c6410 2024-09-16T01:49:21.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.493+0000 7fbe0ea00640 1 -- 172.21.15.146:0/2861598043 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6956 (secure 0 0 0) 0x7fbe00058890 con 0x7fbe180c6410 2024-09-16T01:49:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:21.496 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":15,"btime":"2024-09-16T01:49:20:929876+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:20.929867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4291,4288,4277,4259]},"id":1}]} 2024-09-16T01:49:21.496 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 15 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.495+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbdf406c2b0 msgr2=0x7fbdf406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.495+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbdf406c2b0 0x7fbdf406e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbe08034000 tx=0x7fbe0802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.495+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 msgr2=0x7fbe181f42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 0x7fbe181f42b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000c400 tx=0x7fbe0000c800 comp rx=0 tx=0).stop 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 shutdown_connections 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbdf406c2b0 0x7fbdf406e770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbe18151a90 0x7fbe180c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe180c6410 0x7fbe181f42b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 --2- 172.21.15.146:0/2861598043 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe180c5280 0x7fbe181589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 >> 172.21.15.146:0/2861598043 conn(0x7fbe1806f3c0 msgr2=0x7fbe1806f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 shutdown_connections 2024-09-16T01:49:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.496+0000 7fbe1cc00640 1 -- 172.21.15.146:0/2861598043 wait complete. 2024-09-16T01:49:21.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:21.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.666+0000 7f6e69800640 1 Processor -- start 2024-09-16T01:49:21.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.666+0000 7f6e69800640 1 -- start start 2024-09-16T01:49:21.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.667+0000 7f6e69800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e6415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.667+0000 7f6e69800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6e640c75a0 0x7f6e640cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.667+0000 7f6e69800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6e640cbab0 0x7f6e640cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.667+0000 7f6e69800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6e6415b880 con 0x7f6e640cbab0 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.667+0000 7f6e69800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6e640cab20 con 0x7f6e640c75a0 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.667+0000 7f6e69800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6406d140 con 0x7f6e640c6b70 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6e640cbab0 0x7f6e640cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6e640cbab0 0x7f6e640cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59900/0 (socket says 172.21.15.146:59900) 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 -- 172.21.15.146:0/606518809 learned_addr learned my addr 172.21.15.146:0/606518809 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e63e00640 1 --2- 172.21.15.146:0/606518809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e6415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e63400640 1 --2- 172.21.15.146:0/606518809 >> v2:172.21.15.161:3300/0 conn(0x7f6e640c75a0 0x7f6e640cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 -- 172.21.15.146:0/606518809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 msgr2=0x7f6e6415b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 --2- 172.21.15.146:0/606518809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e6415b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 -- 172.21.15.146:0/606518809 >> v2:172.21.15.161:3300/0 conn(0x7f6e640c75a0 msgr2=0x7f6e640cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 --2- 172.21.15.146:0/606518809 >> v2:172.21.15.161:3300/0 conn(0x7f6e640c75a0 0x7f6e640cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.668+0000 7f6e68e00640 1 -- 172.21.15.146:0/606518809 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6406d420 con 0x7f6e640cbab0 2024-09-16T01:49:21.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.669+0000 7f6e68e00640 1 --2- 172.21.15.146:0/606518809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e640cc040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e6406dda0 tx=0x7f6e5c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf9bb9ea1346060 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:21.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.669+0000 7f6e62a00640 1 -- 172.21.15.146:0/606518809 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c036070 con 0x7f6e640cbab0 2024-09-16T01:49:21.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.669+0000 7f6e62a00640 1 -- 172.21.15.146:0/606518809 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e5c031600 con 0x7f6e640cbab0 2024-09-16T01:49:21.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.669+0000 7f6e62a00640 1 -- 172.21.15.146:0/606518809 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0319d0 con 0x7f6e640cbab0 2024-09-16T01:49:21.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 -- 172.21.15.146:0/606518809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 msgr2=0x7f6e640cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 --2- 172.21.15.146:0/606518809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e640cc040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e6406dda0 tx=0x7f6e5c02a570 comp rx=0 tx=0).stop 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 -- 172.21.15.146:0/606518809 shutdown_connections 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 --2- 172.21.15.146:0/606518809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e640cc040 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 --2- 172.21.15.146:0/606518809 >> v2:172.21.15.161:3300/0 conn(0x7f6e640c75a0 0x7f6e640cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 --2- 172.21.15.146:0/606518809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e6415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 -- 172.21.15.146:0/606518809 >> 172.21.15.146:0/606518809 conn(0x7f6e640ccc20 msgr2=0x7f6e640cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 -- 172.21.15.146:0/606518809 shutdown_connections 2024-09-16T01:49:21.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.670+0000 7f6e69800640 1 -- 172.21.15.146:0/606518809 wait complete. 2024-09-16T01:49:21.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.671+0000 7f6e69800640 1 Processor -- start 2024-09-16T01:49:21.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 -- start start 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e640c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e640c75a0 0x7f6e64151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e64151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e640cb050 con 0x7f6e640cbab0 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6e640cad70 con 0x7f6e640c75a0 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.672+0000 7f6e69800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6e6406cd20 con 0x7f6e640c6b70 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e640c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e640c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44798/0 (socket says 172.21.15.146:44798) 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 -- 172.21.15.146:0/3249826334 learned_addr learned my addr 172.21.15.146:0/3249826334 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e68e00640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e64151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63400640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e640c75a0 0x7f6e64151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 -- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e640c75a0 msgr2=0x7f6e64151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e640c75a0 0x7f6e64151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 -- 172.21.15.146:0/3249826334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 msgr2=0x7f6e64151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e64151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 -- 172.21.15.146:0/3249826334 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6415c6b0 con 0x7f6e640c6b70 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.673+0000 7f6e63e00640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e640c54b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400e8a0 tx=0x7f6e5400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.674+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54017070 con 0x7f6e640c6b70 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.674+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e5400ee90 con 0x7f6e640c6b70 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.674+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5400cb80 con 0x7f6e640c6b70 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.674+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6415c990 con 0x7f6e640c6b70 2024-09-16T01:49:21.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.674+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e6415cf50 con 0x7f6e640c6b70 2024-09-16T01:49:21.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.675+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e640ca280 con 0x7f6e640c6b70 2024-09-16T01:49:21.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.676+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6e5401a070 con 0x7f6e640c6b70 2024-09-16T01:49:21.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.677+0000 7f6e5be00640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6e4406c2b0 0x7f6e4406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:21.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.677+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6e5408b660 con 0x7f6e640c6b70 2024-09-16T01:49:21.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.677+0000 7f6e63400640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6e4406c2b0 0x7f6e4406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:21.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.677+0000 7f6e63400640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6e4406c2b0 0x7f6e4406e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6e641ee430 tx=0x7f6e50009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:21.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.681+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6e54059650 con 0x7f6e640c6b70 2024-09-16T01:49:21.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.895+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e6406da90 con 0x7f6e640c6b70 2024-09-16T01:49:21.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.896+0000 7f6e5be00640 1 -- 172.21.15.146:0/3249826334 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6956 (secure 0 0 0) 0x7f6e5405d950 con 0x7f6e640c6b70 2024-09-16T01:49:21.898 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:21.898 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":15,"btime":"2024-09-16T01:49:20:929876+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:20.929867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4291,4288,4277,4259]},"id":1}]} 2024-09-16T01:49:21.899 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 15 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6e4406c2b0 msgr2=0x7f6e4406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6e4406c2b0 0x7f6e4406e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6e641ee430 tx=0x7f6e50009190 comp rx=0 tx=0).stop 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 msgr2=0x7f6e640c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e640c54b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400e8a0 tx=0x7f6e5400ed60 comp rx=0 tx=0).stop 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 shutdown_connections 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6e4406c2b0 0x7f6e4406e770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e640cbab0 0x7f6e64151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e640c75a0 0x7f6e64151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 --2- 172.21.15.146:0/3249826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e640c6b70 0x7f6e640c54b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 >> 172.21.15.146:0/3249826334 conn(0x7f6e640ccc20 msgr2=0x7f6e640ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 shutdown_connections 2024-09-16T01:49:21.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:21.898+0000 7f6e69800640 1 -- 172.21.15.146:0/3249826334 wait complete. 2024-09-16T01:49:21.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:49:22.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.068+0000 7f81fba00640 1 Processor -- start 2024-09-16T01:49:22.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.068+0000 7f81fba00640 1 -- start start 2024-09-16T01:49:22.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.069+0000 7f81fba00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f81f40cc0c0 0x7f81f415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.069+0000 7f81fba00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81f40cb2f0 0x7f81f40cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.069+0000 7f81fba00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81f415b1d0 con 0x7f81f415e9d0 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.069+0000 7f81fba00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f81f40cbd00 con 0x7f81f40cb2f0 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.069+0000 7f81fba00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f81f415ba70 con 0x7f81f40cc0c0 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.069+0000 7f81ebe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81f40cb2f0 0x7f81f40cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81f9000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f81f40cc0c0 0x7f81f415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81f9a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f81f415e9d0 0x7f81f40cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44438/0 (socket says 172.21.15.146:44438) 2024-09-16T01:49:22.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81f9a00640 1 -- 172.21.15.146:0/1309490486 learned_addr learned my addr 172.21.15.146:0/1309490486 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:22.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81f3e00640 1 -- 172.21.15.146:0/1309490486 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82810137 0 0) 0x7f81f415b1d0 con 0x7f81f415e9d0 2024-09-16T01:49:22.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81ebe00640 1 -- 172.21.15.146:0/1309490486 >> v2:172.21.15.161:3300/0 conn(0x7f81f40cc0c0 msgr2=0x7f81f415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:22.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81ebe00640 1 --2- 172.21.15.146:0/1309490486 >> v2:172.21.15.161:3300/0 conn(0x7f81f40cc0c0 0x7f81f415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81ebe00640 1 -- 172.21.15.146:0/1309490486 >> v1:172.21.15.161:6789/0 conn(0x7f81f415e9d0 legacy=0x7f81f40cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81ebe00640 1 -- 172.21.15.146:0/1309490486 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f415b4b0 con 0x7f81f40cb2f0 2024-09-16T01:49:22.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.070+0000 7f81f9000640 1 --2- 172.21.15.146:0/1309490486 >> v2:172.21.15.161:3300/0 conn(0x7f81f40cc0c0 0x7f81f415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:22.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.071+0000 7f81ebe00640 1 --2- 172.21.15.146:0/1309490486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40cb2f0 0x7f81f40cc4a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f81f40ce150 tx=0x7f81dc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5768f3f7792f8c4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:22.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.071+0000 7f81f3e00640 1 -- 172.21.15.146:0/1309490486 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc037070 con 0x7f81f40cb2f0 2024-09-16T01:49:22.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.071+0000 7f81f3e00640 1 -- 172.21.15.146:0/1309490486 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc032440 con 0x7f81f40cb2f0 2024-09-16T01:49:22.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.071+0000 7f81f3e00640 1 -- 172.21.15.146:0/1309490486 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc032810 con 0x7f81f40cb2f0 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 -- 172.21.15.146:0/1309490486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40cb2f0 msgr2=0x7f81f40cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 --2- 172.21.15.146:0/1309490486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40cb2f0 0x7f81f40cc4a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f81f40ce150 tx=0x7f81dc02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 -- 172.21.15.146:0/1309490486 shutdown_connections 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 --2- 172.21.15.146:0/1309490486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f40cb2f0 0x7f81f40cc4a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 --2- 172.21.15.146:0/1309490486 >> v2:172.21.15.161:3300/0 conn(0x7f81f40cc0c0 0x7f81f415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 -- 172.21.15.146:0/1309490486 >> 172.21.15.146:0/1309490486 conn(0x7f81f406d560 msgr2=0x7f81f406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 -- 172.21.15.146:0/1309490486 shutdown_connections 2024-09-16T01:49:22.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.072+0000 7f81fba00640 1 -- 172.21.15.146:0/1309490486 wait complete. 2024-09-16T01:49:22.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.073+0000 7f81fba00640 1 Processor -- start 2024-09-16T01:49:22.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 -- start start 2024-09-16T01:49:22.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81f40cb2f0 0x7f81f415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 0x7f81f40b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f415e9d0 0x7f81f414af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81f415e1d0 con 0x7f81f415e9d0 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f81f415e540 con 0x7f81f40cc0c0 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.074+0000 7f81fba00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f81f40c5560 con 0x7f81f40cb2f0 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 0x7f81f40b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81f9000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81f40cb2f0 0x7f81f415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 0x7f81f40b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:32870/0 (socket says 172.21.15.146:32870) 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 -- 172.21.15.146:0/2831553154 learned_addr learned my addr 172.21.15.146:0/2831553154 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81f9a00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f415e9d0 0x7f81f414af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 -- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81f40cb2f0 msgr2=0x7f81f415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81f40cb2f0 0x7f81f415bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 -- 172.21.15.146:0/2831553154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f415e9d0 msgr2=0x7f81f414af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f415e9d0 0x7f81f414af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 -- 172.21.15.146:0/2831553154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f41f0c10 con 0x7f81f40cc0c0 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81ebe00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 0x7f81f40b5980 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f81dc000c00 tx=0x7f81dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc037050 con 0x7f81f40cc0c0 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.075+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc038040 con 0x7f81f40cc0c0 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.076+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f415c190 con 0x7f81f40cc0c0 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.076+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc004450 con 0x7f81f40cc0c0 2024-09-16T01:49:22.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.076+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f40b64a0 con 0x7f81f40cc0c0 2024-09-16T01:49:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.077+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f40cbea0 con 0x7f81f40cc0c0 2024-09-16T01:49:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.078+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f81dc03c6d0 con 0x7f81f40cc0c0 2024-09-16T01:49:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.078+0000 7f81f1600640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81d006c2e0 0x7f81d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.078+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f81dc0abd20 con 0x7f81f40cc0c0 2024-09-16T01:49:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.079+0000 7f81f9000640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81d006c2e0 0x7f81d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.079+0000 7f81f9000640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81d006c2e0 0x7f81d006e7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009560 tx=0x7f81e4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:22.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.083+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f81dc079da0 con 0x7f81f40cc0c0 2024-09-16T01:49:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.259+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81f40b6250 con 0x7f81f40cc0c0 2024-09-16T01:49:22.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.261+0000 7f81f1600640 1 -- 172.21.15.146:0/2831553154 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16693 (secure 0 0 0) 0x7f81dc07e0a0 con 0x7f81f40cc0c0 2024-09-16T01:49:22.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:22.263 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":22,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:48:59.816171+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:48:36.639012+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:49:22.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81d006c2e0 msgr2=0x7f81d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81d006c2e0 0x7f81d006e7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009560 tx=0x7f81e4009190 comp rx=0 tx=0).stop 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 msgr2=0x7f81f40b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 0x7f81f40b5980 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f81dc000c00 tx=0x7f81dc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 shutdown_connections 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81d006c2e0 0x7f81d006e7a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81f415e9d0 0x7f81f414af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81f40cc0c0 0x7f81f40b5980 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 --2- 172.21.15.146:0/2831553154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81f40cb2f0 0x7f81f415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 >> 172.21.15.146:0/2831553154 conn(0x7f81f406d560 msgr2=0x7f81f4151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 shutdown_connections 2024-09-16T01:49:22.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.263+0000 7f81fba00640 1 -- 172.21.15.146:0/2831553154 wait complete. 2024-09-16T01:49:22.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:22.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.434+0000 7fb4c9a00640 1 Processor -- start 2024-09-16T01:49:22.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.434+0000 7fb4c9a00640 1 -- start start 2024-09-16T01:49:22.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb4c40c6b70 0x7fb4c415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4c406d140 con 0x7fb4c40cbab0 2024-09-16T01:49:22.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb4c415b880 con 0x7fb4c40c6b70 2024-09-16T01:49:22.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c40cab20 con 0x7fb4c40c75a0 2024-09-16T01:49:22.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb4c40c6b70 0x7fb4c415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb4c40cbab0 0x7fb4c40cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44446/0 (socket says 172.21.15.146:44446) 2024-09-16T01:49:22.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.435+0000 7fb4c9000640 1 -- 172.21.15.146:0/3170662951 learned_addr learned my addr 172.21.15.146:0/3170662951 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:22.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c2a00640 1 -- 172.21.15.146:0/3170662951 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336895206 0 0) 0x7fb4c406d140 con 0x7fb4c40cbab0 2024-09-16T01:49:22.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c2a00640 1 -- 172.21.15.146:0/3170662951 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4a8003520 con 0x7fb4c40cbab0 2024-09-16T01:49:22.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c3400640 1 -- 172.21.15.146:0/3170662951 >> v2:172.21.15.161:3300/0 conn(0x7fb4c40c6b70 msgr2=0x7fb4c415b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:22.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c3400640 1 --2- 172.21.15.146:0/3170662951 >> v2:172.21.15.161:3300/0 conn(0x7fb4c40c6b70 0x7fb4c415b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c3400640 1 -- 172.21.15.146:0/3170662951 >> v1:172.21.15.161:6789/0 conn(0x7fb4c40cbab0 legacy=0x7fb4c40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c3400640 1 -- 172.21.15.146:0/3170662951 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c406d420 con 0x7fb4c40c75a0 2024-09-16T01:49:22.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.436+0000 7fb4c3e00640 1 --2- 172.21.15.146:0/3170662951 >> v2:172.21.15.161:3300/0 conn(0x7fb4c40c6b70 0x7fb4c415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:22.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.437+0000 7fb4c3400640 1 --2- 172.21.15.146:0/3170662951 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c40cb570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00097a0 tx=0x7fb4b002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18682cdd5d12ab37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:22.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.437+0000 7fb4c2a00640 1 -- 172.21.15.146:0/3170662951 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0037070 con 0x7fb4c40c75a0 2024-09-16T01:49:22.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.437+0000 7fb4c2a00640 1 -- 172.21.15.146:0/3170662951 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b002adb0 con 0x7fb4c40c75a0 2024-09-16T01:49:22.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.437+0000 7fb4c2a00640 1 -- 172.21.15.146:0/3170662951 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0032660 con 0x7fb4c40c75a0 2024-09-16T01:49:22.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 -- 172.21.15.146:0/3170662951 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 msgr2=0x7fb4c40cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/3170662951 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c40cb570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00097a0 tx=0x7fb4b002a060 comp rx=0 tx=0).stop 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 -- 172.21.15.146:0/3170662951 shutdown_connections 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/3170662951 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c40cb570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/3170662951 >> v2:172.21.15.161:3300/0 conn(0x7fb4c40c6b70 0x7fb4c415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 -- 172.21.15.146:0/3170662951 >> 172.21.15.146:0/3170662951 conn(0x7fb4c40ccc20 msgr2=0x7fb4c40cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 -- 172.21.15.146:0/3170662951 shutdown_connections 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.438+0000 7fb4c9a00640 1 -- 172.21.15.146:0/3170662951 wait complete. 2024-09-16T01:49:22.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.440+0000 7fb4c9a00640 1 Processor -- start 2024-09-16T01:49:22.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.440+0000 7fb4c9a00640 1 -- start start 2024-09-16T01:49:22.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4c40c6b70 0x7fb4c4159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c41f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 0x7fb4c41f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c40cb050 con 0x7fb4c40cbab0 2024-09-16T01:49:22.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c40cad70 con 0x7fb4c40c6b70 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c406cd20 con 0x7fb4c40c75a0 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 0x7fb4c41f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c41f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4c40c6b70 0x7fb4c4159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 0x7fb4c41f42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59932/0 (socket says 172.21.15.146:59932) 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9000640 1 -- 172.21.15.146:0/752803205 learned_addr learned my addr 172.21.15.146:0/752803205 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3400640 1 -- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4c40c6b70 msgr2=0x7fb4c4159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3400640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4c40c6b70 0x7fb4c4159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3400640 1 -- 172.21.15.146:0/752803205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 msgr2=0x7fb4c41f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3400640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 0x7fb4c41f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c3400640 1 -- 172.21.15.146:0/752803205 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c415abc0 con 0x7fb4c40c75a0 2024-09-16T01:49:22.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.441+0000 7fb4c9000640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 0x7fb4c41f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:22.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.442+0000 7fb4c3400640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c41f3d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40caca0 tx=0x7fb4b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:22.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.442+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0041030 con 0x7fb4c40c75a0 2024-09-16T01:49:22.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.442+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b002abb0 con 0x7fb4c40c75a0 2024-09-16T01:49:22.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.442+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0037040 con 0x7fb4c40c75a0 2024-09-16T01:49:22.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.442+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c4159060 con 0x7fb4c40c75a0 2024-09-16T01:49:22.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.442+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c415ccf0 con 0x7fb4c40c75a0 2024-09-16T01:49:22.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.443+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c406da90 con 0x7fb4c40c75a0 2024-09-16T01:49:22.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.444+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb4b0044070 con 0x7fb4c40c75a0 2024-09-16T01:49:22.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.445+0000 7fb4bbe00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb4a406c2b0 0x7fb4a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:22.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.445+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb4b00ab880 con 0x7fb4c40c75a0 2024-09-16T01:49:22.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.445+0000 7fb4c3e00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb4a406c2b0 0x7fb4a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:22.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.446+0000 7fb4c3e00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb4a406c2b0 0x7fb4a406e770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009560 tx=0x7fb4b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:22.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.450+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb4b0079990 con 0x7fb4c40c75a0 2024-09-16T01:49:22.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.658+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4c40cda30 con 0x7fb4c40c75a0 2024-09-16T01:49:22.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.659+0000 7fb4bbe00640 1 -- 172.21.15.146:0/752803205 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6956 (secure 0 0 0) 0x7fb4b007ee20 con 0x7fb4c40c75a0 2024-09-16T01:49:22.661 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:22.661 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":15,"btime":"2024-09-16T01:49:20:929876+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:20.929867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4291,4288,4277,4259]},"id":1}]} 2024-09-16T01:49:22.661 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 15 2024-09-16T01:49:22.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb4a406c2b0 msgr2=0x7fb4a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb4a406c2b0 0x7fb4a406e770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009560 tx=0x7fb4b4009190 comp rx=0 tx=0).stop 2024-09-16T01:49:22.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 msgr2=0x7fb4c41f3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c41f3d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40caca0 tx=0x7fb4b00046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 shutdown_connections 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb4a406c2b0 0x7fb4a406e770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4c40cbab0 0x7fb4c41f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4c40c75a0 0x7fb4c41f3d90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 --2- 172.21.15.146:0/752803205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4c40c6b70 0x7fb4c4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 >> 172.21.15.146:0/752803205 conn(0x7fb4c40ccc20 msgr2=0x7fb4c40cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 shutdown_connections 2024-09-16T01:49:22.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:22.661+0000 7fb4c9a00640 1 -- 172.21.15.146:0/752803205 wait complete. 2024-09-16T01:49:23.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:48:58.207792+0000', 'modified': '2024-09-16T01:49:20.929867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4288, 'mds_2': 4277, 'mds_3': 4259, 'mds_4': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'd', 'rank': 4, 'incarnation': 15, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.161:6837/3568385147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3568385147}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3568385147}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4259': {'gid': 4259, 'name': 'f', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6839/2836611107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2836611107}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2836611107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4277': {'gid': 4277, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.146:6837/835076388', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 835076388}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 835076388}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4288': {'gid': 4288, 'name': 'a', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.146:6833/1022834380', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6832', 'nonce': 1022834380}, {'type': 'v1', 'addr': '172.21.15.146:6833', 'nonce': 1022834380}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4291': {'gid': 4291, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6836/1807355541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 1807355541}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 1807355541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4291, 'qdb_cluster': [4291, 4288, 4277, 4259]} 2024-09-16T01:49:23.283 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4256:up:creating 2024-09-16T01:49:23.524 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-09-16T01:49:23.522+0000 7f36b7200640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-16T01:49:24.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:24.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.446+0000 7fd2c7a00640 1 Processor -- start 2024-09-16T01:49:24.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.446+0000 7fd2c7a00640 1 -- start start 2024-09-16T01:49:24.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2c7a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2c00c6bb0 0x7fd2c00c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:24.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2c7a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c58c0 0x7fd2c00c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:24.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2c7a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2c0072f50 con 0x7fd2c0152850 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2c7a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c0073210 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2c7a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c015cb50 con 0x7fd2c00c58c0 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2c00c6bb0 0x7fd2c00c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2c00c6bb0 0x7fd2c00c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50190/0 (socket says 172.21.15.146:50190) 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1866749285 learned_addr learned my addr 172.21.15.146:0/1866749285 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2c5a00640 1 --2- 172.21.15.146:0/1866749285 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c58c0 0x7fd2c00c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1866749285 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c58c0 msgr2=0x7fd2c00c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 --2- 172.21.15.146:0/1866749285 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c58c0 0x7fd2c00c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1866749285 >> v1:172.21.15.161:6789/0 conn(0x7fd2c0152850 legacy=0x7fd2c0158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:24.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.448+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1866749285 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c015bd90 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.449+0000 7fd2b7e00640 1 --2- 172.21.15.146:0/1866749285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c6bb0 0x7fd2c00c9700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009870 tx=0x7fd2a802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63c6164934d5cab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:24.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2bfe00640 1 -- 172.21.15.146:0/1866749285 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8037070 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2bfe00640 1 -- 172.21.15.146:0/1866749285 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a8032440 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2bfe00640 1 -- 172.21.15.146:0/1866749285 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8032810 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1866749285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c6bb0 msgr2=0x7fd2c00c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1866749285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c6bb0 0x7fd2c00c9700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009870 tx=0x7fd2a802a250 comp rx=0 tx=0).stop 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1866749285 shutdown_connections 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1866749285 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c58c0 0x7fd2c00c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1866749285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c6bb0 0x7fd2c00c9700 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1866749285 >> 172.21.15.146:0/1866749285 conn(0x7fd2c006f3c0 msgr2=0x7fd2c006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1866749285 shutdown_connections 2024-09-16T01:49:24.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.450+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1866749285 wait complete. 2024-09-16T01:49:24.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.452+0000 7fd2c7a00640 1 Processor -- start 2024-09-16T01:49:24.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.452+0000 7fd2c7a00640 1 -- start start 2024-09-16T01:49:24.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c7a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 0x7fd2c00c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:24.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c7a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c6bb0 0x7fd2c014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:24.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c7a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2c0152850 0x7fd2c0149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c7a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c015c630 con 0x7fd2c00c58c0 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c7a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c015c350 con 0x7fd2c0152850 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c7a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c0070200 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 0x7fd2c00c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 0x7fd2c00c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50194/0 (socket says 172.21.15.146:50194) 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c5000640 1 -- 172.21.15.146:0/1418489525 learned_addr learned my addr 172.21.15.146:0/1418489525 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2b7e00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c6bb0 0x7fd2c014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2c5a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2c0152850 0x7fd2c0149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2c0152850 msgr2=0x7fd2c0149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2b7e00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2c0152850 0x7fd2c0149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1418489525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 msgr2=0x7fd2c00c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.453+0000 7fd2b7e00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 0x7fd2c00c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2b7e00640 1 -- 172.21.15.146:0/1418489525 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c00ca620 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2c5000640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 0x7fd2c00c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2b7e00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c6bb0 0x7fd2c014ad40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8031000 tx=0x7fd2a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8041020 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a8030660 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c00ca900 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8037040 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.454+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c01f22a0 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.455+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c006fcd0 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.456+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd2a8044020 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.457+0000 7fd2bd600640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd29c06c2b0 0x7fd29c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:24.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.457+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd2a80ab980 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.457+0000 7fd2c5000640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd29c06c2b0 0x7fd29c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:24.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.458+0000 7fd2c5000640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd29c06c2b0 0x7fd29c06e770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0007fd0 tx=0x7fd2b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:24.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.462+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd2c006fcd0 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.671+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2c00718a0 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.673+0000 7fd2bd600640 1 -- 172.21.15.146:0/1418489525 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6956 (secure 0 0 0) 0x7fd2a804b030 con 0x7fd2c00c6bb0 2024-09-16T01:49:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:24.675 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":15,"btime":"2024-09-16T01:49:20:929876+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:20.929867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4291,4288,4277,4259]},"id":1}]} 2024-09-16T01:49:24.675 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 15 2024-09-16T01:49:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd29c06c2b0 msgr2=0x7fd29c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd29c06c2b0 0x7fd29c06e770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0007fd0 tx=0x7fd2b0009040 comp rx=0 tx=0).stop 2024-09-16T01:49:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c6bb0 msgr2=0x7fd2c014ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c6bb0 0x7fd2c014ad40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8031000 tx=0x7fd2a80046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 shutdown_connections 2024-09-16T01:49:24.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd29c06c2b0 0x7fd29c06e770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2c0152850 0x7fd2c0149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd2c00c6bb0 0x7fd2c014ad40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 --2- 172.21.15.146:0/1418489525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2c00c58c0 0x7fd2c00c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 >> 172.21.15.146:0/1418489525 conn(0x7fd2c006f3c0 msgr2=0x7fd2c014f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 shutdown_connections 2024-09-16T01:49:24.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:24.675+0000 7fd2c7a00640 1 -- 172.21.15.146:0/1418489525 wait complete. 2024-09-16T01:49:25.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:48:58.207792+0000', 'modified': '2024-09-16T01:49:20.929867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4288, 'mds_2': 4277, 'mds_3': 4259, 'mds_4': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'd', 'rank': 4, 'incarnation': 15, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.161:6837/3568385147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3568385147}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3568385147}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4259': {'gid': 4259, 'name': 'f', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6839/2836611107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2836611107}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2836611107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4277': {'gid': 4277, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.146:6837/835076388', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 835076388}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 835076388}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4288': {'gid': 4288, 'name': 'a', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.146:6833/1022834380', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6832', 'nonce': 1022834380}, {'type': 'v1', 'addr': '172.21.15.146:6833', 'nonce': 1022834380}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4291': {'gid': 4291, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6836/1807355541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 1807355541}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 1807355541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4291, 'qdb_cluster': [4291, 4288, 4277, 4259]} 2024-09-16T01:49:25.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4256:up:creating 2024-09-16T01:49:25.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:49:25.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:49:26.299 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:26.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.461+0000 7f9a9be00640 1 Processor -- start 2024-09-16T01:49:26.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.461+0000 7f9a9be00640 1 -- start start 2024-09-16T01:49:26.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a9be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a940c6410 0x7f9a94158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:26.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a9be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a94151e70 con 0x7f9a94151a90 2024-09-16T01:49:26.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a9be00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a940c8130 con 0x7f9a940c5280 2024-09-16T01:49:26.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a9be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9a940c8410 con 0x7f9a940c6410 2024-09-16T01:49:26.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a99400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9a94151a90 0x7f9a940c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50090/0 (socket says 172.21.15.146:50090) 2024-09-16T01:49:26.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a99400640 1 -- 172.21.15.146:0/2234196438 learned_addr learned my addr 172.21.15.146:0/2234196438 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:26.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.462+0000 7f9a98a00640 1 --2- 172.21.15.146:0/2234196438 >> v2:172.21.15.146:3300/0 conn(0x7f9a940c6410 0x7f9a94158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.463+0000 7f9a93e00640 1 -- 172.21.15.146:0/2234196438 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297341119 0 0) 0x7f9a94151e70 con 0x7f9a94151a90 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.463+0000 7f9a93e00640 1 -- 172.21.15.146:0/2234196438 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a7c005730 con 0x7f9a94151a90 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.463+0000 7f9a98a00640 1 -- 172.21.15.146:0/2234196438 >> v1:172.21.15.161:6789/0 conn(0x7f9a940c5280 legacy=0x7f9a94161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.463+0000 7f9a93e00640 1 -- 172.21.15.146:0/2234196438 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275369715 0 0) 0x7f9a940c8130 con 0x7f9a940c5280 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.463+0000 7f9a98a00640 1 -- 172.21.15.146:0/2234196438 >> v1:172.21.15.146:6789/0 conn(0x7f9a94151a90 legacy=0x7f9a940c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.463+0000 7f9a98a00640 1 -- 172.21.15.146:0/2234196438 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a940c86f0 con 0x7f9a940c6410 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.464+0000 7f9a98a00640 1 --2- 172.21.15.146:0/2234196438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c6410 0x7f9a94158040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a800097a0 tx=0x7f9a8002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d642f870348944b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.464+0000 7f9a93e00640 1 -- 172.21.15.146:0/2234196438 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80037070 con 0x7f9a940c6410 2024-09-16T01:49:26.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.464+0000 7f9a93e00640 1 -- 172.21.15.146:0/2234196438 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a80032440 con 0x7f9a940c6410 2024-09-16T01:49:26.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.464+0000 7f9a93e00640 1 -- 172.21.15.146:0/2234196438 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80032810 con 0x7f9a940c6410 2024-09-16T01:49:26.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 -- 172.21.15.146:0/2234196438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c6410 msgr2=0x7f9a94158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 --2- 172.21.15.146:0/2234196438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c6410 0x7f9a94158040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9a800097a0 tx=0x7f9a8002a230 comp rx=0 tx=0).stop 2024-09-16T01:49:26.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 -- 172.21.15.146:0/2234196438 shutdown_connections 2024-09-16T01:49:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 --2- 172.21.15.146:0/2234196438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c6410 0x7f9a94158040 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 -- 172.21.15.146:0/2234196438 >> 172.21.15.146:0/2234196438 conn(0x7f9a9406f3c0 msgr2=0x7f9a9406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 -- 172.21.15.146:0/2234196438 shutdown_connections 2024-09-16T01:49:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.465+0000 7f9a9be00640 1 -- 172.21.15.146:0/2234196438 wait complete. 2024-09-16T01:49:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.466+0000 7f9a9be00640 1 Processor -- start 2024-09-16T01:49:26.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.467+0000 7f9a9be00640 1 -- start start 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.467+0000 7f9a9be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 0x7f9a94149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.467+0000 7f9a9be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a940c6410 0x7f9a94152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a9be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a94151a90 0x7f9a9414cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a9be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94157b20 con 0x7f9a940c5280 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a9be00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a94157840 con 0x7f9a94151a90 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a9be00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9a94158580 con 0x7f9a940c6410 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 0x7f9a94149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 0x7f9a94149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50212/0 (socket says 172.21.15.146:50212) 2024-09-16T01:49:26.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 -- 172.21.15.146:0/3921159882 learned_addr learned my addr 172.21.15.146:0/3921159882 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a98a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a940c6410 0x7f9a94152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99e00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a94151a90 0x7f9a9414cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 -- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a940c6410 msgr2=0x7f9a94152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a940c6410 0x7f9a94152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 -- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a94151a90 msgr2=0x7f9a9414cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a94151a90 0x7f9a9414cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 -- 172.21.15.146:0/3921159882 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a941ec870 con 0x7f9a940c5280 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.468+0000 7f9a99400640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 0x7f9a94149260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800c3c0 tx=0x7f9a8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.469+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88017070 con 0x7f9a940c5280 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.469+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88005ce0 con 0x7f9a940c5280 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.469+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a941ecb30 con 0x7f9a940c5280 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.469+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a880072f0 con 0x7f9a940c5280 2024-09-16T01:49:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.469+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a94148a60 con 0x7f9a940c5280 2024-09-16T01:49:26.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.470+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a940c71f0 con 0x7f9a940c5280 2024-09-16T01:49:26.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.471+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9a8801a070 con 0x7f9a940c5280 2024-09-16T01:49:26.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.472+0000 7f9a91600640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7406c2b0 0x7f9a7406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:26.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.472+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9a8808a820 con 0x7f9a940c5280 2024-09-16T01:49:26.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.472+0000 7f9a98a00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7406c2b0 0x7f9a7406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:26.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.473+0000 7f9a98a00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7406c2b0 0x7f9a7406e770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9a80034000 tx=0x7f9a8002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:26.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.477+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9a880588a0 con 0x7f9a940c5280 2024-09-16T01:49:26.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.689+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a94157620 con 0x7f9a940c5280 2024-09-16T01:49:26.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.689+0000 7f9a91600640 1 -- 172.21.15.146:0/3921159882 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6956 (secure 0 0 0) 0x7f9a8805cba0 con 0x7f9a940c5280 2024-09-16T01:49:26.691 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:26.692 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":15,"btime":"2024-09-16T01:49:20:929876+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:20.929867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4291,4288,4277,4259]},"id":1}]} 2024-09-16T01:49:26.692 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 15 2024-09-16T01:49:26.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7406c2b0 msgr2=0x7f9a7406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7406c2b0 0x7f9a7406e770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9a80034000 tx=0x7f9a8002e000 comp rx=0 tx=0).stop 2024-09-16T01:49:26.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 msgr2=0x7f9a94149260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:26.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 0x7f9a94149260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800c3c0 tx=0x7f9a8800c880 comp rx=0 tx=0).stop 2024-09-16T01:49:26.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 shutdown_connections 2024-09-16T01:49:26.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7406c2b0 0x7f9a7406e770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a94151a90 0x7f9a9414cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a940c6410 0x7f9a94152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 --2- 172.21.15.146:0/3921159882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a940c5280 0x7f9a94149260 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:26.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 >> 172.21.15.146:0/3921159882 conn(0x7f9a9406f3c0 msgr2=0x7f9a941520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:26.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 shutdown_connections 2024-09-16T01:49:26.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:26.692+0000 7f9a9be00640 1 -- 172.21.15.146:0/3921159882 wait complete. 2024-09-16T01:49:27.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:48:58.207792+0000', 'modified': '2024-09-16T01:49:20.929867+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4288, 'mds_2': 4277, 'mds_3': 4259, 'mds_4': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'd', 'rank': 4, 'incarnation': 15, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.161:6837/3568385147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3568385147}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3568385147}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4259': {'gid': 4259, 'name': 'f', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6839/2836611107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2836611107}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2836611107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4277': {'gid': 4277, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.146:6837/835076388', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 835076388}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 835076388}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4288': {'gid': 4288, 'name': 'a', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.146:6833/1022834380', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6832', 'nonce': 1022834380}, {'type': 'v1', 'addr': '172.21.15.146:6833', 'nonce': 1022834380}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4291': {'gid': 4291, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6836/1807355541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 1807355541}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 1807355541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4291, 'qdb_cluster': [4291, 4288, 4277, 4259]} 2024-09-16T01:49:27.315 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4256:up:creating 2024-09-16T01:49:28.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:49:28.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.478+0000 7f5b64600640 1 Processor -- start 2024-09-16T01:49:28.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.478+0000 7f5b64600640 1 -- start start 2024-09-16T01:49:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b64600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5b5c06ce90 0x7f5b5c15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b64600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b5c15b9f0 con 0x7f5b5c0c6840 2024-09-16T01:49:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b64600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b5c067ba0 con 0x7f5b5c06d270 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b64600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5b5c067950 con 0x7f5b5c06ce90 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b61c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5b5c06ce90 0x7f5b5c15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b62600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5b5c0c6840 0x7f5b5c0725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50098/0 (socket says 172.21.15.146:50098) 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b62600640 1 -- 172.21.15.146:0/984073343 learned_addr learned my addr 172.21.15.146:0/984073343 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b61c00640 1 -- 172.21.15.146:0/984073343 >> v1:172.21.15.161:6789/0 conn(0x7f5b5c06d270 legacy=0x7f5b5c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b61c00640 1 -- 172.21.15.146:0/984073343 >> v1:172.21.15.146:6789/0 conn(0x7f5b5c0c6840 legacy=0x7f5b5c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b61c00640 1 -- 172.21.15.146:0/984073343 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5c067e80 con 0x7f5b5c06ce90 2024-09-16T01:49:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.480+0000 7f5b5be00640 1 -- 172.21.15.146:0/984073343 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938860793 0 0) 0x7f5b54001490 con 0x7f5b5c0c6840 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.481+0000 7f5b61c00640 1 --2- 172.21.15.146:0/984073343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c06ce90 0x7f5b5c15b4b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5b500097a0 tx=0x7f5b5002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0def7da794bdb66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b5be00640 1 -- 172.21.15.146:0/984073343 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50037070 con 0x7f5b5c06ce90 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b5be00640 1 -- 172.21.15.146:0/984073343 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b50032440 con 0x7f5b5c06ce90 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b5be00640 1 -- 172.21.15.146:0/984073343 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50032810 con 0x7f5b5c06ce90 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 -- 172.21.15.146:0/984073343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c06ce90 msgr2=0x7f5b5c15b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 --2- 172.21.15.146:0/984073343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c06ce90 0x7f5b5c15b4b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5b500097a0 tx=0x7f5b5002a230 comp rx=0 tx=0).stop 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 -- 172.21.15.146:0/984073343 shutdown_connections 2024-09-16T01:49:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 --2- 172.21.15.146:0/984073343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c06ce90 0x7f5b5c15b4b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 -- 172.21.15.146:0/984073343 >> 172.21.15.146:0/984073343 conn(0x7f5b5c06d8e0 msgr2=0x7f5b5c06dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 -- 172.21.15.146:0/984073343 shutdown_connections 2024-09-16T01:49:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.482+0000 7f5b64600640 1 -- 172.21.15.146:0/984073343 wait complete. 2024-09-16T01:49:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.484+0000 7f5b64600640 1 Processor -- start 2024-09-16T01:49:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.484+0000 7f5b64600640 1 -- start start 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b64600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b5c06ce90 0x7f5b5c152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b64600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b5c06d270 0x7f5b5c1535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b64600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 0x7f5b5c153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b64600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c06e9c0 con 0x7f5b5c0c6840 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b64600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5c06e6e0 con 0x7f5b5c06d270 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b64600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5b5c0756f0 con 0x7f5b5c06ce90 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 0x7f5b5c153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 0x7f5b5c153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50222/0 (socket says 172.21.15.146:50222) 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 -- 172.21.15.146:0/288283534 learned_addr learned my addr 172.21.15.146:0/288283534 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b61c00640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b5c06ce90 0x7f5b5c152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b61200640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b5c06d270 0x7f5b5c1535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 -- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b5c06ce90 msgr2=0x7f5b5c152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b5c06ce90 0x7f5b5c152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 -- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b5c06d270 msgr2=0x7f5b5c1535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b5c06d270 0x7f5b5c1535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.485+0000 7f5b62600640 1 -- 172.21.15.146:0/288283534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5c0c48e0 con 0x7f5b5c0c6840 2024-09-16T01:49:28.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.486+0000 7f5b62600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 0x7f5b5c153b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c700 tx=0x7f5b5400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:28.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.486+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54018070 con 0x7f5b5c0c6840 2024-09-16T01:49:28.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.486+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b5400ce10 con 0x7f5b5c0c6840 2024-09-16T01:49:28.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.486+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5400a860 con 0x7f5b5c0c6840 2024-09-16T01:49:28.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.486+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5c0cb620 con 0x7f5b5c0c6840 2024-09-16T01:49:28.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.486+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b5c0c4e60 con 0x7f5b5c0c6840 2024-09-16T01:49:28.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.487+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b5c15b4b0 con 0x7f5b5c0c6840 2024-09-16T01:49:28.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.488+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5b5401b070 con 0x7f5b5c0c6840 2024-09-16T01:49:28.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.489+0000 7f5b59600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b3c06c2b0 0x7f5b3c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:28.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.489+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5b5408a910 con 0x7f5b5c0c6840 2024-09-16T01:49:28.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.489+0000 7f5b61c00640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b3c06c2b0 0x7f5b3c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:28.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.490+0000 7f5b61c00640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b3c06c2b0 0x7f5b3c06e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b50034000 tx=0x7f5b5002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:28.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.493+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5b5c15b9f0 con 0x7f5b5c0c6840 2024-09-16T01:49:28.524 INFO:tasks.ceph.mds.c.smithi146.stderr:2024-09-16T01:49:28.522+0000 7f36b7200640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-09-16T01:49:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.704+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b5c150540 con 0x7f5b5c0c6840 2024-09-16T01:49:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.705+0000 7f5b59600640 1 -- 172.21.15.146:0/288283534 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6960 (secure 0 0 0) 0x7f5b54058a20 con 0x7f5b5c0c6840 2024-09-16T01:49:28.707 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:28.707 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":16,"btime":"2024-09-16T01:49:26:945057+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:26.945053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4256,4259,4277,4288,4291]},"id":1}]} 2024-09-16T01:49:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 16 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b3c06c2b0 msgr2=0x7f5b3c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b3c06c2b0 0x7f5b3c06e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b50034000 tx=0x7f5b5002e000 comp rx=0 tx=0).stop 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 msgr2=0x7f5b5c153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 0x7f5b5c153b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c700 tx=0x7f5b5400cbc0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 shutdown_connections 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5b3c06c2b0 0x7f5b3c06e770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b5c0c6840 0x7f5b5c153b30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b5c06d270 0x7f5b5c1535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.707+0000 7f5b64600640 1 --2- 172.21.15.146:0/288283534 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b5c06ce90 0x7f5b5c152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.708+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 >> 172.21.15.146:0/288283534 conn(0x7f5b5c06d8e0 msgr2=0x7f5b5c159e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.708+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 shutdown_connections 2024-09-16T01:49:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:28.708+0000 7f5b64600640 1 -- 172.21.15.146:0/288283534 wait complete. 2024-09-16T01:49:29.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:48:58.207792+0000', 'modified': '2024-09-16T01:49:26.945053+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4291, 'mds_1': 4288, 'mds_2': 4277, 'mds_3': 4259, 'mds_4': 4256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4256': {'gid': 4256, 'name': 'd', 'rank': 4, 'incarnation': 15, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.161:6837/3568385147', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6835', 'nonce': 3568385147}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 3568385147}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4259': {'gid': 4259, 'name': 'f', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.161:6839/2836611107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 2836611107}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 2836611107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4277': {'gid': 4277, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.146:6837/835076388', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 835076388}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 835076388}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4288': {'gid': 4288, 'name': 'a', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.146:6833/1022834380', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6832', 'nonce': 1022834380}, {'type': 'v1', 'addr': '172.21.15.146:6833', 'nonce': 1022834380}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4291': {'gid': 4291, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6836/1807355541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 1807355541}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 1807355541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4291, 'qdb_cluster': [4256, 4259, 4277, 4288, 4291]} 2024-09-16T01:49:29.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-16T01:49:29.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-16T01:49:29.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.485+0000 7f180e000640 1 Processor -- start 2024-09-16T01:49:29.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.485+0000 7f180e000640 1 -- start start 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f180e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f180806c4f0 0x7f18080c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f180e000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1808158630 con 0x7f18080c8ef0 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f180e000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1808073f00 con 0x7f180814f1d0 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f180e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1808158f20 con 0x7f180806c4f0 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f17ffe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f180806c4f0 0x7f18080c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f180d600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f18080c8ef0 0x7f1808162ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50102/0 (socket says 172.21.15.146:50102) 2024-09-16T01:49:29.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f180d600640 1 -- 172.21.15.146:0/1521329643 learned_addr learned my addr 172.21.15.146:0/1521329643 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:29.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f17ffe00640 1 -- 172.21.15.146:0/1521329643 >> v1:172.21.15.161:6789/0 conn(0x7f180814f1d0 legacy=0x7f180815bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f17ffe00640 1 -- 172.21.15.146:0/1521329643 >> v1:172.21.15.146:6789/0 conn(0x7f18080c8ef0 legacy=0x7f1808162ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f1807e00640 1 -- 172.21.15.146:0/1521329643 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1253236184 0 0) 0x7f1808158630 con 0x7f18080c8ef0 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.487+0000 7f17ffe00640 1 -- 172.21.15.146:0/1521329643 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18081593b0 con 0x7f180806c4f0 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.488+0000 7f17ffe00640 1 --2- 172.21.15.146:0/1521329643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f180806c4f0 0x7f18080c89b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097a0 tx=0x7f17f002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0cf9fc09afaa13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f1807e00640 1 -- 172.21.15.146:0/1521329643 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0037070 con 0x7f180806c4f0 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f1807e00640 1 -- 172.21.15.146:0/1521329643 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f0032440 con 0x7f180806c4f0 2024-09-16T01:49:29.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f1807e00640 1 -- 172.21.15.146:0/1521329643 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0032810 con 0x7f180806c4f0 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 -- 172.21.15.146:0/1521329643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f180806c4f0 msgr2=0x7f18080c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 --2- 172.21.15.146:0/1521329643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f180806c4f0 0x7f18080c89b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097a0 tx=0x7f17f002a230 comp rx=0 tx=0).stop 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 -- 172.21.15.146:0/1521329643 shutdown_connections 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 --2- 172.21.15.146:0/1521329643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f180806c4f0 0x7f18080c89b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 -- 172.21.15.146:0/1521329643 >> 172.21.15.146:0/1521329643 conn(0x7f1808067950 msgr2=0x7f1808067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 -- 172.21.15.146:0/1521329643 shutdown_connections 2024-09-16T01:49:29.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.489+0000 7f180e000640 1 -- 172.21.15.146:0/1521329643 wait complete. 2024-09-16T01:49:29.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.491+0000 7f180e000640 1 Processor -- start 2024-09-16T01:49:29.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.491+0000 7f180e000640 1 -- start start 2024-09-16T01:49:29.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f180806c4f0 0x7f18080c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18080c8ef0 0x7f18081f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f180814f1d0 0x7f180813f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1808158bb0 con 0x7f18080c8ef0 2024-09-16T01:49:29.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180e000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f18080c9300 con 0x7f180806c4f0 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180e000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f18080741c0 con 0x7f180814f1d0 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f180806c4f0 0x7f18080c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18080c8ef0 0x7f18081f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f180806c4f0 0x7f18080c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56776/0 (socket says 172.21.15.146:56776) 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180cc00640 1 -- 172.21.15.146:0/3238011622 learned_addr learned my addr 172.21.15.146:0/3238011622 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f180d600640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f180814f1d0 0x7f180813f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f17ffe00640 1 -- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f180814f1d0 msgr2=0x7f180813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f17ffe00640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f180814f1d0 0x7f180813f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f17ffe00640 1 -- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f180806c4f0 msgr2=0x7f18080c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f17ffe00640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f180806c4f0 0x7f18080c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.492+0000 7f17ffe00640 1 -- 172.21.15.146:0/3238011622 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18081f4590 con 0x7f18080c8ef0 2024-09-16T01:49:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.493+0000 7f17ffe00640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18080c8ef0 0x7f18081f3b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f18080b1710 tx=0x7f17f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:29.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.493+0000 7f1805600640 1 -- 172.21.15.146:0/3238011622 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0032e60 con 0x7f18080c8ef0 2024-09-16T01:49:29.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.493+0000 7f1805600640 1 -- 172.21.15.146:0/3238011622 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f00307b0 con 0x7f18080c8ef0 2024-09-16T01:49:29.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.493+0000 7f1805600640 1 -- 172.21.15.146:0/3238011622 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0037040 con 0x7f18080c8ef0 2024-09-16T01:49:29.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.493+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1808141300 con 0x7f18080c8ef0 2024-09-16T01:49:29.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.493+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18081418c0 con 0x7f18080c8ef0 2024-09-16T01:49:29.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.495+0000 7f1805600640 1 -- 172.21.15.146:0/3238011622 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f17f0044020 con 0x7f18080c8ef0 2024-09-16T01:49:29.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.496+0000 7f1805600640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f17e406c2e0 0x7f17e406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.496+0000 7f1805600640 1 -- 172.21.15.146:0/3238011622 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f17f00ab5e0 con 0x7f18080c8ef0 2024-09-16T01:49:29.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.496+0000 7f180cc00640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f17e406c2e0 0x7f17e406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 Processor -- start 2024-09-16T01:49:29.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 -- start start 2024-09-16T01:49:29.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f180cc00640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f17e406c2e0 0x7f17e406e7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f17f8000f60 tx=0x7f17f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:29.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a303f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a303fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3003640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f9a2f643b0 con 0x55f9a3007330 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f9a2f69410 con 0x55f9a3007710 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.497+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55f9a2f11620 con 0x55f9a3004b60 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f180cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a303f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a303fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f180d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3003640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f180cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a303f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50260/0 (socket says 172.21.15.146:50260) 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f180cc00640 1 -- 172.21.15.146:0/132206898 learned_addr learned my addr 172.21.15.146:0/132206898 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a303fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52342/0 (socket says 172.21.15.146:52342) 2024-09-16T01:49:29.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 -- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 msgr2=0x55f9a3003640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3003640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 -- 172.21.15.146:0/132206898 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 msgr2=0x55f9a303f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a303f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 -- 172.21.15.146:0/132206898 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a2c7c2d0 con 0x55f9a3004b60 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f180cc00640 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a303f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17ffe00640 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a303fb00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f17f00a9ad0 tx=0x7f17f0009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6010608a264dfd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.498+0000 7f17fd200640 1 -- 172.21.15.146:0/132206898 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f007d9f0 con 0x55f9a3004b60 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f17fd200640 1 -- 172.21.15.146:0/132206898 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f0082220 con 0x55f9a3004b60 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f17fd200640 1 -- 172.21.15.146:0/132206898 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0075d60 con 0x55f9a3004b60 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 -- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 msgr2=0x55f9a303fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:29.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a303fb00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f17f00a9ad0 tx=0x7f17f0009190 comp rx=0 tx=0).stop 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 -- 172.21.15.146:0/132206898 shutdown_connections 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3003640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a303fb00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 --2- 172.21.15.146:0/132206898 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a303f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 -- 172.21.15.146:0/132206898 >> 172.21.15.146:0/132206898 conn(0x55f9a3005a50 msgr2=0x55f9a3005e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 -- 172.21.15.146:0/132206898 shutdown_connections 2024-09-16T01:49:29.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.499+0000 7f1812421740 1 -- 172.21.15.146:0/132206898 wait complete. 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.500+0000 7f1812421740 1 -- start start 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.500+0000 7f1812421740 1 Processor -- start 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a302bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a302fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a302ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f9a2f24780 con 0x55f9a3007330 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f9a2f653d0 con 0x55f9a3007710 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55f9a3006bc0 con 0x55f9a3004b60 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f1812421740 10 client.0 _ll_register_callbacks cb 0x55f9a2f625d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a302fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a302bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a302fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50272/0 (socket says 172.21.15.146:50272) 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f17ffe00640 1 -- 172.21.15.146:0/2632952308 learned_addr learned my addr 172.21.15.146:0/2632952308 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a302bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52354/0 (socket says 172.21.15.146:52354) 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.501+0000 7f180d600640 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a302ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17ffe00640 1 -- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 msgr2=0x55f9a302bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17ffe00640 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a302bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17ffe00640 1 -- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 msgr2=0x55f9a302ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17ffe00640 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a302ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17ffe00640 1 -- 172.21.15.146:0/2632952308 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a2d466b0 con 0x55f9a3007330 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17ffe00640 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a302fdf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17f00ab010 tx=0x7f17f00a99d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a185737bfab9d14 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17f5600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:29.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17f5600640 1 -- 172.21.15.146:0/2632952308 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f007d9f0 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17f5600640 1 -- 172.21.15.146:0/2632952308 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f17f0079b60 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17f0082220 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f17f5600640 1 -- 172.21.15.146:0/2632952308 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f17f0081d90 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f1812421740 10 client.4578 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.502+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55f9a3029200 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f17f5600640 1 -- 172.21.15.146:0/2632952308 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f0082220 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f17f5600640 1 -- 172.21.15.146:0/2632952308 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7f17f003a030 con 0x55f9a3007330 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f1812421740 10 client.4578 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f1812421740 10 client.4578 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f1812421740 20 client.4578 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f1812421740 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a302cfa0 0x55f9a30655d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f1812421740 4 client.4578 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 --> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9a2f76b40 con 0x55f9a302cfa0 2024-09-16T01:49:29.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.503+0000 7f180cc00640 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a302cfa0 0x55f9a30655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:29.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.504+0000 7f180cc00640 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a302cfa0 0x55f9a30655d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=701b22c44aa4558a server_cookie=9d2905696ba483b in_seq=0 out_seq=0 2024-09-16T01:49:29.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:29.504+0000 7f17f5600640 10 client.4578 ms_handle_connect on v2:172.21.15.161:6835/3568385147 2024-09-16T01:49:31.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.951+0000 7f17f5600640 1 -- 172.21.15.146:0/2632952308 <== mds.4 v2:172.21.15.161:6835/3568385147 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f9a2f76b40 con 0x55f9a302cfa0 2024-09-16T01:49:31.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.951+0000 7f17f5600640 10 client.4578 handle_command_reply: tid=0 2024-09-16T01:49:31.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.951+0000 7f1812421740 1 client.4578 shutdown 2024-09-16T01:49:31.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.951+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 msgr2=0x55f9a302fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:31.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.951+0000 7f1812421740 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a302fdf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17f00ab010 tx=0x7f17f00a99d0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 shutdown_connections 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a302cfa0 0x55f9a30655d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a302ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a302fdf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 --2- 172.21.15.146:0/2632952308 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a302bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f17f5600640 0 client.4578 ms_handle_reset on v2:172.21.15.161:6835/3568385147 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 >> 172.21.15.146:0/2632952308 conn(0x55f9a3005a50 msgr2=0x55f9a3049670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 shutdown_connections 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 1 -- 172.21.15.146:0/2632952308 wait complete. 2024-09-16T01:49:31.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.952+0000 7f1812421740 20 client.4578 trim_cache size 0 max 16384 2024-09-16T01:49:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.973+0000 7f1812421740 1 Processor -- start 2024-09-16T01:49:31.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.973+0000 7f1812421740 1 -- start start 2024-09-16T01:49:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a3000310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f1812421740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a3004050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3004590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f1812421740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f18081fadf0 con 0x55f9a3007330 2024-09-16T01:49:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f9a2f653d0 con 0x55f9a3007710 2024-09-16T01:49:31.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55f9a2f24780 con 0x55f9a3004b60 2024-09-16T01:49:31.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a3004050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f180d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3004590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a3000310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a3004050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50286/0 (socket says 172.21.15.146:50286) 2024-09-16T01:49:31.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 -- 172.21.15.146:0/656314863 learned_addr learned my addr 172.21.15.146:0/656314863 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 -- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 msgr2=0x55f9a3000310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a3000310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 -- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 msgr2=0x55f9a3004590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3004590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f17ffe00640 1 -- 172.21.15.146:0/656314863 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a3029ef0 con 0x55f9a3007330 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.974+0000 7f180d600640 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3004590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.975+0000 7f17ffe00640 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a3004050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097a0 tx=0x7f17f008e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a4da3abf54a16d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:31.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.975+0000 7f17f7e00640 1 -- 172.21.15.146:0/656314863 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f007d9f0 con 0x55f9a3007330 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.975+0000 7f17f7e00640 1 -- 172.21.15.146:0/656314863 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f0082220 con 0x55f9a3007330 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.975+0000 7f17f7e00640 1 -- 172.21.15.146:0/656314863 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0079050 con 0x55f9a3007330 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 -- 172.21.15.146:0/656314863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 msgr2=0x55f9a3004050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a3004050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17f00097a0 tx=0x7f17f008e000 comp rx=0 tx=0).stop 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 -- 172.21.15.146:0/656314863 shutdown_connections 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3007710 0x55f9a3004590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007330 0x55f9a3004050 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 --2- 172.21.15.146:0/656314863 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3004b60 0x55f9a3000310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 -- 172.21.15.146:0/656314863 >> 172.21.15.146:0/656314863 conn(0x55f9a3005a50 msgr2=0x55f9a3034080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 -- 172.21.15.146:0/656314863 shutdown_connections 2024-09-16T01:49:31.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.976+0000 7f1812421740 1 -- 172.21.15.146:0/656314863 wait complete. 2024-09-16T01:49:31.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.977+0000 7f1812421740 1 -- start start 2024-09-16T01:49:31.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.977+0000 7f1812421740 1 Processor -- start 2024-09-16T01:49:31.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 0x55f9a302b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 0x55f9a30be200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 0x55f9a30be740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f9a302dcc0 con 0x55f9a3007710 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55f9a3030330 con 0x55f9a3004b60 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55f9a3059f50 con 0x55f9a3007330 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f1812421740 10 client.0 _ll_register_callbacks cb 0x7f17f0090070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f180d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 0x55f9a30be740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 0x55f9a30be200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 0x55f9a302b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f180cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 0x55f9a302b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56826/0 (socket says 172.21.15.146:56826) 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f17ffe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 0x55f9a30be200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52364/0 (socket says 172.21.15.146:52364) 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f180d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 0x55f9a30be740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50300/0 (socket says 172.21.15.146:50300) 2024-09-16T01:49:31.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.978+0000 7f180cc00640 1 -- 172.21.15.146:0/3737646932 learned_addr learned my addr 172.21.15.146:0/3737646932 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180cc00640 1 -- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 msgr2=0x55f9a30be200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180cc00640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 0x55f9a30be200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180cc00640 1 -- 172.21.15.146:0/3737646932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 msgr2=0x55f9a30be740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180cc00640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 0x55f9a30be740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180cc00640 1 -- 172.21.15.146:0/3737646932 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a3056b30 con 0x55f9a3004b60 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f17ffe00640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 0x55f9a30be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180d600640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 0x55f9a30be740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f180cc00640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 0x55f9a302b7e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17f800fa10 tx=0x7f17f8001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb88db543dcf7813 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f17f5600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-16T01:49:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f17f5600640 1 -- 172.21.15.146:0/3737646932 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8023720 con 0x55f9a3004b60 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f17f5600640 1 -- 172.21.15.146:0/3737646932 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f17f80238c0 con 0x55f9a3004b60 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.979+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17f0082220 con 0x55f9a3004b60 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f17f5600640 1 -- 172.21.15.146:0/3737646932 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f17f8038050 con 0x55f9a3004b60 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 10 client.4654 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55f9a30510c0 con 0x55f9a3004b60 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f17f5600640 1 -- 172.21.15.146:0/3737646932 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f0082220 con 0x55f9a3004b60 2024-09-16T01:49:31.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f17f5600640 1 -- 172.21.15.146:0/3737646932 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7f17f803b020 con 0x55f9a3004b60 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 10 client.4654 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 10 client.4654 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 20 client.4654 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a3048d40 0x55f9a30d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 4 client.4654 mds_command: new command op to 4256 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 --> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] -- command(tid 0: {"prefix": "status"}) -- 0x55f9a2f70e90 con 0x55f9a3048d40 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.980+0000 7f17ffe00640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a3048d40 0x55f9a30d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f17f5600640 10 client.4654 ms_handle_connect on v2:172.21.15.161:6835/3568385147 2024-09-16T01:49:31.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f17ffe00640 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a3048d40 0x55f9a30d0de0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9e851839ee0e80b server_cookie=6213d24afe58d787 in_seq=0 out_seq=0 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f17f5600640 1 -- 172.21.15.146:0/3737646932 <== mds.4 v2:172.21.15.161:6835/3568385147 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55f9a2f70e90 con 0x55f9a3048d40 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f17f5600640 10 client.4654 handle_command_reply: tid=0 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 client.4654 shutdown 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 msgr2=0x55f9a302b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 0x55f9a302b7e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17f800fa10 tx=0x7f17f8001120 comp rx=0 tx=0).stop 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 shutdown_connections 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:6835/3568385147,v1:172.21.15.161:6837/3568385147] conn(0x55f9a3048d40 0x55f9a30d0de0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f9a3007710 0x55f9a30be740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55f9a3007330 0x55f9a30be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f17f5600640 0 client.4654 ms_handle_reset on v2:172.21.15.161:6835/3568385147 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 --2- 172.21.15.146:0/3737646932 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55f9a3004b60 0x55f9a302b7e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.981+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 >> 172.21.15.146:0/3737646932 conn(0x55f9a3005a50 msgr2=0x55f9a3004f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 shutdown_connections 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f1812421740 1 -- 172.21.15.146:0/3737646932 wait complete. 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f1812421740 20 client.4654 trim_cache size 0 max 16384 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 4, 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4256, 2024-09-16T01:49:31.991 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 11.05107531, 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 16, 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 22, 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 22, 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 39.477123458999998 2024-09-16T01:49:31.992 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f17e406c2e0 msgr2=0x7f17e406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f180e000640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f17e406c2e0 0x7f17e406e7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f17f8000f60 tx=0x7f17f8009040 comp rx=0 tx=0).stop 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18080c8ef0 msgr2=0x7f18081f3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.982+0000 7f180e000640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18080c8ef0 0x7f18081f3b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f18080b1710 tx=0x7f17f00046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 shutdown_connections 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f17e406c2e0 0x7f17e406e7a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f180814f1d0 0x7f180813f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18080c8ef0 0x7f18081f3b20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 --2- 172.21.15.146:0/3238011622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f180806c4f0 0x7f18080c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:31.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 >> 172.21.15.146:0/3238011622 conn(0x7f1808067950 msgr2=0x7f18080c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:31.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 shutdown_connections 2024-09-16T01:49:31.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:31.983+0000 7f180e000640 1 -- 172.21.15.146:0/3238011622 wait complete. 2024-09-16T01:49:32.003 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-09-16T01:49:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.159+0000 7f0143e00640 1 Processor -- start 2024-09-16T01:49:32.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.159+0000 7f0143e00640 1 -- start start 2024-09-16T01:49:32.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0143e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01440c6b70 0x7f014415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0143e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c75a0 0x7f01440cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0143e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014406d140 con 0x7f01440cbab0 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0143e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f014415b880 con 0x7f01440c6b70 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0143e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f01440cab20 con 0x7f01440c75a0 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0142a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01440c6b70 0x7f014415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0142a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01440c6b70 0x7f014415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50302/0 (socket says 172.21.15.146:50302) 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0142a00640 1 -- 172.21.15.146:0/2613187618 learned_addr learned my addr 172.21.15.146:0/2613187618 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.160+0000 7f0142000640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c75a0 0x7f01440cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.161+0000 7f0142a00640 1 -- 172.21.15.146:0/2613187618 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c75a0 msgr2=0x7f01440cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.161+0000 7f0142a00640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c75a0 0x7f01440cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.161+0000 7f0142a00640 1 -- 172.21.15.146:0/2613187618 >> v1:172.21.15.161:6789/0 conn(0x7f01440cbab0 legacy=0x7f01440cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.161+0000 7f0141600640 1 -- 172.21.15.146:0/2613187618 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328340110 0 0) 0x7f014406d140 con 0x7f01440cbab0 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.161+0000 7f0142a00640 1 -- 172.21.15.146:0/2613187618 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014406d420 con 0x7f01440c6b70 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.161+0000 7f0142000640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c75a0 0x7f01440cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.162+0000 7f0142a00640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c6b70 0x7f014415b340 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0134009870 tx=0x7f013402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5272f148a6e7425 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.162+0000 7f0141600640 1 -- 172.21.15.146:0/2613187618 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134037070 con 0x7f01440c6b70 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.162+0000 7f0141600640 1 -- 172.21.15.146:0/2613187618 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0134032440 con 0x7f01440c6b70 2024-09-16T01:49:32.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.162+0000 7f0141600640 1 -- 172.21.15.146:0/2613187618 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134032810 con 0x7f01440c6b70 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 -- 172.21.15.146:0/2613187618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c6b70 msgr2=0x7f014415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c6b70 0x7f014415b340 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0134009870 tx=0x7f013402a130 comp rx=0 tx=0).stop 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 -- 172.21.15.146:0/2613187618 shutdown_connections 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c75a0 0x7f01440cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 --2- 172.21.15.146:0/2613187618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c6b70 0x7f014415b340 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 -- 172.21.15.146:0/2613187618 >> 172.21.15.146:0/2613187618 conn(0x7f01440ccc20 msgr2=0x7f01440cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 -- 172.21.15.146:0/2613187618 shutdown_connections 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.163+0000 7f0143e00640 1 -- 172.21.15.146:0/2613187618 wait complete. 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.164+0000 7f0143e00640 1 Processor -- start 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.164+0000 7f0143e00640 1 -- start start 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 0x7f014415ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c75a0 0x7f014414f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01440cbab0 0x7f014414f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f01440cb050 con 0x7f01440c75a0 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01440cad70 con 0x7f01440cbab0 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f014406cd20 con 0x7f01440c6b70 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0142a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 0x7f014415ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0142000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c75a0 0x7f014414f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0142a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 0x7f014415ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52382/0 (socket says 172.21.15.146:52382) 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0142a00640 1 -- 172.21.15.146:0/3218880825 learned_addr learned my addr 172.21.15.146:0/3218880825 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.165+0000 7f0143400640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01440cbab0 0x7f014414f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142000640 1 -- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 msgr2=0x7f014415ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142000640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 0x7f014415ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142000640 1 -- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01440cbab0 msgr2=0x7f014414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142000640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01440cbab0 0x7f014414f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142000640 1 -- 172.21.15.146:0/3218880825 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01440c58b0 con 0x7f01440c75a0 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142a00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 0x7f014415ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0142000640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c75a0 0x7f014414f310 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f013000e870 tx=0x7f013000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f013aa00640 1 -- 172.21.15.146:0/3218880825 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130017070 con 0x7f01440c75a0 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f013aa00640 1 -- 172.21.15.146:0/3218880825 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013000c940 con 0x7f01440c75a0 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f013aa00640 1 -- 172.21.15.146:0/3218880825 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013000cd30 con 0x7f01440c75a0 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01441f1270 con 0x7f01440c75a0 2024-09-16T01:49:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.166+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0144159710 con 0x7f01440c75a0 2024-09-16T01:49:32.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.169+0000 7f013aa00640 1 -- 172.21.15.146:0/3218880825 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f013001a070 con 0x7f01440c75a0 2024-09-16T01:49:32.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.170+0000 7f013aa00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f012406c2e0 0x7f012406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.170+0000 7f013aa00640 1 -- 172.21.15.146:0/3218880825 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f013008b1c0 con 0x7f01440c75a0 2024-09-16T01:49:32.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.170+0000 7f0142a00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f012406c2e0 0x7f012406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.170+0000 7f014c440740 1 Processor -- start 2024-09-16T01:49:32.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.171+0000 7f014c440740 1 -- start start 2024-09-16T01:49:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.171+0000 7f0142a00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f012406c2e0 0x7f012406e7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0134000c00 tx=0x7f013402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.171+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfe330 0x55d33de365c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f014c440740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfbb60 0x55d33de36b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddfa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f014c440740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d33dd5b3b0 con 0x55d33ddfbb60 2024-09-16T01:49:32.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d33dd60410 con 0x55d33ddfe710 2024-09-16T01:49:32.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55d33dd08620 con 0x55d33ddfe330 2024-09-16T01:49:32.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfe330 0x55d33de365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfbb60 0x55d33de36b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfe330 0x55d33de365c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52388/0 (socket says 172.21.15.146:52388) 2024-09-16T01:49:32.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142a00640 1 -- 172.21.15.146:0/2210684572 learned_addr learned my addr 172.21.15.146:0/2210684572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0143400640 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddfa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 -- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfe330 msgr2=0x55d33de365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfe330 0x55d33de365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 -- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 msgr2=0x55d33ddfa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddfa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 -- 172.21.15.146:0/2210684572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d33da732d0 con 0x55d33ddfbb60 2024-09-16T01:49:32.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.172+0000 7f0142000640 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfbb60 0x55d33de36b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0130089620 tx=0x7f0130059300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d8739f1dbd8b1c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f012f000640 1 -- 172.21.15.146:0/2210684572 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130021040 con 0x55d33ddfbb60 2024-09-16T01:49:32.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f012f000640 1 -- 172.21.15.146:0/2210684572 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01300572e0 con 0x55d33ddfbb60 2024-09-16T01:49:32.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f012f000640 1 -- 172.21.15.146:0/2210684572 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130009280 con 0x55d33ddfbb60 2024-09-16T01:49:32.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 -- 172.21.15.146:0/2210684572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfbb60 msgr2=0x55d33de36b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfbb60 0x55d33de36b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0130089620 tx=0x7f0130059300 comp rx=0 tx=0).stop 2024-09-16T01:49:32.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 -- 172.21.15.146:0/2210684572 shutdown_connections 2024-09-16T01:49:32.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddfa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfbb60 0x55d33de36b00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 --2- 172.21.15.146:0/2210684572 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfe330 0x55d33de365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 -- 172.21.15.146:0/2210684572 >> 172.21.15.146:0/2210684572 conn(0x55d33ddfca50 msgr2=0x55d33ddfce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 -- 172.21.15.146:0/2210684572 shutdown_connections 2024-09-16T01:49:32.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.173+0000 7f014c440740 1 -- 172.21.15.146:0/2210684572 wait complete. 2024-09-16T01:49:32.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.174+0000 7f014c440740 1 -- start start 2024-09-16T01:49:32.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.174+0000 7f014c440740 1 Processor -- start 2024-09-16T01:49:32.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33de21ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33de26290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33de2df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d33dd1b780 con 0x55d33ddfe710 2024-09-16T01:49:32.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d33dd5c3d0 con 0x55d33ddfe330 2024-09-16T01:49:32.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55d33ddfdbc0 con 0x55d33ddfbb60 2024-09-16T01:49:32.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f014c440740 10 client.0 _ll_register_callbacks cb 0x55d33dd595d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f0142000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33de26290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f0143400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33de2df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f0142a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33de21ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f0142000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33de26290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56854/0 (socket says 172.21.15.146:56854) 2024-09-16T01:49:32.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.175+0000 7f0142000640 1 -- 172.21.15.146:0/3788198099 learned_addr learned my addr 172.21.15.146:0/3788198099 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0142000640 1 -- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 msgr2=0x55d33de21ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0142000640 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33de21ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0142000640 1 -- 172.21.15.146:0/3788198099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 msgr2=0x55d33de2df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0142000640 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33de2df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0142000640 1 -- 172.21.15.146:0/3788198099 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d33db3d6b0 con 0x55d33ddfe330 2024-09-16T01:49:32.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0142000640 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33de26290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f01300591d0 tx=0x7f0130009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5f4d25f606f7c88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0123400640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-09-16T01:49:32.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0123400640 1 -- 172.21.15.146:0/3788198099 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130021040 con 0x55d33ddfe330 2024-09-16T01:49:32.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0123400640 1 -- 172.21.15.146:0/3788198099 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f013005d170 con 0x55d33ddfe330 2024-09-16T01:49:32.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01300572e0 con 0x55d33ddfe330 2024-09-16T01:49:32.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f0123400640 1 -- 172.21.15.146:0/3788198099 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0130009200 con 0x55d33ddfe330 2024-09-16T01:49:32.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f014c440740 10 client.4666 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.176+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55d33de4db80 con 0x55d33ddfe330 2024-09-16T01:49:32.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f0123400640 1 -- 172.21.15.146:0/3788198099 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01300572e0 con 0x55d33ddfe330 2024-09-16T01:49:32.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f0123400640 1 -- 172.21.15.146:0/3788198099 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7f0130024060 con 0x55d33ddfe330 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f014c440740 10 client.4666 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f014c440740 10 client.4666 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f014c440740 20 client.4666 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f014c440740 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33de2e4a0 0x55d33de5bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f014c440740 4 client.4666 mds_command: new command op to 4259 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 --> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d33dd6db40 con 0x55d33de2e4a0 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f0142a00640 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33de2e4a0 0x55d33de5bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f0142a00640 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33de2e4a0 0x55d33de5bfd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9d754a3408e27c16 server_cookie=b55dadb3dcfa1942 in_seq=0 out_seq=0 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.177+0000 7f0123400640 10 client.4666 ms_handle_connect on v2:172.21.15.161:6838/2836611107 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f0123400640 1 -- 172.21.15.146:0/3788198099 <== mds.3 v2:172.21.15.161:6838/2836611107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d33dd6db40 con 0x55d33de2e4a0 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f0123400640 10 client.4666 handle_command_reply: tid=0 2024-09-16T01:49:32.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 client.4666 shutdown 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 msgr2=0x55d33de26290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33de26290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f01300591d0 tx=0x7f0130009640 comp rx=0 tx=0).stop 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 shutdown_connections 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33de2df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33de26290 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.178+0000 7f014c440740 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33de2e4a0 0x55d33de5bfd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.179+0000 7f014c440740 1 --2- 172.21.15.146:0/3788198099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33de21ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.179+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 >> 172.21.15.146:0/3788198099 conn(0x55d33ddfca50 msgr2=0x55d33de365c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.179+0000 7f0123400640 0 client.4666 ms_handle_reset on v2:172.21.15.161:6838/2836611107 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.179+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 shutdown_connections 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.179+0000 7f014c440740 1 -- 172.21.15.146:0/3788198099 wait complete. 2024-09-16T01:49:32.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.179+0000 7f014c440740 20 client.4666 trim_cache size 0 max 16384 2024-09-16T01:49:32.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.197+0000 7f014c440740 1 Processor -- start 2024-09-16T01:49:32.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.197+0000 7f014c440740 1 -- start start 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33ddfbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f014c440740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 0x55d33ddfc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddf76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f014c440740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0144155fa0 con 0x55d33ddfe330 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d33dd1b780 con 0x55d33ddfe710 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55d33dd5c3d0 con 0x55d33ddfbb60 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f0142000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 0x55d33ddfc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f0142000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 0x55d33ddfc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50352/0 (socket says 172.21.15.146:50352) 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f0142000640 1 -- 172.21.15.146:0/1716757588 learned_addr learned my addr 172.21.15.146:0/1716757588 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f0142a00640 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33ddfbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.198+0000 7f0143400640 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddf76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f0142000640 1 -- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 msgr2=0x55d33ddfbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f0142000640 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33ddfbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f0142000640 1 -- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 msgr2=0x55d33ddf76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f0142000640 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddf76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f0142000640 1 -- 172.21.15.146:0/1716757588 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d33ddf86c0 con 0x55d33ddfe330 2024-09-16T01:49:32.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f0142000640 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 0x55d33ddfc4a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f01300899c0 tx=0x7f0130086000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a71acce704fb0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f012e600640 1 -- 172.21.15.146:0/1716757588 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130021070 con 0x55d33ddfe330 2024-09-16T01:49:32.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f012e600640 1 -- 172.21.15.146:0/1716757588 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013007d070 con 0x55d33ddfe330 2024-09-16T01:49:32.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f012e600640 1 -- 172.21.15.146:0/1716757588 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01300093f0 con 0x55d33ddfe330 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f014c440740 1 -- 172.21.15.146:0/1716757588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 msgr2=0x55d33ddfc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.199+0000 7f014c440740 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 0x55d33ddfc4a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f01300899c0 tx=0x7f0130086000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 -- 172.21.15.146:0/1716757588 shutdown_connections 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe710 0x55d33ddf76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe330 0x55d33ddfc4a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 --2- 172.21.15.146:0/1716757588 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33ddfbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 -- 172.21.15.146:0/1716757588 >> 172.21.15.146:0/1716757588 conn(0x55d33ddfca50 msgr2=0x55d33de365c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 -- 172.21.15.146:0/1716757588 shutdown_connections 2024-09-16T01:49:32.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 -- 172.21.15.146:0/1716757588 wait complete. 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 -- start start 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.200+0000 7f014c440740 1 Processor -- start 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33debbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33ddf76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33ddf7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d33de24ad0 con 0x55d33ddfe710 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55d33de22710 con 0x55d33ddfe330 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55d33de50990 con 0x55d33ddfbb60 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.201+0000 7f014c440740 10 client.0 _ll_register_callbacks cb 0x7f013c0018e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33ddf76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33debbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33ddf76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56862/0 (socket says 172.21.15.146:56862) 2024-09-16T01:49:32.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142000640 1 -- 172.21.15.146:0/2410058307 learned_addr learned my addr 172.21.15.146:0/2410058307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0143400640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33ddf7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 -- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 msgr2=0x55d33ddf76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33ddf76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 -- 172.21.15.146:0/2410058307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 msgr2=0x55d33ddf7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33ddf7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 -- 172.21.15.146:0/2410058307 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55d33debc360 con 0x55d33ddfbb60 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142000640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33ddf76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0142a00640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33debbe20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0134035040 tx=0x7f0134004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=569c553708cd2656 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0123400640 10 client.0 ms_handle_connect on v2:172.21.15.161:3301/0 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0123400640 1 -- 172.21.15.146:0/2410058307 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134037040 con 0x55d33ddfbb60 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.202+0000 7f0123400640 1 -- 172.21.15.146:0/2410058307 <== mon.2 v2:172.21.15.161:3301/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f0134004290 con 0x55d33ddfbb60 2024-09-16T01:49:32.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f013007d070 con 0x55d33ddfbb60 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f0123400640 1 -- 172.21.15.146:0/2410058307 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0134062050 con 0x55d33ddfbb60 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 10 client.4562 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55d33debc570 con 0x55d33ddfbb60 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f0123400640 1 -- 172.21.15.146:0/2410058307 <== mon.2 v2:172.21.15.161:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013007d070 con 0x55d33ddfbb60 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f0123400640 1 -- 172.21.15.146:0/2410058307 <== mon.2 v2:172.21.15.161:3301/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7f0134065020 con 0x55d33ddfbb60 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 10 client.4562 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 10 client.4562 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 20 client.4562 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33debf760 0x55d33dec7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 4 client.4562 mds_command: new command op to 4259 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 --> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] -- command(tid 0: {"prefix": "status"}) -- 0x55d33dd67e90 con 0x55d33debf760 2024-09-16T01:49:32.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.203+0000 7f0142000640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33debf760 0x55d33dec7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f0142000640 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33debf760 0x55d33dec7fa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fe467d181267898b server_cookie=24b26419e4caa4cf in_seq=0 out_seq=0 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f0123400640 10 client.4562 ms_handle_connect on v2:172.21.15.161:6838/2836611107 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f0123400640 1 -- 172.21.15.146:0/2410058307 <== mds.3 v2:172.21.15.161:6838/2836611107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d33dd67e90 con 0x55d33debf760 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f0123400640 10 client.4562 handle_command_reply: tid=0 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 client.4562 shutdown 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 msgr2=0x55d33debbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33debbe20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0134035040 tx=0x7f0134004730 comp rx=0 tx=0).stop 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 shutdown_connections 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d33ddfe710 0x55d33ddf7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d33ddfe330 0x55d33ddf76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:6838/2836611107,v1:172.21.15.161:6839/2836611107] conn(0x55d33debf760 0x55d33dec7fa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 --2- 172.21.15.146:0/2410058307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55d33ddfbb60 0x55d33debbe20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 >> 172.21.15.146:0/2410058307 conn(0x55d33ddfca50 msgr2=0x55d33de27820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f0123400640 0 client.4562 ms_handle_reset on v2:172.21.15.161:6838/2836611107 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 shutdown_connections 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.204+0000 7f014c440740 1 -- 172.21.15.146:0/2410058307 wait complete. 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f014c440740 20 client.4562 trim_cache size 0 max 16384 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 3, 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4259, 2024-09-16T01:49:32.213 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 16.298194415000001, 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 16, 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 22, 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 22, 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 39.698139832999999 2024-09-16T01:49:32.214 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f012406c2e0 msgr2=0x7f012406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f012406c2e0 0x7f012406e7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0134000c00 tx=0x7f013402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c75a0 msgr2=0x7f014414f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c75a0 0x7f014414f310 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f013000e870 tx=0x7f013000ed30 comp rx=0 tx=0).stop 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 shutdown_connections 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f012406c2e0 0x7f012406e7a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01440cbab0 0x7f014414f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01440c75a0 0x7f014414f310 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 --2- 172.21.15.146:0/3218880825 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01440c6b70 0x7f014415ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 >> 172.21.15.146:0/3218880825 conn(0x7f01440ccc20 msgr2=0x7f01440cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 shutdown_connections 2024-09-16T01:49:32.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.205+0000 7f0143e00640 1 -- 172.21.15.146:0/3218880825 wait complete. 2024-09-16T01:49:32.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-09-16T01:49:32.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.374+0000 7fd262c00640 1 Processor -- start 2024-09-16T01:49:32.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.374+0000 7fd262c00640 1 -- start start 2024-09-16T01:49:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd262c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd25c06ce90 0x7fd25c15b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd262c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd25c0c6840 0x7fd25c0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd262c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c15b9f0 con 0x7fd25c06d270 2024-09-16T01:49:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd262c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd25c067950 con 0x7fd25c06ce90 2024-09-16T01:49:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd262c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd25c067ba0 con 0x7fd25c0c6840 2024-09-16T01:49:32.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd253400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd25c06d270 0x7fd25c1548b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50116/0 (socket says 172.21.15.146:50116) 2024-09-16T01:49:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd253400640 1 -- 172.21.15.146:0/2322696431 learned_addr learned my addr 172.21.15.146:0/2322696431 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd25be00640 1 --2- 172.21.15.146:0/2322696431 >> v2:172.21.15.146:3300/0 conn(0x7fd25c06ce90 0x7fd25c15b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.376+0000 7fd260c00640 1 --2- 172.21.15.146:0/2322696431 >> v2:172.21.15.161:3300/0 conn(0x7fd25c0c6840 0x7fd25c0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.377+0000 7fd25be00640 1 -- 172.21.15.146:0/2322696431 >> v2:172.21.15.161:3300/0 conn(0x7fd25c0c6840 msgr2=0x7fd25c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.377+0000 7fd25be00640 1 --2- 172.21.15.146:0/2322696431 >> v2:172.21.15.161:3300/0 conn(0x7fd25c0c6840 0x7fd25c0725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.377+0000 7fd25be00640 1 -- 172.21.15.146:0/2322696431 >> v1:172.21.15.146:6789/0 conn(0x7fd25c06d270 legacy=0x7fd25c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.377+0000 7fd25be00640 1 -- 172.21.15.146:0/2322696431 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c067e80 con 0x7fd25c06ce90 2024-09-16T01:49:32.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.377+0000 7fd25b400640 1 -- 172.21.15.146:0/2322696431 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016457250 0 0) 0x7fd244001110 con 0x7fd25c06d270 2024-09-16T01:49:32.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd25be00640 1 --2- 172.21.15.146:0/2322696431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c15b4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd25c071690 tx=0x7fd24c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ded1c061c0ebea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd25b400640 1 -- 172.21.15.146:0/2322696431 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c007740 con 0x7fd25c06ce90 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd25b400640 1 -- 172.21.15.146:0/2322696431 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24c0078e0 con 0x7fd25c06ce90 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd25b400640 1 -- 172.21.15.146:0/2322696431 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c007cb0 con 0x7fd25c06ce90 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd262c00640 1 -- 172.21.15.146:0/2322696431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 msgr2=0x7fd25c15b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd262c00640 1 --2- 172.21.15.146:0/2322696431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c15b4b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd25c071690 tx=0x7fd24c02a680 comp rx=0 tx=0).stop 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd262c00640 1 -- 172.21.15.146:0/2322696431 shutdown_connections 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd262c00640 1 --2- 172.21.15.146:0/2322696431 >> v2:172.21.15.161:3300/0 conn(0x7fd25c0c6840 0x7fd25c0725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd262c00640 1 --2- 172.21.15.146:0/2322696431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c15b4b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.378+0000 7fd262c00640 1 -- 172.21.15.146:0/2322696431 >> 172.21.15.146:0/2322696431 conn(0x7fd25c06d8e0 msgr2=0x7fd25c06dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.379+0000 7fd262c00640 1 -- 172.21.15.146:0/2322696431 shutdown_connections 2024-09-16T01:49:32.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.379+0000 7fd262c00640 1 -- 172.21.15.146:0/2322696431 wait complete. 2024-09-16T01:49:32.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.380+0000 7fd262c00640 1 Processor -- start 2024-09-16T01:49:32.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.380+0000 7fd262c00640 1 -- start start 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd262c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c14f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd262c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 0x7fd25c14c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd262c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd25c0c6840 0x7fd25c0cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd262c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c06e9c0 con 0x7fd25c06ce90 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd262c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c06e6e0 con 0x7fd25c06d270 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd262c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd25c0756f0 con 0x7fd25c0c6840 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd25be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c14f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd253400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 0x7fd25c14c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd260c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd25c0c6840 0x7fd25c0cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd253400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 0x7fd25c14c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56876/0 (socket says 172.21.15.146:56876) 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.381+0000 7fd253400640 1 -- 172.21.15.146:0/3075698674 learned_addr learned my addr 172.21.15.146:0/3075698674 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd253400640 1 -- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd25c0c6840 msgr2=0x7fd25c0cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd253400640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd25c0c6840 0x7fd25c0cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd253400640 1 -- 172.21.15.146:0/3075698674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 msgr2=0x7fd25c14f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd253400640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c14f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd253400640 1 -- 172.21.15.146:0/3075698674 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c0cad10 con 0x7fd25c06d270 2024-09-16T01:49:32.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd253400640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 0x7fd25c14c250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c780 tx=0x7fd24400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd258c00640 1 -- 172.21.15.146:0/3075698674 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd244018070 con 0x7fd25c06d270 2024-09-16T01:49:32.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd258c00640 1 -- 172.21.15.146:0/3075698674 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2440072d0 con 0x7fd25c06d270 2024-09-16T01:49:32.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd258c00640 1 -- 172.21.15.146:0/3075698674 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2440047e0 con 0x7fd25c06d270 2024-09-16T01:49:32.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c0caa30 con 0x7fd25c06d270 2024-09-16T01:49:32.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.382+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c1f48b0 con 0x7fd25c06d270 2024-09-16T01:49:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.386+0000 7fd258c00640 1 -- 172.21.15.146:0/3075698674 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd244004980 con 0x7fd25c06d270 2024-09-16T01:49:32.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.386+0000 7fd258c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd23806c2b0 0x7fd23806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.387+0000 7fd258c00640 1 -- 172.21.15.146:0/3075698674 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd24408a6e0 con 0x7fd25c06d270 2024-09-16T01:49:32.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.387+0000 7fd265722740 1 Processor -- start 2024-09-16T01:49:32.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.387+0000 7fd25be00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd23806c2b0 0x7fd23806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.387+0000 7fd265722740 1 -- start start 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.387+0000 7fd25be00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd23806c2b0 0x7fd23806e770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034000 tx=0x7fd24c02f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad509410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad509950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd265722740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad4cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd265722740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55aaad42e3b0 con 0x55aaad4d1560 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aaad433410 con 0x55aaad4d1180 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55aaad3db620 con 0x55aaad4ce9b0 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad4cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad4cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50380/0 (socket says 172.21.15.146:50380) 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 -- 172.21.15.146:0/627409707 learned_addr learned my addr 172.21.15.146:0/627409707 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd25be00640 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad509410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd253400640 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad509950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 -- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 msgr2=0x55aaad509950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad509950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 -- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 msgr2=0x55aaad509410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad509410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 -- 172.21.15.146:0/627409707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaad1462d0 con 0x55aaad4d1560 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd253400640 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad509950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.388+0000 7fd260c00640 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad4cd490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd254009bc0 tx=0x7fd254011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfb8211d4459e3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd24be00640 1 -- 172.21.15.146:0/627409707 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254014440 con 0x55aaad4d1560 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd24be00640 1 -- 172.21.15.146:0/627409707 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2540145e0 con 0x55aaad4d1560 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd24be00640 1 -- 172.21.15.146:0/627409707 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254014990 con 0x55aaad4d1560 2024-09-16T01:49:32.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 -- 172.21.15.146:0/627409707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 msgr2=0x55aaad4cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad4cd490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd254009bc0 tx=0x7fd254011690 comp rx=0 tx=0).stop 2024-09-16T01:49:32.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 -- 172.21.15.146:0/627409707 shutdown_connections 2024-09-16T01:49:32.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad4cd490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad509950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 --2- 172.21.15.146:0/627409707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad509410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 -- 172.21.15.146:0/627409707 >> 172.21.15.146:0/627409707 conn(0x55aaad4cf8a0 msgr2=0x55aaad4cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.389+0000 7fd265722740 1 -- 172.21.15.146:0/627409707 shutdown_connections 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.390+0000 7fd265722740 1 -- 172.21.15.146:0/627409707 wait complete. 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.390+0000 7fd265722740 1 -- start start 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.390+0000 7fd265722740 1 Processor -- start 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad524660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad4f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad500db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55aaad3ee780 con 0x55aaad4ce9b0 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aaad42f3d0 con 0x55aaad4d1560 2024-09-16T01:49:32.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55aaad4d0a10 con 0x55aaad4d1180 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd265722740 10 client.0 _ll_register_callbacks cb 0x55aaad42c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.391+0000 7fd25be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad524660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad524660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50396/0 (socket says 172.21.15.146:50396) 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 -- 172.21.15.146:0/536813301 learned_addr learned my addr 172.21.15.146:0/536813301 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd260c00640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad500db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd253400640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad4f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 -- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 msgr2=0x55aaad4f4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad4f4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 -- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 msgr2=0x55aaad500db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad500db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 -- 172.21.15.146:0/536813301 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aaad2106b0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd25be00640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad524660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009e50 tx=0x7fd24c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14501674daa4b34f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd248c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.392+0000 7fd248c00640 1 -- 172.21.15.146:0/536813301 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c046460 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd248c00640 1 -- 172.21.15.146:0/536813301 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd24c053ae0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2540145e0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd248c00640 1 -- 172.21.15.146:0/536813301 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fd24c053ce0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 10 client.4617 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55aaad4f5e00 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd248c00640 1 -- 172.21.15.146:0/536813301 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2540145e0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd248c00640 1 -- 172.21.15.146:0/536813301 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7fd24c044040 con 0x55aaad4ce9b0 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 10 client.4617 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 10 client.4617 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 20 client.4617 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad501470 0x55aaad52f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 4 client.4617 mds_command: new command op to 4277 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:49:32.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.393+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 --> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aaad440b40 con 0x55aaad501470 2024-09-16T01:49:32.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.394+0000 7fd253400640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad501470 0x55aaad52f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.394+0000 7fd253400640 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad501470 0x55aaad52f8f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a260fb9810ca35b0 server_cookie=d54b694bbeb1bf3f in_seq=0 out_seq=0 2024-09-16T01:49:32.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.394+0000 7fd248c00640 10 client.4617 ms_handle_connect on v2:172.21.15.146:6835/835076388 2024-09-16T01:49:32.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd248c00640 1 -- 172.21.15.146:0/536813301 <== mds.2 v2:172.21.15.146:6835/835076388 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aaad440b40 con 0x55aaad501470 2024-09-16T01:49:32.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd248c00640 10 client.4617 handle_command_reply: tid=0 2024-09-16T01:49:32.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 client.4617 shutdown 2024-09-16T01:49:32.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 msgr2=0x55aaad524660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad524660 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009e50 tx=0x7fd24c02c000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 shutdown_connections 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad500db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad501470 0x55aaad52f8f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad4f4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 --2- 172.21.15.146:0/536813301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad524660 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd248c00640 0 client.4617 ms_handle_reset on v2:172.21.15.146:6835/835076388 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 >> 172.21.15.146:0/536813301 conn(0x55aaad4cf8a0 msgr2=0x55aaad509380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 shutdown_connections 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 1 -- 172.21.15.146:0/536813301 wait complete. 2024-09-16T01:49:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.395+0000 7fd265722740 20 client.4617 trim_cache size 0 max 16384 2024-09-16T01:49:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.414+0000 7fd265722740 1 Processor -- start 2024-09-16T01:49:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.414+0000 7fd265722740 1 -- start start 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad4cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad507500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd265722740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad507a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd265722740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c0c5930 con 0x55aaad4d1560 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aaad524e60 con 0x55aaad4d1180 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55aaad4f7260 con 0x55aaad4ce9b0 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad507a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad507a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50398/0 (socket says 172.21.15.146:50398) 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 -- 172.21.15.146:0/3756966182 learned_addr learned my addr 172.21.15.146:0/3756966182 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd253400640 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad507500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd25be00640 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad4cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 -- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 msgr2=0x55aaad4cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad4cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 -- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 msgr2=0x55aaad507500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad507500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.415+0000 7fd260c00640 1 -- 172.21.15.146:0/3756966182 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaad42f590 con 0x55aaad4d1560 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.416+0000 7fd260c00640 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad507a40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd254019000 tx=0x7fd254016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f71c6584f876b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.416+0000 7fd24b400640 1 -- 172.21.15.146:0/3756966182 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254004030 con 0x55aaad4d1560 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.416+0000 7fd24b400640 1 -- 172.21.15.146:0/3756966182 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2540041f0 con 0x55aaad4d1560 2024-09-16T01:49:32.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.416+0000 7fd24b400640 1 -- 172.21.15.146:0/3756966182 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254003610 con 0x55aaad4d1560 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.416+0000 7fd265722740 1 -- 172.21.15.146:0/3756966182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 msgr2=0x55aaad507a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.416+0000 7fd265722740 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad507a40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd254019000 tx=0x7fd254016000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 -- 172.21.15.146:0/3756966182 shutdown_connections 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4d1560 0x55aaad507a40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1180 0x55aaad507500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 --2- 172.21.15.146:0/3756966182 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4ce9b0 0x55aaad4cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 -- 172.21.15.146:0/3756966182 >> 172.21.15.146:0/3756966182 conn(0x55aaad4cf8a0 msgr2=0x55aaad4cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 -- 172.21.15.146:0/3756966182 shutdown_connections 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 -- 172.21.15.146:0/3756966182 wait complete. 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 -- start start 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.417+0000 7fd265722740 1 Processor -- start 2024-09-16T01:49:32.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad513e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad512240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad5211b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55aaad501960 con 0x55aaad4ce9b0 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55aaad5012f0 con 0x55aaad4d1560 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55aaad4f0420 con 0x55aaad4d1180 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.418+0000 7fd265722740 10 client.0 _ll_register_callbacks cb 0x7fd24c007760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad513e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad513e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50412/0 (socket says 172.21.15.146:50412) 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 -- 172.21.15.146:0/2489229645 learned_addr learned my addr 172.21.15.146:0/2489229645 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd253400640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad512240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd260c00640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad5211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 -- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 msgr2=0x55aaad512240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad512240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 -- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 msgr2=0x55aaad5211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad5211b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 -- 172.21.15.146:0/2489229645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aaad501be0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd25be00640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad513e00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd24c007de0 tx=0x7fd24c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6a59df6cbcbf86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd248c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd248c00640 1 -- 172.21.15.146:0/2489229645 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c060030 con 0x55aaad4ce9b0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.419+0000 7fd248c00640 1 -- 172.21.15.146:0/2489229645 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd24c03dd00 con 0x55aaad4ce9b0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2540041f0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd248c00640 1 -- 172.21.15.146:0/2489229645 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fd24c046740 con 0x55aaad4ce9b0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 10 client.4623 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55aaad4f8580 con 0x55aaad4ce9b0 2024-09-16T01:49:32.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd248c00640 1 -- 172.21.15.146:0/2489229645 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2540041f0 con 0x55aaad4ce9b0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd248c00640 1 -- 172.21.15.146:0/2489229645 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7fd24c06d080 con 0x55aaad4ce9b0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 10 client.4623 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 10 client.4623 resolve_mds: resolved name 'e' to daemon mds.e 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 20 client.4623 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad59b8f0 0x55aaad591b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 4 client.4623 mds_command: new command op to 4277 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.420+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 --> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] -- command(tid 0: {"prefix": "status"}) -- 0x55aaad43ae90 con 0x55aaad59b8f0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd253400640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad59b8f0 0x55aaad591b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd253400640 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad59b8f0 0x55aaad591b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8a7cf29bf883156 server_cookie=41c0527c6bce95e2 in_seq=0 out_seq=0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd248c00640 10 client.4623 ms_handle_connect on v2:172.21.15.146:6835/835076388 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd248c00640 1 -- 172.21.15.146:0/2489229645 <== mds.2 v2:172.21.15.146:6835/835076388 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55aaad43ae90 con 0x55aaad59b8f0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd248c00640 10 client.4623 handle_command_reply: tid=0 2024-09-16T01:49:32.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 client.4623 shutdown 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 msgr2=0x55aaad513e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad513e00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd24c007de0 tx=0x7fd24c004ab0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 shutdown_connections 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aaad4d1560 0x55aaad5211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:6835/835076388,v1:172.21.15.146:6837/835076388] conn(0x55aaad59b8f0 0x55aaad591b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55aaad4d1180 0x55aaad512240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 --2- 172.21.15.146:0/2489229645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55aaad4ce9b0 0x55aaad513e00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd248c00640 0 client.4623 ms_handle_reset on v2:172.21.15.146:6835/835076388 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.421+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 >> 172.21.15.146:0/2489229645 conn(0x55aaad4cf8a0 msgr2=0x55aaad500970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 shutdown_connections 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd265722740 1 -- 172.21.15.146:0/2489229645 wait complete. 2024-09-16T01:49:32.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd265722740 20 client.4623 trim_cache size 0 max 16384 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 2, 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4277, 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 22.531465028, 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 16, 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 22, 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 22, 2024-09-16T01:49:32.429 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 39.869212105999999 2024-09-16T01:49:32.430 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd23806c2b0 msgr2=0x7fd23806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd23806c2b0 0x7fd23806e770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd24c034000 tx=0x7fd24c02f000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 msgr2=0x7fd25c14c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 0x7fd25c14c250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c780 tx=0x7fd24400cc40 comp rx=0 tx=0).stop 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 shutdown_connections 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd23806c2b0 0x7fd23806e770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd25c0c6840 0x7fd25c0cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd25c06d270 0x7fd25c14c250 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 --2- 172.21.15.146:0/3075698674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd25c06ce90 0x7fd25c14f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 >> 172.21.15.146:0/3075698674 conn(0x7fd25c06d8e0 msgr2=0x7fd25c0c7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.422+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 shutdown_connections 2024-09-16T01:49:32.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.423+0000 7fd262c00640 1 -- 172.21.15.146:0/3075698674 wait complete. 2024-09-16T01:49:32.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:49:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.595+0000 7f2bee800640 1 Processor -- start 2024-09-16T01:49:32.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.595+0000 7f2bee800640 1 -- start start 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.596+0000 7f2bee800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be80c5280 0x7f2be80c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2bee800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be8151e70 con 0x7f2be8151a90 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2bee800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be80c8130 con 0x7f2be80c6410 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2bee800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80c8410 con 0x7f2be80c5280 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be7400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2be8151a90 0x7f2be80c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:50118/0 (socket says 172.21.15.146:50118) 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be7400640 1 -- 172.21.15.146:0/2673594581 learned_addr learned my addr 172.21.15.146:0/2673594581 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:32.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be7e00640 1 --2- 172.21.15.146:0/2673594581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be80c5280 0x7f2be80c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4148600356 0 0) 0x7f2be8151e70 con 0x7f2be8151a90 2024-09-16T01:49:32.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bcc005730 con 0x7f2be8151a90 2024-09-16T01:49:32.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2637493519 0 0) 0x7f2be80c8130 con 0x7f2be80c6410 2024-09-16T01:49:32.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be8151e70 con 0x7f2be80c6410 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be7e00640 1 -- 172.21.15.146:0/2673594581 >> v1:172.21.15.161:6789/0 conn(0x7f2be80c6410 legacy=0x7f2be8158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be7e00640 1 -- 172.21.15.146:0/2673594581 >> v1:172.21.15.146:6789/0 conn(0x7f2be8151a90 legacy=0x7f2be80c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.597+0000 7f2be7e00640 1 -- 172.21.15.146:0/2673594581 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80c86f0 con 0x7f2be80c5280 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.598+0000 7f2be7e00640 1 --2- 172.21.15.146:0/2673594581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be80c5280 0x7f2be80c5730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009dc0 tx=0x7f2be002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a39427b8992999d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.598+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0007740 con 0x7f2be80c5280 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.598+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be00078e0 con 0x7f2be80c5280 2024-09-16T01:49:32.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.598+0000 7f2be6a00640 1 -- 172.21.15.146:0/2673594581 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0007c90 con 0x7f2be80c5280 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 -- 172.21.15.146:0/2673594581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be80c5280 msgr2=0x7f2be80c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 --2- 172.21.15.146:0/2673594581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be80c5280 0x7f2be80c5730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009dc0 tx=0x7f2be002a6d0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 -- 172.21.15.146:0/2673594581 shutdown_connections 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 --2- 172.21.15.146:0/2673594581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be80c5280 0x7f2be80c5730 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 -- 172.21.15.146:0/2673594581 >> 172.21.15.146:0/2673594581 conn(0x7f2be806f3c0 msgr2=0x7f2be806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 -- 172.21.15.146:0/2673594581 shutdown_connections 2024-09-16T01:49:32.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.599+0000 7f2bee800640 1 -- 172.21.15.146:0/2673594581 wait complete. 2024-09-16T01:49:32.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.601+0000 7f2bee800640 1 Processor -- start 2024-09-16T01:49:32.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.601+0000 7f2bee800640 1 -- start start 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.601+0000 7f2bee800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 0x7f2be80c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.601+0000 7f2bee800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be80c6410 0x7f2be80cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2bee800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be8151a90 0x7f2be81e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2bee800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2be8157b20 con 0x7f2be80c5280 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2bee800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2be8157840 con 0x7f2be80c6410 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2bee800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80c71f0 con 0x7f2be8151a90 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 0x7f2be80c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 0x7f2be80c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50418/0 (socket says 172.21.15.146:50418) 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 -- 172.21.15.146:0/1748012226 learned_addr learned my addr 172.21.15.146:0/1748012226 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7e00640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be8151a90 0x7f2be81e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2bdea00640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be80c6410 0x7f2be80cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 -- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be8151a90 msgr2=0x7f2be81e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be8151a90 0x7f2be81e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 -- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be80c6410 msgr2=0x7f2be80cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be80c6410 0x7f2be80cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 -- 172.21.15.146:0/1748012226 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80ca5f0 con 0x7f2be80c5280 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.602+0000 7f2be7400640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 0x7f2be80c9fb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c410 tx=0x7f2bd800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.603+0000 7f2bdfe00640 1 -- 172.21.15.146:0/1748012226 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8017070 con 0x7f2be80c5280 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.603+0000 7f2bdfe00640 1 -- 172.21.15.146:0/1748012226 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd8005ce0 con 0x7f2be80c5280 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.603+0000 7f2bdfe00640 1 -- 172.21.15.146:0/1748012226 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd80072d0 con 0x7f2be80c5280 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.603+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be814d420 con 0x7f2be80c5280 2024-09-16T01:49:32.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.603+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be814d9e0 con 0x7f2be80c5280 2024-09-16T01:49:32.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.605+0000 7f2bdfe00640 1 -- 172.21.15.146:0/1748012226 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2bd801a070 con 0x7f2be80c5280 2024-09-16T01:49:32.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.606+0000 7f2bdfe00640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2bc406c2b0 0x7f2bc406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.606+0000 7f2bdfe00640 1 -- 172.21.15.146:0/1748012226 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2bd808a6f0 con 0x7f2be80c5280 2024-09-16T01:49:32.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.606+0000 7f2bdea00640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2bc406c2b0 0x7f2bc406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.606+0000 7f2bf165b740 1 Processor -- start 2024-09-16T01:49:32.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.606+0000 7f2bf165b740 1 -- start start 2024-09-16T01:49:32.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bdea00640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2bc406c2b0 0x7f2bc406e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0007fd0 tx=0x7f2bd0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3dd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3dd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af3a1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bf165b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b9af3023b0 con 0x55b9af3a5560 2024-09-16T01:49:32.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b9af307410 con 0x55b9af3a5180 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.607+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55b9af2af620 con 0x55b9af3a29b0 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af3a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2bdea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3dd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3dd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af3a1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50434/0 (socket says 172.21.15.146:50434) 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 -- 172.21.15.146:0/3618371539 learned_addr learned my addr 172.21.15.146:0/3618371539 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 -- 172.21.15.146:0/3618371539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 msgr2=0x55b9af3dd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3dd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 -- 172.21.15.146:0/3618371539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 msgr2=0x55b9af3dd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3dd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 -- 172.21.15.146:0/3618371539 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9af01a2d0 con 0x55b9af3a5560 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2be7e00640 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af3a1490 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009eb0 tx=0x7f2be002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fe1b951994a309 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2bd7e00640 1 -- 172.21.15.146:0/3618371539 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0007800 con 0x55b9af3a5560 2024-09-16T01:49:32.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2bd7e00640 1 -- 172.21.15.146:0/3618371539 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be00079a0 con 0x55b9af3a5560 2024-09-16T01:49:32.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.608+0000 7f2bd7e00640 1 -- 172.21.15.146:0/3618371539 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0007d50 con 0x55b9af3a5560 2024-09-16T01:49:32.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 -- 172.21.15.146:0/3618371539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 msgr2=0x55b9af3a1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af3a1490 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009eb0 tx=0x7f2be002e000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 -- 172.21.15.146:0/3618371539 shutdown_connections 2024-09-16T01:49:32.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af3a1490 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3dd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 --2- 172.21.15.146:0/3618371539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3dd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 -- 172.21.15.146:0/3618371539 >> 172.21.15.146:0/3618371539 conn(0x55b9af3a38a0 msgr2=0x55b9af3a3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 -- 172.21.15.146:0/3618371539 shutdown_connections 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.609+0000 7f2bf165b740 1 -- 172.21.15.146:0/3618371539 wait complete. 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.610+0000 7f2bf165b740 1 -- start start 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.610+0000 7f2bf165b740 1 Processor -- start 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5560 0x55b9af3f5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b9af2c2780 con 0x55b9af3a5180 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b9af3033d0 con 0x55b9af3a5560 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55b9af3a4a10 con 0x55b9af3a29b0 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bf165b740 10 client.0 _ll_register_callbacks cb 0x55b9af3005d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bdea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5560 0x55b9af3f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bdea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50450/0 (socket says 172.21.15.146:50450) 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2bdea00640 1 -- 172.21.15.146:0/458172950 learned_addr learned my addr 172.21.15.146:0/458172950 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.611+0000 7f2be7400640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bdea00640 1 -- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 msgr2=0x55b9af3f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bdea00640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bdea00640 1 -- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5560 msgr2=0x55b9af3f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bdea00640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5560 0x55b9af3f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bdea00640 1 -- 172.21.15.146:0/458172950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9af0e46b0 con 0x55b9af3a5180 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2be7e00640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5560 0x55b9af3f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bdea00640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3cd6d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2be80c6850 tx=0x7f2bd0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8a9724e968f7e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bd4c00640 1 -- 172.21.15.146:0/458172950 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0025020 con 0x55b9af3a5180 2024-09-16T01:49:32.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bd4c00640 1 -- 172.21.15.146:0/458172950 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2bd00216d0 con 0x55b9af3a5180 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be00079a0 con 0x55b9af3a5180 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bd4c00640 1 -- 172.21.15.146:0/458172950 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f2bd002f050 con 0x55b9af3a5180 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bf165b740 10 client.4635 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.612+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55b9af3f8b50 con 0x55b9af3a5180 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bd4c00640 1 -- 172.21.15.146:0/458172950 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be00079a0 con 0x55b9af3a5180 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bd4c00640 1 -- 172.21.15.146:0/458172950 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7f2bd0032020 con 0x55b9af3a5180 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bf165b740 10 client.4635 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bf165b740 10 client.4635 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bf165b740 20 client.4635 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bf165b740 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3a2240 0x55b9af3d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bf165b740 4 client.4635 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 --> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9af314b40 con 0x55b9af3a2240 2024-09-16T01:49:32.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.613+0000 7f2be7400640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3a2240 0x55b9af3d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.614+0000 7f2bd4c00640 10 client.4635 ms_handle_connect on v2:172.21.15.146:6832/1022834380 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.614+0000 7f2be7400640 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3a2240 0x55b9af3d2d10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ba04a150240943f server_cookie=b7a268c93094304d in_seq=0 out_seq=0 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.614+0000 7f2bd4c00640 1 -- 172.21.15.146:0/458172950 <== mds.1 v2:172.21.15.146:6832/1022834380 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b9af314b40 con 0x55b9af3a2240 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.614+0000 7f2bd4c00640 10 client.4635 handle_command_reply: tid=0 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.614+0000 7f2bf165b740 1 client.4635 shutdown 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 msgr2=0x55b9af3cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3cd6d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2be80c6850 tx=0x7f2bd0000ed0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 shutdown_connections 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3a2240 0x55b9af3d2d10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5560 0x55b9af3f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3cd6d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 --2- 172.21.15.146:0/458172950 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bd4c00640 0 client.4635 ms_handle_reset on v2:172.21.15.146:6832/1022834380 2024-09-16T01:49:32.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 >> 172.21.15.146:0/458172950 conn(0x55b9af3a38a0 msgr2=0x55b9af3e74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 shutdown_connections 2024-09-16T01:49:32.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 1 -- 172.21.15.146:0/458172950 wait complete. 2024-09-16T01:49:32.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.615+0000 7f2bf165b740 20 client.4635 trim_cache size 0 max 16384 2024-09-16T01:49:32.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.633+0000 7f2bf165b740 1 Processor -- start 2024-09-16T01:49:32.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.633+0000 7f2bf165b740 1 -- start start 2024-09-16T01:49:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af408770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bf165b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2be81f3b10 con 0x55b9af3a5560 2024-09-16T01:49:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b9af3cde20 con 0x55b9af3a5180 2024-09-16T01:49:32.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55b9af3f5690 con 0x55b9af3a29b0 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af408770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2be7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2be7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3a2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52520/0 (socket says 172.21.15.146:52520) 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2be7400640 1 -- 172.21.15.146:0/1562878994 learned_addr learned my addr 172.21.15.146:0/1562878994 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2bdea00640 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.634+0000 7f2be7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af408770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50458/0 (socket says 172.21.15.146:50458) 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2be7400640 1 -- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 msgr2=0x55b9af3a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2be7400640 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3a32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2be7400640 1 -- 172.21.15.146:0/1562878994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 msgr2=0x55b9af408770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2be7400640 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af408770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2be7400640 1 -- 172.21.15.146:0/1562878994 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9af303590 con 0x55b9af3a29b0 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2bdea00640 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2be7400640 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3a2d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2bd80070a0 tx=0x7f2bd808a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f925d806652abf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2bd7400640 1 -- 172.21.15.146:0/1562878994 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8088c90 con 0x55b9af3a29b0 2024-09-16T01:49:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2bd7400640 1 -- 172.21.15.146:0/1562878994 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd8056b00 con 0x55b9af3a29b0 2024-09-16T01:49:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.635+0000 7f2bd7400640 1 -- 172.21.15.146:0/1562878994 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8007470 con 0x55b9af3a29b0 2024-09-16T01:49:32.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 -- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 msgr2=0x55b9af3a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3a2d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2bd80070a0 tx=0x7f2bd808a040 comp rx=0 tx=0).stop 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 -- 172.21.15.146:0/1562878994 shutdown_connections 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5560 0x55b9af408770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a5180 0x55b9af3a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 --2- 172.21.15.146:0/1562878994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a29b0 0x55b9af3a2d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 -- 172.21.15.146:0/1562878994 >> 172.21.15.146:0/1562878994 conn(0x55b9af3a38a0 msgr2=0x55b9af3e11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 -- 172.21.15.146:0/1562878994 shutdown_connections 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 -- 172.21.15.146:0/1562878994 wait complete. 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.636+0000 7f2bf165b740 1 -- start start 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 Processor -- start 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a29b0 0x55b9af3a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3a2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a5560 0x55b9af45d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b9af3d2b90 con 0x55b9af3a5180 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x55b9af3a20c0 con 0x55b9af3a29b0 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.637+0000 7f2bf165b740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x55b9af3d3260 con 0x55b9af3a5560 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bf165b740 10 client.0 _ll_register_callbacks cb 0x7f2bd0007980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3a2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3a2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50462/0 (socket says 172.21.15.146:50462) 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 -- 172.21.15.146:0/504110821 learned_addr learned my addr 172.21.15.146:0/504110821 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2be7e00640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a5560 0x55b9af45d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2be7400640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a29b0 0x55b9af3a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 -- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a5560 msgr2=0x55b9af45d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a5560 0x55b9af45d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 -- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a29b0 msgr2=0x55b9af3a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a29b0 0x55b9af3a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 -- 172.21.15.146:0/504110821 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9af3cd950 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bdea00640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3a2f80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0007ab0 tx=0x7f2bd0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d12f43a7a75de11 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.638+0000 7f2bd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bd4c00640 1 -- 172.21.15.146:0/504110821 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0025040 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bd4c00640 1 -- 172.21.15.146:0/504110821 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2bd0021920 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bd8056b00 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bd4c00640 1 -- 172.21.15.146:0/504110821 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f2bd0021b20 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 10 client.4641 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55b9af3f7cc0 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bd4c00640 1 -- 172.21.15.146:0/504110821 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd8056b00 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bd4c00640 1 -- 172.21.15.146:0/504110821 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7f2bd0032070 con 0x55b9af3a5180 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 10 client.4641 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 10 client.4641 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 20 client.4641 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3cc190 0x55b9af46ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 4 client.4641 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 --> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] -- command(tid 0: {"prefix": "status"}) -- 0x55b9af30ee90 con 0x55b9af3cc190 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.639+0000 7f2be7400640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3cc190 0x55b9af46ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2be7400640 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3cc190 0x55b9af46ffc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aaedb8fb8e62e7b8 server_cookie=acd1808ea95ad326 in_seq=0 out_seq=0 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bd4c00640 10 client.4641 ms_handle_connect on v2:172.21.15.146:6832/1022834380 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bd4c00640 1 -- 172.21.15.146:0/504110821 <== mds.1 v2:172.21.15.146:6832/1022834380 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55b9af30ee90 con 0x55b9af3cc190 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bd4c00640 10 client.4641 handle_command_reply: tid=0 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 client.4641 shutdown 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 msgr2=0x55b9af3a2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3a2f80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0007ab0 tx=0x7f2bd0014000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 shutdown_connections 2024-09-16T01:49:32.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:6832/1022834380,v1:172.21.15.146:6833/1022834380] conn(0x55b9af3cc190 0x55b9af46ffc0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x55b9af3a5560 0x55b9af45d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9af3a5180 0x55b9af3a2f80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 --2- 172.21.15.146:0/504110821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55b9af3a29b0 0x55b9af3a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bd4c00640 0 client.4641 ms_handle_reset on v2:172.21.15.146:6832/1022834380 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.640+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 >> 172.21.15.146:0/504110821 conn(0x55b9af3a38a0 msgr2=0x55b9af3d88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 shutdown_connections 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bf165b740 1 -- 172.21.15.146:0/504110821 wait complete. 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bf165b740 20 client.4641 trim_cache size 0 max 16384 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:32.647 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 1, 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4288, 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 28.792253929000001, 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 16, 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 22, 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 22, 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 40.094662790999998 2024-09-16T01:49:32.648 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2bc406c2b0 msgr2=0x7f2bc406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2bc406c2b0 0x7f2bc406e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0007fd0 tx=0x7f2bd0009040 comp rx=0 tx=0).stop 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 msgr2=0x7f2be80c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 0x7f2be80c9fb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c410 tx=0x7f2bd800c8d0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 shutdown_connections 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2bc406c2b0 0x7f2bc406e770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2be8151a90 0x7f2be81e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be80c6410 0x7f2be80cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 --2- 172.21.15.146:0/1748012226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2be80c5280 0x7f2be80c9fb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 >> 172.21.15.146:0/1748012226 conn(0x7f2be806f3c0 msgr2=0x7f2be8072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.641+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 shutdown_connections 2024-09-16T01:49:32.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.642+0000 7f2bee800640 1 -- 172.21.15.146:0/1748012226 wait complete. 2024-09-16T01:49:32.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:49:32.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.811+0000 7fdc7de00640 1 Processor -- start 2024-09-16T01:49:32.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.811+0000 7fdc7de00640 1 -- start start 2024-09-16T01:49:32.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7de00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdc780c75a0 0x7fdc7815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780cc040 0x7fdc780cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7de00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc7806d140 con 0x7fdc780c6b70 2024-09-16T01:49:32.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7de00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdc7815b880 con 0x7fdc780c75a0 2024-09-16T01:49:32.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7de00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdc780cab20 con 0x7fdc780cc040 2024-09-16T01:49:32.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc77e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdc780c75a0 0x7fdc7815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780cc040 0x7fdc780cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7ca00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdc780c6b70 0x7fdc780cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:49962/0 (socket says 172.21.15.146:49962) 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.812+0000 7fdc7ca00640 1 -- 172.21.15.146:0/1440330210 learned_addr learned my addr 172.21.15.146:0/1440330210 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.813+0000 7fdc7d400640 1 -- 172.21.15.146:0/1440330210 >> v2:172.21.15.161:3300/0 conn(0x7fdc780c75a0 msgr2=0x7fdc7815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.813+0000 7fdc7d400640 1 --2- 172.21.15.146:0/1440330210 >> v2:172.21.15.161:3300/0 conn(0x7fdc780c75a0 0x7fdc7815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.813+0000 7fdc7d400640 1 -- 172.21.15.146:0/1440330210 >> v1:172.21.15.161:6789/0 conn(0x7fdc780c6b70 legacy=0x7fdc780cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.813+0000 7fdc7d400640 1 -- 172.21.15.146:0/1440330210 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7806d420 con 0x7fdc780cc040 2024-09-16T01:49:32.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.814+0000 7fdc7d400640 1 --2- 172.21.15.146:0/1440330210 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780cc040 0x7fdc780cc4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009d90 tx=0x7fdc7002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1700399ac38b510d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.814+0000 7fdc77400640 1 -- 172.21.15.146:0/1440330210 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70031600 con 0x7fdc780cc040 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.814+0000 7fdc77400640 1 -- 172.21.15.146:0/1440330210 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc700317a0 con 0x7fdc780cc040 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.814+0000 7fdc77400640 1 -- 172.21.15.146:0/1440330210 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70031b70 con 0x7fdc780cc040 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 -- 172.21.15.146:0/1440330210 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780cc040 msgr2=0x7fdc780cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 --2- 172.21.15.146:0/1440330210 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780cc040 0x7fdc780cc4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009d90 tx=0x7fdc7002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 -- 172.21.15.146:0/1440330210 shutdown_connections 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 --2- 172.21.15.146:0/1440330210 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780cc040 0x7fdc780cc4c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 --2- 172.21.15.146:0/1440330210 >> v2:172.21.15.161:3300/0 conn(0x7fdc780c75a0 0x7fdc7815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 -- 172.21.15.146:0/1440330210 >> 172.21.15.146:0/1440330210 conn(0x7fdc780ccc20 msgr2=0x7fdc780cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 -- 172.21.15.146:0/1440330210 shutdown_connections 2024-09-16T01:49:32.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.815+0000 7fdc7de00640 1 -- 172.21.15.146:0/1440330210 wait complete. 2024-09-16T01:49:32.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.817+0000 7fdc7de00640 1 Processor -- start 2024-09-16T01:49:32.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.817+0000 7fdc7de00640 1 -- start start 2024-09-16T01:49:32.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.817+0000 7fdc7de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780c6b70 0x7fdc7815d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.817+0000 7fdc7de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 0x7fdc780abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc7de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc780cc040 0x7fdc780c2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc7de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdc780cb050 con 0x7fdc780c75a0 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc7de00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdc780cad70 con 0x7fdc780cc040 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc7de00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdc780cbc80 con 0x7fdc780c6b70 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 0x7fdc780abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 0x7fdc780abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50478/0 (socket says 172.21.15.146:50478) 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 -- 172.21.15.146:0/3947583146 learned_addr learned my addr 172.21.15.146:0/3947583146 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc7d400640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc780cc040 0x7fdc780c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780c6b70 0x7fdc7815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 -- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780c6b70 msgr2=0x7fdc7815d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780c6b70 0x7fdc7815d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 -- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc780cc040 msgr2=0x7fdc780c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc780cc040 0x7fdc780c2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 -- 172.21.15.146:0/3947583146 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc781f4b00 con 0x7fdc780c75a0 2024-09-16T01:49:32.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.818+0000 7fdc77e00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 0x7fdc780abd40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400e9b0 tx=0x7fdc6400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.819+0000 7fdc74c00640 1 -- 172.21.15.146:0/3947583146 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64017070 con 0x7fdc780c75a0 2024-09-16T01:49:32.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.819+0000 7fdc74c00640 1 -- 172.21.15.146:0/3947583146 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6400c9a0 con 0x7fdc780c75a0 2024-09-16T01:49:32.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.819+0000 7fdc74c00640 1 -- 172.21.15.146:0/3947583146 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6400cd70 con 0x7fdc780c75a0 2024-09-16T01:49:32.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.819+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc781f4dc0 con 0x7fdc780c75a0 2024-09-16T01:49:32.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.819+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7815ba20 con 0x7fdc780c75a0 2024-09-16T01:49:32.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.822+0000 7fdc74c00640 1 -- 172.21.15.146:0/3947583146 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdc6401a070 con 0x7fdc780c75a0 2024-09-16T01:49:32.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.822+0000 7fdc74c00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdc5806c2b0 0x7fdc5806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.823+0000 7fdc74c00640 1 -- 172.21.15.146:0/3947583146 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdc6408b320 con 0x7fdc780c75a0 2024-09-16T01:49:32.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.823+0000 7fdc82276740 1 Processor -- start 2024-09-16T01:49:32.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.823+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdc5806c2b0 0x7fdc5806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.823+0000 7fdc82276740 1 -- start start 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdc5806c2b0 0x7fdc5806e770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c005880 tx=0x7fdc6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc82276740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab485c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab48b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab0c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc82276740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557efaa6d3b0 con 0x557efab10330 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557efaa72410 con 0x557efab10710 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x557efaa1a620 con 0x557efab0db60 2024-09-16T01:49:32.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc7ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab48b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc7d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab0c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab48b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52566/0 (socket says 172.21.15.146:52566) 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc7d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab0c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56962/0 (socket says 172.21.15.146:56962) 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.824+0000 7fdc77e00640 1 -- 172.21.15.146:0/1626419748 learned_addr learned my addr 172.21.15.146:0/1626419748 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc77e00640 1 -- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 msgr2=0x557efab0c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc77e00640 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab0c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc77e00640 1 -- 172.21.15.146:0/1626419748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 msgr2=0x557efab485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc77e00640 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc77e00640 1 -- 172.21.15.146:0/1626419748 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557efa7852d0 con 0x557efab0db60 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc77e00640 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab48b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdc64061a30 tx=0x7fdc64059430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=944cde5c3403ffe7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc69200640 1 -- 172.21.15.146:0/1626419748 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64021040 con 0x557efab0db60 2024-09-16T01:49:32.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc69200640 1 -- 172.21.15.146:0/1626419748 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc64057730 con 0x557efab0db60 2024-09-16T01:49:32.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.825+0000 7fdc69200640 1 -- 172.21.15.146:0/1626419748 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64055aa0 con 0x557efab0db60 2024-09-16T01:49:32.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 -- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 msgr2=0x557efab48b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab48b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdc64061a30 tx=0x7fdc64059430 comp rx=0 tx=0).stop 2024-09-16T01:49:32.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 -- 172.21.15.146:0/1626419748 shutdown_connections 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab0c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab48b00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 --2- 172.21.15.146:0/1626419748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 -- 172.21.15.146:0/1626419748 >> 172.21.15.146:0/1626419748 conn(0x557efab0ea50 msgr2=0x557efab0ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 -- 172.21.15.146:0/1626419748 shutdown_connections 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 -- 172.21.15.146:0/1626419748 wait complete. 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.826+0000 7fdc82276740 1 -- start start 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.827+0000 7fdc82276740 1 Processor -- start 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab31d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab40670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 0x557efab30870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557efaa2d780 con 0x557efab10710 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557efaa6e3d0 con 0x557efab10330 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x557efab0fbc0 con 0x557efab0db60 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc82276740 10 client.0 _ll_register_callbacks cb 0x557efaa6b5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 0x557efab30870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 0x557efab30870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50498/0 (socket says 172.21.15.146:50498) 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 -- 172.21.15.146:0/430987312 learned_addr learned my addr 172.21.15.146:0/430987312 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc77e00640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab40670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab31d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 -- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 msgr2=0x557efab31d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab31d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 -- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 msgr2=0x557efab40670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab40670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.828+0000 7fdc7d400640 1 -- 172.21.15.146:0/430987312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557efa84f6b0 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc7d400640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 0x557efab30870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009e80 tx=0x7fdc70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4e11250d5d67f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc5d600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc5d600640 1 -- 172.21.15.146:0/430987312 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70031610 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc5d600640 1 -- 172.21.15.146:0/430987312 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdc700317b0 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc64057730 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc5d600640 1 -- 172.21.15.146:0/430987312 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fdc70050050 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc82276740 10 client.4650 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x557efab31850 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.829+0000 7fdc5d600640 1 -- 172.21.15.146:0/430987312 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc64057730 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc5d600640 1 -- 172.21.15.146:0/430987312 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7fdc70053020 con 0x557efab10710 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc82276740 10 client.4650 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc82276740 10 client.4650 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc82276740 20 client.4650 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc82276740 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efab5f8f0 0x557efab30db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc82276740 4 client.4650 mds_command: new command op to 4291 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:49:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 --> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557efaa7fb40 con 0x557efab5f8f0 2024-09-16T01:49:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efab5f8f0 0x557efab30db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc5d600640 10 client.4650 ms_handle_connect on v2:172.21.15.146:6834/1807355541 2024-09-16T01:49:32.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.830+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efab5f8f0 0x557efab30db0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=501638b105704ee9 server_cookie=a0ab8839b84d7f1e in_seq=0 out_seq=0 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.831+0000 7fdc5d600640 1 -- 172.21.15.146:0/430987312 <== mds.0 v2:172.21.15.146:6834/1807355541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557efaa7fb40 con 0x557efab5f8f0 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.831+0000 7fdc5d600640 10 client.4650 handle_command_reply: tid=0 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.831+0000 7fdc82276740 1 client.4650 shutdown 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 msgr2=0x557efab30870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 0x557efab30870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009e80 tx=0x7fdc70004ab0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 shutdown_connections 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efab5f8f0 0x557efab30db0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10710 0x557efab30870 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab40670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 --2- 172.21.15.146:0/430987312 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab31d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc5d600640 0 client.4650 ms_handle_reset on v2:172.21.15.146:6834/1807355541 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 >> 172.21.15.146:0/430987312 conn(0x557efab0ea50 msgr2=0x557efab52670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 shutdown_connections 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 1 -- 172.21.15.146:0/430987312 wait complete. 2024-09-16T01:49:32.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.832+0000 7fdc82276740 20 client.4650 trim_cache size 0 max 16384 2024-09-16T01:49:32.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.850+0000 7fdc82276740 1 Processor -- start 2024-09-16T01:49:32.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.850+0000 7fdc82276740 1 -- start start 2024-09-16T01:49:32.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc82276740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 0x557efab0e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab73a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab10710 0x557efab73f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc82276740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdc781fcd50 con 0x557efab0db60 2024-09-16T01:49:32.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdc780c8f10 con 0x557efab10330 2024-09-16T01:49:32.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x557efab35f70 con 0x557efab10710 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc7ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 0x557efab0e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc7ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 0x557efab0e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47140/0 (socket says 172.21.15.146:47140) 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc7ca00640 1 -- 172.21.15.146:0/2957800589 learned_addr learned my addr 172.21.15.146:0/2957800589 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc7d400640 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab10710 0x557efab73f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.851+0000 7fdc77e00640 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab73a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc7ca00640 1 -- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab10710 msgr2=0x557efab73f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab10710 0x557efab73f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc7ca00640 1 -- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 msgr2=0x557efab73a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab73a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc7ca00640 1 -- 172.21.15.146:0/2957800589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557efaa6e590 con 0x557efab0db60 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 0x557efab0e130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c001eb0 tx=0x7fdc6c00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75fe027916dcb34 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc5fe00640 1 -- 172.21.15.146:0/2957800589 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c02f070 con 0x557efab0db60 2024-09-16T01:49:32.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc5fe00640 1 -- 172.21.15.146:0/2957800589 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6c01b040 con 0x557efab0db60 2024-09-16T01:49:32.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.852+0000 7fdc5fe00640 1 -- 172.21.15.146:0/2957800589 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c016ac0 con 0x557efab0db60 2024-09-16T01:49:32.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 -- 172.21.15.146:0/2957800589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 msgr2=0x557efab0e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 0x557efab0e130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c001eb0 tx=0x7fdc6c00a000 comp rx=0 tx=0).stop 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 -- 172.21.15.146:0/2957800589 shutdown_connections 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab10710 0x557efab73f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10330 0x557efab73a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 --2- 172.21.15.146:0/2957800589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab0db60 0x557efab0e130 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 -- 172.21.15.146:0/2957800589 >> 172.21.15.146:0/2957800589 conn(0x557efab0ea50 msgr2=0x557efab403f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 -- 172.21.15.146:0/2957800589 shutdown_connections 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.853+0000 7fdc82276740 1 -- 172.21.15.146:0/2957800589 wait complete. 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.854+0000 7fdc82276740 1 -- start start 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.854+0000 7fdc82276740 1 Processor -- start 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab38070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab385b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557efab63020 con 0x557efab10330 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x557efab40bb0 con 0x557efab10710 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x557efab5fde0 con 0x557efab0db60 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc82276740 10 client.0 _ll_register_callbacks cb 0x7fdc70009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc7ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab38070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc7ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab38070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36506/0 (socket says 172.21.15.146:36506) 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47144/0 (socket says 172.21.15.146:47144) 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc7ca00640 1 -- 172.21.15.146:0/3138610801 learned_addr learned my addr 172.21.15.146:0/3138610801 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc7d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 -- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 msgr2=0x557efab38070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab38070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 -- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 msgr2=0x557efab360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab360f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 -- 172.21.15.146:0/3138610801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557efab0a7e0 con 0x557efab10330 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc7d400640 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc77e00640 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab385b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400e980 tx=0x7fdc64000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8f74fba0570f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.855+0000 7fdc5d600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc5d600640 1 -- 172.21.15.146:0/3138610801 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64021040 con 0x557efab10330 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc5d600640 1 -- 172.21.15.146:0/3138610801 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdc64079dd0 con 0x557efab10330 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc6c01b040 con 0x557efab10330 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc5d600640 1 -- 172.21.15.146:0/3138610801 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fdc640893f0 con 0x557efab10330 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 10 client.4656 fetch_fsmap learned FSMap version 16 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x557efabd23e0 con 0x557efab10330 2024-09-16T01:49:32.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc5d600640 1 -- 172.21.15.146:0/3138610801 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc6c01b040 con 0x557efab10330 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc5d600640 1 -- 172.21.15.146:0/3138610801 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 16) ==== 4792+0+0 (secure 0 0 0) 0x7fdc64024060 con 0x557efab10330 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 10 client.4656 fetch_fsmap finished waiting for FSMap version 16 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 10 client.4656 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 20 client.4656 populate_metadata read hostname 'smithi146' 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efabc78a0 0x557efabc7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 4 client.4656 mds_command: new command op to 4291 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 --> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] -- command(tid 0: {"prefix": "status"}) -- 0x557efaa79e90 con 0x557efabc78a0 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.856+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efabc78a0 0x557efabc7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc7ca00640 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efabc78a0 0x557efabc7c80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7067fd1bb495f7fe server_cookie=de4d827d6c31654b in_seq=0 out_seq=0 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc5d600640 10 client.4656 ms_handle_connect on v2:172.21.15.146:6834/1807355541 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc5d600640 1 -- 172.21.15.146:0/3138610801 <== mds.0 v2:172.21.15.146:6834/1807355541 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x557efaa79e90 con 0x557efabc78a0 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc5d600640 10 client.4656 handle_command_reply: tid=0 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 client.4656 shutdown 2024-09-16T01:49:32.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 msgr2=0x557efab385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab385b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400e980 tx=0x7fdc64000bc0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 shutdown_connections 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:6834/1807355541,v1:172.21.15.146:6836/1807355541] conn(0x557efabc78a0 0x557efabc7c80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557efab10710 0x557efab360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557efab10330 0x557efab385b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 --2- 172.21.15.146:0/3138610801 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x557efab0db60 0x557efab38070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 >> 172.21.15.146:0/3138610801 conn(0x557efab0ea50 msgr2=0x557efab0ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.857+0000 7fdc5d600640 0 client.4656 ms_handle_reset on v2:172.21.15.146:6834/1807355541 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 shutdown_connections 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc82276740 1 -- 172.21.15.146:0/3138610801 wait complete. 2024-09-16T01:49:32.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc82276740 20 client.4656 trim_cache size 0 max 16384 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4291, 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 34.040549726999998, 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 16, 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 22, 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 22, 2024-09-16T01:49:32.865 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 40.305960603000003 2024-09-16T01:49:32.866 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:32.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdc5806c2b0 msgr2=0x7fdc5806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdc5806c2b0 0x7fdc5806e770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c005880 tx=0x7fdc6c009040 comp rx=0 tx=0).stop 2024-09-16T01:49:32.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 msgr2=0x7fdc780abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 0x7fdc780abd40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400e9b0 tx=0x7fdc6400ee70 comp rx=0 tx=0).stop 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 shutdown_connections 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdc5806c2b0 0x7fdc5806e770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc780cc040 0x7fdc780c2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc780c75a0 0x7fdc780abd40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 --2- 172.21.15.146:0/3947583146 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdc780c6b70 0x7fdc7815d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.858+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 >> 172.21.15.146:0/3947583146 conn(0x7fdc780ccc20 msgr2=0x7fdc780cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.859+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 shutdown_connections 2024-09-16T01:49:32.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:32.859+0000 7fdc7de00640 1 -- 172.21.15.146:0/3947583146 wait complete. 2024-09-16T01:49:32.876 INFO:teuthology.run_tasks:Running task sequential... 2024-09-16T01:49:32.886 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-16T01:49:32.886 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-16T01:49:32.887 INFO:teuthology.task.print:Enabling mgr modules 2024-09-16T01:49:32.887 INFO:teuthology.run_tasks:Running task kclient... 2024-09-16T01:49:32.953 INFO:tasks.kclient:Mounting kernel clients... 2024-09-16T01:49:32.954 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-16T01:49:32.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:32.954 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-09-16T01:49:32.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:32.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-09-16T01:49:33.091 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-09-16T01:49:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:49:33.094 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-09-16T01:49:33.094 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:49:33.095 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:33.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:49:33.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.246+0000 7f5c2ac00640 1 Processor -- start 2024-09-16T01:49:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.246+0000 7f5c2ac00640 1 -- start start 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c2ac00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5c240c6410 0x7f5c24158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c2ac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c240c5280 0x7f5c240c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c2ac00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c24151e70 con 0x7f5c24151a90 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c2ac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5c240c8130 con 0x7f5c240c5280 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c2ac00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5c240c8410 con 0x7f5c240c6410 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c23e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5c24151a90 0x7f5c240c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47814/0 (socket says 172.21.15.146:47814) 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c23e00640 1 -- 172.21.15.146:0/2065143272 learned_addr learned my addr 172.21.15.146:0/2065143272 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c28c00640 1 --2- 172.21.15.146:0/2065143272 >> v2:172.21.15.146:3300/0 conn(0x7f5c240c5280 0x7f5c240c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c23400640 1 --2- 172.21.15.146:0/2065143272 >> v2:172.21.15.161:3300/0 conn(0x7f5c240c6410 0x7f5c24158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c22a00640 1 -- 172.21.15.146:0/2065143272 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855474978 0 0) 0x7f5c24151e70 con 0x7f5c24151a90 2024-09-16T01:49:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.248+0000 7f5c22a00640 1 -- 172.21.15.146:0/2065143272 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c08003520 con 0x7f5c24151a90 2024-09-16T01:49:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.249+0000 7f5c28c00640 1 -- 172.21.15.146:0/2065143272 >> v2:172.21.15.161:3300/0 conn(0x7f5c240c6410 msgr2=0x7f5c24158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.249+0000 7f5c28c00640 1 --2- 172.21.15.146:0/2065143272 >> v2:172.21.15.161:3300/0 conn(0x7f5c240c6410 0x7f5c24158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.249+0000 7f5c28c00640 1 -- 172.21.15.146:0/2065143272 >> v1:172.21.15.146:6789/0 conn(0x7f5c24151a90 legacy=0x7f5c240c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.249+0000 7f5c28c00640 1 -- 172.21.15.146:0/2065143272 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c240c86f0 con 0x7f5c240c5280 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c28c00640 1 --2- 172.21.15.146:0/2065143272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c5280 0x7f5c240c56f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5c2406ccb0 tx=0x7f5c1c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a167ac82b68c69 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c22a00640 1 -- 172.21.15.146:0/2065143272 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c037070 con 0x7f5c240c5280 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c22a00640 1 -- 172.21.15.146:0/2065143272 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c1c032440 con 0x7f5c240c5280 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c22a00640 1 -- 172.21.15.146:0/2065143272 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c032810 con 0x7f5c240c5280 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 -- 172.21.15.146:0/2065143272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c5280 msgr2=0x7f5c240c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/2065143272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c5280 0x7f5c240c56f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5c2406ccb0 tx=0x7f5c1c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 -- 172.21.15.146:0/2065143272 shutdown_connections 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/2065143272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c5280 0x7f5c240c56f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/2065143272 >> v2:172.21.15.161:3300/0 conn(0x7f5c240c6410 0x7f5c24158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 -- 172.21.15.146:0/2065143272 >> 172.21.15.146:0/2065143272 conn(0x7f5c2406f3c0 msgr2=0x7f5c2406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:33.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.250+0000 7f5c2ac00640 1 -- 172.21.15.146:0/2065143272 shutdown_connections 2024-09-16T01:49:33.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.251+0000 7f5c2ac00640 1 -- 172.21.15.146:0/2065143272 wait complete. 2024-09-16T01:49:33.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.252+0000 7f5c2ac00640 1 Processor -- start 2024-09-16T01:49:33.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.252+0000 7f5c2ac00640 1 -- start start 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c2ac00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5c240c5280 0x7f5c24147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c2ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 0x7f5c24148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c2ac00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c24151a90 0x7f5c24162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c2ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5c24157b20 con 0x7f5c240c6410 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c2ac00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5c24157840 con 0x7f5c24151a90 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c2ac00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5c24158580 con 0x7f5c240c5280 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c23400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 0x7f5c24148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c23400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 0x7f5c24148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47152/0 (socket says 172.21.15.146:47152) 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c23400640 1 -- 172.21.15.146:0/912737754 learned_addr learned my addr 172.21.15.146:0/912737754 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c23e00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5c240c5280 0x7f5c24147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.253+0000 7f5c28c00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c24151a90 0x7f5c24162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c23400640 1 -- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5c240c5280 msgr2=0x7f5c24147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c23400640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5c240c5280 0x7f5c24147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c23400640 1 -- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c24151a90 msgr2=0x7f5c24162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c23400640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c24151a90 0x7f5c24162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c23400640 1 -- 172.21.15.146:0/912737754 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c241f13b0 con 0x7f5c240c6410 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c23400640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 0x7f5c24148690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000e980 tx=0x7f5c1000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10017070 con 0x7f5c240c6410 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c1000ca00 con 0x7f5c240c6410 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c241f1690 con 0x7f5c240c6410 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1000cdd0 con 0x7f5c240c6410 2024-09-16T01:49:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.254+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c24164e20 con 0x7f5c240c6410 2024-09-16T01:49:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.255+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c240c71f0 con 0x7f5c240c6410 2024-09-16T01:49:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.256+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5c1001a070 con 0x7f5c240c6410 2024-09-16T01:49:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.257+0000 7f5c1be00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5c0406c2b0 0x7f5c0406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:33.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.257+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5c1008b790 con 0x7f5c240c6410 2024-09-16T01:49:33.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.257+0000 7f5c23e00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5c0406c2b0 0x7f5c0406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:33.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.258+0000 7f5c23e00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5c0406c2b0 0x7f5c0406e770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c14005c00 tx=0x7f5c14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:33.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.262+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5c10059810 con 0x7f5c240c6410 2024-09-16T01:49:33.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.471+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5c241547e0 con 0x7f5c240c6410 2024-09-16T01:49:33.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.471+0000 7f5c1be00640 1 -- 172.21.15.146:0/912737754 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f5c1005db10 con 0x7f5c240c6410 2024-09-16T01:49:33.474 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:49:33.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5c0406c2b0 msgr2=0x7f5c0406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:33.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5c0406c2b0 0x7f5c0406e770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c14005c00 tx=0x7f5c14009040 comp rx=0 tx=0).stop 2024-09-16T01:49:33.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 msgr2=0x7f5c24148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:33.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 0x7f5c24148690 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000e980 tx=0x7f5c1000ee40 comp rx=0 tx=0).stop 2024-09-16T01:49:33.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 shutdown_connections 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5c0406c2b0 0x7f5c0406e770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c24151a90 0x7f5c24162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c240c6410 0x7f5c24148690 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 --2- 172.21.15.146:0/912737754 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5c240c5280 0x7f5c24147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 >> 172.21.15.146:0/912737754 conn(0x7f5c2406f3c0 msgr2=0x7f5c2406f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 shutdown_connections 2024-09-16T01:49:33.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33.474+0000 7f5c2ac00640 1 -- 172.21.15.146:0/912737754 wait complete. 2024-09-16T01:49:33.492 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:49:33.493 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:49:33.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:33.494 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-09-16T01:49:33.530 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:49:33.530 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:49:33.530 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-09-16T01:49:33.531 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:49:33.532 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-09-16T01:49:33.532 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:49:33.532 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42667sec preferred_lft 42667sec 2024-09-16T01:49:33.532 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link noprefixroute 2024-09-16T01:49:33.532 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:49:33.532 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T01:49:33.533 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-09-16T01:49:33.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:49:33.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:33.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:33.942 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:49:33.999 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-09-16T01:49:34.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:34.003 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-09-16T01:49:34.058 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-09-16T01:49:34.058 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:49:34.058 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:49:34.058 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:49:34.058 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:49:34.059 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:49:34.059 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-09-16T01:49:34.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:49:34.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:49:34.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:49:34.059 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-09-16T01:49:34.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:49:34.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:34.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:34.636 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-09-16T01:49:34.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:34.673 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-09-16T01:49:34.750 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:49:34.750 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-09-16T01:49:34.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:49:34.750 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-16T01:49:34.750 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-09-16T01:49:34.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:49:34.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:34.927 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-16T01:49:34.928 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-09-16T01:49:35.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:49:35.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:35.167 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:49:35.167 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-09-16T01:49:35.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-09-16T01:49:35.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T01:49:35.168 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-09-16T01:49:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:49:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:35.338 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:49:35.338 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:35.373 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:49:35.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:49:35.374 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:35.450 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-16T01:49:35.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:35.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-16T01:49:35.586 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-16T01:49:35.586 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-16T01:49:36.290 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-16T01:49:36.290 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-16T01:49:36.290 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.146:3300,172.21.15.161:3300,172.21.15.161:3301" 2024-09-16T01:49:36.290 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: trying mount with new device syntax: 0@4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d.cephfs=/ 2024-09-16T01:49:36.290 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.146:3300/172.21.15.161:3300/172.21.15.161:3301" will pass to kernel 2024-09-16T01:49:36.293 INFO:teuthology.orchestra.run.smithi146.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:49:36.293 INFO:teuthology.orchestra.run.smithi146.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:49:36.293 INFO:teuthology.orchestra.run.smithi146.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:49:36.293 INFO:teuthology.orchestra.run.smithi146.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:49:36.293 INFO:teuthology.orchestra.run.smithi146.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:49:36.296 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:49:36.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:49:36.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:36.359 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:49:36.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:49:36.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:36.457 INFO:teuthology.orchestra.run.smithi146.stdout:client4619 2024-09-16T01:49:36.457 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:49:36.457 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:49:36.493 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:49:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:49:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:36.633 INFO:teuthology.orchestra.run.smithi146.stdout:client4619 2024-09-16T01:49:36.633 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:49:36.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:49:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:36.731 INFO:teuthology.orchestra.run.smithi146.stdout:client4619 2024-09-16T01:49:36.731 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d.client4619/status') 2024-09-16T01:49:36.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:49:36.837 INFO:teuthology.orchestra.run.smithi146.stdout:instance: client.4619 (3)172.21.15.146:0/3300930908 2024-09-16T01:49:36.837 INFO:teuthology.orchestra.run.smithi146.stdout:blocklisted: false 2024-09-16T01:49:36.837 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records in 2024-09-16T01:49:36.838 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records out 2024-09-16T01:49:36.838 INFO:teuthology.orchestra.run.smithi146.stderr:71 bytes copied, 5.258e-05 s, 1.4 MB/s 2024-09-16T01:49:36.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:49:36.848 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T01:49:36.880 INFO:tasks.thrashosds:config is {'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T01:49:36.880 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}} 2024-09-16T01:49:36.881 INFO:tasks.thrashosds:config is {'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T01:49:36.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T01:49:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.037+0000 7f5bef200640 1 Processor -- start 2024-09-16T01:49:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.038+0000 7f5bef200640 1 -- start start 2024-09-16T01:49:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bef200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5be8152760 0x7f5be80c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bef200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 0x7f5be80c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bef200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bef200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5be80c8530 con 0x7f5be80c6cc0 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bef200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5be8158890 con 0x7f5be8152760 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bef200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5be8158b00 con 0x7f5be806fee0 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bdf400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 0x7f5be80c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bdf400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 0x7f5be80c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47158/0 (socket says 172.21.15.146:47158) 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bdf400640 1 -- 172.21.15.146:0/3756091157 learned_addr learned my addr 172.21.15.146:0/3756091157 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5bed200640 1 --2- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.039+0000 7f5be7e00640 1 --2- 172.21.15.146:0/3756091157 >> v2:172.21.15.161:3300/0 conn(0x7f5be8152760 0x7f5be80c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.040+0000 7f5be7e00640 1 -- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 msgr2=0x7f5be815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.040+0000 7f5be7e00640 1 --2- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.040+0000 7f5be7e00640 1 -- 172.21.15.146:0/3756091157 >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 msgr2=0x7f5be80c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:37.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.040+0000 7f5be7e00640 1 --2- 172.21.15.146:0/3756091157 >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 0x7f5be80c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.040+0000 7f5be7e00640 1 -- 172.21.15.146:0/3756091157 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be8158de0 con 0x7f5be8152760 2024-09-16T01:49:37.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.040+0000 7f5bdf400640 1 --2- 172.21.15.146:0/3756091157 >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 0x7f5be80c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.041+0000 7f5be7e00640 1 --2- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be80c59d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5be806d830 tx=0x7f5bd802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51d51ac0b44b97ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.041+0000 7f5be7400640 1 -- 172.21.15.146:0/3756091157 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8037070 con 0x7f5be8152760 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.041+0000 7f5be7400640 1 -- 172.21.15.146:0/3756091157 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd8032440 con 0x7f5be8152760 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.041+0000 7f5be7400640 1 -- 172.21.15.146:0/3756091157 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8032810 con 0x7f5be8152760 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.041+0000 7f5bef200640 1 -- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 msgr2=0x7f5be80c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.041+0000 7f5bef200640 1 --2- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be80c59d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5be806d830 tx=0x7f5bd802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 -- 172.21.15.146:0/3756091157 shutdown_connections 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 --2- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 --2- 172.21.15.146:0/3756091157 >> v2:172.21.15.146:3300/0 conn(0x7f5be80c6cc0 0x7f5be80c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 --2- 172.21.15.146:0/3756091157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be80c59d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 -- 172.21.15.146:0/3756091157 >> 172.21.15.146:0/3756091157 conn(0x7f5be806e670 msgr2=0x7f5be806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 -- 172.21.15.146:0/3756091157 shutdown_connections 2024-09-16T01:49:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.042+0000 7f5bef200640 1 -- 172.21.15.146:0/3756091157 wait complete. 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.043+0000 7f5bef200640 1 Processor -- start 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.043+0000 7f5bef200640 1 -- start start 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.044+0000 7f5bef200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be80cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.044+0000 7f5bef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5be80c6cc0 0x7f5be80ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.044+0000 7f5bef200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be814bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.044+0000 7f5bef200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5be8152b40 con 0x7f5be80c6cc0 2024-09-16T01:49:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.044+0000 7f5bef200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5be81593a0 con 0x7f5be8152760 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.044+0000 7f5bef200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5be8152cc0 con 0x7f5be806fee0 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be80cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5bed200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be814bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be80cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36528/0 (socket says 172.21.15.146:36528) 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 -- 172.21.15.146:0/1791124974 learned_addr learned my addr 172.21.15.146:0/1791124974 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5bdf400640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5be80c6cc0 0x7f5be80ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 -- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 msgr2=0x7f5be814bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be814bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 -- 172.21.15.146:0/1791124974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5be80c6cc0 msgr2=0x7f5be80ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5be80c6cc0 0x7f5be80ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 -- 172.21.15.146:0/1791124974 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be80cb8a0 con 0x7f5be806fee0 2024-09-16T01:49:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be7e00640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be80cb360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009890 tx=0x7f5bd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.045+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8041020 con 0x7f5be806fee0 2024-09-16T01:49:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.046+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd803d700 con 0x7f5be806fee0 2024-09-16T01:49:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.046+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be8154980 con 0x7f5be806fee0 2024-09-16T01:49:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.046+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8037040 con 0x7f5be806fee0 2024-09-16T01:49:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.046+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be8154f40 con 0x7f5be806fee0 2024-09-16T01:49:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.047+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be8152e40 con 0x7f5be806fee0 2024-09-16T01:49:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.048+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5bd8044020 con 0x7f5be806fee0 2024-09-16T01:49:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.048+0000 7f5be4c00640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bc406c2b0 0x7f5bc406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.048+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5bd80abb70 con 0x7f5be806fee0 2024-09-16T01:49:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.049+0000 7f5bdf400640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bc406c2b0 0x7f5bc406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.049+0000 7f5bdf400640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bc406c2b0 0x7f5bc406e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5be81e3ee0 tx=0x7f5bd0009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.053+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5be8152e40 con 0x7f5be806fee0 2024-09-16T01:49:37.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.230+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) -- 0x7f5be814ca00 con 0x7f5be806fee0 2024-09-16T01:49:37.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.236+0000 7f5be4c00640 1 -- 172.21.15.146:0/1791124974 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v10)=0 v10) ==== 136+0+0 (secure 0 0 0) 0x7f5be814ca00 con 0x7f5be806fee0 2024-09-16T01:49:37.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bc406c2b0 msgr2=0x7f5bc406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bc406c2b0 0x7f5bc406e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5be81e3ee0 tx=0x7f5bd0009240 comp rx=0 tx=0).stop 2024-09-16T01:49:37.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 msgr2=0x7f5be80cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be80cb360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009890 tx=0x7f5bd80046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 shutdown_connections 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bc406c2b0 0x7f5bc406e770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5be8152760 0x7f5be814bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5be80c6cc0 0x7f5be80ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 --2- 172.21.15.146:0/1791124974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5be806fee0 0x7f5be80cb360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 >> 172.21.15.146:0/1791124974 conn(0x7f5be806e670 msgr2=0x7f5be8151780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 shutdown_connections 2024-09-16T01:49:37.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.239+0000 7f5bef200640 1 -- 172.21.15.146:0/1791124974 wait complete. 2024-09-16T01:49:37.256 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T01:49:37.256 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:49:37.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:37.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.409+0000 7f4282000640 1 Processor -- start 2024-09-16T01:49:37.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.409+0000 7f4282000640 1 -- start start 2024-09-16T01:49:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f4282000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f427c0c6bb0 0x7f427c0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f4282000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f427c0c58c0 0x7f427c0c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f4282000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427c072f50 con 0x7f427c152850 2024-09-16T01:49:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f4282000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f427c073210 con 0x7f427c0c6bb0 2024-09-16T01:49:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f4282000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f427c15cb50 con 0x7f427c0c58c0 2024-09-16T01:49:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f427b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f427c152850 0x7f427c158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47824/0 (socket says 172.21.15.146:47824) 2024-09-16T01:49:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f427aa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f427c0c6bb0 0x7f427c0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.410+0000 7f427b400640 1 -- 172.21.15.146:0/2219218107 learned_addr learned my addr 172.21.15.146:0/2219218107 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.411+0000 7f427be00640 1 --2- 172.21.15.146:0/2219218107 >> v2:172.21.15.161:3300/0 conn(0x7f427c0c58c0 0x7f427c0c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.411+0000 7f427aa00640 1 -- 172.21.15.146:0/2219218107 >> v2:172.21.15.161:3300/0 conn(0x7f427c0c58c0 msgr2=0x7f427c0c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.411+0000 7f427aa00640 1 --2- 172.21.15.146:0/2219218107 >> v2:172.21.15.161:3300/0 conn(0x7f427c0c58c0 0x7f427c0c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.411+0000 7f427aa00640 1 -- 172.21.15.146:0/2219218107 >> v1:172.21.15.146:6789/0 conn(0x7f427c152850 legacy=0x7f427c158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.411+0000 7f427a000640 1 -- 172.21.15.146:0/2219218107 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633145376 0 0) 0x7f427c072f50 con 0x7f427c152850 2024-09-16T01:49:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.411+0000 7f427aa00640 1 -- 172.21.15.146:0/2219218107 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c15bd90 con 0x7f427c0c6bb0 2024-09-16T01:49:37.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.412+0000 7f427aa00640 1 --2- 172.21.15.146:0/2219218107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c6bb0 0x7f427c0c9700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f427c06f350 tx=0x7f426802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95489209c9b7b990 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.412+0000 7f427a000640 1 -- 172.21.15.146:0/2219218107 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268037070 con 0x7f427c0c6bb0 2024-09-16T01:49:37.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.412+0000 7f427a000640 1 -- 172.21.15.146:0/2219218107 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4268032440 con 0x7f427c0c6bb0 2024-09-16T01:49:37.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.412+0000 7f427a000640 1 -- 172.21.15.146:0/2219218107 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268032810 con 0x7f427c0c6bb0 2024-09-16T01:49:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 -- 172.21.15.146:0/2219218107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c6bb0 msgr2=0x7f427c0c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 --2- 172.21.15.146:0/2219218107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c6bb0 0x7f427c0c9700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f427c06f350 tx=0x7f426802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 -- 172.21.15.146:0/2219218107 shutdown_connections 2024-09-16T01:49:37.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 --2- 172.21.15.146:0/2219218107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c6bb0 0x7f427c0c9700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 --2- 172.21.15.146:0/2219218107 >> v2:172.21.15.161:3300/0 conn(0x7f427c0c58c0 0x7f427c0c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 -- 172.21.15.146:0/2219218107 >> 172.21.15.146:0/2219218107 conn(0x7f427c06f3c0 msgr2=0x7f427c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 -- 172.21.15.146:0/2219218107 shutdown_connections 2024-09-16T01:49:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.413+0000 7f4282000640 1 -- 172.21.15.146:0/2219218107 wait complete. 2024-09-16T01:49:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.414+0000 7f4282000640 1 Processor -- start 2024-09-16T01:49:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 -- start start 2024-09-16T01:49:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 0x7f427c146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f427c0c6bb0 0x7f427c1498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f427c152850 0x7f427c14a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f427c15c630 con 0x7f427c0c58c0 2024-09-16T01:49:37.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f427c15c350 con 0x7f427c152850 2024-09-16T01:49:37.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.415+0000 7f4282000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f427c070200 con 0x7f427c0c6bb0 2024-09-16T01:49:37.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 0x7f427c146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 0x7f427c146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47194/0 (socket says 172.21.15.146:47194) 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 -- 172.21.15.146:0/2903576350 learned_addr learned my addr 172.21.15.146:0/2903576350 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427be00640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f427c152850 0x7f427c14a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427aa00640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f427c0c6bb0 0x7f427c1498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 -- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f427c0c6bb0 msgr2=0x7f427c1498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f427c0c6bb0 0x7f427c1498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 -- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f427c152850 msgr2=0x7f427c14a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f427c152850 0x7f427c14a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 -- 172.21.15.146:0/2903576350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c147720 con 0x7f427c0c58c0 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427aa00640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f427c0c6bb0 0x7f427c1498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.416+0000 7f427b400640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 0x7f427c146340 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f426c00c460 tx=0x7f426c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.417+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c018070 con 0x7f427c0c58c0 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.417+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c007500 con 0x7f427c0c58c0 2024-09-16T01:49:37.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.417+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c004a20 con 0x7f427c0c58c0 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.417+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c1f4a80 con 0x7f427c0c58c0 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.417+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c0ccd30 con 0x7f427c0c58c0 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.418+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f427c0c7620 con 0x7f427c0c58c0 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.419+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f426c004bc0 con 0x7f427c0c58c0 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.420+0000 7f4273400640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425c06c2b0 0x7f425c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.420+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f426c08a670 con 0x7f427c0c58c0 2024-09-16T01:49:37.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.420+0000 7f427aa00640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425c06c2b0 0x7f425c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.421+0000 7f427aa00640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425c06c2b0 0x7f425c06e770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4268034000 tx=0x7f426802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.424+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f427c0c7620 con 0x7f427c0c58c0 2024-09-16T01:49:37.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.596+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f427c1f4d60 con 0x7f425c06c2b0 2024-09-16T01:49:37.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.603+0000 7f4273400640 1 -- 172.21.15.146:0/2903576350 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252003 (secure 0 0 0) 0x7f427c1f4d60 con 0x7f425c06c2b0 2024-09-16T01:49:37.607 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:37.609 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425c06c2b0 msgr2=0x7f425c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425c06c2b0 0x7f425c06e770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4268034000 tx=0x7f426802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 msgr2=0x7f427c146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 0x7f427c146340 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f426c00c460 tx=0x7f426c00c920 comp rx=0 tx=0).stop 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 shutdown_connections 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425c06c2b0 0x7f425c06e770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f427c152850 0x7f427c14a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f427c0c6bb0 0x7f427c1498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 --2- 172.21.15.146:0/2903576350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0c58c0 0x7f427c146340 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 >> 172.21.15.146:0/2903576350 conn(0x7f427c06f3c0 msgr2=0x7f427c0c8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 shutdown_connections 2024-09-16T01:49:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.606+0000 7f4282000640 1 -- 172.21.15.146:0/2903576350 wait complete. 2024-09-16T01:49:37.624 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-16T01:49:36.527089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":295,"ondisk_log_size":295,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218304,"kb_used_data":2112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756416,"statfs":{"total":773094113280,"available":772870569984,"internally_reserved":0,"allocated":2162688,"data_stored":1823012,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2028,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003190"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039277+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:49:01.039277+0000","last_peered":"2024-09-16T01:49:01.039277+0000","last_clean":"2024-09-16T01:49:01.039277+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:49:01.039277+0000","last_undegraded":"2024-09-16T01:49:01.039277+0000","last_fullsized":"2024-09-16T01:49:01.039277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039170+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:49:01.039170+0000","last_peered":"2024-09-16T01:49:01.039170+0000","last_clean":"2024-09-16T01:49:01.039170+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:49:01.039170+0000","last_undegraded":"2024-09-16T01:49:01.039170+0000","last_fullsized":"2024-09-16T01:49:01.039170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038749+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:49:01.038749+0000","last_peered":"2024-09-16T01:49:01.038749+0000","last_clean":"2024-09-16T01:49:01.038749+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:49:01.038749+0000","last_undegraded":"2024-09-16T01:49:01.038749+0000","last_fullsized":"2024-09-16T01:49:01.038749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038386+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:49:01.038386+0000","last_peered":"2024-09-16T01:49:01.038386+0000","last_clean":"2024-09-16T01:49:01.038386+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:49:01.038386+0000","last_undegraded":"2024-09-16T01:49:01.038386+0000","last_fullsized":"2024-09-16T01:49:01.038386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:09.260093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038850+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:49:01.038850+0000","last_peered":"2024-09-16T01:49:01.038850+0000","last_clean":"2024-09-16T01:49:01.038850+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:49:01.038850+0000","last_undegraded":"2024-09-16T01:49:01.038850+0000","last_fullsized":"2024-09-16T01:49:01.038850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824333+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:59.824333+0000","last_peered":"2024-09-16T01:48:59.824333+0000","last_clean":"2024-09-16T01:48:59.824333+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:59.824333+0000","last_undegraded":"2024-09-16T01:48:59.824333+0000","last_fullsized":"2024-09-16T01:48:59.824333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:54.394499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038902+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:49:01.038902+0000","last_peered":"2024-09-16T01:49:01.038902+0000","last_clean":"2024-09-16T01:49:01.038902+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:49:01.038902+0000","last_undegraded":"2024-09-16T01:49:01.038902+0000","last_fullsized":"2024-09-16T01:49:01.038902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039312+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:49:01.039312+0000","last_peered":"2024-09-16T01:49:01.039312+0000","last_clean":"2024-09-16T01:49:01.039312+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:49:01.039312+0000","last_undegraded":"2024-09-16T01:49:01.039312+0000","last_fullsized":"2024-09-16T01:49:01.039312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038909+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:49:01.038909+0000","last_peered":"2024-09-16T01:49:01.038909+0000","last_clean":"2024-09-16T01:49:01.038909+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:49:01.038909+0000","last_undegraded":"2024-09-16T01:49:01.038909+0000","last_fullsized":"2024-09-16T01:49:01.038909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038603+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:49:01.038603+0000","last_peered":"2024-09-16T01:49:01.038603+0000","last_clean":"2024-09-16T01:49:01.038603+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:49:01.038603+0000","last_undegraded":"2024-09-16T01:49:01.038603+0000","last_fullsized":"2024-09-16T01:49:01.038603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.863866+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:49:08.863866+0000","last_peered":"2024-09-16T01:49:08.863866+0000","last_clean":"2024-09-16T01:49:08.863866+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:49:08.863866+0000","last_undegraded":"2024-09-16T01:49:08.863866+0000","last_fullsized":"2024-09-16T01:49:08.863866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824444+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:59.824444+0000","last_peered":"2024-09-16T01:48:59.824444+0000","last_clean":"2024-09-16T01:48:59.824444+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:59.824444+0000","last_undegraded":"2024-09-16T01:48:59.824444+0000","last_fullsized":"2024-09-16T01:48:59.824444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:21.869634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917528+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:20.917528+0000","last_peered":"2024-09-16T01:49:20.917528+0000","last_clean":"2024-09-16T01:49:20.917528+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:20.917528+0000","last_undegraded":"2024-09-16T01:49:20.917528+0000","last_fullsized":"2024-09-16T01:49:20.917528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.901405+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:49:14.901405+0000","last_peered":"2024-09-16T01:49:14.901405+0000","last_clean":"2024-09-16T01:49:14.901405+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:49:14.901405+0000","last_undegraded":"2024-09-16T01:49:14.901405+0000","last_fullsized":"2024-09-16T01:49:14.901405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038968+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:49:01.038968+0000","last_peered":"2024-09-16T01:49:01.038968+0000","last_clean":"2024-09-16T01:49:01.038968+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:49:01.038968+0000","last_undegraded":"2024-09-16T01:49:01.038968+0000","last_fullsized":"2024-09-16T01:49:01.038968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918636+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:20.918636+0000","last_peered":"2024-09-16T01:49:20.918636+0000","last_clean":"2024-09-16T01:49:20.918636+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:20.918636+0000","last_undegraded":"2024-09-16T01:49:20.918636+0000","last_fullsized":"2024-09-16T01:49:20.918636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824266+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:59.824266+0000","last_peered":"2024-09-16T01:48:59.824266+0000","last_clean":"2024-09-16T01:48:59.824266+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:59.824266+0000","last_undegraded":"2024-09-16T01:48:59.824266+0000","last_fullsized":"2024-09-16T01:48:59.824266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:31.622539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.866956+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:08.866956+0000","last_peered":"2024-09-16T01:49:08.866956+0000","last_clean":"2024-09-16T01:49:08.866956+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:08.866956+0000","last_undegraded":"2024-09-16T01:49:08.866956+0000","last_fullsized":"2024-09-16T01:49:08.866956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.941828+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:25.941828+0000","last_peered":"2024-09-16T01:49:25.941828+0000","last_clean":"2024-09-16T01:49:25.941828+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:25.941828+0000","last_undegraded":"2024-09-16T01:49:25.941828+0000","last_fullsized":"2024-09-16T01:49:25.941828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039431+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:49:01.039431+0000","last_peered":"2024-09-16T01:49:01.039431+0000","last_clean":"2024-09-16T01:49:01.039431+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:49:01.039431+0000","last_undegraded":"2024-09-16T01:49:01.039431+0000","last_fullsized":"2024-09-16T01:49:01.039431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.863572+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:49:08.863572+0000","last_peered":"2024-09-16T01:49:08.863572+0000","last_clean":"2024-09-16T01:49:08.863572+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:49:08.863572+0000","last_undegraded":"2024-09-16T01:49:08.863572+0000","last_fullsized":"2024-09-16T01:49:08.863572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'10","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.940617+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:49:25.940617+0000","last_peered":"2024-09-16T01:49:25.940617+0000","last_clean":"2024-09-16T01:49:25.940617+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:49:25.940617+0000","last_undegraded":"2024-09-16T01:49:25.940617+0000","last_fullsized":"2024-09-16T01:49:25.940617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.940548+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:49:25.940548+0000","last_peered":"2024-09-16T01:49:25.940548+0000","last_clean":"2024-09-16T01:49:25.940548+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:49:25.940548+0000","last_undegraded":"2024-09-16T01:49:25.940548+0000","last_fullsized":"2024-09-16T01:49:25.940548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918151+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:20.918151+0000","last_peered":"2024-09-16T01:49:20.918151+0000","last_clean":"2024-09-16T01:49:20.918151+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:20.918151+0000","last_undegraded":"2024-09-16T01:49:20.918151+0000","last_fullsized":"2024-09-16T01:49:20.918151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918373+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:20.918373+0000","last_peered":"2024-09-16T01:49:20.918373+0000","last_clean":"2024-09-16T01:49:20.918373+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:20.918373+0000","last_undegraded":"2024-09-16T01:49:20.918373+0000","last_fullsized":"2024-09-16T01:49:20.918373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.916510+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:49:20.916510+0000","last_peered":"2024-09-16T01:49:20.916510+0000","last_clean":"2024-09-16T01:49:20.916510+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:49:20.916510+0000","last_undegraded":"2024-09-16T01:49:20.916510+0000","last_fullsized":"2024-09-16T01:49:20.916510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829332+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:03.829332+0000","last_peered":"2024-09-16T01:49:03.829332+0000","last_clean":"2024-09-16T01:49:03.829332+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:03.829332+0000","last_undegraded":"2024-09-16T01:49:03.829332+0000","last_fullsized":"2024-09-16T01:49:03.829332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.939777+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:49:25.939777+0000","last_peered":"2024-09-16T01:49:25.939777+0000","last_clean":"2024-09-16T01:49:25.939777+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:49:25.939777+0000","last_undegraded":"2024-09-16T01:49:25.939777+0000","last_fullsized":"2024-09-16T01:49:25.939777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917084+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:49:20.917084+0000","last_peered":"2024-09-16T01:49:20.917084+0000","last_clean":"2024-09-16T01:49:20.917084+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:49:20.917084+0000","last_undegraded":"2024-09-16T01:49:20.917084+0000","last_fullsized":"2024-09-16T01:49:20.917084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.916463+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:49:20.916463+0000","last_peered":"2024-09-16T01:49:20.916463+0000","last_clean":"2024-09-16T01:49:20.916463+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:49:20.916463+0000","last_undegraded":"2024-09-16T01:49:20.916463+0000","last_fullsized":"2024-09-16T01:49:20.916463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917307+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:20.917307+0000","last_peered":"2024-09-16T01:49:20.917307+0000","last_clean":"2024-09-16T01:49:20.917307+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:20.917307+0000","last_undegraded":"2024-09-16T01:49:20.917307+0000","last_fullsized":"2024-09-16T01:49:20.917307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824354+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:59.824354+0000","last_peered":"2024-09-16T01:48:59.824354+0000","last_clean":"2024-09-16T01:48:59.824354+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:59.824354+0000","last_undegraded":"2024-09-16T01:48:59.824354+0000","last_fullsized":"2024-09-16T01:48:59.824354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:03.840176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.903518+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:14.903518+0000","last_peered":"2024-09-16T01:49:14.903518+0000","last_clean":"2024-09-16T01:49:14.903518+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:14.903518+0000","last_undegraded":"2024-09-16T01:49:14.903518+0000","last_fullsized":"2024-09-16T01:49:14.903518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.903592+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:49:14.903592+0000","last_peered":"2024-09-16T01:49:14.903592+0000","last_clean":"2024-09-16T01:49:14.903592+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:49:14.903592+0000","last_undegraded":"2024-09-16T01:49:14.903592+0000","last_fullsized":"2024-09-16T01:49:14.903592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038525+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:49:01.038525+0000","last_peered":"2024-09-16T01:49:01.038525+0000","last_clean":"2024-09-16T01:49:01.038525+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:49:01.038525+0000","last_undegraded":"2024-09-16T01:49:01.038525+0000","last_fullsized":"2024-09-16T01:49:01.038525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:20.864782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.941526+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:25.941526+0000","last_peered":"2024-09-16T01:49:25.941526+0000","last_clean":"2024-09-16T01:49:25.941526+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:25.941526+0000","last_undegraded":"2024-09-16T01:49:25.941526+0000","last_fullsized":"2024-09-16T01:49:25.941526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039165+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:49:01.039165+0000","last_peered":"2024-09-16T01:49:01.039165+0000","last_clean":"2024-09-16T01:49:01.039165+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:49:01.039165+0000","last_undegraded":"2024-09-16T01:49:01.039165+0000","last_fullsized":"2024-09-16T01:49:01.039165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917502+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:20.917502+0000","last_peered":"2024-09-16T01:49:20.917502+0000","last_clean":"2024-09-16T01:49:20.917502+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:20.917502+0000","last_undegraded":"2024-09-16T01:49:20.917502+0000","last_fullsized":"2024-09-16T01:49:20.917502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.919024+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:49:20.919024+0000","last_peered":"2024-09-16T01:49:20.919024+0000","last_clean":"2024-09-16T01:49:20.919024+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:49:20.919024+0000","last_undegraded":"2024-09-16T01:49:20.919024+0000","last_fullsized":"2024-09-16T01:49:20.919024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824807+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:59.824807+0000","last_peered":"2024-09-16T01:48:59.824807+0000","last_clean":"2024-09-16T01:48:59.824807+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:59.824807+0000","last_undegraded":"2024-09-16T01:48:59.824807+0000","last_fullsized":"2024-09-16T01:48:59.824807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:08:09.528402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.945020+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:25.945020+0000","last_peered":"2024-09-16T01:49:25.945020+0000","last_clean":"2024-09-16T01:49:25.945020+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:25.945020+0000","last_undegraded":"2024-09-16T01:49:25.945020+0000","last_fullsized":"2024-09-16T01:49:25.945020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824748+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:59.824748+0000","last_peered":"2024-09-16T01:48:59.824748+0000","last_clean":"2024-09-16T01:48:59.824748+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:59.824748+0000","last_undegraded":"2024-09-16T01:48:59.824748+0000","last_fullsized":"2024-09-16T01:48:59.824748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:51:26.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.874853+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:08.874853+0000","last_peered":"2024-09-16T01:49:08.874853+0000","last_clean":"2024-09-16T01:49:08.874853+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:08.874853+0000","last_undegraded":"2024-09-16T01:49:08.874853+0000","last_fullsized":"2024-09-16T01:49:08.874853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.944989+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:49:25.944989+0000","last_peered":"2024-09-16T01:49:25.944989+0000","last_clean":"2024-09-16T01:49:25.944989+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:49:25.944989+0000","last_undegraded":"2024-09-16T01:49:25.944989+0000","last_fullsized":"2024-09-16T01:49:25.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824251+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:59.824251+0000","last_peered":"2024-09-16T01:48:59.824251+0000","last_clean":"2024-09-16T01:48:59.824251+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:59.824251+0000","last_undegraded":"2024-09-16T01:48:59.824251+0000","last_fullsized":"2024-09-16T01:48:59.824251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:26.577467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.914366+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:49:20.914366+0000","last_peered":"2024-09-16T01:49:20.914366+0000","last_clean":"2024-09-16T01:49:20.914366+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:49:20.914366+0000","last_undegraded":"2024-09-16T01:49:20.914366+0000","last_fullsized":"2024-09-16T01:49:20.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038826+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:49:01.038826+0000","last_peered":"2024-09-16T01:49:01.038826+0000","last_clean":"2024-09-16T01:49:01.038826+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:49:01.038826+0000","last_undegraded":"2024-09-16T01:49:01.038826+0000","last_fullsized":"2024-09-16T01:49:01.038826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.887158+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:08.887158+0000","last_peered":"2024-09-16T01:49:08.887158+0000","last_clean":"2024-09-16T01:49:08.887158+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:08.887158+0000","last_undegraded":"2024-09-16T01:49:08.887158+0000","last_fullsized":"2024-09-16T01:49:08.887158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.942546+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:25.942546+0000","last_peered":"2024-09-16T01:49:25.942546+0000","last_clean":"2024-09-16T01:49:25.942546+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:25.942546+0000","last_undegraded":"2024-09-16T01:49:25.942546+0000","last_fullsized":"2024-09-16T01:49:25.942546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'11","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.945415+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:25.945415+0000","last_peered":"2024-09-16T01:49:25.945415+0000","last_clean":"2024-09-16T01:49:25.945415+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:25.945415+0000","last_undegraded":"2024-09-16T01:49:25.945415+0000","last_fullsized":"2024-09-16T01:49:25.945415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039335+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:49:01.039335+0000","last_peered":"2024-09-16T01:49:01.039335+0000","last_clean":"2024-09-16T01:49:01.039335+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:49:01.039335+0000","last_undegraded":"2024-09-16T01:49:01.039335+0000","last_fullsized":"2024-09-16T01:49:01.039335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.828830+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:49:03.828830+0000","last_peered":"2024-09-16T01:49:03.828830+0000","last_clean":"2024-09-16T01:49:03.828830+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:49:03.828830+0000","last_undegraded":"2024-09-16T01:49:03.828830+0000","last_fullsized":"2024-09-16T01:49:03.828830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039274+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:49:01.039274+0000","last_peered":"2024-09-16T01:49:01.039274+0000","last_clean":"2024-09-16T01:49:01.039274+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:49:01.039274+0000","last_undegraded":"2024-09-16T01:49:01.039274+0000","last_fullsized":"2024-09-16T01:49:01.039274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.942146+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:49:25.942146+0000","last_peered":"2024-09-16T01:49:25.942146+0000","last_clean":"2024-09-16T01:49:25.942146+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:49:25.942146+0000","last_undegraded":"2024-09-16T01:49:25.942146+0000","last_fullsized":"2024-09-16T01:49:25.942146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":80998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":79956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509452,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":670058,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":80818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":79330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":80656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":669470,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":81726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:37.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:37.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.816+0000 7f70cca00640 1 Processor -- start 2024-09-16T01:49:37.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.816+0000 7f70cca00640 1 -- start start 2024-09-16T01:49:37.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70cca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70c80cc0c0 0x7f70c815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70cca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cc4a0 0x7f70c8165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70cca00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70c815b1d0 con 0x7f70c80cb2f0 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70cca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f70c80cbd00 con 0x7f70c80cc0c0 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70cca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70c815ba70 con 0x7f70c80cc4a0 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70c80cc0c0 0x7f70c815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c5400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f70c80cb2f0 0x7f70c815e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:45270/0 (socket says 172.21.15.146:45270) 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c5400640 1 -- 172.21.15.146:0/3638543457 learned_addr learned my addr 172.21.15.146:0/3638543457 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c6800640 1 --2- 172.21.15.146:0/3638543457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cc4a0 0x7f70c8165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c4a00640 1 -- 172.21.15.146:0/3638543457 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958688886 0 0) 0x7f70c815b1d0 con 0x7f70c80cb2f0 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c5e00640 1 -- 172.21.15.146:0/3638543457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cc4a0 msgr2=0x7f70c8165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c5e00640 1 --2- 172.21.15.146:0/3638543457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cc4a0 0x7f70c8165430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.818+0000 7f70c5e00640 1 -- 172.21.15.146:0/3638543457 >> v1:172.21.15.161:6789/0 conn(0x7f70c80cb2f0 legacy=0x7f70c815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.819+0000 7f70c5e00640 1 -- 172.21.15.146:0/3638543457 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c815b4b0 con 0x7f70c80cc0c0 2024-09-16T01:49:37.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.820+0000 7f70c5e00640 1 --2- 172.21.15.146:0/3638543457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc0c0 0x7f70c815aab0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f70b80097a0 tx=0x7f70b802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5481926cbe2ce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.820+0000 7f70c4a00640 1 -- 172.21.15.146:0/3638543457 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8037070 con 0x7f70c80cc0c0 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.820+0000 7f70c4a00640 1 -- 172.21.15.146:0/3638543457 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b8032440 con 0x7f70c80cc0c0 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.820+0000 7f70c4a00640 1 -- 172.21.15.146:0/3638543457 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8032810 con 0x7f70c80cc0c0 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.820+0000 7f70cca00640 1 -- 172.21.15.146:0/3638543457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc0c0 msgr2=0x7f70c815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.820+0000 7f70cca00640 1 --2- 172.21.15.146:0/3638543457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc0c0 0x7f70c815aab0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f70b80097a0 tx=0x7f70b802a180 comp rx=0 tx=0).stop 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.821+0000 7f70cca00640 1 -- 172.21.15.146:0/3638543457 shutdown_connections 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.821+0000 7f70cca00640 1 --2- 172.21.15.146:0/3638543457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cc4a0 0x7f70c8165430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.821+0000 7f70cca00640 1 --2- 172.21.15.146:0/3638543457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc0c0 0x7f70c815aab0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.821+0000 7f70cca00640 1 -- 172.21.15.146:0/3638543457 >> 172.21.15.146:0/3638543457 conn(0x7f70c806d560 msgr2=0x7f70c806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.821+0000 7f70cca00640 1 -- 172.21.15.146:0/3638543457 shutdown_connections 2024-09-16T01:49:37.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.821+0000 7f70cca00640 1 -- 172.21.15.146:0/3638543457 wait complete. 2024-09-16T01:49:37.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.822+0000 7f70cca00640 1 Processor -- start 2024-09-16T01:49:37.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.822+0000 7f70cca00640 1 -- start start 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.823+0000 7f70cca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cb2f0 0x7f70c814e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.823+0000 7f70cca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 0x7f70c81f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.823+0000 7f70cca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 0x7f70c80c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.823+0000 7f70cca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f70c815e1d0 con 0x7f70c80cc4a0 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.823+0000 7f70cca00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70c815e540 con 0x7f70c80cc0c0 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.823+0000 7f70cca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70c80cc8f0 con 0x7f70c80cb2f0 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 0x7f70c80c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 0x7f70c81f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 0x7f70c80c9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47214/0 (socket says 172.21.15.146:47214) 2024-09-16T01:49:37.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 0x7f70c81f41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47386/0 (socket says 172.21.15.146:47386) 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c6800640 1 -- 172.21.15.146:0/3281088800 learned_addr learned my addr 172.21.15.146:0/3281088800 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5e00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cb2f0 0x7f70c814e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 -- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cb2f0 msgr2=0x7f70c814e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cb2f0 0x7f70c814e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 -- 172.21.15.146:0/3281088800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 msgr2=0x7f70c80c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 0x7f70c80c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 -- 172.21.15.146:0/3281088800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c81f4c10 con 0x7f70c80cc0c0 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c6800640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 0x7f70c80c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.824+0000 7f70c5400640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 0x7f70c81f41a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c410 tx=0x7f70b400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.825+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40074e0 con 0x7f70c80cc0c0 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.825+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b40047e0 con 0x7f70c80cc0c0 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.825+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c814ca70 con 0x7f70c80cc0c0 2024-09-16T01:49:37.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.825+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4004bd0 con 0x7f70c80cc0c0 2024-09-16T01:49:37.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.825+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c814d030 con 0x7f70c80cc0c0 2024-09-16T01:49:37.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.826+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c80c5560 con 0x7f70c80cc0c0 2024-09-16T01:49:37.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.827+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f70b4011070 con 0x7f70c80cc0c0 2024-09-16T01:49:37.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.828+0000 7f70be000640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f70a806c2b0 0x7f70a806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:37.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.828+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f70b408aea0 con 0x7f70c80cc0c0 2024-09-16T01:49:37.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.828+0000 7f70c5e00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f70a806c2b0 0x7f70a806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:37.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.829+0000 7f70c5e00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f70a806c2b0 0x7f70a806e770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f70b8034000 tx=0x7f70b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:37.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:37.832+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f70b4059070 con 0x7f70c80cc0c0 2024-09-16T01:49:38.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.005+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f70c814d3a0 con 0x7f70a806c2b0 2024-09-16T01:49:38.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.012+0000 7f70be000640 1 -- 172.21.15.146:0/3281088800 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252003 (secure 0 0 0) 0x7f70c814d3a0 con 0x7f70a806c2b0 2024-09-16T01:49:38.017 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:38.018 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f70a806c2b0 msgr2=0x7f70a806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f70a806c2b0 0x7f70a806e770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f70b8034000 tx=0x7f70b802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 msgr2=0x7f70c81f41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 0x7f70c81f41a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c410 tx=0x7f70b400c8d0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 shutdown_connections 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f70a806c2b0 0x7f70a806e770 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70c80cc4a0 0x7f70c80c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70c80cc0c0 0x7f70c81f41a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 --2- 172.21.15.146:0/3281088800 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70c80cb2f0 0x7f70c814e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 >> 172.21.15.146:0/3281088800 conn(0x7f70c806d560 msgr2=0x7f70c80cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:38.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 shutdown_connections 2024-09-16T01:49:38.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.016+0000 7f70cca00640 1 -- 172.21.15.146:0/3281088800 wait complete. 2024-09-16T01:49:38.033 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-16T01:49:36.527089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":295,"ondisk_log_size":295,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218304,"kb_used_data":2112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756416,"statfs":{"total":773094113280,"available":772870569984,"internally_reserved":0,"allocated":2162688,"data_stored":1823012,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2028,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003190"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039277+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:49:01.039277+0000","last_peered":"2024-09-16T01:49:01.039277+0000","last_clean":"2024-09-16T01:49:01.039277+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:49:01.039277+0000","last_undegraded":"2024-09-16T01:49:01.039277+0000","last_fullsized":"2024-09-16T01:49:01.039277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039170+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:49:01.039170+0000","last_peered":"2024-09-16T01:49:01.039170+0000","last_clean":"2024-09-16T01:49:01.039170+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:49:01.039170+0000","last_undegraded":"2024-09-16T01:49:01.039170+0000","last_fullsized":"2024-09-16T01:49:01.039170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038749+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:49:01.038749+0000","last_peered":"2024-09-16T01:49:01.038749+0000","last_clean":"2024-09-16T01:49:01.038749+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:49:01.038749+0000","last_undegraded":"2024-09-16T01:49:01.038749+0000","last_fullsized":"2024-09-16T01:49:01.038749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038386+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:49:01.038386+0000","last_peered":"2024-09-16T01:49:01.038386+0000","last_clean":"2024-09-16T01:49:01.038386+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:49:01.038386+0000","last_undegraded":"2024-09-16T01:49:01.038386+0000","last_fullsized":"2024-09-16T01:49:01.038386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:09.260093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038850+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:49:01.038850+0000","last_peered":"2024-09-16T01:49:01.038850+0000","last_clean":"2024-09-16T01:49:01.038850+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:49:01.038850+0000","last_undegraded":"2024-09-16T01:49:01.038850+0000","last_fullsized":"2024-09-16T01:49:01.038850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824333+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:59.824333+0000","last_peered":"2024-09-16T01:48:59.824333+0000","last_clean":"2024-09-16T01:48:59.824333+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:59.824333+0000","last_undegraded":"2024-09-16T01:48:59.824333+0000","last_fullsized":"2024-09-16T01:48:59.824333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:54.394499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038902+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:49:01.038902+0000","last_peered":"2024-09-16T01:49:01.038902+0000","last_clean":"2024-09-16T01:49:01.038902+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:49:01.038902+0000","last_undegraded":"2024-09-16T01:49:01.038902+0000","last_fullsized":"2024-09-16T01:49:01.038902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039312+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:49:01.039312+0000","last_peered":"2024-09-16T01:49:01.039312+0000","last_clean":"2024-09-16T01:49:01.039312+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:49:01.039312+0000","last_undegraded":"2024-09-16T01:49:01.039312+0000","last_fullsized":"2024-09-16T01:49:01.039312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038909+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:49:01.038909+0000","last_peered":"2024-09-16T01:49:01.038909+0000","last_clean":"2024-09-16T01:49:01.038909+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:49:01.038909+0000","last_undegraded":"2024-09-16T01:49:01.038909+0000","last_fullsized":"2024-09-16T01:49:01.038909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038603+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:49:01.038603+0000","last_peered":"2024-09-16T01:49:01.038603+0000","last_clean":"2024-09-16T01:49:01.038603+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:49:01.038603+0000","last_undegraded":"2024-09-16T01:49:01.038603+0000","last_fullsized":"2024-09-16T01:49:01.038603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.863866+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:49:08.863866+0000","last_peered":"2024-09-16T01:49:08.863866+0000","last_clean":"2024-09-16T01:49:08.863866+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:49:08.863866+0000","last_undegraded":"2024-09-16T01:49:08.863866+0000","last_fullsized":"2024-09-16T01:49:08.863866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824444+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:59.824444+0000","last_peered":"2024-09-16T01:48:59.824444+0000","last_clean":"2024-09-16T01:48:59.824444+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:59.824444+0000","last_undegraded":"2024-09-16T01:48:59.824444+0000","last_fullsized":"2024-09-16T01:48:59.824444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:21.869634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917528+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:20.917528+0000","last_peered":"2024-09-16T01:49:20.917528+0000","last_clean":"2024-09-16T01:49:20.917528+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:20.917528+0000","last_undegraded":"2024-09-16T01:49:20.917528+0000","last_fullsized":"2024-09-16T01:49:20.917528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.901405+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:49:14.901405+0000","last_peered":"2024-09-16T01:49:14.901405+0000","last_clean":"2024-09-16T01:49:14.901405+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:49:14.901405+0000","last_undegraded":"2024-09-16T01:49:14.901405+0000","last_fullsized":"2024-09-16T01:49:14.901405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038968+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:49:01.038968+0000","last_peered":"2024-09-16T01:49:01.038968+0000","last_clean":"2024-09-16T01:49:01.038968+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:49:01.038968+0000","last_undegraded":"2024-09-16T01:49:01.038968+0000","last_fullsized":"2024-09-16T01:49:01.038968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918636+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:20.918636+0000","last_peered":"2024-09-16T01:49:20.918636+0000","last_clean":"2024-09-16T01:49:20.918636+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:20.918636+0000","last_undegraded":"2024-09-16T01:49:20.918636+0000","last_fullsized":"2024-09-16T01:49:20.918636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824266+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:59.824266+0000","last_peered":"2024-09-16T01:48:59.824266+0000","last_clean":"2024-09-16T01:48:59.824266+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:59.824266+0000","last_undegraded":"2024-09-16T01:48:59.824266+0000","last_fullsized":"2024-09-16T01:48:59.824266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:31.622539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.866956+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:08.866956+0000","last_peered":"2024-09-16T01:49:08.866956+0000","last_clean":"2024-09-16T01:49:08.866956+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:08.866956+0000","last_undegraded":"2024-09-16T01:49:08.866956+0000","last_fullsized":"2024-09-16T01:49:08.866956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.941828+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:25.941828+0000","last_peered":"2024-09-16T01:49:25.941828+0000","last_clean":"2024-09-16T01:49:25.941828+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:25.941828+0000","last_undegraded":"2024-09-16T01:49:25.941828+0000","last_fullsized":"2024-09-16T01:49:25.941828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039431+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:49:01.039431+0000","last_peered":"2024-09-16T01:49:01.039431+0000","last_clean":"2024-09-16T01:49:01.039431+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:49:01.039431+0000","last_undegraded":"2024-09-16T01:49:01.039431+0000","last_fullsized":"2024-09-16T01:49:01.039431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.863572+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:49:08.863572+0000","last_peered":"2024-09-16T01:49:08.863572+0000","last_clean":"2024-09-16T01:49:08.863572+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:49:08.863572+0000","last_undegraded":"2024-09-16T01:49:08.863572+0000","last_fullsized":"2024-09-16T01:49:08.863572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'10","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.940617+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:49:25.940617+0000","last_peered":"2024-09-16T01:49:25.940617+0000","last_clean":"2024-09-16T01:49:25.940617+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:49:25.940617+0000","last_undegraded":"2024-09-16T01:49:25.940617+0000","last_fullsized":"2024-09-16T01:49:25.940617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.940548+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:49:25.940548+0000","last_peered":"2024-09-16T01:49:25.940548+0000","last_clean":"2024-09-16T01:49:25.940548+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:49:25.940548+0000","last_undegraded":"2024-09-16T01:49:25.940548+0000","last_fullsized":"2024-09-16T01:49:25.940548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918151+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:20.918151+0000","last_peered":"2024-09-16T01:49:20.918151+0000","last_clean":"2024-09-16T01:49:20.918151+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:20.918151+0000","last_undegraded":"2024-09-16T01:49:20.918151+0000","last_fullsized":"2024-09-16T01:49:20.918151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918373+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:20.918373+0000","last_peered":"2024-09-16T01:49:20.918373+0000","last_clean":"2024-09-16T01:49:20.918373+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:20.918373+0000","last_undegraded":"2024-09-16T01:49:20.918373+0000","last_fullsized":"2024-09-16T01:49:20.918373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.916510+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:49:20.916510+0000","last_peered":"2024-09-16T01:49:20.916510+0000","last_clean":"2024-09-16T01:49:20.916510+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:49:20.916510+0000","last_undegraded":"2024-09-16T01:49:20.916510+0000","last_fullsized":"2024-09-16T01:49:20.916510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829332+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:03.829332+0000","last_peered":"2024-09-16T01:49:03.829332+0000","last_clean":"2024-09-16T01:49:03.829332+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:03.829332+0000","last_undegraded":"2024-09-16T01:49:03.829332+0000","last_fullsized":"2024-09-16T01:49:03.829332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.939777+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:49:25.939777+0000","last_peered":"2024-09-16T01:49:25.939777+0000","last_clean":"2024-09-16T01:49:25.939777+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:49:25.939777+0000","last_undegraded":"2024-09-16T01:49:25.939777+0000","last_fullsized":"2024-09-16T01:49:25.939777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917084+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:49:20.917084+0000","last_peered":"2024-09-16T01:49:20.917084+0000","last_clean":"2024-09-16T01:49:20.917084+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:49:20.917084+0000","last_undegraded":"2024-09-16T01:49:20.917084+0000","last_fullsized":"2024-09-16T01:49:20.917084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.916463+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:49:20.916463+0000","last_peered":"2024-09-16T01:49:20.916463+0000","last_clean":"2024-09-16T01:49:20.916463+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:49:20.916463+0000","last_undegraded":"2024-09-16T01:49:20.916463+0000","last_fullsized":"2024-09-16T01:49:20.916463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917307+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:20.917307+0000","last_peered":"2024-09-16T01:49:20.917307+0000","last_clean":"2024-09-16T01:49:20.917307+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:20.917307+0000","last_undegraded":"2024-09-16T01:49:20.917307+0000","last_fullsized":"2024-09-16T01:49:20.917307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824354+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:59.824354+0000","last_peered":"2024-09-16T01:48:59.824354+0000","last_clean":"2024-09-16T01:48:59.824354+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:59.824354+0000","last_undegraded":"2024-09-16T01:48:59.824354+0000","last_fullsized":"2024-09-16T01:48:59.824354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:03.840176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.903518+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:14.903518+0000","last_peered":"2024-09-16T01:49:14.903518+0000","last_clean":"2024-09-16T01:49:14.903518+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:14.903518+0000","last_undegraded":"2024-09-16T01:49:14.903518+0000","last_fullsized":"2024-09-16T01:49:14.903518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.903592+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:49:14.903592+0000","last_peered":"2024-09-16T01:49:14.903592+0000","last_clean":"2024-09-16T01:49:14.903592+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:49:14.903592+0000","last_undegraded":"2024-09-16T01:49:14.903592+0000","last_fullsized":"2024-09-16T01:49:14.903592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038525+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:49:01.038525+0000","last_peered":"2024-09-16T01:49:01.038525+0000","last_clean":"2024-09-16T01:49:01.038525+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:49:01.038525+0000","last_undegraded":"2024-09-16T01:49:01.038525+0000","last_fullsized":"2024-09-16T01:49:01.038525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:20.864782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.941526+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:25.941526+0000","last_peered":"2024-09-16T01:49:25.941526+0000","last_clean":"2024-09-16T01:49:25.941526+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:25.941526+0000","last_undegraded":"2024-09-16T01:49:25.941526+0000","last_fullsized":"2024-09-16T01:49:25.941526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039165+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:49:01.039165+0000","last_peered":"2024-09-16T01:49:01.039165+0000","last_clean":"2024-09-16T01:49:01.039165+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:49:01.039165+0000","last_undegraded":"2024-09-16T01:49:01.039165+0000","last_fullsized":"2024-09-16T01:49:01.039165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917502+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:20.917502+0000","last_peered":"2024-09-16T01:49:20.917502+0000","last_clean":"2024-09-16T01:49:20.917502+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:20.917502+0000","last_undegraded":"2024-09-16T01:49:20.917502+0000","last_fullsized":"2024-09-16T01:49:20.917502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.919024+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:49:20.919024+0000","last_peered":"2024-09-16T01:49:20.919024+0000","last_clean":"2024-09-16T01:49:20.919024+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:49:20.919024+0000","last_undegraded":"2024-09-16T01:49:20.919024+0000","last_fullsized":"2024-09-16T01:49:20.919024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824807+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:59.824807+0000","last_peered":"2024-09-16T01:48:59.824807+0000","last_clean":"2024-09-16T01:48:59.824807+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:59.824807+0000","last_undegraded":"2024-09-16T01:48:59.824807+0000","last_fullsized":"2024-09-16T01:48:59.824807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:08:09.528402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.945020+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:25.945020+0000","last_peered":"2024-09-16T01:49:25.945020+0000","last_clean":"2024-09-16T01:49:25.945020+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:25.945020+0000","last_undegraded":"2024-09-16T01:49:25.945020+0000","last_fullsized":"2024-09-16T01:49:25.945020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824748+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:59.824748+0000","last_peered":"2024-09-16T01:48:59.824748+0000","last_clean":"2024-09-16T01:48:59.824748+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:59.824748+0000","last_undegraded":"2024-09-16T01:48:59.824748+0000","last_fullsized":"2024-09-16T01:48:59.824748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:51:26.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.874853+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:08.874853+0000","last_peered":"2024-09-16T01:49:08.874853+0000","last_clean":"2024-09-16T01:49:08.874853+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:08.874853+0000","last_undegraded":"2024-09-16T01:49:08.874853+0000","last_fullsized":"2024-09-16T01:49:08.874853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.944989+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:49:25.944989+0000","last_peered":"2024-09-16T01:49:25.944989+0000","last_clean":"2024-09-16T01:49:25.944989+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:49:25.944989+0000","last_undegraded":"2024-09-16T01:49:25.944989+0000","last_fullsized":"2024-09-16T01:49:25.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824251+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:59.824251+0000","last_peered":"2024-09-16T01:48:59.824251+0000","last_clean":"2024-09-16T01:48:59.824251+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:59.824251+0000","last_undegraded":"2024-09-16T01:48:59.824251+0000","last_fullsized":"2024-09-16T01:48:59.824251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:26.577467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.914366+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:49:20.914366+0000","last_peered":"2024-09-16T01:49:20.914366+0000","last_clean":"2024-09-16T01:49:20.914366+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:49:20.914366+0000","last_undegraded":"2024-09-16T01:49:20.914366+0000","last_fullsized":"2024-09-16T01:49:20.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038826+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:49:01.038826+0000","last_peered":"2024-09-16T01:49:01.038826+0000","last_clean":"2024-09-16T01:49:01.038826+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:49:01.038826+0000","last_undegraded":"2024-09-16T01:49:01.038826+0000","last_fullsized":"2024-09-16T01:49:01.038826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.887158+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:08.887158+0000","last_peered":"2024-09-16T01:49:08.887158+0000","last_clean":"2024-09-16T01:49:08.887158+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:08.887158+0000","last_undegraded":"2024-09-16T01:49:08.887158+0000","last_fullsized":"2024-09-16T01:49:08.887158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.942546+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:25.942546+0000","last_peered":"2024-09-16T01:49:25.942546+0000","last_clean":"2024-09-16T01:49:25.942546+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:25.942546+0000","last_undegraded":"2024-09-16T01:49:25.942546+0000","last_fullsized":"2024-09-16T01:49:25.942546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'11","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.945415+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:25.945415+0000","last_peered":"2024-09-16T01:49:25.945415+0000","last_clean":"2024-09-16T01:49:25.945415+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:25.945415+0000","last_undegraded":"2024-09-16T01:49:25.945415+0000","last_fullsized":"2024-09-16T01:49:25.945415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039335+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:49:01.039335+0000","last_peered":"2024-09-16T01:49:01.039335+0000","last_clean":"2024-09-16T01:49:01.039335+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:49:01.039335+0000","last_undegraded":"2024-09-16T01:49:01.039335+0000","last_fullsized":"2024-09-16T01:49:01.039335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.828830+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:49:03.828830+0000","last_peered":"2024-09-16T01:49:03.828830+0000","last_clean":"2024-09-16T01:49:03.828830+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:49:03.828830+0000","last_undegraded":"2024-09-16T01:49:03.828830+0000","last_fullsized":"2024-09-16T01:49:03.828830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039274+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:49:01.039274+0000","last_peered":"2024-09-16T01:49:01.039274+0000","last_clean":"2024-09-16T01:49:01.039274+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:49:01.039274+0000","last_undegraded":"2024-09-16T01:49:01.039274+0000","last_fullsized":"2024-09-16T01:49:01.039274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.942146+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:49:25.942146+0000","last_peered":"2024-09-16T01:49:25.942146+0000","last_clean":"2024-09-16T01:49:25.942146+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:49:25.942146+0000","last_undegraded":"2024-09-16T01:49:25.942146+0000","last_fullsized":"2024-09-16T01:49:25.942146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":80998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":79956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509452,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":670058,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":80818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":79330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":80656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":669470,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":81726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:38.037 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:49:38.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T01:49:38.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.224+0000 7f93df800640 1 Processor -- start 2024-09-16T01:49:38.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.224+0000 7f93df800640 1 -- start start 2024-09-16T01:49:38.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93df800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93d80c6410 0x7f93d8158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93df800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c5280 0x7f93d80c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93df800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d8151e70 con 0x7f93d8151a90 2024-09-16T01:49:38.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93df800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f93d80c8130 con 0x7f93d80c6410 2024-09-16T01:49:38.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93df800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f93d80c8410 con 0x7f93d80c5280 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93d80c6410 0x7f93d8158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93dd800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c5280 0x7f93d80c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93d80c6410 0x7f93d8158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47394/0 (socket says 172.21.15.146:47394) 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 -- 172.21.15.146:0/3236658473 learned_addr learned my addr 172.21.15.146:0/3236658473 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93dce00640 1 --1- 172.21.15.146:0/3236658473 >> v1:172.21.15.161:6789/0 conn(0x7f93d8151a90 0x7f93d80c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:45284/0 (socket says 172.21.15.146:45284) 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 -- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c5280 msgr2=0x7f93d80c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 --2- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c5280 0x7f93d80c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 -- 172.21.15.146:0/3236658473 >> v1:172.21.15.161:6789/0 conn(0x7f93d8151a90 legacy=0x7f93d80c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93cfe00640 1 -- 172.21.15.146:0/3236658473 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d80c86f0 con 0x7f93d80c6410 2024-09-16T01:49:38.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.226+0000 7f93dd800640 1 --2- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c5280 0x7f93d80c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.227+0000 7f93cfe00640 1 --2- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c6410 0x7f93d8158040 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009870 tx=0x7f93c002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db747e648c5d6825 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93d7e00640 1 -- 172.21.15.146:0/3236658473 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0037070 con 0x7f93d80c6410 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93d7e00640 1 -- 172.21.15.146:0/3236658473 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c0032440 con 0x7f93d80c6410 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93d7e00640 1 -- 172.21.15.146:0/3236658473 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0032810 con 0x7f93d80c6410 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 -- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c6410 msgr2=0x7f93d8158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 --2- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c6410 0x7f93d8158040 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009870 tx=0x7f93c002a250 comp rx=0 tx=0).stop 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 -- 172.21.15.146:0/3236658473 shutdown_connections 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 --2- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c5280 0x7f93d80c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 --2- 172.21.15.146:0/3236658473 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c6410 0x7f93d8158040 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 -- 172.21.15.146:0/3236658473 >> 172.21.15.146:0/3236658473 conn(0x7f93d806f3c0 msgr2=0x7f93d806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:38.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 -- 172.21.15.146:0/3236658473 shutdown_connections 2024-09-16T01:49:38.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.228+0000 7f93df800640 1 -- 172.21.15.146:0/3236658473 wait complete. 2024-09-16T01:49:38.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.230+0000 7f93df800640 1 Processor -- start 2024-09-16T01:49:38.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.230+0000 7f93df800640 1 -- start start 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93df800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 0x7f93d81ecc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93df800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c6410 0x7f93d81e2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93df800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d8151a90 0x7f93d8147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93df800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f93d8157b20 con 0x7f93d8151a90 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93df800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f93d8157840 con 0x7f93d80c5280 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93df800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f93d8158580 con 0x7f93d80c6410 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 0x7f93d81ecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93cfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c6410 0x7f93d81e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 0x7f93d81ecc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47402/0 (socket says 172.21.15.146:47402) 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 -- 172.21.15.146:0/529269876 learned_addr learned my addr 172.21.15.146:0/529269876 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dd800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d8151a90 0x7f93d8147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 -- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c6410 msgr2=0x7f93d81e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c6410 0x7f93d81e2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 -- 172.21.15.146:0/529269876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d8151a90 msgr2=0x7f93d8147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.231+0000 7f93dce00640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d8151a90 0x7f93d8147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93dce00640 1 -- 172.21.15.146:0/529269876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d8150520 con 0x7f93d80c5280 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93cfe00640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c6410 0x7f93d81e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93dce00640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 0x7f93d81ecc90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c3c0 tx=0x7f93c800c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80074e0 con 0x7f93d80c5280 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c8007680 con 0x7f93d80c5280 2024-09-16T01:49:38.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d80ccf90 con 0x7f93d80c5280 2024-09-16T01:49:38.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8004a00 con 0x7f93d80c5280 2024-09-16T01:49:38.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.232+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d81507e0 con 0x7f93d80c5280 2024-09-16T01:49:38.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.233+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d8072940 con 0x7f93d80c5280 2024-09-16T01:49:38.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.234+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f93c801a020 con 0x7f93d80c5280 2024-09-16T01:49:38.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.235+0000 7f93d5600640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f93b406c2b0 0x7f93b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.235+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f93c808ab00 con 0x7f93d80c5280 2024-09-16T01:49:38.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.235+0000 7f93cfe00640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f93b406c2b0 0x7f93b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.236+0000 7f93cfe00640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f93b406c2b0 0x7f93b406e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93c0034000 tx=0x7f93c002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.242+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f93d8072940 con 0x7f93d80c5280 2024-09-16T01:49:38.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.414+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f93d8155710 con 0x7f93d80c5280 2024-09-16T01:49:38.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.416+0000 7f93d5600640 1 -- 172.21.15.146:0/529269876 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v22) ==== 56+0+3251 (secure 0 0 0) 0x7f93c808ae20 con 0x7f93d80c5280 2024-09-16T01:49:38.417 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 22 2024-09-16T01:49:38.417 INFO:teuthology.orchestra.run.smithi146.stdout:fsid 4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d 2024-09-16T01:49:38.417 INFO:teuthology.orchestra.run.smithi146.stdout:created 2024-09-16T01:48:34.399638+0000 2024-09-16T01:49:38.417 INFO:teuthology.orchestra.run.smithi146.stdout:modified 2024-09-16T01:48:59.816171+0000 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:crush_version 6 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:full_ratio 0.95 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:backfillfull_ratio 0.9 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:nearfull_ratio 0.85 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:require_min_compat_client luminous 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:min_compat_client jewel 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:require_osd_release squid 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:stretch_mode_enabled false 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-09-16T01:49:38.418 INFO:teuthology.orchestra.run.smithi146.stdout:pool 3 'cephfs_data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:max_osd 8 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] [v2:172.21.15.146:6826/61440064,v1:172.21.15.146:6827/61440064] exists,up e7758ddf-75ff-43c7-abae-3647b170f7c0 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] [v2:172.21.15.146:6810/2375201615,v1:172.21.15.146:6811/2375201615] exists,up a065e77e-26f0-4245-abe8-e36c2fb4436b 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] [v2:172.21.15.146:6802/1818492127,v1:172.21.15.146:6803/1818492127] exists,up fefb6d8f-6799-4344-a977-85ab983ec0cb 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] [v2:172.21.15.146:6818/1633804394,v1:172.21.15.146:6819/1633804394] exists,up 9d1a2f96-efcb-4a73-b5db-ec5e2a63122a 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4 up in weight 1 up_from 13 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] [v2:172.21.15.161:6802/154390265,v1:172.21.15.161:6803/154390265] exists,up b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5 up in weight 1 up_from 14 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] [v2:172.21.15.161:6826/356357279,v1:172.21.15.161:6827/356357279] exists,up 11562dd5-d537-4d88-a282-e8687caebad6 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6 up in weight 1 up_from 14 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] [v2:172.21.15.161:6810/3661001615,v1:172.21.15.161:6811/3661001615] exists,up b31adf01-6768-49cf-a232-127b9d0d2dc8 2024-09-16T01:49:38.419 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7 up in weight 1 up_from 14 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] [v2:172.21.15.161:6818/1279166573,v1:172.21.15.161:6819/1279166573] exists,up 67c95c28-b600-430c-9474-a902cf46a73b 2024-09-16T01:49:38.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f93b406c2b0 msgr2=0x7f93b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f93b406c2b0 0x7f93b406e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93c0034000 tx=0x7f93c002e000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 msgr2=0x7f93d81ecc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 0x7f93d81ecc90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c3c0 tx=0x7f93c800c7c0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 shutdown_connections 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f93b406c2b0 0x7f93b406e770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93d8151a90 0x7f93d8147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93d80c6410 0x7f93d81e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 --2- 172.21.15.146:0/529269876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93d80c5280 0x7f93d81ecc90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 >> 172.21.15.146:0/529269876 conn(0x7f93d806f3c0 msgr2=0x7f93d8156d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 shutdown_connections 2024-09-16T01:49:38.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.417+0000 7f93df800640 1 -- 172.21.15.146:0/529269876 wait complete. 2024-09-16T01:49:38.435 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] [v2:172.21.15.146:6826/61440064,v1:172.21.15.146:6827/61440064] exists,up e7758ddf-75ff-43c7-abae-3647b170f7c0', 'osd.1 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] [v2:172.21.15.146:6810/2375201615,v1:172.21.15.146:6811/2375201615] exists,up a065e77e-26f0-4245-abe8-e36c2fb4436b', 'osd.2 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] [v2:172.21.15.146:6802/1818492127,v1:172.21.15.146:6803/1818492127] exists,up fefb6d8f-6799-4344-a977-85ab983ec0cb', 'osd.3 up in weight 1 up_from 15 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] [v2:172.21.15.146:6818/1633804394,v1:172.21.15.146:6819/1633804394] exists,up 9d1a2f96-efcb-4a73-b5db-ec5e2a63122a', 'osd.4 up in weight 1 up_from 13 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] [v2:172.21.15.161:6802/154390265,v1:172.21.15.161:6803/154390265] exists,up b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c', 'osd.5 up in weight 1 up_from 14 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] [v2:172.21.15.161:6826/356357279,v1:172.21.15.161:6827/356357279] exists,up 11562dd5-d537-4d88-a282-e8687caebad6', 'osd.6 up in weight 1 up_from 14 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] [v2:172.21.15.161:6810/3661001615,v1:172.21.15.161:6811/3661001615] exists,up b31adf01-6768-49cf-a232-127b9d0d2dc8', 'osd.7 up in weight 1 up_from 14 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] [v2:172.21.15.161:6818/1279166573,v1:172.21.15.161:6819/1279166573] exists,up 67c95c28-b600-430c-9474-a902cf46a73b'] 2024-09-16T01:49:38.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T01:49:38.588 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mon.a", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster": "ceph", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "admin_socket_mode": "", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_client_required": "cephx, none", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_cluster_required": "cephx", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_debug": "true", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T01:49:38.589 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_service_required": "cephx", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "auth_supported": "cephx", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_aio": "true", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_aio_reap_max": "16", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_async_discard": "false", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_block_size": "4096", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_debug_aio": "false", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T01:49:38.590 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_enable_discard": "false", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_flock_retry": "3", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_inject_crash": "0", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_ioring": "false", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_ioring_hipri": "false", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T01:49:38.591 INFO:teuthology.orchestra.run.smithi146.stdout: "bdev_type": "", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_allocator": "hybrid", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_buffered_io": "true", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T01:49:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_replay_recovery": "false", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_sync_write": "false", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_allocator": "hybrid", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T01:49:38.593 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_create": "true", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_db_create": "false", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_db_path": "", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_db_size": "0", 2024-09-16T01:49:38.594 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_path": "", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_size": "107374182400", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_wal_create": "false", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_wal_path": "", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bluefs": "true", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T01:49:38.595 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_autotune": "true", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_size": "0", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_type": "2q", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T01:49:38.596 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_clone_cow": "true", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_mode": "none", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_freelist": "false", 2024-09-16T01:49:38.597 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T01:49:38.598 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fail_eio": "false", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T01:49:38.599 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T01:49:38.600 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_max_blob_size": "0", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T01:49:38.601 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T01:49:38.602 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_spdk_mem": "512", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T01:49:38.603 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_tracing": "false", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T01:49:38.604 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T01:49:38.605 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_cluster_require_version": "2", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_require_signatures": "false", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_require_version": "2", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_service_require_signatures": "false", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_service_require_version": "2", 2024-09-16T01:49:38.606 INFO:teuthology.orchestra.run.smithi146.stdout: "cephx_sign_messages": "true", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "chdir": "", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_acl_type": "", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_asio_thread_count": "2", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_cache_mid": "0.750000", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_cache_size": "16384", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_caps_release_delay": "5", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_check_pool_perm": "true", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_debug_force_sync_read": "false", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_debug_getattr_caps": "false", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_debug_inject_features": "", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T01:49:38.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_die_on_failed_remount": "false", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_dirsize_rbytes": "true", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_force_lazyio": "false", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_fs": "", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_inject_release_failure": "false", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_max_inline_size": "4096", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_mds_namespace": "", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_metadata": "", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_mount_gid": "-1", 2024-09-16T01:49:38.608 INFO:teuthology.orchestra.run.smithi146.stdout: "client_mount_timeout": "300", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_mount_uid": "-1", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_mountpoint": "/", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_notify_timeout": "10", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_oc": "true", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_oc_max_objects": "1000", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_oc_size": "209715200", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_permissions": "true", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_quota": "true", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_quota_df": "true", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_readahead_max_bytes": "0", 2024-09-16T01:49:38.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_readahead_max_periods": "4", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_readahead_min": "131072", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_reconnect_stale": "false", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_shutdown_timeout": "30", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_snapdir": ".snap", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_tick_interval": "1", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_trace": "", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_use_faked_inos": "false", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "client_use_random_mds": "false", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_graylog": "false", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_graylog_port": "12201", 2024-09-16T01:49:38.610 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_monitors": "default=true", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_syslog": "false", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "clog_to_syslog_level": "info", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "-", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_network": "", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_network_interface": "", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "compressor_zlib_isal": "false", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "compressor_zlib_level": "5", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "compressor_zstd_level": "1", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T01:49:38.611 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_location": "", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_location_hook": "", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_location_hook_timeout": "10", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "daemonize": "false", 2024-09-16T01:49:38.612 INFO:teuthology.orchestra.run.smithi146.stdout: "daos_pool": "tank", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_alienstore": "0/5", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_asok": "1/5", 2024-09-16T01:49:38.613 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_asok_assert_abort": "false", 2024-09-16T01:49:38.614 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T01:49:38.614 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_auth": "20/20", 2024-09-16T01:49:38.614 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_bdev": "1/3", 2024-09-16T01:49:38.614 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_bluefs": "1/5", 2024-09-16T01:49:38.614 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_bluestore": "1/5", 2024-09-16T01:49:38.614 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_buffer": "0/1", 2024-09-16T01:49:38.615 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T01:49:38.615 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T01:49:38.615 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_cephsqlite": "0/5", 2024-09-16T01:49:38.615 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_client": "0/5", 2024-09-16T01:49:38.615 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_compressor": "1/5", 2024-09-16T01:49:38.615 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_context": "0/1", 2024-09-16T01:49:38.616 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T01:49:38.616 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_crush": "1/1", 2024-09-16T01:49:38.616 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_crypto": "1/5", 2024-09-16T01:49:38.616 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_cyanstore": "0/5", 2024-09-16T01:49:38.616 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T01:49:38.616 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T01:49:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_dpdk": "1/5", 2024-09-16T01:49:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_eventtrace": "1/5", 2024-09-16T01:49:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_filer": "0/1", 2024-09-16T01:49:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_filestore": "1/3", 2024-09-16T01:49:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_finisher": "1/1", 2024-09-16T01:49:38.617 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_fuse": "1/5", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_javaclient": "1/5", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_journal": "1/3", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_journaler": "0/5", 2024-09-16T01:49:38.618 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_kstore": "1/5", 2024-09-16T01:49:38.619 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_lockdep": "0/1", 2024-09-16T01:49:38.619 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mclock": "1/5", 2024-09-16T01:49:38.619 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds": "1/5", 2024-09-16T01:49:38.619 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds_balancer": "1/5", 2024-09-16T01:49:38.619 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds_locker": "1/5", 2024-09-16T01:49:38.619 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds_log": "1/5", 2024-09-16T01:49:38.620 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T01:49:38.620 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds_migrator": "1/5", 2024-09-16T01:49:38.620 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T01:49:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_memstore": "1/5", 2024-09-16T01:49:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mgr": "2/5", 2024-09-16T01:49:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mgrc": "1/5", 2024-09-16T01:49:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_mon": "20/20", 2024-09-16T01:49:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_monc": "0/10", 2024-09-16T01:49:38.621 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_ms": "1/1", 2024-09-16T01:49:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_none": "0/5", 2024-09-16T01:49:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_objclass": "0/5", 2024-09-16T01:49:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_objectcacher": "0/5", 2024-09-16T01:49:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_objecter": "0/1", 2024-09-16T01:49:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_optracker": "0/5", 2024-09-16T01:49:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_osd": "1/5", 2024-09-16T01:49:38.623 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_paxos": "20/20", 2024-09-16T01:49:38.623 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_perfcounter": "1/5", 2024-09-16T01:49:38.623 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_prioritycache": "1/5", 2024-09-16T01:49:38.623 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rados": "0/5", 2024-09-16T01:49:38.623 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rbd": "0/5", 2024-09-16T01:49:38.623 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rbd_replay": "0/5", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_refs": "0/0", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_reserver": "1/1", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw": "1/5", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw_access": "1/5", 2024-09-16T01:49:38.624 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T01:49:38.625 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T01:49:38.625 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw_flight": "1/5", 2024-09-16T01:49:38.625 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T01:49:38.625 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rgw_sync": "1/5", 2024-09-16T01:49:38.625 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_rocksdb": "4/5", 2024-09-16T01:49:38.625 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore": "0/5", 2024-09-16T01:49:38.626 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_backref": "0/5", 2024-09-16T01:49:38.626 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_cache": "0/5", 2024-09-16T01:49:38.626 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T01:49:38.626 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_device": "0/5", 2024-09-16T01:49:38.626 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_epm": "0/5", 2024-09-16T01:49:38.626 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T01:49:38.627 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_journal": "0/5", 2024-09-16T01:49:38.627 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_lba": "0/5", 2024-09-16T01:49:38.627 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_odata": "0/5", 2024-09-16T01:49:38.627 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_omap": "0/5", 2024-09-16T01:49:38.627 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_onode": "0/5", 2024-09-16T01:49:38.627 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_t": "0/5", 2024-09-16T01:49:38.628 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_seastore_tm": "0/5", 2024-09-16T01:49:38.628 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_striper": "0/1", 2024-09-16T01:49:38.628 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_test": "0/5", 2024-09-16T01:49:38.628 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_throttle": "1/1", 2024-09-16T01:49:38.628 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_timer": "0/1", 2024-09-16T01:49:38.628 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_tp": "0/5", 2024-09-16T01:49:38.629 INFO:teuthology.orchestra.run.smithi146.stdout: "debug_trace": "1/5", 2024-09-16T01:49:38.629 INFO:teuthology.orchestra.run.smithi146.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T01:49:38.629 INFO:teuthology.orchestra.run.smithi146.stdout: "device_failure_prediction_mode": "none", 2024-09-16T01:49:38.629 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T01:49:38.629 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T01:49:38.629 INFO:teuthology.orchestra.run.smithi146.stdout: "err_to_graylog": "false", 2024-09-16T01:49:38.630 INFO:teuthology.orchestra.run.smithi146.stdout: "err_to_journald": "false", 2024-09-16T01:49:38.630 INFO:teuthology.orchestra.run.smithi146.stdout: "err_to_stderr": "true", 2024-09-16T01:49:38.630 INFO:teuthology.orchestra.run.smithi146.stdout: "err_to_syslog": "false", 2024-09-16T01:49:38.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event_tracing": "false", 2024-09-16T01:49:38.630 INFO:teuthology.orchestra.run.smithi146.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T01:49:38.631 INFO:teuthology.orchestra.run.smithi146.stdout: "exporter_http_port": "9926", 2024-09-16T01:49:38.631 INFO:teuthology.orchestra.run.smithi146.stdout: "exporter_prio_limit": "5", 2024-09-16T01:49:38.631 INFO:teuthology.orchestra.run.smithi146.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T01:49:38.631 INFO:teuthology.orchestra.run.smithi146.stdout: "exporter_sort_metrics": "true", 2024-09-16T01:49:38.631 INFO:teuthology.orchestra.run.smithi146.stdout: "exporter_stats_period": "5", 2024-09-16T01:49:38.631 INFO:teuthology.orchestra.run.smithi146.stdout: "fake_statfs_for_testing": "0", 2024-09-16T01:49:38.632 INFO:teuthology.orchestra.run.smithi146.stdout: "fatal_signal_handlers": "true", 2024-09-16T01:49:38.632 INFO:teuthology.orchestra.run.smithi146.stdout: "filer_max_purge_ops": "10", 2024-09-16T01:49:38.632 INFO:teuthology.orchestra.run.smithi146.stdout: "filer_max_truncate_ops": "128", 2024-09-16T01:49:38.632 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T01:49:38.632 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_blackhole": "false", 2024-09-16T01:49:38.633 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T01:49:38.633 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_btrfs_snap": "true", 2024-09-16T01:49:38.633 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_caller_concurrency": "10", 2024-09-16T01:49:38.633 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T01:49:38.633 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T01:49:38.633 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T01:49:38.634 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_debug_omap_check": "false", 2024-09-16T01:49:38.634 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T01:49:38.634 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_debug_verify_split": "false", 2024-09-16T01:49:38.634 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_dump_file": "", 2024-09-16T01:49:38.634 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T01:49:38.634 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T01:49:38.635 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fadvise": "true", 2024-09-16T01:49:38.635 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fail_eio": "true", 2024-09-16T01:49:38.635 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T01:49:38.635 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fd_cache_size": "128", 2024-09-16T01:49:38.635 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fiemap": "false", 2024-09-16T01:49:38.635 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T01:49:38.636 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T01:49:38.636 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T01:49:38.636 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_inject_stall": "0", 2024-09-16T01:49:38.636 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_journal_parallel": "false", 2024-09-16T01:49:38.636 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_journal_trailing": "false", 2024-09-16T01:49:38.636 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_journal_writeahead": "false", 2024-09-16T01:49:38.637 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_kill_at": "0", 2024-09-16T01:49:38.637 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T01:49:38.637 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T01:49:38.637 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T01:49:38.637 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T01:49:38.638 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_merge_threshold": "-10", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_odsync_write": "false", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_omap_backend_path": "", 2024-09-16T01:49:38.639 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_op_threads": "2", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_punch_hole": "false", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T01:49:38.640 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T01:49:38.641 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_queue_max_ops": "50", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_seek_data_hole": "false", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_sloppy_crc": "false", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_splice": "false", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_split_multiple": "2", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_split_rand_factor": "20", 2024-09-16T01:49:38.642 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_update_to": "1000", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T01:49:38.643 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_xfs_extsize": "false", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "filestore_zfs_snap": "false", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fio_dir": "/tmp/fio", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_allow_other": "true", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_big_writes": "true", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_debug": "false", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_default_permissions": "false", 2024-09-16T01:49:38.644 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_disable_pagecache": "false", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_max_write": "0", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_multithreaded": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_require_active_mds": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_set_user_groups": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_splice_move": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_splice_read": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_splice_write": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "gss_target_name": "ceph", 2024-09-16T01:49:38.645 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_file": "", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_inject_failure": "0", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_interval": "5", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "host": "smithi146", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T01:49:38.646 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "inject_early_sigterm": "false", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "jaeger_agent_port": "6799", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "jaeger_tracing_enable": "false", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_aio": "true", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_align_min_size": "65536", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_block_align": "true", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_block_size": "4096", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_dio": "true", 2024-09-16T01:49:38.647 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_discard": "false", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_force_aio": "false", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_ignore_corruption": "false", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_max_write_entries": "100", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_replay_from": "0", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_write_header_frequency": "0", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journal_zero_on_create": "false", 2024-09-16T01:49:38.648 INFO:teuthology.orchestra.run.smithi146.stdout: "journaler_prefetch_periods": "10", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "journaler_prezero_periods": "5", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "journaler_write_head_interval": "15", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "key": "", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "keyfile": "", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_backend": "rocksdb", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_max_bytes": "67108864", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_max_ops": "512", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_onode_map_size": "1024", 2024-09-16T01:49:38.649 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "kstore_sync_transaction": "false", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "librados_thread_count": "2", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "lockdep": "false", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "lockdep_force_backtrace": "false", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "log_coarse_timestamps": "true", 2024-09-16T01:49:38.650 INFO:teuthology.orchestra.run.smithi146.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_flush_on_exit": "false", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_graylog_port": "12201", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_max_new": "1000", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_max_recent": "10000", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_stderr_prefix": "", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": "true", 2024-09-16T01:49:38.651 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_graylog": "false", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_journald": "false", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_stderr": "false", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_syslog": "false", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_action_on_write_error": "1", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_alternate_name_max": "8192", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_asio_thread_count": "2", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_export_pin": "true", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T01:49:38.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_interval": "10", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_max": "-1", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_max_until": "-1", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_merge_size": "50", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_mode": "0", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T01:49:38.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_split_bits": "3", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_split_size": "10000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T01:49:38.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_mid": "0.700000", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T01:49:38.655 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_trim_interval": "1", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_debug_auth_pins": "false", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_debug_frag": "false", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_debug_scatterstat": "false", 2024-09-16T01:49:38.656 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_debug_subtrees": "false", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_default_dir_hash": "2", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_defer_session_stale": "true", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dir_max_entries": "0", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dir_prefetch": "true", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T01:49:38.657 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_early_reply": "true", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_enable_op_tracker": "true", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_enforce_unique_name": "true", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T01:49:38.658 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_inject_health_dummy": "false", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_join_fs": "", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_journal_format": "1", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T01:49:38.659 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_create_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_export_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_import_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_journal_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_link_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_openc_at": "0", 2024-09-16T01:49:38.660 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_rename_at": "0", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_max_events": "-1", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_max_segments": "128", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_pause": "false", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_segment_size": "0", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T01:49:38.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_trim_threshold": "128", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_completed_requests": "100000", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_export_size": "20971520", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_file_recover": "32", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_purge_files": "64", 2024-09-16T01:49:38.662 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_purge_ops": "8192", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_metrics_update_interval": "2", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_min_caps_per_client": "100", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_numa_node": "-1", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_op_history_duration": "600", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_op_history_size": "20", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T01:49:38.663 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_op_log_threshold": "5", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_ping_grace": "15", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_ping_interval": "5", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_recall_max_caps": "30000", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T01:49:38.664 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_replay_interval": "1.000000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_root_ino_gid": "0", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_root_ino_uid": "0", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T01:49:38.665 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_shutdown_check": "0", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_skip_ino": "0", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_snap_min_uid": "0", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_snap_rstat": "false", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_symlink_recovery": "true", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_thrash_exports": "0", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_thrash_fragments": "0", 2024-09-16T01:49:38.666 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_tick_interval": "5.000000", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_valgrind_exit": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_verify_backtrace": "1", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_verify_scatter": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_wipe_sessions": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mempool_debug": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "memstore_page_set": "false", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "memstore_page_size": "65536", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_client_bytes": "134217728", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_client_messages": "512", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T01:49:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_disabled_modules": "", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_mds_messages": "128", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T01:49:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_mon_messages": "128", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_op_history_duration": "600", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_op_history_size": "20", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T01:49:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_op_log_threshold": "5", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_osd_messages": "8192", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_pool": "true", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_standby_modules": "true", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_stats_period": "5", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_stats_threshold": "5", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_tick_period": "2", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_allow_pool_delete": "true", 2024-09-16T01:49:38.670 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_bytes": "104857600", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_directed_command_retry": "2", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T01:49:38.671 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_client_target_rank": "-1", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T01:49:38.672 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_file_level": "debug", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T01:49:38.673 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_compact_on_start": "false", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_compact_on_trim": "true", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_cpu_threads": "4", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_data_avail_crit": "5", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_data_avail_warn": "5", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T01:49:38.674 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_dump_json": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_extra_checks": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_down_mkfs_grace": "300", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_down_uptime_grace": "60", 2024-09-16T01:49:38.675 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_election_default_strategy": "1", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_election_timeout": "5.000000", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_enable_op_tracker": "true", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_fake_pool_delete": "false", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_force_quorum_join": "false", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T01:49:38.676 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_health_log_update_period": "5", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_health_max_detail": "50", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_health_to_clog": "true", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_host": "172.21.15.146,172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790]", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_host_override": "", 2024-09-16T01:49:38.677 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_initial_members": "", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_lease": "5.000000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_log_full_interval": "50", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_log_max": "10000", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_log_max_summary": "50", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_log_epochs": "500", 2024-09-16T01:49:38.678 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_osd": "10000", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_memory_autotune": "true", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_memory_target": "2147483648", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T01:49:38.679 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mgr_digest_period": "5", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_op_complaint_time": "120", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_op_history_duration": "600", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_op_history_size": "20", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_op_log_threshold": "5", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T01:49:38.680 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_cache_size": "500", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T01:49:38.681 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T01:49:38.682 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_report_timeout": "900", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T01:49:38.683 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T01:49:38.684 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_reweight_max_osds": "4", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_scrub_interval": "86400", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_scrub_max_keys": "100", 2024-09-16T01:49:38.685 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_scrub_timeout": "300", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_session_timeout": "300", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_smart_report_timeout": "5", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_stretch_pool_size": "4", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_sync_debug": "false", 2024-09-16T01:49:38.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_tick_interval": "5", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T01:49:38.687 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_older_version": "true", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T01:49:38.688 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "monmap": "", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "motr_tracing_enabled": "false", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_op_threads": "3", 2024-09-16T01:49:38.689 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_cm": "false", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_device_name": "", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T01:49:38.690 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_sl": "3", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_rdma_type": "ib", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_async_reap_threshold": "5", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_before_connect": "false", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_ipv4": "true", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_ipv6": "false", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_msgr1": "true", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_msgr2": "true", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_port_max": "7568", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_port_min": "6800", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_retry_count": "3", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_bind_retry_delay": "5", 2024-09-16T01:49:38.691 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_blackhole_client": "false", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_blackhole_mds": "false", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_blackhole_mgr": "false", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_blackhole_mon": "false", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_blackhole_osd": "false", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_client_mode": "crc secure", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T01:49:38.692 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_cluster_type": "", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_compress_secure": "false", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_crc_data": "true", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_crc_header": "true", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_die_on_bug": "true", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_die_on_old_message": "true", 2024-09-16T01:49:38.693 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_hugepages": "", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_lro": "true", 2024-09-16T01:49:38.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_pmd": "", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_port_id": "0", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_dump_on_send": "false", 2024-09-16T01:49:38.695 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_inject_delay_type": "", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_inject_network_congestion": "0", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_inject_socket_failures": "0", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_max_accept_failures": "4", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_max_backoff": "15.000000", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T01:49:38.696 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_osd_compress_mode": "none", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_pq_min_cost": "65536", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_public_type": "", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_service_mode": "crc secure", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_tcp_nodelay": "true", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "ms_type": "async+posix", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "no_config_file": "false", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T01:49:38.697 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_inflight_ops": "1024", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter_timeout": "10.000000", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objectstore_blackhole": "false", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "openssl_engine_opts": "", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T01:49:38.698 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_max_ops": "4", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_agent_slop": "0.020000", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_asio_thread_count": "2", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_backfill_scan_max": "512", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_backfill_scan_min": "64", 2024-09-16T01:49:38.699 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_backoff_on_peering": "false", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_beacon_report_interval": "300", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bench_duration": "30", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_blkin_trace_all": "false", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T01:49:38.700 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_class_update_on_start": "true", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_client_message_cap": "256", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_client_op_priority": "63", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_client_watch_timeout": "30", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_command_max_records": "256", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_compact_on_start": "false", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T01:49:38.701 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_crush_update_on_start": "true", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T01:49:38.702 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T01:49:38.703 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T01:49:38.703 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T01:49:38.703 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T01:49:38.703 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T01:49:38.703 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_op_order": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_shutdown": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_trim_objects": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T01:49:38.704 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_default_notify_timeout": "30", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T01:49:38.705 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_enable_op_tracker": "true", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_fast_info": "true", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_fast_shutdown": "true", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T01:49:38.706 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_function_tracing": "false", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_grace": "20", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_interval": "6", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_stale": "600", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T01:49:38.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_journal_size": "5120", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_kill_backfill_at": "0", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_map_cache_size": "50", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_map_dedup": "true", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_map_message_max": "40", 2024-09-16T01:49:38.708 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_attr_name_len": "100", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_attr_size": "0", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_backfills": "1", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_markdown_count": "5", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_markdown_period": "600", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_object_name_len": "2048", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_object_size": "134217728", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T01:49:38.709 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_pgls": "1024", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_push_cost": "8388608", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_push_objects": "10", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_scrubs": "3", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_max_write_size": "90", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T01:49:38.710 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_profile": "balanced", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T01:49:38.711 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_base": "805306368", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_target": "4294967296", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_target_autotune": "false", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_min_recovery_priority": "0", 2024-09-16T01:49:38.712 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mon_report_interval": "5", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_num_cache_shards": "32", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_numa_node": "-1", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T01:49:38.713 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_objecter_finishers": "1", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_objectstore": "bluestore", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_objectstore_fuse": "false", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_objectstore_tracing": "false", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_history_duration": "600", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_history_size": "20", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T01:49:38.714 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_log_threshold": "5", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_num_shards": "0", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_op_thread_timeout": "15", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_open_classes_on_start": "true", 2024-09-16T01:49:38.715 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_os_flags": "0", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_peering_op_priority": "255", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_delete_priority": "5", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T01:49:38.716 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_crimson": "false", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T01:49:38.717 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T01:49:38.720 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T01:49:38.720 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T01:49:38.720 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T01:49:38.720 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T01:49:38.720 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_flags": "0", 2024-09-16T01:49:38.721 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T01:49:38.721 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_min_size": "0", 2024-09-16T01:49:38.721 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T01:49:38.721 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T01:49:38.721 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T01:49:38.721 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_size": "2", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_default_type": "replicated", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_cost": "20971520", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_max_active": "0", 2024-09-16T01:49:38.722 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_op_priority": "3", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_priority": "5", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T01:49:38.723 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T01:49:38.724 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T01:49:38.724 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_repair_during_recovery": "false", 2024-09-16T01:49:38.724 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T01:49:38.724 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T01:49:38.724 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_cost": "52428800", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_end_hour": "0", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T01:49:38.725 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_priority": "5", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T01:49:38.726 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_skip_data_digest": "false", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_smart_report_timeout": "5", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_snap_trim_priority": "5", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_target_transaction_size": "30", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T01:49:38.727 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_tracing": "false", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_use_stale_snap": "false", 2024-09-16T01:49:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_kill_at": "0", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_max_join_drift": "10", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_min": "500", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_min_wait": "0.050000", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_service_trim_max": "500", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T01:49:38.729 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_service_trim_min": "250", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_stash_full_interval": "25", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_trim_max": "500", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "paxos_trim_min": "250", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "perf": "true", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "-", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrv": "", 2024-09-16T01:49:38.730 INFO:teuthology.orchestra.run.smithi146.stdout: "public_bind_addr": "-", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "public_network": "", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "public_network_interface": "", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "qat_compressor_enabled": "false", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rados_mon_op_timeout": "0", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rados_osd_op_timeout": "0", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rados_tracing": "false", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_atime_update_interval": "60", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T01:49:38.731 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache": "true", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_policy": "writearound", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_size": "33554432", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T01:49:38.732 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_compression_hint": "none", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_clone_format": "auto", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_data_pool": "", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_features": "61", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_format": "2", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_map_options": "", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_order": "22", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_pool": "rbd", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_stripe_count": "0", 2024-09-16T01:49:38.733 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_io_scheduler": "simple", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T01:49:38.734 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_order": "24", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_pool": "", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_journal_splay_width": "4", 2024-09-16T01:49:38.735 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T01:49:38.736 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T01:49:38.737 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T01:49:38.738 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_op_threads": "1", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_plugins": "", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T01:49:38.739 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T01:49:38.740 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_tracing": "false", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_validate_names": "true", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_validate_pool": "true", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "restapi_base_url": "", 2024-09-16T01:49:38.741 INFO:teuthology.orchestra.run.smithi146.stdout: "restapi_log_level": "", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_admin_entry": "admin", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_backend_store": "rados", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_barbican_url": "", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_beast_enable_async": "true", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T01:49:38.742 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_cache_enabled": "true", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_config_store": "rados", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_content_length_compat": "false", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_cross_domain_policy": "", 2024-09-16T01:49:38.743 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T01:49:38.744 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T01:49:38.745 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T01:49:38.746 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_d4n_port": "6379", 2024-09-16T01:49:38.747 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_log_window": "30", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T01:49:38.748 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T01:49:38.749 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dns_name": "", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dns_s3website_name": "", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T01:49:38.750 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_ops_log": "false", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_static_website": "false", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enable_usage_log": "false", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_expose_bucket": "false", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_extended_http_attrs": "", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_filter": "none", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T01:49:38.751 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_max_objs": "32", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T01:49:38.752 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_init_timeout": "300", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_password": "", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T01:49:38.753 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_project": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_token": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_admin_user": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_api_version": "2", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T01:49:38.754 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T01:49:38.755 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T01:49:38.755 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T01:49:38.755 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T01:49:38.755 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_url": "", 2024-09-16T01:49:38.755 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T01:49:38.755 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_max_objs": "32", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_max_worker": "3", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T01:49:38.756 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ldap_uri": "", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T01:49:38.757 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T01:49:38.758 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T01:49:38.758 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_log_http_headers": "", 2024-09-16T01:49:38.758 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T01:49:38.758 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T01:49:38.758 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T01:49:38.758 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T01:49:38.759 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T01:49:38.759 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T01:49:38.759 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_attr_size": "0", 2024-09-16T01:49:38.759 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T01:49:38.759 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T01:49:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T01:49:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T01:49:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T01:49:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_listing_results": "1000", 2024-09-16T01:49:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_notify_retries": "10", 2024-09-16T01:49:38.760 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T01:49:38.761 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T01:49:38.762 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T01:49:38.763 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T01:49:38.764 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T01:49:38.764 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T01:49:38.764 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T01:49:38.764 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T01:49:38.764 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T01:49:38.764 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_num_control_oids": "8", 2024-09-16T01:49:38.765 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_numa_node": "-1", 2024-09-16T01:49:38.765 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T01:49:38.765 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T01:49:38.765 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T01:49:38.765 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T01:49:38.765 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_op_tracing": "false", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_opa_token": "", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_opa_url": "", 2024-09-16T01:49:38.766 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T01:49:38.767 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T01:49:38.767 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T01:49:38.767 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ops_log_rados": "false", 2024-09-16T01:49:38.767 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T01:49:38.767 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T01:49:38.767 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T01:49:38.768 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T01:49:38.769 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T01:49:38.769 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T01:49:38.769 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T01:49:38.769 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T01:49:38.769 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_print_continue": "true", 2024-09-16T01:49:38.769 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T01:49:38.770 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T01:49:38.770 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T01:49:38.770 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T01:49:38.770 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T01:49:38.770 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_rados_tracing": "false", 2024-09-16T01:49:38.770 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_realm": "", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_realm_id": "", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_region": "", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T01:49:38.771 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T01:49:38.772 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_request_uri": "", 2024-09-16T01:49:38.772 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T01:49:38.772 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T01:49:38.772 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T01:49:38.772 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T01:49:38.772 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_resolve_cname": "false", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_run_sync_thread": "true", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T01:49:38.773 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T01:49:38.774 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T01:49:38.774 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T01:49:38.774 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T01:49:38.774 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T01:49:38.774 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T01:49:38.774 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_script_uri": "", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_service_provider_name": "", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_client_id": "", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_client_secret": "", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_entry": "sts", 2024-09-16T01:49:38.775 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_key": "sts", 2024-09-16T01:49:38.776 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T01:49:38.776 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T01:49:38.776 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T01:49:38.776 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T01:49:38.776 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T01:49:38.776 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_auth_url": "", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_custom_header": "", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_need_stats": "true", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_tenant_name": "", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_url": "", 2024-09-16T01:49:38.777 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_lease_period": "120", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T01:49:38.778 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T01:49:38.779 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T01:49:38.779 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T01:49:38.779 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T01:49:38.779 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T01:49:38.779 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T01:49:38.779 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_thread_pool_size": "512", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_comment": "", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_createby": "", 2024-09-16T01:49:38.780 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_encoding": "", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_flag": "false", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_origin": "", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_torrent_tracker": "", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T01:49:38.781 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_usage_max_shards": "32", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_use_opa_authz": "false", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_counters_cache": "false", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T01:49:38.782 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T01:49:38.783 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_user_unique_email": "true", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_verify_ssl": "true", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_zone": "", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_zone_id": "", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_zonegroup": "", 2024-09-16T01:49:38.784 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_zonegroup_id": "", 2024-09-16T01:49:38.785 INFO:teuthology.orchestra.run.smithi146.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T01:49:38.785 INFO:teuthology.orchestra.run.smithi146.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T01:49:38.785 INFO:teuthology.orchestra.run.smithi146.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T01:49:38.785 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_block_size": "4096", 2024-09-16T01:49:38.785 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T01:49:38.785 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T01:49:38.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T01:49:38.787 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T01:49:38.787 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T01:49:38.787 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T01:49:38.787 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T01:49:38.787 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T01:49:38.787 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_partition_filters": "false", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_perf": "false", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T01:49:38.788 INFO:teuthology.orchestra.run.smithi146.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T01:49:38.789 INFO:teuthology.orchestra.run.smithi146.stdout: "run_dir": "/var/run/ceph", 2024-09-16T01:49:38.789 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_block_create": "true", 2024-09-16T01:49:38.789 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T01:49:38.789 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T01:49:38.789 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T01:49:38.789 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_device_size": "53687091200", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_full_integrity_check": "false", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T01:49:38.790 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_main_device_type": "SSD", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T01:49:38.791 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T01:49:38.792 INFO:teuthology.orchestra.run.smithi146.stdout: "seastore_segment_size": "67108864", 2024-09-16T01:49:38.792 INFO:teuthology.orchestra.run.smithi146.stdout: "set_keepcaps": "false", 2024-09-16T01:49:38.792 INFO:teuthology.orchestra.run.smithi146.stdout: "setgroup": "", 2024-09-16T01:49:38.792 INFO:teuthology.orchestra.run.smithi146.stdout: "setuser": "", 2024-09-16T01:49:38.792 INFO:teuthology.orchestra.run.smithi146.stdout: "setuser_match_path": "", 2024-09-16T01:49:38.792 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T01:49:38.793 INFO:teuthology.orchestra.run.smithi146.stdout: "thp": "false", 2024-09-16T01:49:38.793 INFO:teuthology.orchestra.run.smithi146.stdout: "threadpool_default_timeout": "60", 2024-09-16T01:49:38.793 INFO:teuthology.orchestra.run.smithi146.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T01:49:38.793 INFO:teuthology.orchestra.run.smithi146.stdout: "throttler_perf_counter": "true" 2024-09-16T01:49:38.793 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:38.796 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T01:49:38.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.949+0000 7f6d7ec00640 1 Processor -- start 2024-09-16T01:49:38.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.949+0000 7f6d7ec00640 1 -- start start 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.950+0000 7f6d7ec00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d780c7980 0x7f6d780c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d7ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d781585b0 con 0x7f6d78151820 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d7ec00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d781582d0 con 0x7f6d78071a10 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d7ec00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6d781580a0 con 0x7f6d780c7980 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d77e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6d78151820 0x7f6d780c8070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47834/0 (socket says 172.21.15.146:47834) 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d77e00640 1 -- 172.21.15.146:0/2780710082 learned_addr learned my addr 172.21.15.146:0/2780710082 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:38.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d6f400640 1 --2- 172.21.15.146:0/2780710082 >> v2:172.21.15.161:3300/0 conn(0x7f6d780c7980 0x7f6d780c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d77400640 1 -- 172.21.15.146:0/2780710082 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1305700709 0 0) 0x7f6d781585b0 con 0x7f6d78151820 2024-09-16T01:49:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d77400640 1 -- 172.21.15.146:0/2780710082 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d5c003520 con 0x7f6d78151820 2024-09-16T01:49:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d6f400640 1 -- 172.21.15.146:0/2780710082 >> v1:172.21.15.161:6789/0 conn(0x7f6d78071a10 legacy=0x7f6d780c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d6f400640 1 -- 172.21.15.146:0/2780710082 >> v1:172.21.15.146:6789/0 conn(0x7f6d78151820 legacy=0x7f6d780c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.951+0000 7f6d6f400640 1 -- 172.21.15.146:0/2780710082 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d780c6bc0 con 0x7f6d780c7980 2024-09-16T01:49:38.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.952+0000 7f6d6f400640 1 --2- 172.21.15.146:0/2780710082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d780c7980 0x7f6d780c36b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ac397e9d37b137 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d77400640 1 -- 172.21.15.146:0/2780710082 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d60037070 con 0x7f6d780c7980 2024-09-16T01:49:38.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d77400640 1 -- 172.21.15.146:0/2780710082 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d60032640 con 0x7f6d780c7980 2024-09-16T01:49:38.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d77400640 1 -- 172.21.15.146:0/2780710082 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d60032a10 con 0x7f6d780c7980 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2780710082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d780c7980 msgr2=0x7f6d780c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2780710082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d780c7980 0x7f6d780c36b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002a2c0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2780710082 shutdown_connections 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2780710082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d780c7980 0x7f6d780c36b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2780710082 >> 172.21.15.146:0/2780710082 conn(0x7f6d7806bc10 msgr2=0x7f6d7806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2780710082 shutdown_connections 2024-09-16T01:49:38.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.953+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2780710082 wait complete. 2024-09-16T01:49:38.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.955+0000 7f6d7ec00640 1 Processor -- start 2024-09-16T01:49:38.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.955+0000 7f6d7ec00640 1 -- start start 2024-09-16T01:49:38.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7ec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78071a10 0x7f6d7814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7ec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d780c7980 0x7f6d781e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d78151820 0x7f6d781e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6d78071e80 con 0x7f6d78151820 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7ec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d780c6680 con 0x7f6d78071a10 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7ec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6d780c74f0 con 0x7f6d780c7980 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d7cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d78151820 0x7f6d781e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78071a10 0x7f6d7814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d6f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d780c7980 0x7f6d781e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78071a10 0x7f6d7814ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47410/0 (socket says 172.21.15.146:47410) 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 -- 172.21.15.146:0/2792238980 learned_addr learned my addr 172.21.15.146:0/2792238980 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d780c7980 msgr2=0x7f6d781e1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d780c7980 0x7f6d781e1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d78151820 msgr2=0x7f6d781e17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d78151820 0x7f6d781e17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.956+0000 7f6d77e00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d781ef8e0 con 0x7f6d78071a10 2024-09-16T01:49:38.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.957+0000 7f6d77e00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78071a10 0x7f6d7814ae50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800c500 tx=0x7f6d6800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.957+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d68017070 con 0x7f6d78071a10 2024-09-16T01:49:38.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.957+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d68005ce0 con 0x7f6d78071a10 2024-09-16T01:49:38.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.957+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7814ba80 con 0x7f6d78071a10 2024-09-16T01:49:38.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.957+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d68007330 con 0x7f6d78071a10 2024-09-16T01:49:38.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.957+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d780cb240 con 0x7f6d78071a10 2024-09-16T01:49:38.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.958+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6d7814bfa0 con 0x7f6d78071a10 2024-09-16T01:49:38.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.959+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6d6801a070 con 0x7f6d78071a10 2024-09-16T01:49:38.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.960+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6d5406c2b0 0x7f6d5406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.960+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6d6808aa70 con 0x7f6d78071a10 2024-09-16T01:49:38.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.960+0000 7f6d6f400640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6d5406c2b0 0x7f6d5406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.960+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+1289 (secure 0 0 0) 0x7f6d7814bfa0 con 0x7f6d78071a10 2024-09-16T01:49:38.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.961+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 0x7f6d780cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.961+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6d78153190 con 0x7f6d780cca20 2024-09-16T01:49:38.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.961+0000 7f6d6f400640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6d5406c2b0 0x7f6d5406e770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d60034000 tx=0x7f6d6002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.961+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 0x7f6d780cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.961+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f6d780c64a0 con 0x7f6d780cca20 2024-09-16T01:49:38.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 0x7f6d780cb760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000f7e0 tx=0x7f6d7000fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70003b80 con 0x7f6d780cca20 2024-09-16T01:49:38.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.0 v2:172.21.15.146:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d70018070 con 0x7f6d780cca20 2024-09-16T01:49:38.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 msgr2=0x7f6d780cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 0x7f6d780cb760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000f7e0 tx=0x7f6d7000fca0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 msgr2=0x7f6d780cb760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:49:38.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.962+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cca20 0x7f6d780cb760 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000f7e0 tx=0x7f6d7000fca0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.968+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 0x7f6d78147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.968+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6d78152590 con 0x7f6d780cbca0 2024-09-16T01:49:38.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.969+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 0x7f6d78147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.969+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f6d781575a0 con 0x7f6d780cbca0 2024-09-16T01:49:38.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.969+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 0x7f6d78147ff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000f7e0 tx=0x7f6d70001400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.970+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7002c0a0 con 0x7f6d780cbca0 2024-09-16T01:49:38.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.0 v2:172.21.15.146:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f6d70006d60 con 0x7f6d780cbca0 2024-09-16T01:49:38.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 msgr2=0x7f6d78147ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 0x7f6d78147ff0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000f7e0 tx=0x7f6d70001400 comp rx=0 tx=0).stop 2024-09-16T01:49:38.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 msgr2=0x7f6d78147ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:49:38.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d780cbca0 0x7f6d78147ff0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000f7e0 tx=0x7f6d70001400 comp rx=0 tx=0).stop 2024-09-16T01:49:38.974 INFO:teuthology.orchestra.run.smithi146.stdout:mon.a: {} 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 0x7f6d781f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.971+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d780cc080 con 0x7f6d78148530 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.972+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 0x7f6d781f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.972+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f6d780c7e30 con 0x7f6d78148530 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.972+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 0x7f6d781f22a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6d78152710 tx=0x7f6d70009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.972+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7003a070 con 0x7f6d78148530 2024-09-16T01:49:38.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.973+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d70031040 con 0x7f6d78148530 2024-09-16T01:49:38.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.973+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 msgr2=0x7f6d781f22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.973+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 0x7f6d781f22a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6d78152710 tx=0x7f6d70009040 comp rx=0 tx=0).stop 2024-09-16T01:49:38.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.973+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 msgr2=0x7f6d781f22a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:49:38.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.973+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 reap_dead start 2024-09-16T01:49:38.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.973+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148530 0x7f6d781f22a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.979+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 0x7f6d781e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.979+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d781e4020 con 0x7f6d78148940 2024-09-16T01:49:38.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.979+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 0x7f6d781e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.979+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f6d781e3d40 con 0x7f6d78148940 2024-09-16T01:49:38.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.980+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 0x7f6d781e1280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000fce0 tx=0x7f6d7002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.980+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7003a050 con 0x7f6d78148940 2024-09-16T01:49:38.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f6d7003b040 con 0x7f6d78148940 2024-09-16T01:49:38.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 msgr2=0x7f6d781e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 0x7f6d781e1280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000fce0 tx=0x7f6d7002a000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 msgr2=0x7f6d781e1280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:49:38.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78148940 0x7f6d781e1280 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000fce0 tx=0x7f6d7002a000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.984 INFO:teuthology.orchestra.run.smithi146.stdout:mon.b: {} 2024-09-16T01:49:38.985 INFO:teuthology.orchestra.run.smithi146.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T01:49:38.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 0x7f6d781f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6d781e1ba0 con 0x7f6d781e17c0 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.982+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 0x7f6d781f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f6d780cb480 con 0x7f6d781e17c0 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 0x7f6d781f27e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e010 tx=0x7f6d70009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7003c480 con 0x7f6d781e17c0 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.2 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d70007030 con 0x7f6d781e17c0 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 msgr2=0x7f6d781f27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 0x7f6d781f27e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e010 tx=0x7f6d70009040 comp rx=0 tx=0).stop 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 msgr2=0x7f6d781f27e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:49:38.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.983+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781e17c0 0x7f6d781f27e0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e010 tx=0x7f6d70009040 comp rx=0 tx=0).stop 2024-09-16T01:49:38.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.989+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 0x7f6d780c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:38.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.989+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6d781f3100 con 0x7f6d781f2d20 2024-09-16T01:49:38.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.990+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 0x7f6d780c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:38.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.990+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f6d780008d0 con 0x7f6d781f2d20 2024-09-16T01:49:38.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.991+0000 7f6d7cc00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 0x7f6d780c9da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d70001330 tx=0x7f6d70044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:38.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.991+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70052020 con 0x7f6d781f2d20 2024-09-16T01:49:38.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.992+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 <== mon.2 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f6d700079e0 con 0x7f6d781f2d20 2024-09-16T01:49:38.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.992+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 msgr2=0x7f6d780c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.992+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 0x7f6d780c9da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d70001330 tx=0x7f6d70044000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.992+0000 7f6d74c00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 msgr2=0x7f6d780c9da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:49:38.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.992+0000 7f6d74c00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6d781f2d20 0x7f6d780c9da0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d70001330 tx=0x7f6d70044000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.995 INFO:teuthology.orchestra.run.smithi146.stdout:mon.c: {} 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6d5406c2b0 msgr2=0x7f6d5406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6d5406c2b0 0x7f6d5406e770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6d60034000 tx=0x7f6d6002e000 comp rx=0 tx=0).stop 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78071a10 msgr2=0x7f6d7814ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 --2- 172.21.15.146:0/2792238980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d78071a10 0x7f6d7814ae50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800c500 tx=0x7f6d6800c9c0 comp rx=0 tx=0).stop 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7cc00640 1 -- 172.21.15.146:0/2792238980 reap_dead start 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 shutdown_connections 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 >> 172.21.15.146:0/2792238980 conn(0x7f6d7806bc10 msgr2=0x7f6d78151e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:38.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 shutdown_connections 2024-09-16T01:49:38.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:38.993+0000 7f6d7ec00640 1 -- 172.21.15.146:0/2792238980 wait complete. 2024-09-16T01:49:39.010 DEBUG:teuthology.orchestra.run.smithi146:> type ceph-objectstore-tool 2024-09-16T01:49:39.043 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T01:49:39.044 DEBUG:teuthology.orchestra.run.smithi161:> type ceph-objectstore-tool 2024-09-16T01:49:39.062 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T01:49:39.063 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T01:49:39.077 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T01:49:39.078 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T01:49:39.078 INFO:tasks.workunit:timeout=3h 2024-09-16T01:49:39.078 INFO:tasks.workunit:cleanup=True 2024-09-16T01:49:39.078 DEBUG:teuthology.orchestra.run.smithi146:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:39.079 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T01:49:39.079 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:49:39.079 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:49:39.079 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T01:49:39.079 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T01:49:39.080 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T01:49:39.080 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T01:49:39.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:49:39.080 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T01:49:39.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:49:39.081 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T01:49:39.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:49:39.103 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:39.102+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 2bh/43d Inode: 1 Links: 2 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-16 01:49:03.820159778 +0000 2024-09-16T01:49:39.129 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-16 01:49:36.349712194 +0000 2024-09-16T01:49:39.130 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-09-16 01:49:03.820159778 +0000 2024-09-16T01:49:39.130 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-16T01:49:39.130 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-16T01:49:39.219 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:39.217+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.266+0000 7f75b4c00640 1 Processor -- start 2024-09-16T01:49:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.266+0000 7f75b4c00640 1 -- start start 2024-09-16T01:49:39.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.267+0000 7f75b4c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75b00cca70 0x7f75b015b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75b4c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b015e0b0 con 0x7f75b00cafa0 2024-09-16T01:49:39.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75b4c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b015e390 con 0x7f75b0154c10 2024-09-16T01:49:39.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75b4c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f75b015de60 con 0x7f75b00cca70 2024-09-16T01:49:39.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75ae800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f75b00cafa0 0x7f75b00cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47836/0 (socket says 172.21.15.146:47836) 2024-09-16T01:49:39.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75ae800640 1 -- 172.21.15.146:0/3947612089 learned_addr learned my addr 172.21.15.146:0/3947612089 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:39.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75a5400640 1 --2- 172.21.15.146:0/3947612089 >> v2:172.21.15.146:3300/0 conn(0x7f75b00cca70 0x7f75b015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.268+0000 7f75ad400640 1 -- 172.21.15.146:0/3947612089 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292295761 0 0) 0x7f75b015e0b0 con 0x7f75b00cafa0 2024-09-16T01:49:39.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.269+0000 7f75ad400640 1 -- 172.21.15.146:0/3947612089 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7594003520 con 0x7f75b00cafa0 2024-09-16T01:49:39.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.269+0000 7f75ad400640 1 -- 172.21.15.146:0/3947612089 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120708117 0 0) 0x7f75b015e390 con 0x7f75b0154c10 2024-09-16T01:49:39.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.269+0000 7f75a5400640 1 -- 172.21.15.146:0/3947612089 >> v1:172.21.15.161:6789/0 conn(0x7f75b0154c10 legacy=0x7f75b0073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.269+0000 7f75a5400640 1 -- 172.21.15.146:0/3947612089 >> v1:172.21.15.146:6789/0 conn(0x7f75b00cafa0 legacy=0x7f75b00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.269+0000 7f75a5400640 1 -- 172.21.15.146:0/3947612089 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b015e670 con 0x7f75b00cca70 2024-09-16T01:49:39.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.270+0000 7f75a5400640 1 --2- 172.21.15.146:0/3947612089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cca70 0x7f75b015b370 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7598009790 tx=0x7f759802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d578c2b0808c03fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.271+0000 7f75ad400640 1 -- 172.21.15.146:0/3947612089 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598037070 con 0x7f75b00cca70 2024-09-16T01:49:39.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.271+0000 7f75ad400640 1 -- 172.21.15.146:0/3947612089 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f759802acc0 con 0x7f75b00cca70 2024-09-16T01:49:39.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.271+0000 7f75ad400640 1 -- 172.21.15.146:0/3947612089 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8001670 con 0x7f75b00cca70 2024-09-16T01:49:39.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.272+0000 7f75b4c00640 1 -- 172.21.15.146:0/3947612089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cca70 msgr2=0x7f75b015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.272+0000 7f75b4c00640 1 --2- 172.21.15.146:0/3947612089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cca70 0x7f75b015b370 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7598009790 tx=0x7f759802a230 comp rx=0 tx=0).stop 2024-09-16T01:49:39.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.272+0000 7f75b4c00640 1 -- 172.21.15.146:0/3947612089 shutdown_connections 2024-09-16T01:49:39.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.272+0000 7f75b4c00640 1 --2- 172.21.15.146:0/3947612089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cca70 0x7f75b015b370 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.272+0000 7f75b4c00640 1 -- 172.21.15.146:0/3947612089 >> 172.21.15.146:0/3947612089 conn(0x7f75b006ce80 msgr2=0x7f75b006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:39.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.273+0000 7f75b4c00640 1 -- 172.21.15.146:0/3947612089 shutdown_connections 2024-09-16T01:49:39.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.273+0000 7f75b4c00640 1 -- 172.21.15.146:0/3947612089 wait complete. 2024-09-16T01:49:39.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.274+0000 7f75b4c00640 1 Processor -- start 2024-09-16T01:49:39.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 -- start start 2024-09-16T01:49:39.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 0x7f75b01f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b00cca70 0x7f75b0158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b0154c10 0x7f75b00cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0154ff0 con 0x7f75b0154c10 2024-09-16T01:49:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75b015ec30 con 0x7f75b00cca70 2024-09-16T01:49:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.275+0000 7f75b4c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75b0073a50 con 0x7f75b00cafa0 2024-09-16T01:49:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 0x7f75b01f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 0x7f75b01f87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36630/0 (socket says 172.21.15.146:36630) 2024-09-16T01:49:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 -- 172.21.15.146:0/307170086 learned_addr learned my addr 172.21.15.146:0/307170086 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b00cca70 0x7f75b0158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.280 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T01:49:39.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7feebe000640 1 Processor -- start 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b00cca70 msgr2=0x7f75b0158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b00cca70 0x7f75b0158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b0154c10 msgr2=0x7f75b00cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b0154c10 0x7f75b00cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75ade00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b0155d10 con 0x7f75b00cafa0 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b00cca70 0x7f75b0158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.277+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 0x7f75b01f87a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f75a000c780 tx=0x7f75a000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.276+0000 7feebe000640 1 -- start start 2024-09-16T01:49:39.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.277+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003da0 con 0x7f75b00cafa0 2024-09-16T01:49:39.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.277+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a0005070 con 0x7f75b00cafa0 2024-09-16T01:49:39.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.277+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0005420 con 0x7f75b00cafa0 2024-09-16T01:49:39.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b01f8ce0 con 0x7f75b00cafa0 2024-09-16T01:49:39.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b015a4a0 con 0x7f75b00cafa0 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebe000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feeb80c6b70 0x7feeb815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebe000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80c75a0 0x7feeb80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebe000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb806d140 con 0x7feeb80cbab0 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebe000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7feeb815b880 con 0x7feeb80c6b70 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebe000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7feeb80cab20 con 0x7feeb80c75a0 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebcc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feeb80c6b70 0x7feeb815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebcc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feeb80c6b70 0x7feeb815b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47456/0 (socket says 172.21.15.146:47456) 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.278+0000 7feebcc00640 1 -- 172.21.15.146:0/3948085806 learned_addr learned my addr 172.21.15.146:0/3948085806 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.279+0000 7feeb7e00640 1 --2- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80c75a0 0x7feeb80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.279+0000 7feebcc00640 1 -- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80c75a0 msgr2=0x7feeb80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:39.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.279+0000 7feebcc00640 1 --2- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80c75a0 0x7feeb80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.279+0000 7feebcc00640 1 -- 172.21.15.146:0/3948085806 >> v1:172.21.15.161:6789/0 conn(0x7feeb80cbab0 legacy=0x7feeb80cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:39.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.279+0000 7feebcc00640 1 -- 172.21.15.146:0/3948085806 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb806d420 con 0x7feeb80c6b70 2024-09-16T01:49:39.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.279+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f75b0153b30 con 0x7f75b00cafa0 2024-09-16T01:49:39.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.280+0000 7f1a73200640 1 Processor -- start 2024-09-16T01:49:39.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.280+0000 7feebcc00640 1 --2- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb815b340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feeac009870 tx=0x7feeac02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c47d60d7f4cfe07a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.280+0000 7f1a73200640 1 -- start start 2024-09-16T01:49:39.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.280+0000 7feeb7400640 1 -- 172.21.15.146:0/3948085806 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac037070 con 0x7feeb80c6b70 2024-09-16T01:49:39.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.280+0000 7feeb7400640 1 -- 172.21.15.146:0/3948085806 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeac032440 con 0x7feeb80c6b70 2024-09-16T01:49:39.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.280+0000 7feeb7400640 1 -- 172.21.15.146:0/3948085806 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac032810 con 0x7feeb80c6b70 2024-09-16T01:49:39.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f75a00055c0 con 0x7f75b00cafa0 2024-09-16T01:49:39.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 -- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 msgr2=0x7feeb815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 --2- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb815b340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feeac009870 tx=0x7feeac02a130 comp rx=0 tx=0).stop 2024-09-16T01:49:39.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 -- 172.21.15.146:0/3948085806 shutdown_connections 2024-09-16T01:49:39.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 --2- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80c75a0 0x7feeb80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 --2- 172.21.15.146:0/3948085806 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb815b340 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 -- 172.21.15.146:0/3948085806 >> 172.21.15.146:0/3948085806 conn(0x7feeb80ccc20 msgr2=0x7feeb80cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:39.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7f75a6a00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f758c06c2b0 0x7f758c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 -- 172.21.15.146:0/3948085806 shutdown_connections 2024-09-16T01:49:39.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.281+0000 7feebe000640 1 -- 172.21.15.146:0/3948085806 wait complete. 2024-09-16T01:49:39.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7f75a008b5c0 con 0x7f75b00cafa0 2024-09-16T01:49:39.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f758c06c2b0 0x7f758c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v23) ==== 54+0+16 (secure 0 0 0) 0x7f75b0153b30 con 0x7f75b00cafa0 2024-09-16T01:49:39.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b015a780 0x7f75b0158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75b015b370 con 0x7f75b015a780 2024-09-16T01:49:39.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f1a73200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a740cd210 0x7f1a74073c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.292 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:39.286+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:39.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f758c06c2b0 0x7f758c06e770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7598034000 tx=0x7f759802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f1a73200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a740c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f1a73200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a740741c0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f1a73200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1a74072f00 con 0x7f1a740cd210 2024-09-16T01:49:39.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.282+0000 7f1a73200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7415f410 con 0x7f1a7415ca70 2024-09-16T01:49:39.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a740cd210 0x7f1a74073c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a740cd210 0x7f1a74073c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47466/0 (socket says 172.21.15.146:47466) 2024-09-16T01:49:39.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 -- 172.21.15.146:0/1940869507 learned_addr learned my addr 172.21.15.146:0/1940869507 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:39.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f75ae800640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b015a780 0x7f75b0158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 -- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 msgr2=0x7f1a740c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:39.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 --2- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a740c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 -- 172.21.15.146:0/1940869507 >> v1:172.21.15.146:6789/0 conn(0x7f1a740cd5f0 legacy=0x7f1a7415c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:39.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.283+0000 7f1a71e00640 1 -- 172.21.15.146:0/1940869507 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7415f6f0 con 0x7f1a740cd210 2024-09-16T01:49:39.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7feebe000640 1 Processor -- start 2024-09-16T01:49:39.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7f75ae800640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b015a780 0x7f75b0158550 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7f1a71e00640 1 --2- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd210 0x7f1a74073c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009870 tx=0x7f1a6402a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26642e71dcba2bfc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.296 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:39.296 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:39.296 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:39.296 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:39.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7feebe000640 1 -- start start 2024-09-16T01:49:39.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7f1a70a00640 1 -- 172.21.15.146:0/1940869507 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64037070 con 0x7f1a740cd210 2024-09-16T01:49:39.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7f1a70a00640 1 -- 172.21.15.146:0/1940869507 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a64032440 con 0x7f1a740cd210 2024-09-16T01:49:39.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.284+0000 7f1a70a00640 1 -- 172.21.15.146:0/1940869507 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64032810 con 0x7f1a740cd210 2024-09-16T01:49:39.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b015b370 con 0x7f75b015a780 2024-09-16T01:49:39.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 -- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd210 msgr2=0x7f1a74073c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 --2- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd210 0x7f1a74073c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009870 tx=0x7f1a6402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 -- 172.21.15.146:0/1940869507 shutdown_connections 2024-09-16T01:49:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 --2- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a740c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7feebe000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb8150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7feebe000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80c75a0 0x7feeb80c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7feebe000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80cbab0 0x7feeb80c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7feebe000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feeb80cb050 con 0x7feeb80c75a0 2024-09-16T01:49:39.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7feebe000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feeb80cad70 con 0x7feeb80c6b70 2024-09-16T01:49:39.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7feebe000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7feeb806cd20 con 0x7feeb80cbab0 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 --2- 172.21.15.146:0/1940869507 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd210 0x7f1a74073c80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 -- 172.21.15.146:0/1940869507 >> 172.21.15.146:0/1940869507 conn(0x7f1a7406cd20 msgr2=0x7f1a7406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 -- 172.21.15.146:0/1940869507 shutdown_connections 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.285+0000 7f1a73200640 1 -- 172.21.15.146:0/1940869507 wait complete. 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7f1a73200640 1 Processor -- start 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7f1a73200640 1 -- start start 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.288+0000 7f1a73200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a740cd210 0x7f1a741f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a73200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 0x7f1a7415a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a73200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a74153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a73200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7415f0c0 con 0x7f1a740cd210 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a73200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7415fcb0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a73200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1a74072ae0 con 0x7f1a7415ca70 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a71400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 0x7f1a7415a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a71400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 0x7f1a7415a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47486/0 (socket says 172.21.15.146:47486) 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a71400640 1 -- 172.21.15.146:0/1902698526 learned_addr learned my addr 172.21.15.146:0/1902698526 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a72800640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a74153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7f1a71e00640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a740cd210 0x7f1a741f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71400640 1 -- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 msgr2=0x7f1a74153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71400640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a74153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71400640 1 -- 172.21.15.146:0/1902698526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a740cd210 msgr2=0x7f1a741f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71400640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a740cd210 0x7f1a741f4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71400640 1 -- 172.21.15.146:0/1902698526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a741f4f50 con 0x7f1a740cd5f0 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71e00640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a740cd210 0x7f1a741f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb8150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb8150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47476/0 (socket says 172.21.15.146:47476) 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 -- 172.21.15.146:0/396444615 learned_addr learned my addr 172.21.15.146:0/396444615 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:39.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feeb7e00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80c75a0 0x7feeb80c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 -- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80cbab0 msgr2=0x7feeb80c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80cbab0 0x7feeb80c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 -- 172.21.15.146:0/396444615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80c75a0 msgr2=0x7feeb80c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80c75a0 0x7feeb80c1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.286+0000 7feebcc00640 1 -- 172.21.15.146:0/396444615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb81f06d0 con 0x7feeb80c6b70 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7feebcc00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb8150700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac041020 con 0x7feeb80c6b70 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeac0306c0 con 0x7feeb80c6b70 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac037040 con 0x7feeb80c6b70 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb8159610 con 0x7feeb80c6b70 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.287+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb8159bd0 con 0x7feeb80c6b70 2024-09-16T01:49:39.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.289+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb806da90 con 0x7feeb80c6b70 2024-09-16T01:49:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7feeac044020 con 0x7feeb80c6b70 2024-09-16T01:49:39.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.290+0000 7f1a71400640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 0x7f1a7415a060 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000c400 tx=0x7f1a6000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.291+0000 7feeb4c00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fee9806c2b0 0x7fee9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.291+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7feeac0abda0 con 0x7feeb80c6b70 2024-09-16T01:49:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.291+0000 7feeb7e00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fee9806c2b0 0x7fee9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.292+0000 7feeb7e00640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fee9806c2b0 0x7fee9806e770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feea4006fd0 tx=0x7feea4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.295+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60011790 con 0x7f1a740cd5f0 2024-09-16T01:49:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.295+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a60017070 con 0x7f1a740cd5f0 2024-09-16T01:49:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.295+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60011bb0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.295+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a741523b0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.295+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a74156e40 con 0x7f1a740cd5f0 2024-09-16T01:49:39.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.295+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7feeac04b030 con 0x7feeb80c6b70 2024-09-16T01:49:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.296+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a74072680 con 0x7f1a740cd5f0 2024-09-16T01:49:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.298+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1a6001c460 con 0x7f1a740cd5f0 2024-09-16T01:49:39.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.300+0000 7f1a6a000640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1a5406c330 0x7f1a5406e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:49:39.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.300+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7f1a6008b0f0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.300+0000 7f1a71e00640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1a5406c330 0x7f1a5406e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.301+0000 7f1a71e00640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1a5406c330 0x7f1a5406e7f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a64034000 tx=0x7f1a6402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.305+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b00c6270 con 0x7f75b015a780 2024-09-16T01:49:39.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.305+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1a74072680 con 0x7f1a740cd5f0 2024-09-16T01:49:39.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.308+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b00c6270 con 0x7f75b015a780 2024-09-16T01:49:39.311 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:39.312 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:39.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.308+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b0156410 0x7f75b014ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.308+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f75b0158a90 con 0x7f75b0156410 2024-09-16T01:49:39.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.309+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b0156410 0x7f75b014ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.309+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b0156410 0x7f75b014ab00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.310+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b0158a90 con 0x7f75b0156410 2024-09-16T01:49:39.333 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T01:49:39.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.341+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b0151d50 con 0x7f75b0156410 2024-09-16T01:49:39.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.344+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b0151d50 con 0x7f75b0156410 2024-09-16T01:49:39.346 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:39.346 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:39.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.344+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b00b2ed0 0x7f75b01f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.344+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f75b01f5280 con 0x7f75b00b2ed0 2024-09-16T01:49:39.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.344+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b00b2ed0 0x7f75b01f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.345+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b00b2ed0 0x7f75b01f4d40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.347+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b01f5280 con 0x7f75b00b2ed0 2024-09-16T01:49:39.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.375+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b0062630 con 0x7f75b00b2ed0 2024-09-16T01:49:39.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.378+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b0062630 con 0x7f75b00b2ed0 2024-09-16T01:49:39.381 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:39.381 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:39.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.379+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b0063110 0x7f75b015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.379+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f75b0063510 con 0x7f75b0063110 2024-09-16T01:49:39.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.379+0000 7f75ae800640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b0063110 0x7f75b015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.380+0000 7f75ae800640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b0063110 0x7f75b015ba50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.381+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b0063510 con 0x7f75b0063110 2024-09-16T01:49:39.388 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:39.387+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:39.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.409+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b015c260 con 0x7f75b0063110 2024-09-16T01:49:39.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.411+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b015c260 con 0x7f75b0063110 2024-09-16T01:49:39.413 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:39.414 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:39.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.411+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b002c640 0x7f75b002ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.411+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f75b015c1e0 con 0x7f75b002c640 2024-09-16T01:49:39.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.412+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b002c640 0x7f75b002ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.412+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b002c640 0x7f75b002ca80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.413+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b015c1e0 con 0x7f75b002c640 2024-09-16T01:49:39.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.442+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b015bf90 con 0x7f75b002c640 2024-09-16T01:49:39.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.444+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b015bf90 con 0x7f75b002c640 2024-09-16T01:49:39.446 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:39.447 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:39.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.445+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b002d4d0 0x7f75b020b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.445+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f75b002d930 con 0x7f75b002d4d0 2024-09-16T01:49:39.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.445+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b002d4d0 0x7f75b020b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.446+0000 7f75a5400640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b002d4d0 0x7f75b020b1d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.446+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b002d930 con 0x7f75b002d4d0 2024-09-16T01:49:39.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.478+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b002d930 con 0x7f75b002d4d0 2024-09-16T01:49:39.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.480+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b002d930 con 0x7f75b002d4d0 2024-09-16T01:49:39.482 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:39.483 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:39.483 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:39.483 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:39.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.480+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b020bc20 0x7f75b020e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.480+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f75b0210de0 con 0x7f75b020bc20 2024-09-16T01:49:39.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.481+0000 7f75ae800640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b020bc20 0x7f75b020e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.482+0000 7f75ae800640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b020bc20 0x7f75b020e010 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.482+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b0210de0 con 0x7f75b020bc20 2024-09-16T01:49:39.489 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:39.488+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:39.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:49:39.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.509+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b020e550 con 0x7f75b020bc20 2024-09-16T01:49:39.516 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:39.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.511+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b020e550 con 0x7f75b020bc20 2024-09-16T01:49:39.516 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:39.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.512+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b020ec70 0x7f75b020f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.512+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f75b02141d0 con 0x7f75b020ec70 2024-09-16T01:49:39.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.512+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b020ec70 0x7f75b020f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.513+0000 7f75ade00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b020ec70 0x7f75b020f0d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.514+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b02141d0 con 0x7f75b020ec70 2024-09-16T01:49:39.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.516+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7feeb80cda30 con 0x7feeb80c6b70 2024-09-16T01:49:39.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.545+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b020f6d0 con 0x7f75b020ec70 2024-09-16T01:49:39.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.547+0000 7f75a6a00640 1 -- 172.21.15.146:0/307170086 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b020f6d0 con 0x7f75b020ec70 2024-09-16T01:49:39.549 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:39.550 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.547+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b015a780 msgr2=0x7f75b0158550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b015a780 0x7f75b0158550 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b0156410 msgr2=0x7f75b014ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b0156410 0x7f75b014ab00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b00b2ed0 msgr2=0x7f75b01f4d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b00b2ed0 0x7f75b01f4d40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b0063110 msgr2=0x7f75b015ba50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b0063110 0x7f75b015ba50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b002c640 msgr2=0x7f75b002ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b002c640 0x7f75b002ca80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b002d4d0 msgr2=0x7f75b020b1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b002d4d0 0x7f75b020b1d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75ae800640 1 -- 172.21.15.146:0/307170086 reap_dead start 2024-09-16T01:49:39.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b020bc20 msgr2=0x7f75b020e010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b020bc20 0x7f75b020e010 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75ae800640 1 -- 172.21.15.146:0/307170086 reap_dead start 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b020ec70 msgr2=0x7f75b020f0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b020ec70 0x7f75b020f0d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75ae800640 1 -- 172.21.15.146:0/307170086 reap_dead start 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75ae800640 1 -- 172.21.15.146:0/307170086 reap_dead start 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f758c06c2b0 msgr2=0x7f758c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f758c06c2b0 0x7f758c06e770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7598034000 tx=0x7f759802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 msgr2=0x7f75b01f87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 0x7f75b01f87a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f75a000c780 tx=0x7f75a000cc40 comp rx=0 tx=0).stop 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 shutdown_connections 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f758c06c2b0 0x7f758c06e770 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 --2- 172.21.15.146:0/307170086 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cafa0 0x7f75b01f87a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 >> 172.21.15.146:0/307170086 conn(0x7f75b006ce80 msgr2=0x7f75b0073210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:39.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 shutdown_connections 2024-09-16T01:49:39.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.548+0000 7f75b4c00640 1 -- 172.21.15.146:0/307170086 wait complete. 2024-09-16T01:49:39.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.603+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f1a740728c0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.605 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:39.604+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:39.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.605+0000 7f1a6a000640 1 -- 172.21.15.146:0/1902698526 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v23) ==== 81+0+34415 (secure 0 0 0) 0x7f1a600599b0 con 0x7f1a740cd5f0 2024-09-16T01:49:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:39.607 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 23, 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "created": "2024-09-16T01:48:34.399638+0000", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "modified": "2024-09-16T01:49:39.236304+0000", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "last_up_change": "2024-09-16T01:48:45.748664+0000", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "last_in_change": "2024-09-16T01:48:36.639012+0000", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_num": 5799936, 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_set": [ 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "pglog_hardlimit", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "purged_snapdirs", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_deletes", 2024-09-16T01:49:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "sortbitwise" 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_version": 6, 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_snapshot": "", 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_max": 3, 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd": 8, 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "require_min_compat_client": "luminous", 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "min_compat_client": "luminous", 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "require_osd_release": "squid", 2024-09-16T01:49:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_crimson": false, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "pools": [ 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 1, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": ".mgr", 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:46.570355+0000", 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:49:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 1, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 1, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 1, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 1, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 1, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:49:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "18", 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:49:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:49:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:49:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_max": 32, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 1 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": {} 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:49:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 8, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 8, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.25, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 1, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 1 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 2, 2024-09-16T01:49:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_metadata", 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:56.008811+0000", 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:49:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 32, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 32, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 32, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 32, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 32, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:49:39.618 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "21", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:49:39.619 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:49:39.620 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:49:39.621 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_bias": 4, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 16, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_priority": 5 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": "cephfs" 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.622 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 2.25, 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 2.25, 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1, 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2.25, 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2.25, 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 1, 2024-09-16T01:49:39.623 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 1 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 3, 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_data", 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:57.207121+0000", 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:49:39.624 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 64, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 64, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 64, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 64, 2024-09-16T01:49:39.625 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 64, 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "22", 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:49:39.626 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:49:39.627 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.628 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 64, 2024-09-16T01:49:39.629 INFO:teuthology.orchestra.run.smithi146.stdout: "target_size_ratio": 0.89999997615814209 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "data": "cephfs" 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.5, 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.5, 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1, 2024-09-16T01:49:39.630 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 1.5, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 1.5, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 1, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 1 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "osds": [ 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.631 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.632 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6824", 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6825", 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6826", 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6827", 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6830", 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6831", 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.635 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6828", 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6829", 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.636 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6825/61440064", 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6827/61440064", 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6831/61440064", 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6829/61440064", 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.637 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6808", 2024-09-16T01:49:39.638 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6809", 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6810", 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6811", 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6814", 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6815", 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.641 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6812", 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6813", 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6809/2375201615", 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6811/2375201615", 2024-09-16T01:49:39.642 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6815/2375201615", 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6813/2375201615", 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.643 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.644 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6800", 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6801", 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6802", 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6803", 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6806", 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6807", 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6804", 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6805", 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.648 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6801/1818492127", 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6803/1818492127", 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6807/1818492127", 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6805/1818492127", 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.649 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.650 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6816", 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6817", 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6818", 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6819", 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6822", 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6823", 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6820", 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6821", 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.654 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6817/1633804394", 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6819/1633804394", 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1633804394", 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6821/1633804394", 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.655 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 13, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.656 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6800", 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6801", 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6802", 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6803", 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6806", 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6807", 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6804", 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6805", 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6801/154390265", 2024-09-16T01:49:39.660 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6803/154390265", 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6807/154390265", 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6805/154390265", 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.661 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6824", 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6825", 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6826", 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6827", 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.664 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6830", 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6831", 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6828", 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6829", 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.666 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6825/356357279", 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6827/356357279", 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6831/356357279", 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6829/356357279", 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:49:39.667 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6808", 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6809", 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.669 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6810", 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6811", 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.670 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6814", 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6815", 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6812", 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6813", 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:49:39.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6809/3661001615", 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6811/3661001615", 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6815/3661001615", 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6813/3661001615", 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 20, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:49:39.674 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6816", 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6817", 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6818", 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6819", 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6822", 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6823", 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6820", 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6821", 2024-09-16T01:49:39.678 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6817/1279166573", 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6819/1279166573", 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6823/1279166573", 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6821/1279166573", 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.679 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_xinfo": [ 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.135308+0000", 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.290750+0000", 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.681 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.712744+0000", 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.682 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.988678+0000", 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.198114+0000", 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.085332+0000", 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.684 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.394321+0000", 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:49:39.685 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.236983+0000", 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap": [], 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_items": [ 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.3f", 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 0, 2024-09-16T01:49:39.686 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 7 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_primaries": [], 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_temp": [], 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_temp": [], 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklist": {}, 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "range_blocklist": {}, 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profiles": { 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "default": { 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-failure-domain": "osd", 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "k": "2", 2024-09-16T01:49:39.687 INFO:teuthology.orchestra.run.smithi146.stdout: "m": "1", 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin": "jerasure", 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "technique": "reed_sol_van" 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps_queue": [], 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "new_removed_snaps": [], 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "new_purged_snaps": [], 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_node_flags": {}, 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "device_class_flags": {}, 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode": { 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_enabled": false, 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_bucket_count": 0, 2024-09-16T01:49:39.688 INFO:teuthology.orchestra.run.smithi146.stdout: "degraded_stretch_mode": 0, 2024-09-16T01:49:39.689 INFO:teuthology.orchestra.run.smithi146.stdout: "recovering_stretch_mode": 0, 2024-09-16T01:49:39.689 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_bucket": 0 2024-09-16T01:49:39.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:39.689 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1a5406c330 msgr2=0x7f1a5406e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1a5406c330 0x7f1a5406e7f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1a64034000 tx=0x7f1a6402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 msgr2=0x7f1a7415a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 0x7f1a7415a060 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000c400 tx=0x7f1a6000c8c0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 shutdown_connections 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1a5406c330 0x7f1a5406e7f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1a7415ca70 0x7f1a74153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a740cd5f0 0x7f1a7415a060 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 --2- 172.21.15.146:0/1902698526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a740cd210 0x7f1a741f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 >> 172.21.15.146:0/1902698526 conn(0x7f1a7406cd20 msgr2=0x7f1a7415b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 shutdown_connections 2024-09-16T01:49:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.607+0000 7f1a73200640 1 -- 172.21.15.146:0/1902698526 wait complete. 2024-09-16T01:49:39.692 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': '4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d', 'created': '2024-09-16T01:48:34.399638+0000', 'modified': '2024-09-16T01:49:39.236304+0000', 'last_up_change': '2024-09-16T01:48:45.748664+0000', 'last_in_change': '2024-09-16T01:48:36.639012+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:48:46.570355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:48:56.008811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:48:57.207121+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e7758ddf-75ff-43c7-abae-3647b170f7c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 61440064}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 61440064}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 61440064}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 61440064}]}, 'public_addr': '172.21.15.146:6825/61440064', 'cluster_addr': '172.21.15.146:6827/61440064', 'heartbeat_back_addr': '172.21.15.146:6831/61440064', 'heartbeat_front_addr': '172.21.15.146:6829/61440064', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a065e77e-26f0-4245-abe8-e36c2fb4436b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 2375201615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 2375201615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 2375201615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 2375201615}]}, 'public_addr': '172.21.15.146:6809/2375201615', 'cluster_addr': '172.21.15.146:6811/2375201615', 'heartbeat_back_addr': '172.21.15.146:6815/2375201615', 'heartbeat_front_addr': '172.21.15.146:6813/2375201615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fefb6d8f-6799-4344-a977-85ab983ec0cb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 1818492127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 1818492127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 1818492127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 1818492127}]}, 'public_addr': '172.21.15.146:6801/1818492127', 'cluster_addr': '172.21.15.146:6803/1818492127', 'heartbeat_back_addr': '172.21.15.146:6807/1818492127', 'heartbeat_front_addr': '172.21.15.146:6805/1818492127', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9d1a2f96-efcb-4a73-b5db-ec5e2a63122a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 1633804394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 1633804394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1633804394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 1633804394}]}, 'public_addr': '172.21.15.146:6817/1633804394', 'cluster_addr': '172.21.15.146:6819/1633804394', 'heartbeat_back_addr': '172.21.15.146:6823/1633804394', 'heartbeat_front_addr': '172.21.15.146:6821/1633804394', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6800', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6801', 'nonce': 154390265}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6802', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6803', 'nonce': 154390265}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6806', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6807', 'nonce': 154390265}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6804', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6805', 'nonce': 154390265}]}, 'public_addr': '172.21.15.161:6801/154390265', 'cluster_addr': '172.21.15.161:6803/154390265', 'heartbeat_back_addr': '172.21.15.161:6807/154390265', 'heartbeat_front_addr': '172.21.15.161:6805/154390265', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '11562dd5-d537-4d88-a282-e8687caebad6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6824', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6825', 'nonce': 356357279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6826', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6827', 'nonce': 356357279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6830', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6831', 'nonce': 356357279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6828', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6829', 'nonce': 356357279}]}, 'public_addr': '172.21.15.161:6825/356357279', 'cluster_addr': '172.21.15.161:6827/356357279', 'heartbeat_back_addr': '172.21.15.161:6831/356357279', 'heartbeat_front_addr': '172.21.15.161:6829/356357279', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b31adf01-6768-49cf-a232-127b9d0d2dc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6808', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6809', 'nonce': 3661001615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6810', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6811', 'nonce': 3661001615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6814', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6815', 'nonce': 3661001615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6812', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6813', 'nonce': 3661001615}]}, 'public_addr': '172.21.15.161:6809/3661001615', 'cluster_addr': '172.21.15.161:6811/3661001615', 'heartbeat_back_addr': '172.21.15.161:6815/3661001615', 'heartbeat_front_addr': '172.21.15.161:6813/3661001615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '67c95c28-b600-430c-9474-a902cf46a73b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6816', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6817', 'nonce': 1279166573}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6818', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6819', 'nonce': 1279166573}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6822', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6823', 'nonce': 1279166573}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6820', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6821', 'nonce': 1279166573}]}, 'public_addr': '172.21.15.161:6817/1279166573', 'cluster_addr': '172.21.15.161:6819/1279166573', 'heartbeat_back_addr': '172.21.15.161:6823/1279166573', 'heartbeat_front_addr': '172.21.15.161:6821/1279166573', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.135308+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.290750+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.712744+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.988678+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.198114+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.085332+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.394321+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.236983+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3f', 'mappings': [{'from': 0, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T01:49:39.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:49:39.693 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:39.693 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:39.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:49:39.696 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:39.693+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:39.795 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:39.794+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.893+0000 7faf39e00640 1 Processor -- start 2024-09-16T01:49:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.893+0000 7faf39e00640 1 -- start start 2024-09-16T01:49:39.896 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:39.895+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.894+0000 7faf39e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf39e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf3415b1d0 con 0x7faf340cb2f0 2024-09-16T01:49:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf39e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf3415b4b0 con 0x7faf340cc4a0 2024-09-16T01:49:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf39e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7faf340cbd00 con 0x7faf340cc0c0 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf32a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf2a000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7faf340cb2f0 0x7faf3415e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47842/0 (socket says 172.21.15.146:47842) 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf33400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7faf340cc4a0 0x7faf34165430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:45316/0 (socket says 172.21.15.146:45316) 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf2a000640 1 -- 172.21.15.146:0/2700868860 learned_addr learned my addr 172.21.15.146:0/2700868860 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf32a00640 1 -- 172.21.15.146:0/2700868860 >> v1:172.21.15.161:6789/0 conn(0x7faf340cc4a0 legacy=0x7faf34165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf32a00640 1 -- 172.21.15.146:0/2700868860 >> v1:172.21.15.146:6789/0 conn(0x7faf340cb2f0 legacy=0x7faf3415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf32000640 1 -- 172.21.15.146:0/2700868860 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086530101 0 0) 0x7faf3415b4b0 con 0x7faf340cc4a0 2024-09-16T01:49:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.895+0000 7faf32a00640 1 -- 172.21.15.146:0/2700868860 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3415b790 con 0x7faf340cc0c0 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.896+0000 7faf32a00640 1 --2- 172.21.15.146:0/2700868860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3415aab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf240097d0 tx=0x7faf2402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e789d6487688f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf32000640 1 -- 172.21.15.146:0/2700868860 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24037070 con 0x7faf340cc0c0 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf32000640 1 -- 172.21.15.146:0/2700868860 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf24032440 con 0x7faf340cc0c0 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf32000640 1 -- 172.21.15.146:0/2700868860 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24032810 con 0x7faf340cc0c0 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf39e00640 1 -- 172.21.15.146:0/2700868860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 msgr2=0x7faf3415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf39e00640 1 --2- 172.21.15.146:0/2700868860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3415aab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf240097d0 tx=0x7faf2402a210 comp rx=0 tx=0).stop 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf39e00640 1 -- 172.21.15.146:0/2700868860 shutdown_connections 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf39e00640 1 --2- 172.21.15.146:0/2700868860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3415aab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf39e00640 1 -- 172.21.15.146:0/2700868860 >> 172.21.15.146:0/2700868860 conn(0x7faf3406d560 msgr2=0x7faf3406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:39.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.897+0000 7faf39e00640 1 -- 172.21.15.146:0/2700868860 shutdown_connections 2024-09-16T01:49:39.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.898+0000 7faf39e00640 1 -- 172.21.15.146:0/2700868860 wait complete. 2024-09-16T01:49:39.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.899+0000 7faf39e00640 1 Processor -- start 2024-09-16T01:49:39.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.899+0000 7faf39e00640 1 -- start start 2024-09-16T01:49:39.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf39e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 0x7faf340c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf39e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3414b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf39e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf340cc4a0 0x7faf340b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf39e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf3415e540 con 0x7faf340cb2f0 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf39e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faf3415ba70 con 0x7faf340cc4a0 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf39e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7faf340cc8b0 con 0x7faf340cc0c0 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf32a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 0x7faf340c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf32a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 0x7faf340c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47292/0 (socket says 172.21.15.146:47292) 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf32a00640 1 -- 172.21.15.146:0/3923629713 learned_addr learned my addr 172.21.15.146:0/3923629713 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.900+0000 7faf2a000640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3414b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf33400640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf340cc4a0 0x7faf340b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf32a00640 1 -- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 msgr2=0x7faf3414b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf32a00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3414b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf32a00640 1 -- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf340cc4a0 msgr2=0x7faf340b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf32a00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf340cc4a0 0x7faf340b5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf32a00640 1 -- 172.21.15.146:0/3923629713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3414b8e0 con 0x7faf340cb2f0 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf2a000640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3414b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.901+0000 7faf32a00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 0x7faf340c7c70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faf240097d0 tx=0x7faf240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.902+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24037040 con 0x7faf340cb2f0 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.902+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf24030770 con 0x7faf340cb2f0 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.902+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24030b60 con 0x7faf340cb2f0 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.902+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf341f12b0 con 0x7faf340cb2f0 2024-09-16T01:49:39.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.902+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf340b5170 con 0x7faf340cb2f0 2024-09-16T01:49:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.903+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf340c5560 con 0x7faf340cb2f0 2024-09-16T01:49:39.906 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:39.906 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.905+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faf24044020 con 0x7faf340cb2f0 2024-09-16T01:49:39.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.906+0000 7faf2be00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7faf1006c2e0 0x7faf1006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:39.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.906+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5738+0+0 (secure 0 0 0) 0x7faf240ab680 con 0x7faf340cb2f0 2024-09-16T01:49:39.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.907+0000 7faf2a000640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7faf1006c2e0 0x7faf1006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:39.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.907+0000 7faf2a000640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7faf1006c2e0 0x7faf1006e7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faf3414f3e0 tx=0x7faf1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:39.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:39.915+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faf2404b020 con 0x7faf340cb2f0 2024-09-16T01:49:39.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:49:39.997 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:39.995+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:40.099 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:40.097+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:40.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.102+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faf341f1590 con 0x7faf1006c2e0 2024-09-16T01:49:40.108 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:40.108 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:40.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.109+0000 7faf2be00640 1 -- 172.21.15.146:0/3923629713 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252037 (secure 0 0 0) 0x7faf341f1590 con 0x7faf1006c2e0 2024-09-16T01:49:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:49:40.116 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:49:40.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7faf1006c2e0 msgr2=0x7faf1006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7faf1006c2e0 0x7faf1006e7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faf3414f3e0 tx=0x7faf1c009040 comp rx=0 tx=0).stop 2024-09-16T01:49:40.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 msgr2=0x7faf340c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 0x7faf340c7c70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faf240097d0 tx=0x7faf240046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 shutdown_connections 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7faf1006c2e0 0x7faf1006e7a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf340cc4a0 0x7faf340b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faf340cc0c0 0x7faf3414b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 --2- 172.21.15.146:0/3923629713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf340cb2f0 0x7faf340c7c70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 >> 172.21.15.146:0/3923629713 conn(0x7faf3406d560 msgr2=0x7faf34153850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 shutdown_connections 2024-09-16T01:49:40.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.114+0000 7faf39e00640 1 -- 172.21.15.146:0/3923629713 wait complete. 2024-09-16T01:49:40.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:49:40.132 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-16T01:49:39.245140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598920,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":367,"num_write_kb":2481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218304,"kb_used_data":2112,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756416,"statfs":{"total":773094113280,"available":772870569984,"internally_reserved":0,"allocated":2162688,"data_stored":1823271,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":630,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.720224"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039530+0000","last_change":"2024-09-16T01:48:58.842434+0000","last_active":"2024-09-16T01:49:01.039530+0000","last_peered":"2024-09-16T01:49:01.039530+0000","last_clean":"2024-09-16T01:49:01.039530+0000","last_became_active":"2024-09-16T01:48:58.842043+0000","last_became_peered":"2024-09-16T01:48:58.842043+0000","last_unstale":"2024-09-16T01:49:01.039530+0000","last_undegraded":"2024-09-16T01:49:01.039530+0000","last_fullsized":"2024-09-16T01:49:01.039530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039277+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:49:01.039277+0000","last_peered":"2024-09-16T01:49:01.039277+0000","last_clean":"2024-09-16T01:49:01.039277+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:49:01.039277+0000","last_undegraded":"2024-09-16T01:49:01.039277+0000","last_fullsized":"2024-09-16T01:49:01.039277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039170+0000","last_change":"2024-09-16T01:48:58.827408+0000","last_active":"2024-09-16T01:49:01.039170+0000","last_peered":"2024-09-16T01:49:01.039170+0000","last_clean":"2024-09-16T01:49:01.039170+0000","last_became_active":"2024-09-16T01:48:58.827101+0000","last_became_peered":"2024-09-16T01:48:58.827101+0000","last_unstale":"2024-09-16T01:49:01.039170+0000","last_undegraded":"2024-09-16T01:49:01.039170+0000","last_fullsized":"2024-09-16T01:49:01.039170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823984+0000","last_change":"2024-09-16T01:48:58.833321+0000","last_active":"2024-09-16T01:48:59.823984+0000","last_peered":"2024-09-16T01:48:59.823984+0000","last_clean":"2024-09-16T01:48:59.823984+0000","last_became_active":"2024-09-16T01:48:58.828915+0000","last_became_peered":"2024-09-16T01:48:58.828915+0000","last_unstale":"2024-09-16T01:48:59.823984+0000","last_undegraded":"2024-09-16T01:48:59.823984+0000","last_fullsized":"2024-09-16T01:48:59.823984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:46.083905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039337+0000","last_change":"2024-09-16T01:48:58.832881+0000","last_active":"2024-09-16T01:49:01.039337+0000","last_peered":"2024-09-16T01:49:01.039337+0000","last_clean":"2024-09-16T01:49:01.039337+0000","last_became_active":"2024-09-16T01:48:58.832257+0000","last_became_peered":"2024-09-16T01:48:58.832257+0000","last_unstale":"2024-09-16T01:49:01.039337+0000","last_undegraded":"2024-09-16T01:49:01.039337+0000","last_fullsized":"2024-09-16T01:49:01.039337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823923+0000","last_change":"2024-09-16T01:48:58.830375+0000","last_active":"2024-09-16T01:48:59.823923+0000","last_peered":"2024-09-16T01:48:59.823923+0000","last_clean":"2024-09-16T01:48:59.823923+0000","last_became_active":"2024-09-16T01:48:58.830066+0000","last_became_peered":"2024-09-16T01:48:58.830066+0000","last_unstale":"2024-09-16T01:48:59.823923+0000","last_undegraded":"2024-09-16T01:48:59.823923+0000","last_fullsized":"2024-09-16T01:48:59.823923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:47.490801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038683+0000","last_change":"2024-09-16T01:48:58.828028+0000","last_active":"2024-09-16T01:49:01.038683+0000","last_peered":"2024-09-16T01:49:01.038683+0000","last_clean":"2024-09-16T01:49:01.038683+0000","last_became_active":"2024-09-16T01:48:58.827768+0000","last_became_peered":"2024-09-16T01:48:58.827768+0000","last_unstale":"2024-09-16T01:49:01.038683+0000","last_undegraded":"2024-09-16T01:49:01.038683+0000","last_fullsized":"2024-09-16T01:49:01.038683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038749+0000","last_change":"2024-09-16T01:48:58.822570+0000","last_active":"2024-09-16T01:49:01.038749+0000","last_peered":"2024-09-16T01:49:01.038749+0000","last_clean":"2024-09-16T01:49:01.038749+0000","last_became_active":"2024-09-16T01:48:58.822161+0000","last_became_peered":"2024-09-16T01:48:58.822161+0000","last_unstale":"2024-09-16T01:49:01.038749+0000","last_undegraded":"2024-09-16T01:49:01.038749+0000","last_fullsized":"2024-09-16T01:49:01.038749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:42:58.003875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040721+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:49:01.040721+0000","last_peered":"2024-09-16T01:49:01.040721+0000","last_clean":"2024-09-16T01:49:01.040721+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:49:01.040721+0000","last_undegraded":"2024-09-16T01:49:01.040721+0000","last_fullsized":"2024-09-16T01:49:01.040721+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039415+0000","last_change":"2024-09-16T01:48:58.840151+0000","last_active":"2024-09-16T01:49:01.039415+0000","last_peered":"2024-09-16T01:49:01.039415+0000","last_clean":"2024-09-16T01:49:01.039415+0000","last_became_active":"2024-09-16T01:48:58.839644+0000","last_became_peered":"2024-09-16T01:48:58.839644+0000","last_unstale":"2024-09-16T01:49:01.039415+0000","last_undegraded":"2024-09-16T01:49:01.039415+0000","last_fullsized":"2024-09-16T01:49:01.039415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039105+0000","last_change":"2024-09-16T01:48:58.842316+0000","last_active":"2024-09-16T01:49:01.039105+0000","last_peered":"2024-09-16T01:49:01.039105+0000","last_clean":"2024-09-16T01:49:01.039105+0000","last_became_active":"2024-09-16T01:48:58.842053+0000","last_became_peered":"2024-09-16T01:48:58.842053+0000","last_unstale":"2024-09-16T01:49:01.039105+0000","last_undegraded":"2024-09-16T01:49:01.039105+0000","last_fullsized":"2024-09-16T01:49:01.039105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038994+0000","last_change":"2024-09-16T01:48:58.834417+0000","last_active":"2024-09-16T01:49:01.038994+0000","last_peered":"2024-09-16T01:49:01.038994+0000","last_clean":"2024-09-16T01:49:01.038994+0000","last_became_active":"2024-09-16T01:48:58.829945+0000","last_became_peered":"2024-09-16T01:48:58.829945+0000","last_unstale":"2024-09-16T01:49:01.038994+0000","last_undegraded":"2024-09-16T01:49:01.038994+0000","last_fullsized":"2024-09-16T01:49:01.038994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825793+0000","last_change":"2024-09-16T01:48:58.827459+0000","last_active":"2024-09-16T01:48:59.825793+0000","last_peered":"2024-09-16T01:48:59.825793+0000","last_clean":"2024-09-16T01:48:59.825793+0000","last_became_active":"2024-09-16T01:48:58.826394+0000","last_became_peered":"2024-09-16T01:48:58.826394+0000","last_unstale":"2024-09-16T01:48:59.825793+0000","last_undegraded":"2024-09-16T01:48:59.825793+0000","last_fullsized":"2024-09-16T01:48:59.825793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:47.685752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825485+0000","last_change":"2024-09-16T01:48:58.825966+0000","last_active":"2024-09-16T01:48:59.825485+0000","last_peered":"2024-09-16T01:48:59.825485+0000","last_clean":"2024-09-16T01:48:59.825485+0000","last_became_active":"2024-09-16T01:48:58.825539+0000","last_became_peered":"2024-09-16T01:48:58.825539+0000","last_unstale":"2024-09-16T01:48:59.825485+0000","last_undegraded":"2024-09-16T01:48:59.825485+0000","last_fullsized":"2024-09-16T01:48:59.825485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:41.220686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038386+0000","last_change":"2024-09-16T01:48:58.828877+0000","last_active":"2024-09-16T01:49:01.038386+0000","last_peered":"2024-09-16T01:49:01.038386+0000","last_clean":"2024-09-16T01:49:01.038386+0000","last_became_active":"2024-09-16T01:48:58.828596+0000","last_became_peered":"2024-09-16T01:48:58.828596+0000","last_unstale":"2024-09-16T01:49:01.038386+0000","last_undegraded":"2024-09-16T01:49:01.038386+0000","last_fullsized":"2024-09-16T01:49:01.038386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:09.260093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038366+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:49:01.038366+0000","last_peered":"2024-09-16T01:49:01.038366+0000","last_clean":"2024-09-16T01:49:01.038366+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:49:01.038366+0000","last_undegraded":"2024-09-16T01:49:01.038366+0000","last_fullsized":"2024-09-16T01:49:01.038366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040029+0000","last_change":"2024-09-16T01:48:58.835435+0000","last_active":"2024-09-16T01:49:01.040029+0000","last_peered":"2024-09-16T01:49:01.040029+0000","last_clean":"2024-09-16T01:49:01.040029+0000","last_became_active":"2024-09-16T01:48:58.834336+0000","last_became_peered":"2024-09-16T01:48:58.834336+0000","last_unstale":"2024-09-16T01:49:01.040029+0000","last_undegraded":"2024-09-16T01:49:01.040029+0000","last_fullsized":"2024-09-16T01:49:01.040029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:52.620859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038850+0000","last_change":"2024-09-16T01:48:58.832278+0000","last_active":"2024-09-16T01:49:01.038850+0000","last_peered":"2024-09-16T01:49:01.038850+0000","last_clean":"2024-09-16T01:49:01.038850+0000","last_became_active":"2024-09-16T01:48:58.832006+0000","last_became_peered":"2024-09-16T01:48:58.832006+0000","last_unstale":"2024-09-16T01:49:01.038850+0000","last_undegraded":"2024-09-16T01:49:01.038850+0000","last_fullsized":"2024-09-16T01:49:01.038850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040246+0000","last_change":"2024-09-16T01:48:58.829765+0000","last_active":"2024-09-16T01:49:01.040246+0000","last_peered":"2024-09-16T01:49:01.040246+0000","last_clean":"2024-09-16T01:49:01.040246+0000","last_became_active":"2024-09-16T01:48:58.829499+0000","last_became_peered":"2024-09-16T01:48:58.829499+0000","last_unstale":"2024-09-16T01:49:01.040246+0000","last_undegraded":"2024-09-16T01:49:01.040246+0000","last_fullsized":"2024-09-16T01:49:01.040246+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:15.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039007+0000","last_change":"2024-09-16T01:48:58.840676+0000","last_active":"2024-09-16T01:49:01.039007+0000","last_peered":"2024-09-16T01:49:01.039007+0000","last_clean":"2024-09-16T01:49:01.039007+0000","last_became_active":"2024-09-16T01:48:58.840441+0000","last_became_peered":"2024-09-16T01:48:58.840441+0000","last_unstale":"2024-09-16T01:49:01.039007+0000","last_undegraded":"2024-09-16T01:49:01.039007+0000","last_fullsized":"2024-09-16T01:49:01.039007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824333+0000","last_change":"2024-09-16T01:48:58.835972+0000","last_active":"2024-09-16T01:48:59.824333+0000","last_peered":"2024-09-16T01:48:59.824333+0000","last_clean":"2024-09-16T01:48:59.824333+0000","last_became_active":"2024-09-16T01:48:58.835678+0000","last_became_peered":"2024-09-16T01:48:58.835678+0000","last_unstale":"2024-09-16T01:48:59.824333+0000","last_undegraded":"2024-09-16T01:48:59.824333+0000","last_fullsized":"2024-09-16T01:48:59.824333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:54.394499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040447+0000","last_change":"2024-09-16T01:48:58.825281+0000","last_active":"2024-09-16T01:49:01.040447+0000","last_peered":"2024-09-16T01:49:01.040447+0000","last_clean":"2024-09-16T01:49:01.040447+0000","last_became_active":"2024-09-16T01:48:58.824946+0000","last_became_peered":"2024-09-16T01:48:58.824946+0000","last_unstale":"2024-09-16T01:49:01.040447+0000","last_undegraded":"2024-09-16T01:49:01.040447+0000","last_fullsized":"2024-09-16T01:49:01.040447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:00.162120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038902+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:49:01.038902+0000","last_peered":"2024-09-16T01:49:01.038902+0000","last_clean":"2024-09-16T01:49:01.038902+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:49:01.038902+0000","last_undegraded":"2024-09-16T01:49:01.038902+0000","last_fullsized":"2024-09-16T01:49:01.038902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038514+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:49:01.038514+0000","last_peered":"2024-09-16T01:49:01.038514+0000","last_clean":"2024-09-16T01:49:01.038514+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:49:01.038514+0000","last_undegraded":"2024-09-16T01:49:01.038514+0000","last_fullsized":"2024-09-16T01:49:01.038514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039312+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:49:01.039312+0000","last_peered":"2024-09-16T01:49:01.039312+0000","last_clean":"2024-09-16T01:49:01.039312+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:49:01.039312+0000","last_undegraded":"2024-09-16T01:49:01.039312+0000","last_fullsized":"2024-09-16T01:49:01.039312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824280+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:48:59.824280+0000","last_peered":"2024-09-16T01:48:59.824280+0000","last_clean":"2024-09-16T01:48:59.824280+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:48:59.824280+0000","last_undegraded":"2024-09-16T01:48:59.824280+0000","last_fullsized":"2024-09-16T01:48:59.824280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825795+0000","last_change":"2024-09-16T01:48:58.828564+0000","last_active":"2024-09-16T01:48:59.825795+0000","last_peered":"2024-09-16T01:48:59.825795+0000","last_clean":"2024-09-16T01:48:59.825795+0000","last_became_active":"2024-09-16T01:48:58.827235+0000","last_became_peered":"2024-09-16T01:48:58.827235+0000","last_unstale":"2024-09-16T01:48:59.825795+0000","last_undegraded":"2024-09-16T01:48:59.825795+0000","last_fullsized":"2024-09-16T01:48:59.825795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:32.154501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825659+0000","last_change":"2024-09-16T01:48:58.838396+0000","last_active":"2024-09-16T01:48:59.825659+0000","last_peered":"2024-09-16T01:48:59.825659+0000","last_clean":"2024-09-16T01:48:59.825659+0000","last_became_active":"2024-09-16T01:48:58.838055+0000","last_became_peered":"2024-09-16T01:48:58.838055+0000","last_unstale":"2024-09-16T01:48:59.825659+0000","last_undegraded":"2024-09-16T01:48:59.825659+0000","last_fullsized":"2024-09-16T01:48:59.825659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:21.170063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038909+0000","last_change":"2024-09-16T01:48:58.827677+0000","last_active":"2024-09-16T01:49:01.038909+0000","last_peered":"2024-09-16T01:49:01.038909+0000","last_clean":"2024-09-16T01:49:01.038909+0000","last_became_active":"2024-09-16T01:48:58.827294+0000","last_became_peered":"2024-09-16T01:48:58.827294+0000","last_unstale":"2024-09-16T01:49:01.038909+0000","last_undegraded":"2024-09-16T01:49:01.038909+0000","last_fullsized":"2024-09-16T01:49:01.038909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:03.532878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038995+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:49:01.038995+0000","last_peered":"2024-09-16T01:49:01.038995+0000","last_clean":"2024-09-16T01:49:01.038995+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:49:01.038995+0000","last_undegraded":"2024-09-16T01:49:01.038995+0000","last_fullsized":"2024-09-16T01:49:01.038995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038603+0000","last_change":"2024-09-16T01:48:58.825892+0000","last_active":"2024-09-16T01:49:01.038603+0000","last_peered":"2024-09-16T01:49:01.038603+0000","last_clean":"2024-09-16T01:49:01.038603+0000","last_became_active":"2024-09-16T01:48:58.825589+0000","last_became_peered":"2024-09-16T01:48:58.825589+0000","last_unstale":"2024-09-16T01:49:01.038603+0000","last_undegraded":"2024-09-16T01:49:01.038603+0000","last_fullsized":"2024-09-16T01:49:01.038603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.823887+0000","last_change":"2024-09-16T01:48:58.826211+0000","last_active":"2024-09-16T01:48:59.823887+0000","last_peered":"2024-09-16T01:48:59.823887+0000","last_clean":"2024-09-16T01:48:59.823887+0000","last_became_active":"2024-09-16T01:48:58.825739+0000","last_became_peered":"2024-09-16T01:48:58.825739+0000","last_unstale":"2024-09-16T01:48:59.823887+0000","last_undegraded":"2024-09-16T01:48:59.823887+0000","last_fullsized":"2024-09-16T01:48:59.823887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:22.083769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040379+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:49:01.040379+0000","last_peered":"2024-09-16T01:49:01.040379+0000","last_clean":"2024-09-16T01:49:01.040379+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:49:01.040379+0000","last_undegraded":"2024-09-16T01:49:01.040379+0000","last_fullsized":"2024-09-16T01:49:01.040379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.863866+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:49:08.863866+0000","last_peered":"2024-09-16T01:49:08.863866+0000","last_clean":"2024-09-16T01:49:08.863866+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:49:08.863866+0000","last_undegraded":"2024-09-16T01:49:08.863866+0000","last_fullsized":"2024-09-16T01:49:08.863866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824444+0000","last_change":"2024-09-16T01:48:58.838352+0000","last_active":"2024-09-16T01:48:59.824444+0000","last_peered":"2024-09-16T01:48:59.824444+0000","last_clean":"2024-09-16T01:48:59.824444+0000","last_became_active":"2024-09-16T01:48:58.838047+0000","last_became_peered":"2024-09-16T01:48:58.838047+0000","last_unstale":"2024-09-16T01:48:59.824444+0000","last_undegraded":"2024-09-16T01:48:59.824444+0000","last_fullsized":"2024-09-16T01:48:59.824444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:21.869634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917528+0000","last_change":"2024-09-16T01:48:57.822919+0000","last_active":"2024-09-16T01:49:20.917528+0000","last_peered":"2024-09-16T01:49:20.917528+0000","last_clean":"2024-09-16T01:49:20.917528+0000","last_became_active":"2024-09-16T01:48:57.822556+0000","last_became_peered":"2024-09-16T01:48:57.822556+0000","last_unstale":"2024-09-16T01:49:20.917528+0000","last_undegraded":"2024-09-16T01:49:20.917528+0000","last_fullsized":"2024-09-16T01:49:20.917528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:00.230928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.901405+0000","last_change":"2024-09-16T01:48:57.817340+0000","last_active":"2024-09-16T01:49:14.901405+0000","last_peered":"2024-09-16T01:49:14.901405+0000","last_clean":"2024-09-16T01:49:14.901405+0000","last_became_active":"2024-09-16T01:48:57.817051+0000","last_became_peered":"2024-09-16T01:48:57.817051+0000","last_unstale":"2024-09-16T01:49:14.901405+0000","last_undegraded":"2024-09-16T01:49:14.901405+0000","last_fullsized":"2024-09-16T01:49:14.901405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038968+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:49:01.038968+0000","last_peered":"2024-09-16T01:49:01.038968+0000","last_clean":"2024-09-16T01:49:01.038968+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:49:01.038968+0000","last_undegraded":"2024-09-16T01:49:01.038968+0000","last_fullsized":"2024-09-16T01:49:01.038968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918636+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:49:20.918636+0000","last_peered":"2024-09-16T01:49:20.918636+0000","last_clean":"2024-09-16T01:49:20.918636+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:49:20.918636+0000","last_undegraded":"2024-09-16T01:49:20.918636+0000","last_fullsized":"2024-09-16T01:49:20.918636+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824266+0000","last_change":"2024-09-16T01:48:58.828330+0000","last_active":"2024-09-16T01:48:59.824266+0000","last_peered":"2024-09-16T01:48:59.824266+0000","last_clean":"2024-09-16T01:48:59.824266+0000","last_became_active":"2024-09-16T01:48:58.827988+0000","last_became_peered":"2024-09-16T01:48:58.827988+0000","last_unstale":"2024-09-16T01:48:59.824266+0000","last_undegraded":"2024-09-16T01:48:59.824266+0000","last_fullsized":"2024-09-16T01:48:59.824266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:31.622539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.866956+0000","last_change":"2024-09-16T01:48:57.799212+0000","last_active":"2024-09-16T01:49:08.866956+0000","last_peered":"2024-09-16T01:49:08.866956+0000","last_clean":"2024-09-16T01:49:08.866956+0000","last_became_active":"2024-09-16T01:48:57.798917+0000","last_became_peered":"2024-09-16T01:48:57.798917+0000","last_unstale":"2024-09-16T01:49:08.866956+0000","last_undegraded":"2024-09-16T01:49:08.866956+0000","last_fullsized":"2024-09-16T01:49:08.866956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824375+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:48:59.824375+0000","last_peered":"2024-09-16T01:48:59.824375+0000","last_clean":"2024-09-16T01:48:59.824375+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:48:59.824375+0000","last_undegraded":"2024-09-16T01:48:59.824375+0000","last_fullsized":"2024-09-16T01:48:59.824375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.941828+0000","last_change":"2024-09-16T01:48:57.824637+0000","last_active":"2024-09-16T01:49:25.941828+0000","last_peered":"2024-09-16T01:49:25.941828+0000","last_clean":"2024-09-16T01:49:25.941828+0000","last_became_active":"2024-09-16T01:48:57.824388+0000","last_became_peered":"2024-09-16T01:48:57.824388+0000","last_unstale":"2024-09-16T01:49:25.941828+0000","last_undegraded":"2024-09-16T01:49:25.941828+0000","last_fullsized":"2024-09-16T01:49:25.941828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039431+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:49:01.039431+0000","last_peered":"2024-09-16T01:49:01.039431+0000","last_clean":"2024-09-16T01:49:01.039431+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:49:01.039431+0000","last_undegraded":"2024-09-16T01:49:01.039431+0000","last_fullsized":"2024-09-16T01:49:01.039431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.863572+0000","last_change":"2024-09-16T01:48:57.804123+0000","last_active":"2024-09-16T01:49:08.863572+0000","last_peered":"2024-09-16T01:49:08.863572+0000","last_clean":"2024-09-16T01:49:08.863572+0000","last_became_active":"2024-09-16T01:48:57.803337+0000","last_became_peered":"2024-09-16T01:48:57.803337+0000","last_unstale":"2024-09-16T01:49:08.863572+0000","last_undegraded":"2024-09-16T01:49:08.863572+0000","last_fullsized":"2024-09-16T01:49:08.863572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039746+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:49:01.039746+0000","last_peered":"2024-09-16T01:49:01.039746+0000","last_clean":"2024-09-16T01:49:01.039746+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:49:01.039746+0000","last_undegraded":"2024-09-16T01:49:01.039746+0000","last_fullsized":"2024-09-16T01:49:01.039746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:36.354725+0000","last_change":"2024-09-16T01:48:57.809143+0000","last_active":"2024-09-16T01:49:36.354725+0000","last_peered":"2024-09-16T01:49:36.354725+0000","last_clean":"2024-09-16T01:49:36.354725+0000","last_became_active":"2024-09-16T01:48:57.805959+0000","last_became_peered":"2024-09-16T01:48:57.805959+0000","last_unstale":"2024-09-16T01:49:36.354725+0000","last_undegraded":"2024-09-16T01:49:36.354725+0000","last_fullsized":"2024-09-16T01:49:36.354725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:47.945990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1871,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039283+0000","last_change":"2024-09-16T01:48:58.836179+0000","last_active":"2024-09-16T01:49:01.039283+0000","last_peered":"2024-09-16T01:49:01.039283+0000","last_clean":"2024-09-16T01:49:01.039283+0000","last_became_active":"2024-09-16T01:48:58.833712+0000","last_became_peered":"2024-09-16T01:48:58.833712+0000","last_unstale":"2024-09-16T01:49:01.039283+0000","last_undegraded":"2024-09-16T01:49:01.039283+0000","last_fullsized":"2024-09-16T01:49:01.039283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.940548+0000","last_change":"2024-09-16T01:48:57.811512+0000","last_active":"2024-09-16T01:49:25.940548+0000","last_peered":"2024-09-16T01:49:25.940548+0000","last_clean":"2024-09-16T01:49:25.940548+0000","last_became_active":"2024-09-16T01:48:57.811153+0000","last_became_peered":"2024-09-16T01:48:57.811153+0000","last_unstale":"2024-09-16T01:49:25.940548+0000","last_undegraded":"2024-09-16T01:49:25.940548+0000","last_fullsized":"2024-09-16T01:49:25.940548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825560+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:48:59.825560+0000","last_peered":"2024-09-16T01:48:59.825560+0000","last_clean":"2024-09-16T01:48:59.825560+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:48:59.825560+0000","last_undegraded":"2024-09-16T01:48:59.825560+0000","last_fullsized":"2024-09-16T01:48:59.825560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918151+0000","last_change":"2024-09-16T01:48:57.810678+0000","last_active":"2024-09-16T01:49:20.918151+0000","last_peered":"2024-09-16T01:49:20.918151+0000","last_clean":"2024-09-16T01:49:20.918151+0000","last_became_active":"2024-09-16T01:48:57.809489+0000","last_became_peered":"2024-09-16T01:48:57.809489+0000","last_unstale":"2024-09-16T01:49:20.918151+0000","last_undegraded":"2024-09-16T01:49:20.918151+0000","last_fullsized":"2024-09-16T01:49:20.918151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:20.163119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039730+0000","last_change":"2024-09-16T01:48:58.831719+0000","last_active":"2024-09-16T01:49:01.039730+0000","last_peered":"2024-09-16T01:49:01.039730+0000","last_clean":"2024-09-16T01:49:01.039730+0000","last_became_active":"2024-09-16T01:48:58.831446+0000","last_became_peered":"2024-09-16T01:48:58.831446+0000","last_unstale":"2024-09-16T01:49:01.039730+0000","last_undegraded":"2024-09-16T01:49:01.039730+0000","last_fullsized":"2024-09-16T01:49:01.039730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:21:57.461246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.918952+0000","last_change":"2024-09-16T01:48:57.821124+0000","last_active":"2024-09-16T01:49:20.918952+0000","last_peered":"2024-09-16T01:49:20.918952+0000","last_clean":"2024-09-16T01:49:20.918952+0000","last_became_active":"2024-09-16T01:48:57.820814+0000","last_became_peered":"2024-09-16T01:48:57.820814+0000","last_unstale":"2024-09-16T01:49:20.918952+0000","last_undegraded":"2024-09-16T01:49:20.918952+0000","last_fullsized":"2024-09-16T01:49:20.918952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:42.831872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039751+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:49:01.039751+0000","last_peered":"2024-09-16T01:49:01.039751+0000","last_clean":"2024-09-16T01:49:01.039751+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:49:01.039751+0000","last_undegraded":"2024-09-16T01:49:01.039751+0000","last_fullsized":"2024-09-16T01:49:01.039751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.916510+0000","last_change":"2024-09-16T01:48:57.804686+0000","last_active":"2024-09-16T01:49:20.916510+0000","last_peered":"2024-09-16T01:49:20.916510+0000","last_clean":"2024-09-16T01:49:20.916510+0000","last_became_active":"2024-09-16T01:48:57.804374+0000","last_became_peered":"2024-09-16T01:48:57.804374+0000","last_unstale":"2024-09-16T01:49:20.916510+0000","last_undegraded":"2024-09-16T01:49:20.916510+0000","last_fullsized":"2024-09-16T01:49:20.916510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:44.257949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038452+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:49:01.038452+0000","last_peered":"2024-09-16T01:49:01.038452+0000","last_clean":"2024-09-16T01:49:01.038452+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:49:01.038452+0000","last_undegraded":"2024-09-16T01:49:01.038452+0000","last_fullsized":"2024-09-16T01:49:01.038452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.829332+0000","last_change":"2024-09-16T01:48:57.819384+0000","last_active":"2024-09-16T01:49:03.829332+0000","last_peered":"2024-09-16T01:49:03.829332+0000","last_clean":"2024-09-16T01:49:03.829332+0000","last_became_active":"2024-09-16T01:48:57.819122+0000","last_became_peered":"2024-09-16T01:48:57.819122+0000","last_unstale":"2024-09-16T01:49:03.829332+0000","last_undegraded":"2024-09-16T01:49:03.829332+0000","last_fullsized":"2024-09-16T01:49:03.829332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824405+0000","last_change":"2024-09-16T01:48:58.833884+0000","last_active":"2024-09-16T01:48:59.824405+0000","last_peered":"2024-09-16T01:48:59.824405+0000","last_clean":"2024-09-16T01:48:59.824405+0000","last_became_active":"2024-09-16T01:48:58.830361+0000","last_became_peered":"2024-09-16T01:48:58.830361+0000","last_unstale":"2024-09-16T01:48:59.824405+0000","last_undegraded":"2024-09-16T01:48:59.824405+0000","last_fullsized":"2024-09-16T01:48:59.824405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:22.709795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.939777+0000","last_change":"2024-09-16T01:48:57.807506+0000","last_active":"2024-09-16T01:49:25.939777+0000","last_peered":"2024-09-16T01:49:25.939777+0000","last_clean":"2024-09-16T01:49:25.939777+0000","last_became_active":"2024-09-16T01:48:57.807210+0000","last_became_peered":"2024-09-16T01:48:57.807210+0000","last_unstale":"2024-09-16T01:49:25.939777+0000","last_undegraded":"2024-09-16T01:49:25.939777+0000","last_fullsized":"2024-09-16T01:49:25.939777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:41.749202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039929+0000","last_change":"2024-09-16T01:48:58.820141+0000","last_active":"2024-09-16T01:49:01.039929+0000","last_peered":"2024-09-16T01:49:01.039929+0000","last_clean":"2024-09-16T01:49:01.039929+0000","last_became_active":"2024-09-16T01:48:58.819801+0000","last_became_peered":"2024-09-16T01:48:58.819801+0000","last_unstale":"2024-09-16T01:49:01.039929+0000","last_undegraded":"2024-09-16T01:49:01.039929+0000","last_fullsized":"2024-09-16T01:49:01.039929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917084+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:49:20.917084+0000","last_peered":"2024-09-16T01:49:20.917084+0000","last_clean":"2024-09-16T01:49:20.917084+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:49:20.917084+0000","last_undegraded":"2024-09-16T01:49:20.917084+0000","last_fullsized":"2024-09-16T01:49:20.917084+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039154+0000","last_change":"2024-09-16T01:48:58.829614+0000","last_active":"2024-09-16T01:49:01.039154+0000","last_peered":"2024-09-16T01:49:01.039154+0000","last_clean":"2024-09-16T01:49:01.039154+0000","last_became_active":"2024-09-16T01:48:58.829315+0000","last_became_peered":"2024-09-16T01:48:58.829315+0000","last_unstale":"2024-09-16T01:49:01.039154+0000","last_undegraded":"2024-09-16T01:49:01.039154+0000","last_fullsized":"2024-09-16T01:49:01.039154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:13.507324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.916463+0000","last_change":"2024-09-16T01:48:57.821741+0000","last_active":"2024-09-16T01:49:20.916463+0000","last_peered":"2024-09-16T01:49:20.916463+0000","last_clean":"2024-09-16T01:49:20.916463+0000","last_became_active":"2024-09-16T01:48:57.821383+0000","last_became_peered":"2024-09-16T01:48:57.821383+0000","last_unstale":"2024-09-16T01:49:20.916463+0000","last_undegraded":"2024-09-16T01:49:20.916463+0000","last_fullsized":"2024-09-16T01:49:20.916463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040242+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:49:01.040242+0000","last_peered":"2024-09-16T01:49:01.040242+0000","last_clean":"2024-09-16T01:49:01.040242+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:49:01.040242+0000","last_undegraded":"2024-09-16T01:49:01.040242+0000","last_fullsized":"2024-09-16T01:49:01.040242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039845+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:49:01.039845+0000","last_peered":"2024-09-16T01:49:01.039845+0000","last_clean":"2024-09-16T01:49:01.039845+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:49:01.039845+0000","last_undegraded":"2024-09-16T01:49:01.039845+0000","last_fullsized":"2024-09-16T01:49:01.039845+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917307+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:49:20.917307+0000","last_peered":"2024-09-16T01:49:20.917307+0000","last_clean":"2024-09-16T01:49:20.917307+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:49:20.917307+0000","last_undegraded":"2024-09-16T01:49:20.917307+0000","last_fullsized":"2024-09-16T01:49:20.917307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824354+0000","last_change":"2024-09-16T01:48:58.836061+0000","last_active":"2024-09-16T01:48:59.824354+0000","last_peered":"2024-09-16T01:48:59.824354+0000","last_clean":"2024-09-16T01:48:59.824354+0000","last_became_active":"2024-09-16T01:48:58.835813+0000","last_became_peered":"2024-09-16T01:48:58.835813+0000","last_unstale":"2024-09-16T01:48:59.824354+0000","last_undegraded":"2024-09-16T01:48:59.824354+0000","last_fullsized":"2024-09-16T01:48:59.824354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:03.840176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.903518+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:49:14.903518+0000","last_peered":"2024-09-16T01:49:14.903518+0000","last_clean":"2024-09-16T01:49:14.903518+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:49:14.903518+0000","last_undegraded":"2024-09-16T01:49:14.903518+0000","last_fullsized":"2024-09-16T01:49:14.903518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825452+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:48:59.825452+0000","last_peered":"2024-09-16T01:48:59.825452+0000","last_clean":"2024-09-16T01:48:59.825452+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:48:59.825452+0000","last_undegraded":"2024-09-16T01:48:59.825452+0000","last_fullsized":"2024-09-16T01:48:59.825452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:14.903592+0000","last_change":"2024-09-16T01:48:57.809778+0000","last_active":"2024-09-16T01:49:14.903592+0000","last_peered":"2024-09-16T01:49:14.903592+0000","last_clean":"2024-09-16T01:49:14.903592+0000","last_became_active":"2024-09-16T01:48:57.809455+0000","last_became_peered":"2024-09-16T01:48:57.809455+0000","last_unstale":"2024-09-16T01:49:14.903592+0000","last_undegraded":"2024-09-16T01:49:14.903592+0000","last_fullsized":"2024-09-16T01:49:14.903592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038525+0000","last_change":"2024-09-16T01:48:58.828651+0000","last_active":"2024-09-16T01:49:01.038525+0000","last_peered":"2024-09-16T01:49:01.038525+0000","last_clean":"2024-09-16T01:49:01.038525+0000","last_became_active":"2024-09-16T01:48:58.827662+0000","last_became_peered":"2024-09-16T01:48:58.827662+0000","last_unstale":"2024-09-16T01:49:01.038525+0000","last_undegraded":"2024-09-16T01:49:01.038525+0000","last_fullsized":"2024-09-16T01:49:01.038525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:20.864782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.941526+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:49:25.941526+0000","last_peered":"2024-09-16T01:49:25.941526+0000","last_clean":"2024-09-16T01:49:25.941526+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:49:25.941526+0000","last_undegraded":"2024-09-16T01:49:25.941526+0000","last_fullsized":"2024-09-16T01:49:25.941526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039202+0000","last_change":"2024-09-16T01:48:58.836793+0000","last_active":"2024-09-16T01:49:01.039202+0000","last_peered":"2024-09-16T01:49:01.039202+0000","last_clean":"2024-09-16T01:49:01.039202+0000","last_became_active":"2024-09-16T01:48:58.833946+0000","last_became_peered":"2024-09-16T01:48:58.833946+0000","last_unstale":"2024-09-16T01:49:01.039202+0000","last_undegraded":"2024-09-16T01:49:01.039202+0000","last_fullsized":"2024-09-16T01:49:01.039202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039165+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:49:01.039165+0000","last_peered":"2024-09-16T01:49:01.039165+0000","last_clean":"2024-09-16T01:49:01.039165+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:49:01.039165+0000","last_undegraded":"2024-09-16T01:49:01.039165+0000","last_fullsized":"2024-09-16T01:49:01.039165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824289+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:48:59.824289+0000","last_peered":"2024-09-16T01:48:59.824289+0000","last_clean":"2024-09-16T01:48:59.824289+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:48:59.824289+0000","last_undegraded":"2024-09-16T01:48:59.824289+0000","last_fullsized":"2024-09-16T01:48:59.824289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.917502+0000","last_change":"2024-09-16T01:48:57.816552+0000","last_active":"2024-09-16T01:49:20.917502+0000","last_peered":"2024-09-16T01:49:20.917502+0000","last_clean":"2024-09-16T01:49:20.917502+0000","last_became_active":"2024-09-16T01:48:57.816191+0000","last_became_peered":"2024-09-16T01:48:57.816191+0000","last_unstale":"2024-09-16T01:49:20.917502+0000","last_undegraded":"2024-09-16T01:49:20.917502+0000","last_fullsized":"2024-09-16T01:49:20.917502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:59.155177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.825818+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:48:59.825818+0000","last_peered":"2024-09-16T01:48:59.825818+0000","last_clean":"2024-09-16T01:48:59.825818+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:48:59.825818+0000","last_undegraded":"2024-09-16T01:48:59.825818+0000","last_fullsized":"2024-09-16T01:48:59.825818+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.919024+0000","last_change":"2024-09-16T01:48:57.821684+0000","last_active":"2024-09-16T01:49:20.919024+0000","last_peered":"2024-09-16T01:49:20.919024+0000","last_clean":"2024-09-16T01:49:20.919024+0000","last_became_active":"2024-09-16T01:48:57.821385+0000","last_became_peered":"2024-09-16T01:48:57.821385+0000","last_unstale":"2024-09-16T01:49:20.919024+0000","last_undegraded":"2024-09-16T01:49:20.919024+0000","last_fullsized":"2024-09-16T01:49:20.919024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:08.298822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824807+0000","last_change":"2024-09-16T01:48:58.829723+0000","last_active":"2024-09-16T01:48:59.824807+0000","last_peered":"2024-09-16T01:48:59.824807+0000","last_clean":"2024-09-16T01:48:59.824807+0000","last_became_active":"2024-09-16T01:48:58.829456+0000","last_became_peered":"2024-09-16T01:48:58.829456+0000","last_unstale":"2024-09-16T01:48:59.824807+0000","last_undegraded":"2024-09-16T01:48:59.824807+0000","last_fullsized":"2024-09-16T01:48:59.824807+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:08:09.528402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.945020+0000","last_change":"2024-09-16T01:48:57.822361+0000","last_active":"2024-09-16T01:49:25.945020+0000","last_peered":"2024-09-16T01:49:25.945020+0000","last_clean":"2024-09-16T01:49:25.945020+0000","last_became_active":"2024-09-16T01:48:57.822143+0000","last_became_peered":"2024-09-16T01:48:57.822143+0000","last_unstale":"2024-09-16T01:49:25.945020+0000","last_undegraded":"2024-09-16T01:49:25.945020+0000","last_fullsized":"2024-09-16T01:49:25.945020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:35:31.147558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824748+0000","last_change":"2024-09-16T01:48:58.827035+0000","last_active":"2024-09-16T01:48:59.824748+0000","last_peered":"2024-09-16T01:48:59.824748+0000","last_clean":"2024-09-16T01:48:59.824748+0000","last_became_active":"2024-09-16T01:48:58.826718+0000","last_became_peered":"2024-09-16T01:48:58.826718+0000","last_unstale":"2024-09-16T01:48:59.824748+0000","last_undegraded":"2024-09-16T01:48:59.824748+0000","last_fullsized":"2024-09-16T01:48:59.824748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:51:26.788666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.874853+0000","last_change":"2024-09-16T01:48:57.822511+0000","last_active":"2024-09-16T01:49:08.874853+0000","last_peered":"2024-09-16T01:49:08.874853+0000","last_clean":"2024-09-16T01:49:08.874853+0000","last_became_active":"2024-09-16T01:48:57.820900+0000","last_became_peered":"2024-09-16T01:48:57.820900+0000","last_unstale":"2024-09-16T01:49:08.874853+0000","last_undegraded":"2024-09-16T01:49:08.874853+0000","last_fullsized":"2024-09-16T01:49:08.874853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038471+0000","last_change":"2024-09-16T01:48:58.829321+0000","last_active":"2024-09-16T01:49:01.038471+0000","last_peered":"2024-09-16T01:49:01.038471+0000","last_clean":"2024-09-16T01:49:01.038471+0000","last_became_active":"2024-09-16T01:48:58.827552+0000","last_became_peered":"2024-09-16T01:48:58.827552+0000","last_unstale":"2024-09-16T01:49:01.038471+0000","last_undegraded":"2024-09-16T01:49:01.038471+0000","last_fullsized":"2024-09-16T01:49:01.038471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:32.910546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.944989+0000","last_change":"2024-09-16T01:48:57.815365+0000","last_active":"2024-09-16T01:49:25.944989+0000","last_peered":"2024-09-16T01:49:25.944989+0000","last_clean":"2024-09-16T01:49:25.944989+0000","last_became_active":"2024-09-16T01:48:57.815052+0000","last_became_peered":"2024-09-16T01:48:57.815052+0000","last_unstale":"2024-09-16T01:49:25.944989+0000","last_undegraded":"2024-09-16T01:49:25.944989+0000","last_fullsized":"2024-09-16T01:49:25.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:21.528570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:48:59.824251+0000","last_change":"2024-09-16T01:48:58.827626+0000","last_active":"2024-09-16T01:48:59.824251+0000","last_peered":"2024-09-16T01:48:59.824251+0000","last_clean":"2024-09-16T01:48:59.824251+0000","last_became_active":"2024-09-16T01:48:58.827375+0000","last_became_peered":"2024-09-16T01:48:58.827375+0000","last_unstale":"2024-09-16T01:48:59.824251+0000","last_undegraded":"2024-09-16T01:48:59.824251+0000","last_fullsized":"2024-09-16T01:48:59.824251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:53:26.577467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:20.914366+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:49:20.914366+0000","last_peered":"2024-09-16T01:49:20.914366+0000","last_clean":"2024-09-16T01:49:20.914366+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:49:20.914366+0000","last_undegraded":"2024-09-16T01:49:20.914366+0000","last_fullsized":"2024-09-16T01:49:20.914366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.038826+0000","last_change":"2024-09-16T01:48:58.825853+0000","last_active":"2024-09-16T01:49:01.038826+0000","last_peered":"2024-09-16T01:49:01.038826+0000","last_clean":"2024-09-16T01:49:01.038826+0000","last_became_active":"2024-09-16T01:48:58.825343+0000","last_became_peered":"2024-09-16T01:48:58.825343+0000","last_unstale":"2024-09-16T01:49:01.038826+0000","last_undegraded":"2024-09-16T01:49:01.038826+0000","last_fullsized":"2024-09-16T01:49:01.038826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:08.887158+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:49:08.887158+0000","last_peered":"2024-09-16T01:49:08.887158+0000","last_clean":"2024-09-16T01:49:08.887158+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:49:08.887158+0000","last_undegraded":"2024-09-16T01:49:08.887158+0000","last_fullsized":"2024-09-16T01:49:08.887158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039191+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:49:01.039191+0000","last_peered":"2024-09-16T01:49:01.039191+0000","last_clean":"2024-09-16T01:49:01.039191+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:49:01.039191+0000","last_undegraded":"2024-09-16T01:49:01.039191+0000","last_fullsized":"2024-09-16T01:49:01.039191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040060+0000","last_change":"2024-09-16T01:48:58.837478+0000","last_active":"2024-09-16T01:49:01.040060+0000","last_peered":"2024-09-16T01:49:01.040060+0000","last_clean":"2024-09-16T01:49:01.040060+0000","last_became_active":"2024-09-16T01:48:58.837160+0000","last_became_peered":"2024-09-16T01:48:58.837160+0000","last_unstale":"2024-09-16T01:49:01.040060+0000","last_undegraded":"2024-09-16T01:49:01.040060+0000","last_fullsized":"2024-09-16T01:49:01.040060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:20.671303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.942546+0000","last_change":"2024-09-16T01:48:57.822430+0000","last_active":"2024-09-16T01:49:25.942546+0000","last_peered":"2024-09-16T01:49:25.942546+0000","last_clean":"2024-09-16T01:49:25.942546+0000","last_became_active":"2024-09-16T01:48:57.821970+0000","last_became_peered":"2024-09-16T01:48:57.821970+0000","last_unstale":"2024-09-16T01:49:25.942546+0000","last_undegraded":"2024-09-16T01:49:25.942546+0000","last_fullsized":"2024-09-16T01:49:25.942546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:19.990505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.040444+0000","last_change":"2024-09-16T01:48:58.836375+0000","last_active":"2024-09-16T01:49:01.040444+0000","last_peered":"2024-09-16T01:49:01.040444+0000","last_clean":"2024-09-16T01:49:01.040444+0000","last_became_active":"2024-09-16T01:48:58.836077+0000","last_became_peered":"2024-09-16T01:48:58.836077+0000","last_unstale":"2024-09-16T01:49:01.040444+0000","last_undegraded":"2024-09-16T01:49:01.040444+0000","last_fullsized":"2024-09-16T01:49:01.040444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:20.350285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'12","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.945520+0000","last_change":"2024-09-16T01:48:57.825178+0000","last_active":"2024-09-16T01:49:25.945520+0000","last_peered":"2024-09-16T01:49:25.945520+0000","last_clean":"2024-09-16T01:49:25.945520+0000","last_became_active":"2024-09-16T01:48:57.824946+0000","last_became_peered":"2024-09-16T01:48:57.824946+0000","last_unstale":"2024-09-16T01:49:25.945520+0000","last_undegraded":"2024-09-16T01:49:25.945520+0000","last_fullsized":"2024-09-16T01:49:25.945520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039335+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:49:01.039335+0000","last_peered":"2024-09-16T01:49:01.039335+0000","last_clean":"2024-09-16T01:49:01.039335+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:49:01.039335+0000","last_undegraded":"2024-09-16T01:49:01.039335+0000","last_fullsized":"2024-09-16T01:49:01.039335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:03.828830+0000","last_change":"2024-09-16T01:48:57.808318+0000","last_active":"2024-09-16T01:49:03.828830+0000","last_peered":"2024-09-16T01:49:03.828830+0000","last_clean":"2024-09-16T01:49:03.828830+0000","last_became_active":"2024-09-16T01:48:57.807106+0000","last_became_peered":"2024-09-16T01:48:57.807106+0000","last_unstale":"2024-09-16T01:49:03.828830+0000","last_undegraded":"2024-09-16T01:49:03.828830+0000","last_fullsized":"2024-09-16T01:49:03.828830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:47.341651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:01.039274+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:49:01.039274+0000","last_peered":"2024-09-16T01:49:01.039274+0000","last_clean":"2024-09-16T01:49:01.039274+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:49:01.039274+0000","last_undegraded":"2024-09-16T01:49:01.039274+0000","last_fullsized":"2024-09-16T01:49:01.039274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-16T01:49:25.942146+0000","last_change":"2024-09-16T01:48:57.815555+0000","last_active":"2024-09-16T01:49:25.942146+0000","last_peered":"2024-09-16T01:49:25.942146+0000","last_clean":"2024-09-16T01:49:25.942146+0000","last_became_active":"2024-09-16T01:48:57.815315+0000","last_became_peered":"2024-09-16T01:48:57.815315+0000","last_unstale":"2024-09-16T01:49:25.942146+0000","last_undegraded":"2024-09-16T01:49:25.942146+0000","last_fullsized":"2024-09-16T01:49:25.942146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:01.234203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8552,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":182,"ondisk_log_size":182,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509454,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":80998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509454,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":80215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509453,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":670058,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":80818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":79330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":80656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":669470,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":81726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:49:40.137 INFO:tasks.thrashosds.thrasher:Setting 3.20 to [0, 4, 6, 5] 2024-09-16T01:49:40.137 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.20', '0', '4', '6', '5'] 2024-09-16T01:49:40.137 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.20 0 4 6 5 2024-09-16T01:49:40.220 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:40.218+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:40.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.250+0000 7feeb4c00640 1 -- 172.21.15.146:0/396444615 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v24) ==== 87+0+0 (secure 0 0 0) 0x7feeac07a3c0 con 0x7feeb80c6b70 2024-09-16T01:49:40.253 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:49:40.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fee9806c2b0 msgr2=0x7fee9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fee9806c2b0 0x7fee9806e770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feea4006fd0 tx=0x7feea4008040 comp rx=0 tx=0).stop 2024-09-16T01:49:40.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 msgr2=0x7feeb8150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb8150700 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 shutdown_connections 2024-09-16T01:49:40.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fee9806c2b0 0x7fee9806e770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feeb80cbab0 0x7feeb80c23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80c75a0 0x7feeb80c1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 --2- 172.21.15.146:0/396444615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feeb80c6b70 0x7feeb8150700 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 >> 172.21.15.146:0/396444615 conn(0x7feeb80ccc20 msgr2=0x7feeb8151760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 shutdown_connections 2024-09-16T01:49:40.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.253+0000 7feebe000640 1 -- 172.21.15.146:0/396444615 wait complete. 2024-09-16T01:49:40.316 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:40.313+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:40.340 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:40.340 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.338+0000 7f13cd400640 1 Processor -- start 2024-09-16T01:49:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.338+0000 7f13cd400640 1 -- start start 2024-09-16T01:49:40.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.339+0000 7f13cd400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f13c8154ea0 0x7f13c80cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:40.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.339+0000 7f13cd400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c815ae30 con 0x7f13c815a940 2024-09-16T01:49:40.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.339+0000 7f13cd400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c815b110 con 0x7f13c80b3160 2024-09-16T01:49:40.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.339+0000 7f13cd400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f13c806c7b0 con 0x7f13c8154ea0 2024-09-16T01:49:40.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c7200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f13c815a940 0x7f13c8165240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47854/0 (socket says 172.21.15.146:47854) 2024-09-16T01:49:40.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c7200640 1 -- 172.21.15.146:0/533088967 learned_addr learned my addr 172.21.15.146:0/533088967 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:40.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c6800640 1 --2- 172.21.15.146:0/533088967 >> v2:172.21.15.161:3300/0 conn(0x7f13c8154ea0 0x7f13c80cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789258077 0 0) 0x7f13c815b110 con 0x7f13c80b3160 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13ac003520 con 0x7f13c80b3160 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531957433 0 0) 0x7f13c815ae30 con 0x7f13c815a940 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c815b110 con 0x7f13c815a940 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c6800640 1 -- 172.21.15.146:0/533088967 >> v1:172.21.15.161:6789/0 conn(0x7f13c80b3160 legacy=0x7f13c806cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c6800640 1 -- 172.21.15.146:0/533088967 >> v1:172.21.15.146:6789/0 conn(0x7f13c815a940 legacy=0x7f13c8165240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.340+0000 7f13c6800640 1 -- 172.21.15.146:0/533088967 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c815b3f0 con 0x7f13c8154ea0 2024-09-16T01:49:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.342+0000 7f13c6800640 1 --2- 172.21.15.146:0/533088967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c8154ea0 0x7f13c80cbe70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f13b80097a0 tx=0x7f13b802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ba87e2957446494 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:40.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.342+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8037070 con 0x7f13c8154ea0 2024-09-16T01:49:40.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.342+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b8032640 con 0x7f13c8154ea0 2024-09-16T01:49:40.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.342+0000 7f13c5400640 1 -- 172.21.15.146:0/533088967 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8032a10 con 0x7f13c8154ea0 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 -- 172.21.15.146:0/533088967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c8154ea0 msgr2=0x7f13c80cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 --2- 172.21.15.146:0/533088967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c8154ea0 0x7f13c80cbe70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f13b80097a0 tx=0x7f13b802a2c0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 -- 172.21.15.146:0/533088967 shutdown_connections 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 --2- 172.21.15.146:0/533088967 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c8154ea0 0x7f13c80cbe70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 -- 172.21.15.146:0/533088967 >> 172.21.15.146:0/533088967 conn(0x7f13c8067b30 msgr2=0x7f13c8067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 -- 172.21.15.146:0/533088967 shutdown_connections 2024-09-16T01:49:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.343+0000 7f13cd400640 1 -- 172.21.15.146:0/533088967 wait complete. 2024-09-16T01:49:40.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.345+0000 7f13cd400640 1 Processor -- start 2024-09-16T01:49:40.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.345+0000 7f13cd400640 1 -- start start 2024-09-16T01:49:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13cd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c80b3160 0x7f13c81e7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13cd400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c8154ea0 0x7f13c80c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13cd400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 0x7f13c80cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13cd400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f13c80ccbd0 con 0x7f13c80b3160 2024-09-16T01:49:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13cd400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f13c80cca50 con 0x7f13c815a940 2024-09-16T01:49:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13cd400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f13c806d630 con 0x7f13c8154ea0 2024-09-16T01:49:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c80b3160 0x7f13c81e7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13c7200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 0x7f13c80cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c8154ea0 0x7f13c80c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13c7200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 0x7f13c80cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47528/0 (socket says 172.21.15.146:47528) 2024-09-16T01:49:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c80b3160 0x7f13c81e7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47294/0 (socket says 172.21.15.146:47294) 2024-09-16T01:49:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.346+0000 7f13c7200640 1 -- 172.21.15.146:0/3963789341 learned_addr learned my addr 172.21.15.146:0/3963789341 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c7200640 1 -- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c8154ea0 msgr2=0x7f13c80c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c7200640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c8154ea0 0x7f13c80c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c7200640 1 -- 172.21.15.146:0/3963789341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c80b3160 msgr2=0x7f13c81e7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c7200640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c80b3160 0x7f13c81e7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c7200640 1 -- 172.21.15.146:0/3963789341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c81f3d00 con 0x7f13c815a940 2024-09-16T01:49:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c5e00640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c8154ea0 0x7f13c80c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13c7200640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 0x7f13c80cf210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13c000cad0 tx=0x7f13c000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0017070 con 0x7f13c815a940 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c000ac30 con 0x7f13c815a940 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0012400 con 0x7f13c815a940 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c815ece0 con 0x7f13c815a940 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.347+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c80b5950 con 0x7f13c815a940 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.348+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c80cb340 con 0x7f13c815a940 2024-09-16T01:49:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.349+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f13c001a070 con 0x7f13c815a940 2024-09-16T01:49:40.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.350+0000 7f13bea00640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a806c2b0 0x7f13a806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:40.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.350+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5738+0+0 (secure 0 0 0) 0x7f13c008b290 con 0x7f13c815a940 2024-09-16T01:49:40.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.351+0000 7f13c6800640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a806c2b0 0x7f13a806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:40.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:49:40.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.351+0000 7f13c6800640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a806c2b0 0x7f13a806e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f13b8034000 tx=0x7f13b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:40.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.355+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f13c0059260 con 0x7f13c815a940 2024-09-16T01:49:40.403 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:40.401+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:40.502 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:40.500+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:40.520 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:40.521 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:40.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:49:40.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:40.552+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.20", "id": [0, 4, 6, 5]} v 0) -- 0x7f13c814ef20 con 0x7f13c815a940 2024-09-16T01:49:40.603 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:40.602+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:40.704 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:40.703+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:40.709 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:40.710 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:40.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:49:40.805 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:40.803+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:40.880 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:40.881 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:40.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:49:40.934 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:40.933+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:41.007 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:41.005+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:41.090 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:41.090 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:41.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:49:41.107 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:41.106+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:41.209 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:41.208+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:41.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.254+0000 7f13bea00640 1 -- 172.21.15.146:0/3963789341 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.20", "id": [0, 4, 6, 5]}]=0 set 3.20 pg_upmap_items mapping to [0->4,6->5] v25) ==== 148+0+0 (secure 0 0 0) 0x7f13c005d560 con 0x7f13c815a940 2024-09-16T01:49:41.256 INFO:teuthology.orchestra.run.smithi146.stderr:set 3.20 pg_upmap_items mapping to [0->4,6->5] 2024-09-16T01:49:41.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a806c2b0 msgr2=0x7f13a806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a806c2b0 0x7f13a806e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f13b8034000 tx=0x7f13b802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:41.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 msgr2=0x7f13c80cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 0x7f13c80cf210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13c000cad0 tx=0x7f13c000cf90 comp rx=0 tx=0).stop 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 shutdown_connections 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a806c2b0 0x7f13a806e770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c815a940 0x7f13c80cf210 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c8154ea0 0x7f13c80c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 --2- 172.21.15.146:0/3963789341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c80b3160 0x7f13c81e7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 >> 172.21.15.146:0/3963789341 conn(0x7f13c8067b30 msgr2=0x7f13c80c5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 shutdown_connections 2024-09-16T01:49:41.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.257+0000 7f13cd400640 1 -- 172.21.15.146:0/3963789341 wait complete. 2024-09-16T01:49:41.304 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:41.304 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:41.309 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:41.308+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:41.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:49:41.411 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:41.409+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:41.512 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:41.510+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:41.514 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:41.514 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:41.526 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:49:41.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:49:41.618 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:41.616+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:41.642 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:41.642 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:41.651 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:49:41.713 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:41.712+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:41.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.721+0000 7f22a8400640 1 Processor -- start 2024-09-16T01:49:41.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.721+0000 7f22a8400640 1 -- start start 2024-09-16T01:49:41.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a8400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22a0152c30 0x7f22a00c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a8400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a00c6e90 0x7f22a00c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a8400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a00c4240 con 0x7f22a00c6ab0 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a8400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f22a00c4da0 con 0x7f22a0152c30 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a8400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f22a00c4500 con 0x7f22a00c6e90 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a5000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f22a00c6ab0 0x7f22a00c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47868/0 (socket says 172.21.15.146:47868) 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.722+0000 7f22a5000640 1 -- 172.21.15.146:0/565465902 learned_addr learned my addr 172.21.15.146:0/565465902 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f22a6400640 1 --2- 172.21.15.146:0/565465902 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a00c6e90 0x7f22a00c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f22a5a00640 1 --2- 172.21.15.146:0/565465902 >> v2:172.21.15.161:3300/0 conn(0x7f22a0152c30 0x7f22a00c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f22a6400640 1 -- 172.21.15.146:0/565465902 >> v2:172.21.15.161:3300/0 conn(0x7f22a0152c30 msgr2=0x7f22a00c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:41.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f22a6400640 1 --2- 172.21.15.146:0/565465902 >> v2:172.21.15.161:3300/0 conn(0x7f22a0152c30 0x7f22a00c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f22a6400640 1 -- 172.21.15.146:0/565465902 >> v1:172.21.15.146:6789/0 conn(0x7f22a00c6ab0 legacy=0x7f22a00c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f22a6400640 1 -- 172.21.15.146:0/565465902 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a00c47e0 con 0x7f22a00c6e90 2024-09-16T01:49:41.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.723+0000 7f229fe00640 1 -- 172.21.15.146:0/565465902 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628828137 0 0) 0x7f22a00c4240 con 0x7f22a00c6ab0 2024-09-16T01:49:41.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.724+0000 7f22a6400640 1 --2- 172.21.15.146:0/565465902 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a00c6e90 0x7f22a00c7ff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2298009d90 tx=0x7f229802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96edfe8b6a0f5d2f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.724+0000 7f229fe00640 1 -- 172.21.15.146:0/565465902 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298007740 con 0x7f22a00c6e90 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.724+0000 7f229fe00640 1 -- 172.21.15.146:0/565465902 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22980078e0 con 0x7f22a00c6e90 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.724+0000 7f229fe00640 1 -- 172.21.15.146:0/565465902 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298007cb0 con 0x7f22a00c6e90 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 -- 172.21.15.146:0/565465902 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a00c6e90 msgr2=0x7f22a00c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 --2- 172.21.15.146:0/565465902 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a00c6e90 0x7f22a00c7ff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2298009d90 tx=0x7f229802a660 comp rx=0 tx=0).stop 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 -- 172.21.15.146:0/565465902 shutdown_connections 2024-09-16T01:49:41.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 --2- 172.21.15.146:0/565465902 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a00c6e90 0x7f22a00c7ff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 --2- 172.21.15.146:0/565465902 >> v2:172.21.15.161:3300/0 conn(0x7f22a0152c30 0x7f22a00c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 -- 172.21.15.146:0/565465902 >> 172.21.15.146:0/565465902 conn(0x7f22a006bc10 msgr2=0x7f22a006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:41.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 -- 172.21.15.146:0/565465902 shutdown_connections 2024-09-16T01:49:41.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.725+0000 7f22a8400640 1 -- 172.21.15.146:0/565465902 wait complete. 2024-09-16T01:49:41.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.726+0000 7f22a8400640 1 Processor -- start 2024-09-16T01:49:41.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.727+0000 7f22a8400640 1 -- start start 2024-09-16T01:49:41.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.727+0000 7f22a8400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c6ab0 0x7f22a01e3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.727+0000 7f22a8400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22a00c6e90 0x7f22a01f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.727+0000 7f22a8400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a0152c30 0x7f22a01489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.727+0000 7f22a8400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f22a00c9350 con 0x7f22a00c6e90 2024-09-16T01:49:41.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.727+0000 7f22a8400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22a00c6630 con 0x7f22a00c6ab0 2024-09-16T01:49:41.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a8400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f22a00c94d0 con 0x7f22a0152c30 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22a00c6e90 0x7f22a01f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a5a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c6ab0 0x7f22a01e3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a0152c30 0x7f22a01489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22a00c6e90 0x7f22a01f0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47298/0 (socket says 172.21.15.146:47298) 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a5000640 1 -- 172.21.15.146:0/2993132528 learned_addr learned my addr 172.21.15.146:0/2993132528 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c6ab0 msgr2=0x7f22a01e3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00c6ab0 0x7f22a01e3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22a00c6e90 msgr2=0x7f22a01f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22a00c6e90 0x7f22a01f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a01e6560 con 0x7f22a0152c30 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22a00c6e90 0x7f22a01f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:41.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.728+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a0152c30 0x7f22a01489b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2298031000 tx=0x7f2298004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.729+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229803f070 con 0x7f22a0152c30 2024-09-16T01:49:41.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.729+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2298031bc0 con 0x7f22a0152c30 2024-09-16T01:49:41.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.729+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a01e6840 con 0x7f22a0152c30 2024-09-16T01:49:41.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.729+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298009450 con 0x7f22a0152c30 2024-09-16T01:49:41.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.729+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a00bddd0 con 0x7f22a0152c30 2024-09-16T01:49:41.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.730+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f22a00c36b0 con 0x7f22a0152c30 2024-09-16T01:49:41.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.731+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f229803b410 con 0x7f22a0152c30 2024-09-16T01:49:41.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f229d600640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f228006c2b0 0x7f228006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5767+0+0 (secure 0 0 0) 0x7f22980aca50 con 0x7f22a0152c30 2024-09-16T01:49:41.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f228006c2b0 0x7f228006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v25) ==== 54+0+16 (secure 0 0 0) 0x7f22980b1220 con 0x7f22a0152c30 2024-09-16T01:49:41.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f22a01f09b0 0x7f22a006f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22a0149400 con 0x7f22a01f09b0 2024-09-16T01:49:41.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.732+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f22a01f09b0 0x7f22a006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.733+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f228006c2b0 0x7f228006e770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2294009b40 tx=0x7f2294009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.733+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f22a01f09b0 0x7f22a006f790 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.734+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a0149400 con 0x7f22a01f09b0 2024-09-16T01:49:41.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.748+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a01e6a10 con 0x7f22a01f09b0 2024-09-16T01:49:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.750+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a01e6a10 con 0x7f22a01f09b0 2024-09-16T01:49:41.752 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:41.752 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.750+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f22a01f3ef0 0x7f22a01f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.750+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f22a00c8cb0 con 0x7f22a01f3ef0 2024-09-16T01:49:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.750+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f22a01f3ef0 0x7f22a01f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.751+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f22a01f3ef0 0x7f22a01f4310 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.752+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a00c8cb0 con 0x7f22a01f3ef0 2024-09-16T01:49:41.756 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:41.756 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:41.765 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:49:41.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.767+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a006c370 con 0x7f22a01f3ef0 2024-09-16T01:49:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.769+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a006c370 con 0x7f22a01f3ef0 2024-09-16T01:49:41.771 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:41.772 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:41.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.769+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f22a0062b40 0x7f22a0062f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.769+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f22a01e4330 con 0x7f22a0062b40 2024-09-16T01:49:41.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.770+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f22a0062b40 0x7f22a0062f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.770+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f22a0062b40 0x7f22a0062f20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.771+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a01e4330 con 0x7f22a0062b40 2024-09-16T01:49:41.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.785+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a00c8820 con 0x7f22a0062b40 2024-09-16T01:49:41.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.787+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a00c8820 con 0x7f22a0062b40 2024-09-16T01:49:41.789 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:41.789 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:41.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.787+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f22a0150960 0x7f22a0150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.787+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f22a01512e0 con 0x7f22a0150960 2024-09-16T01:49:41.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.787+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f22a0150960 0x7f22a0150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.787+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f22a0150960 0x7f22a0150da0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.788+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a01512e0 con 0x7f22a0150960 2024-09-16T01:49:41.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.802+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a01512e0 con 0x7f22a0150960 2024-09-16T01:49:41.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.803+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a01512e0 con 0x7f22a0150960 2024-09-16T01:49:41.805 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:41.806 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:41.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.803+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f22a002cb50 0x7f22a002cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.803+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f22a0149930 con 0x7f22a002cb50 2024-09-16T01:49:41.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.804+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f22a002cb50 0x7f22a002cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.804+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f22a002cb50 0x7f22a002cf70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.805+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a0149930 con 0x7f22a002cb50 2024-09-16T01:49:41.814 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:41.813+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:41.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.819+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a0149930 con 0x7f22a002cb50 2024-09-16T01:49:41.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.821+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a0149930 con 0x7f22a002cb50 2024-09-16T01:49:41.823 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:41.824 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:41.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.821+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f22a0204d60 0x7f22a0207150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.821+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f22a0149930 con 0x7f22a0204d60 2024-09-16T01:49:41.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.821+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f22a0204d60 0x7f22a0207150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.822+0000 7f22a5000640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f22a0204d60 0x7f22a0207150 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.823+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a0149930 con 0x7f22a0204d60 2024-09-16T01:49:41.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.836+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a0149930 con 0x7f22a0204d60 2024-09-16T01:49:41.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.838+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a0149930 con 0x7f22a0204d60 2024-09-16T01:49:41.840 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:41.841 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:41.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.838+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f22a0207ba0 0x7f22a0209f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.838+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f22a002d9c0 con 0x7f22a0207ba0 2024-09-16T01:49:41.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.838+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f22a0207ba0 0x7f22a0209f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.839+0000 7f22a6400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f22a0207ba0 0x7f22a0209f90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.840+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a002d9c0 con 0x7f22a0207ba0 2024-09-16T01:49:41.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.853+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a002d9c0 con 0x7f22a0207ba0 2024-09-16T01:49:41.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.856+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a002d9c0 con 0x7f22a0207ba0 2024-09-16T01:49:41.857 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:41.858 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:41.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.856+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f22a020a9e0 0x7f22a020cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:41.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.856+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f22a020fba0 con 0x7f22a020a9e0 2024-09-16T01:49:41.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.856+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f22a020a9e0 0x7f22a020cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:41.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.856+0000 7f22a5a00640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f22a020a9e0 0x7f22a020cdd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:41.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.857+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22a020fba0 con 0x7f22a020a9e0 2024-09-16T01:49:41.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.871+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22a020d5e0 con 0x7f22a020a9e0 2024-09-16T01:49:41.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f229d600640 1 -- 172.21.15.146:0/2993132528 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22a020d5e0 con 0x7f22a020a9e0 2024-09-16T01:49:41.875 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f22a01f09b0 msgr2=0x7f22a006f790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f22a01f09b0 0x7f22a006f790 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f22a01f3ef0 msgr2=0x7f22a01f4310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f22a01f3ef0 0x7f22a01f4310 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f22a0062b40 msgr2=0x7f22a0062f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f22a0062b40 0x7f22a0062f20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f22a0150960 msgr2=0x7f22a0150da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f22a0150960 0x7f22a0150da0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f22a002cb50 msgr2=0x7f22a002cf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f22a002cb50 0x7f22a002cf70 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 reap_dead start 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f22a0204d60 msgr2=0x7f22a0207150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f22a0204d60 0x7f22a0207150 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 reap_dead start 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 reap_dead start 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f22a0207ba0 msgr2=0x7f22a0209f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f22a0207ba0 0x7f22a0209f90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f22a020a9e0 msgr2=0x7f22a020cdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f22a020a9e0 0x7f22a020cdd0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f228006c2b0 msgr2=0x7f228006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f228006c2b0 0x7f228006e770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2294009b40 tx=0x7f2294009190 comp rx=0 tx=0).stop 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a0152c30 msgr2=0x7f22a01489b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:41.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 --2- 172.21.15.146:0/2993132528 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22a0152c30 0x7f22a01489b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2298031000 tx=0x7f2298004a40 comp rx=0 tx=0).stop 2024-09-16T01:49:41.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a6400640 1 -- 172.21.15.146:0/2993132528 reap_dead start 2024-09-16T01:49:41.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 shutdown_connections 2024-09-16T01:49:41.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 >> 172.21.15.146:0/2993132528 conn(0x7f22a006bc10 msgr2=0x7f22a014d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:41.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 shutdown_connections 2024-09-16T01:49:41.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:41.874+0000 7f22a8400640 1 -- 172.21.15.146:0/2993132528 wait complete. 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.08659703, 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073505619999999997, 2024-09-16T01:49:41.914 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:49:41.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825023+0000", 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1150000000000004e-06 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825036+0000", 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3022e-05 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3779e-05 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3620000000000006e-06 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825161+0000", 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010298600000000001 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825279+0000", 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118087 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825418+0000", 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013899299999999999 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829035+0000", 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036175130000000002 2024-09-16T01:49:41.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832264+0000", 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003228908 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832322+0000", 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7751000000000003e-05 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832369+0000", 2024-09-16T01:49:41.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7046000000000003e-05 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.086510025000003, 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0091091320000000007, 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:49:41.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4900000000000005e-07 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825113+0000", 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.229e-06 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825125+0000", 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2493e-05 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825133+0000", 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6340000000000001e-06 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825778+0000", 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064556499999999996 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825887+0000", 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108416 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826017+0000", 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013044 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829740+0000", 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037227200000000001 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834119+0000", 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043795930000000002 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834175+0000", 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5492999999999999e-05 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834215+0000", 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9700000000000003e-05 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.925 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.086315812999999, 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061249809999999998, 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:49:41.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.34e-07 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825303+0000", 2024-09-16T01:49:41.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.96e-06 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825311+0000", 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6000000000000001e-06 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825317+0000", 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6139999999999997e-06 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828779+0000", 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034614210000000001 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829036+0000", 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00025713999999999999 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829173+0000", 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000136797 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831180+0000", 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020068859999999998 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831366+0000", 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018597399999999999 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831398+0000", 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2518999999999998e-05 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831425+0000", 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6636e-05 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.08629002, 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006225062, 2024-09-16T01:49:41.931 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2e-07 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825335+0000", 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4339999999999998e-06 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825342+0000", 2024-09-16T01:49:41.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6900000000000003e-06 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825348+0000", 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5470000000000004e-06 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829429+0000", 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040815720000000003 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829541+0000", 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112285 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829647+0000", 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105592 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831435+0000", 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017880330000000001 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831474+0000", 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831518+0000", 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3516999999999997e-05 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831551+0000", 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3062000000000003e-05 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.936 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.086260305000003, 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062900040000000001, 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825356+0000", 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825359+0000", 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0620000000000001e-06 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825366+0000", 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6899999999999992e-06 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825380+0000", 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3193000000000001e-05 2024-09-16T01:49:41.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829978+0000", 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045982879999999999 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830056+0000", 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8466000000000004e-05 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830226+0000", 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000169944 2024-09-16T01:49:41.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831440+0000", 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012132790000000001 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831590+0000", 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015076400000000001 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831621+0000", 2024-09-16T01:49:41.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0440999999999999e-05 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831645+0000", 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4383e-05 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.085306826999997, 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080128709999999995, 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:49:41.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826310+0000", 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3200000000000003e-07 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826315+0000", 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6980000000000003e-06 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826328+0000", 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2568999999999999e-05 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826336+0000", 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9670000000000001e-06 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826916+0000", 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058005899999999996 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827058+0000", 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000142533 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827196+0000", 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013736299999999999 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829940+0000", 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027445600000000001 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834254+0000", 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043136210000000001 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834292+0000", 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8266000000000003e-05 2024-09-16T01:49:41.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834322+0000", 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9303e-05 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.085260918000003, 2024-09-16T01:49:41.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080982150000000006, 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5799999999999999e-07 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826368+0000", 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3152999999999999e-05 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826378+0000", 2024-09-16T01:49:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826385+0000", 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5440000000000004e-06 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827629+0000", 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001243886 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827855+0000", 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000226615 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834453+0000", 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006597548 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:14 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.051063831, 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013360132, 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6600000000000002e-07 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860560+0000", 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7919999999999993e-06 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860569+0000", 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3999999999999998e-06 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860576+0000", 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8589999999999999e-06 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860646+0000", 2024-09-16T01:49:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9676000000000002e-05 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143559 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860917+0000", 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012731900000000001 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862638+0000", 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017212670000000001 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873802+0000", 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011164074 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873866+0000", 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4381999999999994e-05 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873912+0000", 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5337999999999999e-05 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:31 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.050875609000002, 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013335217, 2024-09-16T01:49:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860741+0000", 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9899999999999999e-07 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860748+0000", 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3250000000000002e-06 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860761+0000", 2024-09-16T01:49:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2958e-05 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860769+0000", 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2230000000000007e-06 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861199+0000", 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000429178 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861286+0000", 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7436999999999996e-05 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861415+0000", 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012907000000000001 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863136+0000", 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001720627 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873957+0000", 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010821809999999999 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874029+0000", 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1641000000000001e-05 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874075+0000", 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6048999999999999e-05 2024-09-16T01:49:41.963 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:19 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.050831438000003, 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013415967000000001, 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860785+0000", 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6300000000000005e-07 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5279999999999997e-06 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860797+0000", 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.418e-06 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860802+0000", 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.344e-06 2024-09-16T01:49:41.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861655+0000", 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085269000000000002 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861767+0000", 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011184900000000001 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861914+0000", 2024-09-16T01:49:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014756700000000001 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863730+0000", 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001815564 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874127+0000", 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010396924 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874163+0000", 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6285999999999998e-05 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874200+0000", 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7134000000000001e-05 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:6 2.1f 2:fa28013d:::615.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.969 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.014976986000001, 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006133548, 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896640+0000", 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896645+0000", 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6540000000000004e-06 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896655+0000", 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0316000000000001e-05 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896664+0000", 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2090000000000005e-06 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897019+0000", 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035541199999999999 2024-09-16T01:49:41.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897088+0000", 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9388999999999996e-05 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897195+0000", 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010608299999999999 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900354+0000", 2024-09-16T01:49:41.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031591919999999999 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902602+0000", 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002248468 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902659+0000", 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6889999999999999e-05 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902772+0000", 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112972 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:9 2.6 2:62978f0a:::617.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.014185756, 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063594389999999997, 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.975 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897431+0000", 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8899999999999998e-07 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897437+0000", 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3060000000000003e-06 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897447+0000", 2024-09-16T01:49:41.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9920000000000002e-06 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897455+0000", 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.627e-06 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897523+0000", 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7912999999999998e-05 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897652+0000", 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000129259 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897777+0000", 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012468599999999999 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901149+0000", 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033725389999999999 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903683+0000", 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025335539999999999 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903730+0000", 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7339999999999997e-05 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903789+0000", 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9234000000000002e-05 2024-09-16T01:49:41.980 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:10 2.6 2:62978f0a:::617.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.013892725000002, 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061735380000000001, 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897724+0000", 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0139999999999999e-06 2024-09-16T01:49:41.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897730+0000", 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7629999999999997e-06 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897740+0000", 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0499999999999999e-05 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897748+0000", 2024-09-16T01:49:41.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1519999999999992e-06 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898076+0000", 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00032781800000000001 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898143+0000", 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6905999999999999e-05 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898267+0000", 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123979 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901274+0000", 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00300716 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903796+0000", 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025223210000000001 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903859+0000", 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2638000000000006e-05 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903896+0000", 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7286999999999997e-05 2024-09-16T01:49:41.986 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.013801010000002, 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060644100000000001, 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5099999999999999e-07 2024-09-16T01:49:41.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897831+0000", 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5534000000000001e-05 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897850+0000", 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8825999999999999e-05 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897858+0000", 2024-09-16T01:49:41.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3939999999999996e-06 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898448+0000", 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059040600000000003 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898526+0000", 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7600000000000002e-05 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:49:41.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898646+0000", 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011967400000000001 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903214+0000", 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045682589999999999 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903787+0000", 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00057291199999999997 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903842+0000", 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4747999999999998e-05 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903879+0000", 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7305999999999998e-05 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:8 2.6 2:6642f7a8:::620.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.992 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.999800341, 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0071307610000000002, 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:49:41.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911816+0000", 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2299999999999999e-07 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911822+0000", 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0460000000000002e-06 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:41.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911846+0000", 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4193999999999999e-05 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911857+0000", 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.042e-05 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912714+0000", 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085704200000000003 2024-09-16T01:49:41.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912790+0000", 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6019999999999994e-05 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912934+0000", 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014454 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916495+0000", 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035605770000000001 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918865+0000", 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002370451 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918906+0000", 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0123999999999997e-05 2024-09-16T01:49:41.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918946+0000", 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:12 2.1f 2:ffe2d39d:::622.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.999357893999999, 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060451970000000004, 2024-09-16T01:49:41.998 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:41.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912259+0000", 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0729999999999999e-06 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912265+0000", 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4409999999999998e-06 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912276+0000", 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0798000000000001e-05 2024-09-16T01:49:42.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912286+0000", 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0176999999999999e-05 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912581+0000", 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00029479199999999999 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912692+0000", 2024-09-16T01:49:42.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110874 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913326+0000", 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00063410300000000001 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916590+0000", 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003264543 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918212+0000", 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016214440000000001 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918252+0000", 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0562999999999998e-05 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918303+0000", 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0389000000000002e-05 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.003 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:21 2.6 2:62eebe7c:::627.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.999315493000001, 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0067672629999999999, 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:49:42.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912301+0000", 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1999999999999999e-07 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912305+0000", 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9770000000000002e-06 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912312+0000", 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9920000000000001e-06 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912318+0000", 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9089999999999997e-06 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084597199999999996 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913267+0000", 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103433 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913378+0000", 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011124300000000001 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916553+0000", 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031750900000000002 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918992+0000", 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00243837 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919023+0000", 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1498999999999998e-05 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919067+0000", 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4158e-05 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.999289604000001, 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0068352660000000004, 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.009 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8299999999999997e-07 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912336+0000", 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0079e-05 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912345+0000", 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0639999999999994e-06 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912351+0000", 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.686e-06 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913855+0000", 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0015034829999999999 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913971+0000", 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011639 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914107+0000", 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013549999999999999 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917094+0000", 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002987132 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919106+0000", 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020125389999999998 2024-09-16T01:49:42.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919139+0000", 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2670999999999999e-05 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919161+0000", 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2238999999999998e-05 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:9 2.9 2:954a9a2b:::62b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.974045909000001, 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074502429999999996, 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937571+0000", 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1480000000000001e-06 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937578+0000", 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1890000000000001e-06 2024-09-16T01:49:42.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937593+0000", 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4613999999999999e-05 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937602+0000", 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937662+0000", 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9778000000000003e-05 2024-09-16T01:49:42.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937799+0000", 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013690299999999999 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937967+0000", 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016826100000000001 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940308+0000", 2024-09-16T01:49:42.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023405750000000001 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944953+0000", 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046456860000000004 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944987+0000", 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3997e-05 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945020+0000", 2024-09-16T01:49:42.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2653000000000001e-05 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.973990964, 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080442459999999997, 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8999999999999996e-07 2024-09-16T01:49:42.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937632+0000", 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7399999999999998e-06 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937640+0000", 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.354e-06 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937647+0000", 2024-09-16T01:49:42.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5050000000000004e-06 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939098+0000", 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001451473 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939454+0000", 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035574400000000001 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939603+0000", 2024-09-16T01:49:42.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000149228 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940646+0000", 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001042821 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945559+0000", 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049129070000000002 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945623+0000", 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4072999999999995e-05 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945669+0000", 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5711000000000003e-05 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.025 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:42.027 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:49:42.027 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:42.026+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:42.127 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:42.126+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:42.171 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:42.171 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:42.171 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:49:42.171 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:42.179 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:49:42.263 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:42.262+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:42.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:49:42.325 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:42.326 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:42.326 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:42.326 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:42.326 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:42.329 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:42.327+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:42.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:49:42.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.427+0000 7f1c11400640 1 Processor -- start 2024-09-16T01:49:42.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.427+0000 7f1c11400640 1 -- start start 2024-09-16T01:49:42.431 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:42.429+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.428+0000 7f1c11400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 0x7f1c0c0c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.428+0000 7f1c11400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c0c0c3f00 0x7f1c0c152660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.428+0000 7f1c11400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c06eb30 con 0x7f1c0c06ce90 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.428+0000 7f1c11400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1c0c06e6a0 con 0x7f1c0c0c3f00 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.428+0000 7f1c11400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c0c8f40 con 0x7f1c0c06d270 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c09e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 0x7f1c0c0c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c09e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 0x7f1c0c0c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36702/0 (socket says 172.21.15.146:36702) 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c09e00640 1 -- 172.21.15.146:0/2699252325 learned_addr learned my addr 172.21.15.146:0/2699252325 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c0b200640 1 --2- 172.21.15.146:0/2699252325 >> v2:172.21.15.161:3300/0 conn(0x7f1c0c0c3f00 0x7f1c0c152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c0a800640 1 --1- 172.21.15.146:0/2699252325 >> v1:172.21.15.161:6789/0 conn(0x7f1c0c06ce90 0x7f1c0c158810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:45326/0 (socket says 172.21.15.146:45326) 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c0b200640 1 -- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 msgr2=0x7f1c0c0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c0b200640 1 --2- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 0x7f1c0c0c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c0b200640 1 -- 172.21.15.146:0/2699252325 >> v1:172.21.15.161:6789/0 conn(0x7f1c0c06ce90 legacy=0x7f1c0c158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c0b200640 1 -- 172.21.15.146:0/2699252325 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c067ad0 con 0x7f1c0c0c3f00 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c09400640 1 -- 172.21.15.146:0/2699252325 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607787819 0 0) 0x7f1c0c06eb30 con 0x7f1c0c06ce90 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.429+0000 7f1c09e00640 1 --2- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 0x7f1c0c0c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.430+0000 7f1c0b200640 1 --2- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c0c3f00 0x7f1c0c152660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009d90 tx=0x7f1c0402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74850f3d4544a00b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.430+0000 7f1c09400640 1 -- 172.21.15.146:0/2699252325 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04007740 con 0x7f1c0c0c3f00 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.430+0000 7f1c09400640 1 -- 172.21.15.146:0/2699252325 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c040078e0 con 0x7f1c0c0c3f00 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.430+0000 7f1c09400640 1 -- 172.21.15.146:0/2699252325 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04007cb0 con 0x7f1c0c0c3f00 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 -- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c0c3f00 msgr2=0x7f1c0c152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 --2- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c0c3f00 0x7f1c0c152660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009d90 tx=0x7f1c0402a660 comp rx=0 tx=0).stop 2024-09-16T01:49:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 -- 172.21.15.146:0/2699252325 shutdown_connections 2024-09-16T01:49:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 --2- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c0c3f00 0x7f1c0c152660 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 --2- 172.21.15.146:0/2699252325 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c06d270 0x7f1c0c0c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 -- 172.21.15.146:0/2699252325 >> 172.21.15.146:0/2699252325 conn(0x7f1c0c06d8e0 msgr2=0x7f1c0c06dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 -- 172.21.15.146:0/2699252325 shutdown_connections 2024-09-16T01:49:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.431+0000 7f1c11400640 1 -- 172.21.15.146:0/2699252325 wait complete. 2024-09-16T01:49:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.433+0000 7f1c11400640 1 Processor -- start 2024-09-16T01:49:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.433+0000 7f1c11400640 1 -- start start 2024-09-16T01:49:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c11400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ce90 0x7f1c0c147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c11400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 0x7f1c0c149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c11400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c0c3f00 0x7f1c0c1f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c11400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c158f20 con 0x7f1c0c06ce90 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c11400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c0c69c0 con 0x7f1c0c06d270 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c11400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c158d10 con 0x7f1c0c0c3f00 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c0b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c0c3f00 0x7f1c0c1f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 0x7f1c0c149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c0a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ce90 0x7f1c0c147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c0b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c0c3f00 0x7f1c0c1f1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36704/0 (socket says 172.21.15.146:36704) 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 0x7f1c0c149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47550/0 (socket says 172.21.15.146:47550) 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c0b200640 1 -- 172.21.15.146:0/298163149 learned_addr learned my addr 172.21.15.146:0/298163149 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 -- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c0c3f00 msgr2=0x7f1c0c1f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c0c3f00 0x7f1c0c1f1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 -- 172.21.15.146:0/298163149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ce90 msgr2=0x7f1c0c147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ce90 0x7f1c0c147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 -- 172.21.15.146:0/298163149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c1f1aa0 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.434+0000 7f1c09e00640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 0x7f1c0c149a10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c071560 tx=0x7f1bf800ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.435+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8004030 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.435+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf80041d0 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.435+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf800c940 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.435+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c158a30 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.435+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c153510 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.436+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0c1520f0 con 0x7f1c0c06d270 2024-09-16T01:49:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.437+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1bf800cae0 con 0x7f1c0c06d270 2024-09-16T01:49:42.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.438+0000 7f1c02a00640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bec06c2b0 0x7f1bec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:42.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.438+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5767+0+0 (secure 0 0 0) 0x7f1bf808b790 con 0x7f1c0c06d270 2024-09-16T01:49:42.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.438+0000 7f1c0a800640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bec06c2b0 0x7f1bec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:42.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.438+0000 7f1c0a800640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bec06c2b0 0x7f1bec06e770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000f60 tx=0x7f1bfc007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:42.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.442+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1c0c1f1aa0 con 0x7f1c0c06d270 2024-09-16T01:49:42.481 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:42.481 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:42.481 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:42.481 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:74 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'48, mlcod=17'43)", 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.201352+0000", 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.278017708999997, 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059279409999999999, 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201352+0000", 2024-09-16T01:49:42.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201352+0000", 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201353+0000", 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8700000000000005e-07 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201360+0000", 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6599999999999998e-06 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201368+0000", 2024-09-16T01:49:42.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5469999999999997e-06 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201373+0000", 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6779999999999999e-06 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201414+0000", 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1084000000000002e-05 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201427+0000", 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3331e-05 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.207246+0000", 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058184680000000003 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.207280+0000", 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4486000000000002e-05 2024-09-16T01:49:42.487 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:82 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'52, mlcod=17'50)", 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.219972+0000", 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.259397673000002, 2024-09-16T01:49:42.488 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006421459, 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219972+0000", 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219972+0000", 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219973+0000", 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7999999999999995e-07 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219980+0000", 2024-09-16T01:49:42.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.64e-06 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219990+0000", 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8989999999999996e-06 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219995+0000", 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0220000000000004e-06 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220033+0000", 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8810999999999998e-05 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220047+0000", 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3876999999999999e-05 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226360+0000", 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063126700000000003 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226394+0000", 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3859999999999998e-05 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:83 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'53, mlcod=17'50)", 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.220300+0000", 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.259069818999997, 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061152699999999999, 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.493 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220300+0000", 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220300+0000", 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220301+0000", 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8400000000000004e-07 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220307+0000", 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8449999999999999e-06 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220314+0000", 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2760000000000003e-06 2024-09-16T01:49:42.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220318+0000", 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9609999999999999e-06 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220362+0000", 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4365e-05 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220381+0000", 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9199999999999999e-05 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226398+0000", 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060167570000000002 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226415+0000", 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7282000000000002e-05 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.497 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:84 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.220645+0000", 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.258724991000001, 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057886719999999999, 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220645+0000", 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220645+0000", 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220646+0000", 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220707+0000", 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1493000000000002e-05 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220717+0000", 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0030999999999999e-05 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220721+0000", 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1180000000000002e-06 2024-09-16T01:49:42.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220754+0000", 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2499999999999997e-05 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220768+0000", 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3824e-05 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226417+0000", 2024-09-16T01:49:42.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056497759999999996 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226434+0000", 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6339000000000001e-05 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:85 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.221055+0000", 2024-09-16T01:49:42.502 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.258314943999999, 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005392188, 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221055+0000", 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221055+0000", 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221056+0000", 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-07 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221061+0000", 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7950000000000004e-06 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221069+0000", 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2869999999999999e-06 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221072+0000", 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7220000000000001e-06 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223322+0000", 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002249675 2024-09-16T01:49:42.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223336+0000", 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3617e-05 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226435+0000", 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003099667 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226447+0000", 2024-09-16T01:49:42.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1895e-05 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.257942466000003, 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076983349999999997, 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.06e-07 2024-09-16T01:49:42.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221436+0000", 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221446+0000", 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0903999999999999e-05 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221451+0000", 2024-09-16T01:49:42.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9130000000000002e-06 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223533+0000", 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020812259999999998 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223554+0000", 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1747999999999999e-05 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229094+0000", 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055398820000000003 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229126+0000", 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1642e-05 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'50)", 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.221732+0000", 2024-09-16T01:49:42.511 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.257637584999998, 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074182919999999999, 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221732+0000", 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.512 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221732+0000", 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221733+0000", 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2699999999999999e-07 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221772+0000", 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9201e-05 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221782+0000", 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7450000000000007e-06 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221786+0000", 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0369999999999996e-06 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223814+0000", 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020279600000000001 2024-09-16T01:49:42.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223827+0000", 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3002e-05 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229127+0000", 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00530045 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229151+0000", 2024-09-16T01:49:42.515 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3370000000000002e-05 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:88 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'58, mlcod=17'50)", 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.222408+0000", 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.256962133999998, 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0069243669999999998, 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222408+0000", 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222408+0000", 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222409+0000", 2024-09-16T01:49:42.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1600000000000001e-07 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222417+0000", 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4840000000000004e-06 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222427+0000", 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0305e-05 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.518 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222433+0000", 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7830000000000004e-06 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226475+0000", 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040419139999999998 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226489+0000", 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3971999999999999e-05 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229315+0000", 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028261060000000001 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229332+0000", 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7087000000000001e-05 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.520 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:116 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'78, mlcod=17'68)", 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.471060+0000", 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.008310088999998, 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0085754279999999995, 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471060+0000", 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471060+0000", 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471061+0000", 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8999999999999996e-07 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471068+0000", 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4240000000000004e-06 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471077+0000", 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471083+0000", 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.297e-06 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471421+0000", 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00033748300000000002 2024-09-16T01:49:42.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471439+0000", 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8131e-05 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479591+0000", 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0081523910000000002 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479635+0000", 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4156000000000002e-05 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:117 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'79, mlcod=17'68)", 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.007957288999997, 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0082421309999999998, 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.525 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471419+0000", 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2519999999999999e-06 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471426+0000", 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4680000000000003e-06 2024-09-16T01:49:42.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471432+0000", 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8449999999999999e-06 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471667+0000", 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00023510999999999999 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.528 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471679+0000", 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2302e-05 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479638+0000", 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0079587300000000007 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479655+0000", 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6833e-05 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.529 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:139 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'97, mlcod=17'89)", 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.041023+0000", 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.438346738999996, 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056538869999999998, 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.530 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041023+0000", 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041023+0000", 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041024+0000", 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5900000000000002e-07 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041028+0000", 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2710000000000003e-06 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041034+0000", 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041038+0000", 2024-09-16T01:49:42.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7529999999999999e-06 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041072+0000", 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4354999999999999e-05 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041085+0000", 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2466999999999999e-05 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046641+0000", 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055564880000000001 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046677+0000", 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5799000000000001e-05 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.534 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:140 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'98, mlcod=17'89)", 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.438018403000001, 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053458910000000002, 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5899999999999998e-07 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041412+0000", 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0520999999999999e-05 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041419+0000", 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7769999999999997e-06 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041422+0000", 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2229999999999999e-06 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041446+0000", 2024-09-16T01:49:42.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4046999999999999e-05 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041458+0000", 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.164e-05 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046679+0000", 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005221275 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046697+0000", 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8148999999999999e-05 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:143 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'101, mlcod=17'98)", 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.048358+0000", 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.431012015999997, 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065183130000000004, 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.539 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048358+0000", 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048358+0000", 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048359+0000", 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-07 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048422+0000", 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3003000000000006e-05 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048427+0000", 2024-09-16T01:49:42.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.93e-06 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048431+0000", 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2789999999999999e-06 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048464+0000", 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2839999999999997e-05 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048479+0000", 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5664000000000001e-05 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054749+0000", 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062695320000000004 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054876+0000", 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012753499999999999 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.543 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:144 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'102, mlcod=17'98)", 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.048708+0000", 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.430661778000001, 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061860739999999997, 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048708+0000", 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048708+0000", 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048709+0000", 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0699999999999998e-07 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048761+0000", 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2342000000000001e-05 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048768+0000", 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6259999999999997e-06 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048771+0000", 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.208e-06 2024-09-16T01:49:42.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048797+0000", 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6361999999999999e-05 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048809+0000", 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1697e-05 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054879+0000", 2024-09-16T01:49:42.547 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060705220000000001 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054894+0000", 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.491e-05 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:145 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'103, mlcod=17'98)", 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.049177+0000", 2024-09-16T01:49:42.548 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.430192929, 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059428930000000003, 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049177+0000", 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049177+0000", 2024-09-16T01:49:42.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049178+0000", 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.58e-07 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049267+0000", 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9957999999999995e-05 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049273+0000", 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6110000000000001e-06 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049277+0000", 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1509999999999997e-06 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049310+0000", 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2486e-05 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049323+0000", 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3349999999999999e-05 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.055098+0000", 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057749769999999997 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.055120+0000", 2024-09-16T01:49:42.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1902000000000001e-05 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825295+0000", 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.654075401999997, 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073553409999999996, 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.553 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825295+0000", 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825295+0000", 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9740000000000001e-06 2024-09-16T01:49:42.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825308+0000", 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9129999999999992e-06 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825328+0000", 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.999e-05 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825338+0000", 2024-09-16T01:49:42.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825911+0000", 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00057331000000000005 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826029+0000", 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011776499999999999 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826161+0000", 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013176900000000001 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827819+0000", 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016579030000000001 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832580+0000", 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047607259999999998 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832618+0000", 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8398000000000001e-05 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832650+0000", 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1903000000000003e-05 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.558 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.896865+0000", 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.582504637, 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063761060000000003, 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896865+0000", 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.559 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896865+0000", 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896868+0000", 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4119999999999999e-06 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896874+0000", 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.596e-06 2024-09-16T01:49:42.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896887+0000", 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2351000000000001e-05 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896895+0000", 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4030000000000007e-06 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896962+0000", 2024-09-16T01:49:42.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7176999999999997e-05 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897082+0000", 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119671 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903241+0000", 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061594960000000004 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:5 2.0 e22/19 2:03625d61:::614.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.581960200000001, 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005944024, 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8299999999999997e-07 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897417+0000", 2024-09-16T01:49:42.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7789999999999998e-06 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897427+0000", 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0271e-05 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897435+0000", 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7230000000000004e-06 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.565 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897513+0000", 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7733999999999998e-05 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897532+0000", 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9072e-05 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903278+0000", 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057462040000000004 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.566 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903354+0000", 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5658e-05 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:1 2.1 e22/19 2:85bbe569:::202.00000000:head v 22'4, mlcod=0'0)", 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898125+0000", 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.581244824999999, 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0051968099999999996, 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.567 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898125+0000", 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898125+0000", 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898126+0000", 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3699999999999999e-07 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898134+0000", 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9340000000000006e-06 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898151+0000", 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7206000000000001e-05 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898160+0000", 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0489999999999993e-06 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898280+0000", 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011925399999999999 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898298+0000", 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610999999999999e-05 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903276+0000", 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049776760000000003 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903322+0000", 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6143000000000002e-05 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.571 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:29 2.d 2:b2057508:::204.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.936198+0000", 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.543172330000001, 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0091713439999999997, 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936198+0000", 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936198+0000", 2024-09-16T01:49:42.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936200+0000", 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5610000000000001e-06 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936207+0000", 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1350000000000003e-06 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936224+0000", 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6254000000000001e-05 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936235+0000", 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1094000000000001e-05 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936314+0000", 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9560000000000004e-05 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936461+0000", 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014680499999999999 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945369+0000", 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0089079350000000005 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.578 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:42.580 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:49:42.580 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:42.579+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:42.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:42.625+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1c0c0cc220 con 0x7f1c0c06d270 2024-09-16T01:49:42.681 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:42.679+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:42.724 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:42.724 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:42.724 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:49:42.725 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:42.734 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:49:42.781 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:42.780+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:42.837 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:42.837 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:42.837 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:42.837 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:42.838 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:42.846 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:49:42.883 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:42.881+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 0 entries 22'1 (0'0) error 2:ddb4e4d7:::200.00000002:head by mds.0.4:33 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:42.952 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.12399053, 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001054845, 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.57e-06 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826088+0000", 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5670000000000001e-05 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826118+0000", 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9603000000000001e-05 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826136+0000", 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.88e-05 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826229+0000", 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2366000000000003e-05 2024-09-16T01:49:42.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827126+0000", 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00089683600000000003 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.1.9:17 2.1b e22/19 2:dbae76b5:::mds1_inotable:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.089450255999999, 2024-09-16T01:49:42.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016283739999999999, 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860613+0000", 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.446e-06 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860625+0000", 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2549e-05 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860645+0000", 2024-09-16T01:49:42.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9545e-05 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860655+0000", 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9550000000000004e-06 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860902+0000", 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024690900000000002 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860935+0000", 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3788999999999998e-05 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862198+0000", 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012625169999999999 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862239+0000", 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1664e-05 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.960 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 4 entries 22'3 (0'0) error 2:db457a8a:::201.00000001:head by mds.1.9:20 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.088134451999998, 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011507889999999999, 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:42.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861928+0000", 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6099999999999999e-07 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861940+0000", 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1817999999999999e-05 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861962+0000", 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2078000000000001e-05 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861976+0000", 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4338000000000001e-05 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862056+0000", 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0253999999999994e-05 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863078+0000", 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213399999999999 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.964 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 5 entries 22'4 (0'0) error 2:d80f8093:::201.00000002:head by mds.1.9:21 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.087806409999999, 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014125819999999999, 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862256+0000", 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5000000000000003e-07 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862264+0000", 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.789e-06 2024-09-16T01:49:42.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862280+0000", 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5633000000000001e-05 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862294+0000", 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4316e-05 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862755+0000", 2024-09-16T01:49:42.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00046123399999999998 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863668+0000", 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00091206000000000002 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.1.9:34 2.1b e22/19 2:db457a8a:::201.00000001:head v 22'5, mlcod=22'3)", 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.080578273999997, 2024-09-16T01:49:42.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00078421100000000004, 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869484+0000", 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0699999999999999e-06 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869492+0000", 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0860000000000002e-06 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869505+0000", 2024-09-16T01:49:42.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.308e-05 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869512+0000", 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3010000000000003e-06 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869573+0000", 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0956999999999999e-05 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869595+0000", 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2311999999999999e-05 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.870232+0000", 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000637456 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.870267+0000", 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4949e-05 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.972 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:35 2.a 2:54ba6923:::mds1_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.059722428000001, 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014025660000000001, 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:42.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890341+0000", 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.607e-06 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890347+0000", 2024-09-16T01:49:42.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9530000000000002e-06 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890358+0000", 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0910000000000001e-05 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890368+0000", 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6330000000000008e-06 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.975 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890418+0000", 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0290999999999997e-05 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890538+0000", 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011986599999999999 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890640+0000", 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101625 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891432+0000", 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00079213799999999996 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891662+0000", 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000230237 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891697+0000", 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4866999999999997e-05 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891741+0000", 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4438999999999999e-05 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:11 2.c 2:36924187:::618.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.054345067, 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055247409999999997, 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.978 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895717+0000", 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1289999999999999e-06 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895724+0000", 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0769999999999996e-06 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895733+0000", 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4210000000000001e-06 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895741+0000", 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0290000000000005e-06 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895816+0000", 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5190000000000003e-05 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895955+0000", 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138737 2024-09-16T01:49:42.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896073+0000", 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011843799999999999 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897963+0000", 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001890061 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901132+0000", 2024-09-16T01:49:42.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031683089999999998 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901196+0000", 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4132000000000002e-05 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901241+0000", 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5218e-05 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:12 2.c 2:36924187:::618.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.054292772, 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056235809999999999, 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895782+0000", 2024-09-16T01:49:42.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2383e-05 2024-09-16T01:49:42.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:42.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0747999999999999e-05 2024-09-16T01:49:42.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895799+0000", 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.849e-06 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896381+0000", 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058177900000000004 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896459+0000", 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8511999999999996e-05 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897012+0000", 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055262699999999998 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900008+0000", 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029960410000000001 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901291+0000", 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00128261 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901351+0000", 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0603000000000002e-05 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901392+0000", 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0799e-05 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.991 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.054247496999999, 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055047550000000001, 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895815+0000", 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.85e-07 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895824+0000", 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4350000000000003e-06 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895842+0000", 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8264999999999999e-05 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895850+0000", 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8339999999999999e-06 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895962+0000", 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111644 2024-09-16T01:49:42.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896088+0000", 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012620200000000001 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896219+0000", 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000130782 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:42.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898012+0000", 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001793474 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901212+0000", 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031999390000000002 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9030999999999998e-05 2024-09-16T01:49:42.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901319+0000", 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7364000000000003e-05 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.054189310999998, 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055742070000000003, 2024-09-16T01:49:42.997 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:42.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895873+0000", 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8100000000000001e-07 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895877+0000", 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3710000000000002e-06 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895886+0000", 2024-09-16T01:49:42.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0739999999999999e-06 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895893+0000", 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3550000000000001e-06 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896519+0000", 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00062635700000000002 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896612+0000", 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3059000000000005e-05 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896737+0000", 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012455499999999999 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900030+0000", 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032929740000000002 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901357+0000", 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001327204 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901403+0000", 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6508000000000003e-05 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901446+0000", 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2868999999999999e-05 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.002 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.054141305000002, 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0070037900000000002, 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895921+0000", 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8899999999999999e-07 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895926+0000", 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.023e-06 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895934+0000", 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1939999999999998e-06 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895941+0000", 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0770000000000002e-06 2024-09-16T01:49:43.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897276+0000", 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001334614 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897409+0000", 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000133215 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897545+0000", 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013621099999999999 2024-09-16T01:49:43.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901441+0000", 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003895459 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902804+0000", 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013631120000000001 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902877+0000", 2024-09-16T01:49:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3647999999999997e-05 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902924+0000", 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6548000000000002e-05 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.2 epoch 22/19 rep_tid 20 entries 22'1 (0'0) error 2:40dbbc66:::202.00000003:head by mds.2.11:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.052931906000001, 2024-09-16T01:49:43.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029206929999999998, 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:43.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897130+0000", 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897142+0000", 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1095e-05 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897165+0000", 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3278999999999999e-05 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897180+0000", 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5613000000000001e-05 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897269+0000", 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8749e-05 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900050+0000", 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027809940000000002 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:28 2.c 2:35dcb8d2:::203.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.038284807, 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047880470000000001, 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.012 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911779+0000", 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3609999999999999e-06 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911799+0000", 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9721e-05 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911817+0000", 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.802e-05 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911851+0000", 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4289000000000003e-05 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911931+0000", 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0142000000000006e-05 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912062+0000", 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013077100000000001 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916565+0000", 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004502743 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:33 2.a 2:556620e9:::203.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.038164158000001, 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026481180000000001, 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.016 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911898+0000", 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2799999999999995e-07 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911912+0000", 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4003e-05 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911922+0000", 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.045e-05 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911929+0000", 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6850000000000004e-06 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911993+0000", 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4196999999999995e-05 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912089+0000", 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5383000000000005e-05 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914545+0000", 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024565720000000002 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.020 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:34 2.c 2:35dcb8d2:::203.00000001:head [write 0~588] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.033100429000001, 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041374530000000001, 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916963+0000", 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.64e-06 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916972+0000", 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7659999999999993e-06 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916990+0000", 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7445e-05 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916999+0000", 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3859999999999996e-06 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917077+0000", 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8307999999999996e-05 2024-09-16T01:49:43.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917172+0000", 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4617000000000006e-05 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917357+0000", 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018458799999999999 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918065+0000", 2024-09-16T01:49:43.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070841800000000005 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.920973+0000", 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029077769999999998 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.921071+0000", 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7918000000000007e-05 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.921098+0000", 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7589999999999998e-05 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:35 2.a 2:579bf162:::mds3_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.020197533000001, 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001587256, 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.026 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929867+0000", 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.858e-06 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929878+0000", 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1651000000000001e-05 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929898+0000", 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9828999999999999e-05 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929907+0000", 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1239999999999994e-06 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929986+0000", 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8608000000000004e-05 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.930169+0000", 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018343600000000001 2024-09-16T01:49:43.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.930319+0000", 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014941299999999999 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931104+0000", 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00078539600000000005 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931374+0000", 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027029000000000002 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931415+0000", 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0068999999999999e-05 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931451+0000", 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6582000000000003e-05 2024-09-16T01:49:43.031 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:24 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.013622230999999, 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048886399999999997, 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.032 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936441+0000", 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0600000000000002e-06 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.033 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936464+0000", 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3161999999999999e-05 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936480+0000", 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5537e-05 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936490+0000", 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6889999999999999e-06 2024-09-16T01:49:43.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936578+0000", 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8596000000000004e-05 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936729+0000", 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015100000000000001 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936946+0000", 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000216586 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939283+0000", 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023372559999999998 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941230+0000", 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001946677 2024-09-16T01:49:43.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941279+0000", 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9478999999999998e-05 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941328+0000", 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8597999999999998e-05 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.4.15:11 2.2 e22/19 2:447438e6:::62c.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.012772870999999, 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002990131, 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937290+0000", 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.299e-06 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937298+0000", 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4400000000000005e-06 2024-09-16T01:49:43.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937309+0000", 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1049e-05 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3410000000000003e-06 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937382+0000", 2024-09-16T01:49:43.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4307999999999995e-05 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937403+0000", 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0664000000000001e-05 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940230+0000", 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028275319999999998 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940279+0000", 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8498000000000002e-05 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.4.15:12 2.2 e22/19 2:447438e6:::62c.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.011951677999999, 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0022086779999999999, 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.042 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938111+0000", 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.95e-07 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938119+0000", 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9370000000000004e-06 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938137+0000", 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8360999999999999e-05 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938145+0000", 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4060000000000005e-06 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938218+0000", 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2739999999999998e-05 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938238+0000", 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9738999999999999e-05 2024-09-16T01:49:43.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940284+0000", 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020457850000000001 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940318+0000", 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4715000000000002e-05 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.046 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.2 epoch 22/19 rep_tid 37 entries 22'4 (0'0) error 2:4182a243:::204.00000001:head by mds.4.15:28 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.011886523000001, 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002163842, 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.047 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5599999999999995e-07 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938191+0000", 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5967000000000001e-05 2024-09-16T01:49:43.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938224+0000", 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2169000000000003e-05 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938239+0000", 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5063e-05 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.049 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938468+0000", 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000229082 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940339+0000", 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001871005 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.050 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:43.052 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:49:43.052 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:43.051+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:43.153 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:43.152+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:43.156 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:43.156 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:43.156 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:49:43.156 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:43.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:49:43.253 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:43.252+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:43.268 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:43.268 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:43.268 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:43.268 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:43.268 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:43.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.270+0000 7f1c02a00640 1 -- 172.21.15.146:0/298163149 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v27) ==== 97+0+0 (secure 0 0 0) 0x7f1bf8017020 con 0x7f1c0c06d270 2024-09-16T01:49:43.272 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.272+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bec06c2b0 msgr2=0x7f1bec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.272+0000 7f1c11400640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bec06c2b0 0x7f1bec06e770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000f60 tx=0x7f1bfc007d50 comp rx=0 tx=0).stop 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 msgr2=0x7f1c0c149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 0x7f1c0c149a10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c071560 tx=0x7f1bf800ed90 comp rx=0 tx=0).stop 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 shutdown_connections 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bec06c2b0 0x7f1bec06e770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c0c0c3f00 0x7f1c0c1f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c0c06d270 0x7f1c0c149a10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 --2- 172.21.15.146:0/298163149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c0c06ce90 0x7f1c0c147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:43.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 >> 172.21.15.146:0/298163149 conn(0x7f1c0c06d8e0 msgr2=0x7f1c0c075110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:43.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 shutdown_connections 2024-09-16T01:49:43.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:43.273+0000 7f1c11400640 1 -- 172.21.15.146:0/298163149 wait complete. 2024-09-16T01:49:43.279 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:49:43.354 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:43.353+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:43.383 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:43.383 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:43.383 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.556518029999999, 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055002239999999997, 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.384 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824857+0000", 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1457000000000001e-05 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824870+0000", 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2483999999999999e-05 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824885+0000", 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4758e-05 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824894+0000", 2024-09-16T01:49:43.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5319999999999995e-06 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015500299999999999 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825242+0000", 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000192986 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:43.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825387+0000", 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000144986 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828137+0000", 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027499 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830217+0000", 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020800369999999999 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.388 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830297+0000", 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0011000000000004e-05 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830346+0000", 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.389 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.556445678999999, 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055792059999999997, 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.390 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824919+0000", 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7000000000000005e-07 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824922+0000", 2024-09-16T01:49:43.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824931+0000", 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824935+0000", 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825813+0000", 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00087810899999999996 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825889+0000", 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6157999999999998e-05 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826034+0000", 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014488599999999999 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828143+0000", 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002108408 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830415+0000", 2024-09-16T01:49:43.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00227247 2024-09-16T01:49:43.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830467+0000", 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830498+0000", 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0956e-05 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.396 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.556400600000003, 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0072987340000000003, 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:43.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824964+0000", 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2800000000000002e-07 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824974+0000", 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0627e-05 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824982+0000", 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3159999999999999e-06 2024-09-16T01:49:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2280000000000001e-06 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.400 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825059+0000", 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9754999999999999e-05 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825232+0000", 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000172986 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.401 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825435+0000", 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00020313199999999999 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828256+0000", 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028206009999999998 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832187+0000", 2024-09-16T01:49:43.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039315920000000002 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832228+0000", 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0420999999999998e-05 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832262+0000", 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4647999999999998e-05 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.556353455999997, 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073664890000000004, 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:43.404 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825015+0000", 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7129999999999999e-06 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825024+0000", 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5629999999999992e-06 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825029+0000", 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9269999999999996e-06 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825830+0000", 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00080144400000000001 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825970+0000", 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014016899999999999 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826100+0000", 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012975500000000001 2024-09-16T01:49:43.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828596+0000", 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024965830000000001 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832306+0000", 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037092140000000002 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832349+0000", 2024-09-16T01:49:43.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3210000000000001e-05 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832377+0000", 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8161e-05 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.556317430999997, 2024-09-16T01:49:43.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059226180000000002, 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:43.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3399999999999999e-07 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825061+0000", 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4144999999999999e-05 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825069+0000", 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5209999999999997e-06 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825074+0000", 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8450000000000002e-06 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826274+0000", 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001200586 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826528+0000", 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000253446 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826691+0000", 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016368900000000001 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828551+0000", 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0018598 2024-09-16T01:49:43.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830882+0000", 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023311180000000001 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830929+0000", 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6882999999999998e-05 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830969+0000", 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0051e-05 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.520829141999997, 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.415 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:49:43.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.520730635, 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:43.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:49:43.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:49:43.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:7 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:43.427 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.485739481, 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057447150000000001, 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895627+0000", 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0480000000000001e-06 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895632+0000", 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5879999999999997e-06 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895650+0000", 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7322000000000001e-05 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895657+0000", 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1949999999999997e-06 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895729+0000", 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2120999999999996e-05 2024-09-16T01:49:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895880+0000", 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015082600000000001 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896034+0000", 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015426799999999999 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897859+0000", 2024-09-16T01:49:43.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001824736 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034126600000000001 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901329+0000", 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7836e-05 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901369+0000", 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114999999999998e-05 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:8 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.485690298000002, 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058392840000000001, 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.433 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9900000000000001e-07 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895677+0000", 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1e-06 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895684+0000", 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5769999999999999e-06 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895699+0000", 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.473e-05 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896845+0000", 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011460960000000001 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896914+0000", 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9115999999999994e-05 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897029+0000", 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114974 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900398+0000", 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033696749999999999 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.437 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901420+0000", 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213030000000001 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901470+0000", 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0083999999999997e-05 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.438 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901513+0000", 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3130000000000002e-05 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.439 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.485637183000001, 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048597969999999999, 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895737+0000", 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2070000000000001e-06 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895743+0000", 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3430000000000001e-06 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895747+0000", 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0940000000000001e-06 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895803+0000", 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6339999999999999e-05 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895947+0000", 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143199 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896084+0000", 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000137126 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900351+0000", 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042672250000000004 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900515+0000", 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016424500000000001 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900552+0000", 2024-09-16T01:49:43.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6804999999999997e-05 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900587+0000", 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4582999999999997e-05 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.445 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.485606220000001, 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049417419999999998, 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895761+0000", 2024-09-16T01:49:43.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4760000000000001e-06 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895765+0000", 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2540000000000003e-06 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895772+0000", 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4679999999999997e-06 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896412+0000", 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064042899999999998 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896492+0000", 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0500000000000005e-05 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896615+0000", 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000122194 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900360+0000", 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037456830000000001 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900635+0000", 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027509799999999999 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900673+0000", 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7265999999999999e-05 2024-09-16T01:49:43.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900700+0000", 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.688e-05 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.485584607, 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0078497150000000002, 2024-09-16T01:49:43.451 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895780+0000", 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1699999999999998e-07 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895783+0000", 2024-09-16T01:49:43.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9239999999999999e-06 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895789+0000", 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8080000000000001e-06 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4089999999999999e-06 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897342+0000", 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001550287 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897499+0000", 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015634399999999999 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897630+0000", 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013139099999999999 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902981+0000", 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053504950000000003 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903540+0000", 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055920099999999995 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903591+0000", 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1224999999999998e-05 2024-09-16T01:49:43.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903629+0000", 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8114000000000001e-05 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1e epoch 22/19 rep_tid 7 entries 22'2 (0'0) error 2:7d92db23:::202.00000002:head by mds.2.11:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:43.457 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.484025658, 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056237750000000001, 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897339+0000", 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.029e-06 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897350+0000", 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0913e-05 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898029+0000", 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00067867600000000002 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898053+0000", 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3799999999999999e-05 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898142+0000", 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9368000000000002e-05 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902962+0000", 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048199890000000002 2024-09-16T01:49:43.461 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:17 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'11, mlcod=22'9)", 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.482771613000001, 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043821320000000004, 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.462 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.463 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898593+0000", 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.023e-06 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898603+0000", 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2490000000000007e-06 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898619+0000", 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6073e-05 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898627+0000", 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3950000000000001e-06 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898688+0000", 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0545999999999999e-05 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898704+0000", 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5903999999999999e-05 2024-09-16T01:49:43.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902933+0000", 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042294389999999998 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902975+0000", 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1502999999999999e-05 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:18 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'12, mlcod=22'9)", 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.482624018999999, 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042640050000000004, 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.467 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898741+0000", 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8999999999999995e-07 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898747+0000", 2024-09-16T01:49:43.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3029999999999996e-06 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898756+0000", 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3969999999999994e-06 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898764+0000", 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.469 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898924+0000", 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016043599999999999 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898948+0000", 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4068999999999999e-05 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902977+0000", 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040289449999999999 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903004+0000", 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6784e-05 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:26 2.10 e22/19 2:091e33df:::502.00000000:head v 22'14, mlcod=22'9)", 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.481667391999999, 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039915870000000004, 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.471 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899698+0000", 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9999999999999995e-07 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899706+0000", 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.96e-06 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899791+0000", 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5702e-05 2024-09-16T01:49:43.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899800+0000", 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6840000000000002e-06 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899870+0000", 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9617999999999997e-05 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899888+0000", 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8046e-05 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903646+0000", 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037583310000000002 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903688+0000", 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2246000000000002e-05 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.475 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:26 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.469448738000001, 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050127030000000003, 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.476 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911918+0000", 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3479999999999998e-06 2024-09-16T01:49:43.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911934+0000", 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6659000000000001e-05 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911948+0000", 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3334e-05 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911957+0000", 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8390000000000002e-06 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912051+0000", 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3702999999999994e-05 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912182+0000", 2024-09-16T01:49:43.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013119100000000001 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000179147 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.480 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913761+0000", 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001399382 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916815+0000", 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030542350000000002 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.481 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916884+0000", 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8823000000000002e-05 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916928+0000", 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4042000000000003e-05 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:24 2.13 2:cbf94b35:::mds3_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.482 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.469368413000002, 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057070769999999996, 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912001+0000", 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2320000000000003e-06 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912015+0000", 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4562000000000001e-05 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912022+0000", 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.165e-06 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912138+0000", 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116098 2024-09-16T01:49:43.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912260+0000", 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121224 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101796 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913434+0000", 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010729610000000001 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917594+0000", 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041594010000000001 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917652+0000", 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8357999999999999e-05 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917703+0000", 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0460000000000001e-05 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:13 2.13 2:c81eaac8:::62d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.445663884999998, 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004875094, 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.488 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935702+0000", 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.277e-06 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935711+0000", 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6100000000000006e-06 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935729+0000", 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8224e-05 2024-09-16T01:49:43.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935742+0000", 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2405999999999999e-05 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935832+0000", 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0382000000000003e-05 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936059+0000", 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022734599999999999 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936207+0000", 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014755500000000001 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938029+0000", 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001821694 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940508+0000", 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024797479999999999 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8040000000000002e-05 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940575+0000", 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8812000000000001e-05 2024-09-16T01:49:43.493 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:14 2.13 2:c81eaac8:::62d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.445501212, 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055486820000000001, 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935864+0000", 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935868+0000", 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7639999999999996e-06 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935877+0000", 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7560000000000005e-06 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935888+0000", 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1184e-05 2024-09-16T01:49:43.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937525+0000", 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001637023 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937611+0000", 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6114999999999998e-05 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937759+0000", 2024-09-16T01:49:43.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014728699999999999 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938463+0000", 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070381299999999996 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941309+0000", 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002846374 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941366+0000", 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6848000000000002e-05 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941411+0000", 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5698000000000002e-05 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.499 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.500 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.500 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:43.500 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:43.500 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:43.501 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:43.500+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:43.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:49:43.602 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:43.601+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:43.666 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:43.666 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:43.666 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:43.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:49:43.725 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:43.723+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:43.805 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:43.803+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:43.835 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:43.836 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:43.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:49:43.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:49:43.905 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:43.904+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:44.007 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:44.004+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:44.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.047+0000 7fc879600640 1 Processor -- start 2024-09-16T01:49:44.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.047+0000 7fc879600640 1 -- start start 2024-09-16T01:49:44.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc879600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc874154f40 0x7fc8740c7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc879600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc87415eac0 0x7fc8740d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc879600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8740cb600 con 0x7fc8740d0810 2024-09-16T01:49:44.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc879600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc8740cbbc0 con 0x7fc874154f40 2024-09-16T01:49:44.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc879600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc874155320 con 0x7fc87415eac0 2024-09-16T01:49:44.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc873400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc8740d0810 0x7fc87415ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:48866/0 (socket says 172.21.15.146:48866) 2024-09-16T01:49:44.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc87415eac0 0x7fc8740d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc873400640 1 -- 172.21.15.146:0/1013932415 learned_addr learned my addr 172.21.15.146:0/1013932415 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:44.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872a00640 1 --2- 172.21.15.146:0/1013932415 >> v2:172.21.15.161:3300/0 conn(0x7fc874154f40 0x7fc8740c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc871600640 1 -- 172.21.15.146:0/1013932415 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233172433 0 0) 0x7fc8740cb600 con 0x7fc8740d0810 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc871600640 1 -- 172.21.15.146:0/1013932415 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc858003520 con 0x7fc8740d0810 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872000640 1 -- 172.21.15.146:0/1013932415 >> v2:172.21.15.161:3300/0 conn(0x7fc874154f40 msgr2=0x7fc8740c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872000640 1 --2- 172.21.15.146:0/1013932415 >> v2:172.21.15.161:3300/0 conn(0x7fc874154f40 0x7fc8740c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872000640 1 -- 172.21.15.146:0/1013932415 >> v1:172.21.15.161:6789/0 conn(0x7fc8740d0810 legacy=0x7fc87415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872000640 1 -- 172.21.15.146:0/1013932415 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8740cb8e0 con 0x7fc87415eac0 2024-09-16T01:49:44.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.050+0000 7fc872a00640 1 --2- 172.21.15.146:0/1013932415 >> v2:172.21.15.161:3300/0 conn(0x7fc874154f40 0x7fc8740c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:44.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.051+0000 7fc872000640 1 --2- 172.21.15.146:0/1013932415 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc87415eac0 0x7fc8740d02d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc8600097a0 tx=0x7fc86002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=839c69a595488ce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.051+0000 7fc871600640 1 -- 172.21.15.146:0/1013932415 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860037070 con 0x7fc87415eac0 2024-09-16T01:49:44.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.051+0000 7fc871600640 1 -- 172.21.15.146:0/1013932415 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc860032440 con 0x7fc87415eac0 2024-09-16T01:49:44.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.051+0000 7fc871600640 1 -- 172.21.15.146:0/1013932415 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860032810 con 0x7fc87415eac0 2024-09-16T01:49:44.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 -- 172.21.15.146:0/1013932415 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc87415eac0 msgr2=0x7fc8740d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 --2- 172.21.15.146:0/1013932415 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc87415eac0 0x7fc8740d02d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc8600097a0 tx=0x7fc86002a180 comp rx=0 tx=0).stop 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 -- 172.21.15.146:0/1013932415 shutdown_connections 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 --2- 172.21.15.146:0/1013932415 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc87415eac0 0x7fc8740d02d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 --2- 172.21.15.146:0/1013932415 >> v2:172.21.15.161:3300/0 conn(0x7fc874154f40 0x7fc8740c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 -- 172.21.15.146:0/1013932415 >> 172.21.15.146:0/1013932415 conn(0x7fc87406cb50 msgr2=0x7fc87406e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 -- 172.21.15.146:0/1013932415 shutdown_connections 2024-09-16T01:49:44.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.052+0000 7fc879600640 1 -- 172.21.15.146:0/1013932415 wait complete. 2024-09-16T01:49:44.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.053+0000 7fc879600640 1 Processor -- start 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 -- start start 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8740d0810 0x7fc8741592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc874154f40 0x7fc8741e78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc87415eac0 0x7fc8741f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc87415eee0 con 0x7fc8740d0810 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8740b6210 con 0x7fc87415eac0 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.054+0000 7fc879600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc87415b280 con 0x7fc874154f40 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8740d0810 0x7fc8741592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc874154f40 0x7fc8741e78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc873400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc87415eac0 0x7fc8741f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8740d0810 0x7fc8741592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59732/0 (socket says 172.21.15.146:59732) 2024-09-16T01:49:44.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872a00640 1 -- 172.21.15.146:0/734517860 learned_addr learned my addr 172.21.15.146:0/734517860 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc87415eac0 msgr2=0x7fc8741f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc87415eac0 0x7fc8741f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8740d0810 msgr2=0x7fc8741592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8740d0810 0x7fc8741592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87415e070 con 0x7fc874154f40 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc8740d0810 0x7fc8741592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc87415eac0 0x7fc8741f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.055+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc874154f40 0x7fc8741e78a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc860031000 tx=0x7fc8600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.056+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860041030 con 0x7fc874154f40 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.056+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86003d4e0 con 0x7fc874154f40 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.056+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860037040 con 0x7fc874154f40 2024-09-16T01:49:44.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.056+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8741582d0 con 0x7fc874154f40 2024-09-16T01:49:44.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.056+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87415bd50 con 0x7fc874154f40 2024-09-16T01:49:44.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.057+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc87415a470 con 0x7fc874154f40 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.232162774999999, 2024-09-16T01:49:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:49:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:49:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:49:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.232056726000003, 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:49:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:49:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.232002100000003, 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:49:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:49:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.231933767000001, 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:49:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.197678564, 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:49:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:49:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.197495365999998, 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:49:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:49:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:49:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.197441357999999, 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:49:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:49:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.197408918000001, 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:49:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.194040917000002, 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:49:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.193970241000002, 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:49:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.193933866000002, 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:49:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.193306122999999, 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:49:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.193251515999997, 2024-09-16T01:49:44.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:44.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.133 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.121113107999999, 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:49:44.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:49:44.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.121054795999999, 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:44.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.120980720999999, 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.120922100000001, 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:44.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:44.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.120900165999998, 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:44.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:49:44.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:49:44.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:49:44.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.120869806000002, 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:49:44.165 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:49:44.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.120843227000002, 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:44.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:44.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:49:44.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:49:44.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:44.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:44.178 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:44.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.058+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc86003d8b0 con 0x7fc874154f40 2024-09-16T01:49:44.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.059+0000 7fc86aa00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.059+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5767+0+0 (secure 0 0 0) 0x7fc8600ab7e0 con 0x7fc874154f40 2024-09-16T01:49:44.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.059+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.059+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v27) ==== 54+0+16 (secure 0 0 0) 0x7fc86004b020 con 0x7fc874154f40 2024-09-16T01:49:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.060+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc874159b20 0x7fc8741f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.060+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc874158570 con 0x7fc874159b20 2024-09-16T01:49:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.060+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc864009560 tx=0x7fc864009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.060+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc874159b20 0x7fc8741f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.061+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc874159b20 0x7fc8741f3d70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.061+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc874158570 con 0x7fc874159b20 2024-09-16T01:49:44.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:49:44.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.077+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc874154760 con 0x7fc874159b20 2024-09-16T01:49:44.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.078+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc874154760 con 0x7fc874159b20 2024-09-16T01:49:44.182 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:44.183 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:44.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.078+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc87414f770 0x7fc87414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.078+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc8740cd6d0 con 0x7fc87414f770 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.079+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc87414f770 0x7fc87414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.079+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc87414f770 0x7fc87414fb50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.080+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8740cd6d0 con 0x7fc87414f770 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.094+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc8740d0ea0 con 0x7fc87414f770 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.096+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8740d0ea0 con 0x7fc87414f770 2024-09-16T01:49:44.184 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:44.185 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:44.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.096+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc874062630 0x7fc874062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.096+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc8741505a0 con 0x7fc874062630 2024-09-16T01:49:44.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.097+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc874062630 0x7fc874062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.097+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc874062630 0x7fc874062a80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.098+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8741505a0 con 0x7fc874062630 2024-09-16T01:49:44.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.112+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc8741e7de0 con 0x7fc874062630 2024-09-16T01:49:44.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.114+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8741e7de0 con 0x7fc874062630 2024-09-16T01:49:44.186 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.114+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc8741530d0 0x7fc874153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.114+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc87415c5b0 con 0x7fc8741530d0 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.114+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc8741530d0 0x7fc874153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.114+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc8741530d0 0x7fc874153510 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.114+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc87415c5b0 con 0x7fc8741530d0 2024-09-16T01:49:44.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.128+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc87415c5b0 con 0x7fc8741530d0 2024-09-16T01:49:44.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.129+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc87415c5b0 con 0x7fc8741530d0 2024-09-16T01:49:44.188 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.130+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc87402c640 0x7fc87402ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.130+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc874153f60 con 0x7fc87402c640 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.130+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc87402c640 0x7fc87402ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.131+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc87402c640 0x7fc87402ca80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.131+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc874153f60 con 0x7fc87402c640 2024-09-16T01:49:44.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.146+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc874153f60 con 0x7fc87402c640 2024-09-16T01:49:44.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.148+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc874153f60 con 0x7fc87402c640 2024-09-16T01:49:44.190 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.148+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc87402d4d0 0x7fc874206dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.148+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc87402d920 con 0x7fc87402d4d0 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.148+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc87402d4d0 0x7fc874206dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.148+0000 7fc872000640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc87402d4d0 0x7fc874206dd0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.148+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc87402d920 con 0x7fc87402d4d0 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.161+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc874153f60 con 0x7fc87402d4d0 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.164+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc874153f60 con 0x7fc87402d4d0 2024-09-16T01:49:44.191 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:44.192 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:44.182+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.164+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc87402d920 0x7fc874209830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.164+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc87420c8f0 con 0x7fc87402d920 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.164+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc87402d920 0x7fc874209830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.165+0000 7fc873400640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc87402d920 0x7fc874209830 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.165+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc87420c8f0 con 0x7fc87402d920 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.179+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc87420c880 con 0x7fc87402d920 2024-09-16T01:49:44.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.182+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc87420c880 con 0x7fc87402d920 2024-09-16T01:49:44.194 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:44.195 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:44.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.182+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc87420bae0 0x7fc87420bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:44.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.182+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc87420fcc0 con 0x7fc87420bae0 2024-09-16T01:49:44.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.182+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc87420bae0 0x7fc87420bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:44.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.182+0000 7fc872a00640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc87420bae0 0x7fc87420bf20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:44.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.183+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc87420fcc0 con 0x7fc87420bae0 2024-09-16T01:49:44.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.196+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc874209e90 con 0x7fc87420bae0 2024-09-16T01:49:44.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc86aa00640 1 -- 172.21.15.146:0/734517860 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc874209e90 con 0x7fc87420bae0 2024-09-16T01:49:44.200 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:44.201 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:44.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc874159b20 msgr2=0x7fc8741f3d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc874159b20 0x7fc8741f3d70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc87414f770 msgr2=0x7fc87414fb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc87414f770 0x7fc87414fb50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc874062630 msgr2=0x7fc874062a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc874062630 0x7fc874062a80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc8741530d0 msgr2=0x7fc874153510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc8741530d0 0x7fc874153510 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc87402c640 msgr2=0x7fc87402ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc87402c640 0x7fc87402ca80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc87402d4d0 msgr2=0x7fc874206dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc87402d4d0 0x7fc874206dd0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc87402d920 msgr2=0x7fc874209830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc87402d920 0x7fc874209830 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc87420bae0 msgr2=0x7fc87420bf20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc87420bae0 0x7fc87420bf20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc873400640 1 -- 172.21.15.146:0/734517860 reap_dead start 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc873400640 1 -- 172.21.15.146:0/734517860 reap_dead start 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc873400640 1 -- 172.21.15.146:0/734517860 reap_dead start 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc873400640 1 -- 172.21.15.146:0/734517860 reap_dead start 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc873400640 1 -- 172.21.15.146:0/734517860 reap_dead start 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 msgr2=0x7fc85406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc864009560 tx=0x7fc864009190 comp rx=0 tx=0).stop 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc874154f40 msgr2=0x7fc8741e78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc874154f40 0x7fc8741e78a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc860031000 tx=0x7fc8600046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 shutdown_connections 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 --2- 172.21.15.146:0/734517860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc874154f40 0x7fc8741e78a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:44.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 >> 172.21.15.146:0/734517860 conn(0x7fc87406cb50 msgr2=0x7fc874152760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:44.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 shutdown_connections 2024-09-16T01:49:44.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:44.199+0000 7fc879600640 1 -- 172.21.15.146:0/734517860 wait complete. 2024-09-16T01:49:44.284 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:44.283+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:44.353 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:44.353 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:44.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:49:44.385 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:44.384+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:44.486 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:44.485+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:44.525 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:44.525 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:44.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:49:44.586 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:44.585+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:44.687 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:44.685+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:44.697 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:44.697 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:44.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:49:44.796 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:44.794+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:44.889 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:44.888+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:44.909 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:44.909 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:44.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:49:44.990 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:44.989+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:45.091 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:45.089+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:45.121 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:45.122 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:45.134 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:49:45.221 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:45.219+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:45.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:49:45.293 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:45.293+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:45.336 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:45.337 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:45.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:49:45.394 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:45.393+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:45.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.450+0000 7f68b5800640 1 Processor -- start 2024-09-16T01:49:45.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.450+0000 7f68b5800640 1 -- start start 2024-09-16T01:49:45.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.451+0000 7f68b5800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68b0151a90 0x7f68b00c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:45.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.451+0000 7f68b5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68b00c6410 0x7f68b0158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:45.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68b5800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68b0151e70 con 0x7f68b00c5280 2024-09-16T01:49:45.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68b5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f68b00c8130 con 0x7f68b00c6410 2024-09-16T01:49:45.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68b5800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f68b00c8410 con 0x7f68b0151a90 2024-09-16T01:49:45.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68a7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68b00c6410 0x7f68b0158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68af400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f68b00c5280 0x7f68b015aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47106/0 (socket says 172.21.15.146:47106) 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68af400640 1 -- 172.21.15.146:0/1856203328 learned_addr learned my addr 172.21.15.146:0/1856203328 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68aea00640 1 --2- 172.21.15.146:0/1856203328 >> v2:172.21.15.161:3300/0 conn(0x7f68b0151a90 0x7f68b00c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68a7e00640 1 -- 172.21.15.146:0/1856203328 >> v2:172.21.15.161:3300/0 conn(0x7f68b0151a90 msgr2=0x7f68b00c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68a7e00640 1 --2- 172.21.15.146:0/1856203328 >> v2:172.21.15.161:3300/0 conn(0x7f68b0151a90 0x7f68b00c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68a7e00640 1 -- 172.21.15.146:0/1856203328 >> v1:172.21.15.146:6789/0 conn(0x7f68b00c5280 legacy=0x7f68b015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68a7e00640 1 -- 172.21.15.146:0/1856203328 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b00c86f0 con 0x7f68b00c6410 2024-09-16T01:49:45.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.452+0000 7f68ae000640 1 -- 172.21.15.146:0/1856203328 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560203056 0 0) 0x7f68b0151e70 con 0x7f68b00c5280 2024-09-16T01:49:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.453+0000 7f68a7e00640 1 --2- 172.21.15.146:0/1856203328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b00c6410 0x7f68b0158040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68b0069950 tx=0x7f689802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528bfbeffe63fea5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.454+0000 7f68ae000640 1 -- 172.21.15.146:0/1856203328 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898037070 con 0x7f68b00c6410 2024-09-16T01:49:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.454+0000 7f68ae000640 1 -- 172.21.15.146:0/1856203328 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6898032440 con 0x7f68b00c6410 2024-09-16T01:49:45.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.454+0000 7f68ae000640 1 -- 172.21.15.146:0/1856203328 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898032810 con 0x7f68b00c6410 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.454+0000 7f68b5800640 1 -- 172.21.15.146:0/1856203328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b00c6410 msgr2=0x7f68b0158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.454+0000 7f68b5800640 1 --2- 172.21.15.146:0/1856203328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b00c6410 0x7f68b0158040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68b0069950 tx=0x7f689802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.455+0000 7f68b5800640 1 -- 172.21.15.146:0/1856203328 shutdown_connections 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.455+0000 7f68b5800640 1 --2- 172.21.15.146:0/1856203328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b00c6410 0x7f68b0158040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.455+0000 7f68b5800640 1 --2- 172.21.15.146:0/1856203328 >> v2:172.21.15.161:3300/0 conn(0x7f68b0151a90 0x7f68b00c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.455+0000 7f68b5800640 1 -- 172.21.15.146:0/1856203328 >> 172.21.15.146:0/1856203328 conn(0x7f68b006f3c0 msgr2=0x7f68b006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.455+0000 7f68b5800640 1 -- 172.21.15.146:0/1856203328 shutdown_connections 2024-09-16T01:49:45.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.455+0000 7f68b5800640 1 -- 172.21.15.146:0/1856203328 wait complete. 2024-09-16T01:49:45.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.457+0000 7f68b5800640 1 Processor -- start 2024-09-16T01:49:45.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.457+0000 7f68b5800640 1 -- start start 2024-09-16T01:49:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68b5800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 0x7f68b00bb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68b5800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68b00c6410 0x7f68b0071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68b5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b0151a90 0x7f68b01527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68b5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0157b20 con 0x7f68b0151a90 2024-09-16T01:49:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68b5800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0157840 con 0x7f68b00c6410 2024-09-16T01:49:45.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68b5800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f68b00c3bf0 con 0x7f68b00c5280 2024-09-16T01:49:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68aea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 0x7f68b00bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68aea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 0x7f68b00bb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:33054/0 (socket says 172.21.15.146:33054) 2024-09-16T01:49:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68a7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68b00c6410 0x7f68b0071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68aea00640 1 -- 172.21.15.146:0/958028913 learned_addr learned my addr 172.21.15.146:0/958028913 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:45.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.459+0000 7f68af400640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b0151a90 0x7f68b01527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68aea00640 1 -- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68b00c6410 msgr2=0x7f68b0071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68aea00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68b00c6410 0x7f68b0071fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68aea00640 1 -- 172.21.15.146:0/958028913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b0151a90 msgr2=0x7f68b01527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68aea00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b0151a90 0x7f68b01527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68aea00640 1 -- 172.21.15.146:0/958028913 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b01e5ba0 con 0x7f68b00c5280 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68a7e00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68b00c6410 0x7f68b0071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68aea00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 0x7f68b00bb820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68a000e870 tx=0x7f68a000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0017070 con 0x7f68b00c5280 2024-09-16T01:49:45.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a000c940 con 0x7f68b00c5280 2024-09-16T01:49:45.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b01f0f50 con 0x7f68b00c5280 2024-09-16T01:49:45.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a000cd30 con 0x7f68b00c5280 2024-09-16T01:49:45.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.460+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b01e58c0 con 0x7f68b00c5280 2024-09-16T01:49:45.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.462+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68b00c71f0 con 0x7f68b00c5280 2024-09-16T01:49:45.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.462+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f68a001a070 con 0x7f68b00c5280 2024-09-16T01:49:45.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.464+0000 7f68a6a00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f688c06c2b0 0x7f688c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:45.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.464+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5767+0+0 (secure 0 0 0) 0x7f68a008b2a0 con 0x7f68b00c5280 2024-09-16T01:49:45.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.464+0000 7f68a7e00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f688c06c2b0 0x7f688c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:45.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.465+0000 7f68a7e00640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f688c06c2b0 0x7f688c06e770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6898034000 tx=0x7f689802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:45.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.468+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f68a0059370 con 0x7f68b00c5280 2024-09-16T01:49:45.495 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:45.494+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:45.557 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:45.557 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:45.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:49:45.596 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:45.595+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:45.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:45.661+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f68b00c5910 con 0x7f68b00c5280 2024-09-16T01:49:45.697 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:45.696+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:45.727 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:45.727 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:45.740 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:49:45.797 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:45.796+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:45.898 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:45.897+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:45.899 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:45.899 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:45.912 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:49:45.999 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:45.999+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:46.028 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:46.028 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:46.037 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:49:46.101 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:46.100+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:46.141 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:46.141 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:46.149 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:49:46.202 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:46.201+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:46.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:49:46.253 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:46.253 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:46.261 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:49:46.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.273+0000 7f68a6a00640 1 -- 172.21.15.146:0/958028913 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v28) ==== 91+0+0 (secure 0 0 0) 0x7f68a005d670 con 0x7f68b00c5280 2024-09-16T01:49:46.274 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:49:46.276 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:49:46.276 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:49:46.276 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T01:49:46.276 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:49:46.276 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:49:46.276 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-09-16T01:49:46.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f688c06c2b0 msgr2=0x7f688c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f688c06c2b0 0x7f688c06e770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6898034000 tx=0x7f689802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:46.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 msgr2=0x7f68b00bb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 0x7f68b00bb820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f68a000e870 tx=0x7f68a000ed30 comp rx=0 tx=0).stop 2024-09-16T01:49:46.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 shutdown_connections 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f688c06c2b0 0x7f688c06e770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68b0151a90 0x7f68b01527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68b00c6410 0x7f68b0071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 --2- 172.21.15.146:0/958028913 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f68b00c5280 0x7f68b00bb820 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 >> 172.21.15.146:0/958028913 conn(0x7f68b006f3c0 msgr2=0x7f68b0156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 shutdown_connections 2024-09-16T01:49:46.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.276+0000 7f68b5800640 1 -- 172.21.15.146:0/958028913 wait complete. 2024-09-16T01:49:46.305 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:46.304+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:46.370 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:46.370 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.371+0000 7fd390a00640 1 Processor -- start 2024-09-16T01:49:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.371+0000 7fd390a00640 1 -- start start 2024-09-16T01:49:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd390a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd38c151820 0x7fd38c0c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd390a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd38c071a10 0x7fd38c0c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd390a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c1585b0 con 0x7fd38c0c7980 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd390a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd38c1580a0 con 0x7fd38c071a10 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd390a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd38c0c6680 con 0x7fd38c151820 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd389e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd38c151820 0x7fd38c0c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd381400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd38c0c7980 0x7fd38c0c8070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47112/0 (socket says 172.21.15.146:47112) 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd381400640 1 -- 172.21.15.146:0/1515701880 learned_addr learned my addr 172.21.15.146:0/1515701880 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.372+0000 7fd38a800640 1 --2- 172.21.15.146:0/1515701880 >> v2:172.21.15.146:3300/0 conn(0x7fd38c071a10 0x7fd38c0c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.373+0000 7fd389e00640 1 -- 172.21.15.146:0/1515701880 >> v2:172.21.15.146:3300/0 conn(0x7fd38c071a10 msgr2=0x7fd38c0c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.373+0000 7fd389e00640 1 --2- 172.21.15.146:0/1515701880 >> v2:172.21.15.146:3300/0 conn(0x7fd38c071a10 0x7fd38c0c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.373+0000 7fd389e00640 1 -- 172.21.15.146:0/1515701880 >> v1:172.21.15.146:6789/0 conn(0x7fd38c0c7980 legacy=0x7fd38c0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.373+0000 7fd389e00640 1 -- 172.21.15.146:0/1515701880 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c1582d0 con 0x7fd38c151820 2024-09-16T01:49:46.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.374+0000 7fd389e00640 1 --2- 172.21.15.146:0/1515701880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c151820 0x7fd38c0c36b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd38c071690 tx=0x7fd37c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=570897220dd375b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.374+0000 7fd389400640 1 -- 172.21.15.146:0/1515701880 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c007740 con 0x7fd38c151820 2024-09-16T01:49:46.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.374+0000 7fd389400640 1 -- 172.21.15.146:0/1515701880 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c0078e0 con 0x7fd38c151820 2024-09-16T01:49:46.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.374+0000 7fd389400640 1 -- 172.21.15.146:0/1515701880 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c007cb0 con 0x7fd38c151820 2024-09-16T01:49:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 -- 172.21.15.146:0/1515701880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c151820 msgr2=0x7fd38c0c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 --2- 172.21.15.146:0/1515701880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c151820 0x7fd38c0c36b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd38c071690 tx=0x7fd37c02a710 comp rx=0 tx=0).stop 2024-09-16T01:49:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 -- 172.21.15.146:0/1515701880 shutdown_connections 2024-09-16T01:49:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 --2- 172.21.15.146:0/1515701880 >> v2:172.21.15.146:3300/0 conn(0x7fd38c071a10 0x7fd38c0c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 --2- 172.21.15.146:0/1515701880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c151820 0x7fd38c0c36b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 -- 172.21.15.146:0/1515701880 >> 172.21.15.146:0/1515701880 conn(0x7fd38c06bc10 msgr2=0x7fd38c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 -- 172.21.15.146:0/1515701880 shutdown_connections 2024-09-16T01:49:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.375+0000 7fd390a00640 1 -- 172.21.15.146:0/1515701880 wait complete. 2024-09-16T01:49:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.376+0000 7fd390a00640 1 Processor -- start 2024-09-16T01:49:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.376+0000 7fd390a00640 1 -- start start 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.377+0000 7fd390a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 0x7fd38c153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.377+0000 7fd390a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd38c0c7980 0x7fd38c073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.377+0000 7fd390a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd38c151820 0x7fd38c1485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.377+0000 7fd390a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c0c74f0 con 0x7fd38c0c7980 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.377+0000 7fd390a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c071e80 con 0x7fd38c071a10 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.377+0000 7fd390a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd38c0c6ea0 con 0x7fd38c151820 2024-09-16T01:49:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 0x7fd38c153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 0x7fd38c153cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:58058/0 (socket says 172.21.15.146:58058) 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd38a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd38c151820 0x7fd38c1485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 -- 172.21.15.146:0/325188722 learned_addr learned my addr 172.21.15.146:0/325188722 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd38c0c7980 0x7fd38c073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd38c151820 msgr2=0x7fd38c1485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd38c151820 0x7fd38c1485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd38c0c7980 msgr2=0x7fd38c073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd38c0c7980 0x7fd38c073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c14c310 con 0x7fd38c071a10 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd38c151820 0x7fd38c1485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.378+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 0x7fd38c153cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009e80 tx=0x7fd37c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.382 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:49:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.379+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c040070 con 0x7fd38c071a10 2024-09-16T01:49:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.379+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c031c40 con 0x7fd38c071a10 2024-09-16T01:49:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.379+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c152c20 con 0x7fd38c071a10 2024-09-16T01:49:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.379+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c009450 con 0x7fd38c071a10 2024-09-16T01:49:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.379+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c1546a0 con 0x7fd38c071a10 2024-09-16T01:49:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.379+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd38c0c7180 con 0x7fd38c071a10 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.381+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd37c039040 con 0x7fd38c071a10 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.381+0000 7fd382a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd36806c2b0 0x7fd36806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5767+0+0 (secure 0 0 0) 0x7fd37c0acef0 con 0x7fd38c071a10 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd36806c2b0 0x7fd36806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v28) ==== 54+0+16 (secure 0 0 0) 0x7fd37c0ad2a0 con 0x7fd38c071a10 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd38c148b20 0x7fd38c1f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd38c0c7e30 con 0x7fd38c148b20 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd36806c2b0 0x7fd36806e770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd38c14b950 tx=0x7fd374007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.382+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd38c148b20 0x7fd38c1f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.383+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd38c148b20 0x7fd38c1f25e0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.384+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c0c7e30 con 0x7fd38c148b20 2024-09-16T01:49:46.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.398+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c14b980 con 0x7fd38c148b20 2024-09-16T01:49:46.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.400+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c14b980 con 0x7fd38c148b20 2024-09-16T01:49:46.401 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.400+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd38c141100 0x7fd38c145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.400+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd38c14b6a0 con 0x7fd38c141100 2024-09-16T01:49:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.400+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd38c141100 0x7fd38c145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.401+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd38c141100 0x7fd38c145cb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.401+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c14b6a0 con 0x7fd38c141100 2024-09-16T01:49:46.403 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:46.402+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:46.412 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:46.412 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T01:49:46.412 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:46.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.417+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c1556e0 con 0x7fd38c141100 2024-09-16T01:49:46.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.419+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c1556e0 con 0x7fd38c141100 2024-09-16T01:49:46.420 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:46.421 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:46.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.419+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd38c146700 0x7fd38c062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.419+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd38c146ae0 con 0x7fd38c146700 2024-09-16T01:49:46.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.419+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd38c146700 0x7fd38c062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.420+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd38c146700 0x7fd38c062630 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.421+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c146ae0 con 0x7fd38c146700 2024-09-16T01:49:46.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.434+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c062e40 con 0x7fd38c146700 2024-09-16T01:49:46.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.436+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c062e40 con 0x7fd38c146700 2024-09-16T01:49:46.437 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:46.437 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:46.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.436+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd38c0633b0 0x7fd38c02cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.436+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd38c141690 con 0x7fd38c0633b0 2024-09-16T01:49:46.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.436+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd38c0633b0 0x7fd38c02cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.436+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd38c0633b0 0x7fd38c02cb50 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.437+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c141690 con 0x7fd38c0633b0 2024-09-16T01:49:46.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.451+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c141690 con 0x7fd38c0633b0 2024-09-16T01:49:46.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.452+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c141690 con 0x7fd38c0633b0 2024-09-16T01:49:46.453 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:46.454 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:46.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.453+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd38c02d850 0x7fd38c149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.453+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd38c14a8f0 con 0x7fd38c02d850 2024-09-16T01:49:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.453+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd38c02d850 0x7fd38c149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.453+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd38c02d850 0x7fd38c149230 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.454+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c14a8f0 con 0x7fd38c02d850 2024-09-16T01:49:46.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.468+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c149770 con 0x7fd38c02d850 2024-09-16T01:49:46.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.470+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c149770 con 0x7fd38c02d850 2024-09-16T01:49:46.471 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.470+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd38c149e90 0x7fd38c14a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.470+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd38c14a830 con 0x7fd38c149e90 2024-09-16T01:49:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.471+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd38c149e90 0x7fd38c14a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.471+0000 7fd381400640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd38c149e90 0x7fd38c14a2f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.472+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c14a830 con 0x7fd38c149e90 2024-09-16T01:49:46.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.485+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c14a830 con 0x7fd38c149e90 2024-09-16T01:49:46.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.487+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c14a830 con 0x7fd38c149e90 2024-09-16T01:49:46.488 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:46.489 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:46.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.487+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd38c2057c0 0x7fd38c207bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.487+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd38c20aa00 con 0x7fd38c2057c0 2024-09-16T01:49:46.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.488+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd38c2057c0 0x7fd38c207bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.488+0000 7fd38a800640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd38c2057c0 0x7fd38c207bb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.490 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:46.490 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:46.490 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:46.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:46.490 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:46.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.489+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c20aa00 con 0x7fd38c2057c0 2024-09-16T01:49:46.499 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:49:46.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.502+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c2083c0 con 0x7fd38c2057c0 2024-09-16T01:49:46.504 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:46.503+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:46.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.504+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c2083c0 con 0x7fd38c2057c0 2024-09-16T01:49:46.505 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:46.507 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:46.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.504+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd38c208ae0 0x7fd38c208f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:46.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.504+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd38c20ddd0 con 0x7fd38c208ae0 2024-09-16T01:49:46.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.504+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd38c208ae0 0x7fd38c208f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:46.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.505+0000 7fd389e00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd38c208ae0 0x7fd38c208f00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:46.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.506+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd38c20ddd0 con 0x7fd38c208ae0 2024-09-16T01:49:46.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.519+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd38c209440 con 0x7fd38c208ae0 2024-09-16T01:49:46.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd382a00640 1 -- 172.21.15.146:0/325188722 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd38c209440 con 0x7fd38c208ae0 2024-09-16T01:49:46.522 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd38c148b20 msgr2=0x7fd38c1f25e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd38c148b20 0x7fd38c1f25e0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd38c141100 msgr2=0x7fd38c145cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd38c141100 0x7fd38c145cb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd38c146700 msgr2=0x7fd38c062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd38c146700 0x7fd38c062630 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd38c0633b0 msgr2=0x7fd38c02cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.521+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd38c0633b0 0x7fd38c02cb50 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd38c02d850 msgr2=0x7fd38c149230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd38c02d850 0x7fd38c149230 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd38c149e90 msgr2=0x7fd38c14a2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd38c149e90 0x7fd38c14a2f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd38c2057c0 msgr2=0x7fd38c207bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd38c2057c0 0x7fd38c207bb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd38c208ae0 msgr2=0x7fd38c208f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd38c208ae0 0x7fd38c208f00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd38a800640 1 -- 172.21.15.146:0/325188722 reap_dead start 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd38a800640 1 -- 172.21.15.146:0/325188722 reap_dead start 2024-09-16T01:49:46.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd38a800640 1 -- 172.21.15.146:0/325188722 reap_dead start 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd38a800640 1 -- 172.21.15.146:0/325188722 reap_dead start 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd38a800640 1 -- 172.21.15.146:0/325188722 reap_dead start 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd36806c2b0 msgr2=0x7fd36806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd36806c2b0 0x7fd36806e770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd38c14b950 tx=0x7fd374007e80 comp rx=0 tx=0).stop 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 msgr2=0x7fd38c153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 0x7fd38c153cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd37c009e80 tx=0x7fd37c004a40 comp rx=0 tx=0).stop 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 shutdown_connections 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd36806c2b0 0x7fd36806e770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 --2- 172.21.15.146:0/325188722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd38c071a10 0x7fd38c153cd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 >> 172.21.15.146:0/325188722 conn(0x7fd38c06bc10 msgr2=0x7fd38c150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 shutdown_connections 2024-09-16T01:49:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:46.522+0000 7fd390a00640 1 -- 172.21.15.146:0/325188722 wait complete. 2024-09-16T01:49:46.604 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:74 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'48, mlcod=17'43)", 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.201352+0000", 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.401032575000002, 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059279409999999999, 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201352+0000", 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201352+0000", 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201353+0000", 2024-09-16T01:49:46.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8700000000000005e-07 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201360+0000", 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6599999999999998e-06 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201368+0000", 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5469999999999997e-06 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201373+0000", 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6779999999999999e-06 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201414+0000", 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1084000000000002e-05 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.201427+0000", 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3331e-05 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.207246+0000", 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058184680000000003 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.207280+0000", 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4486000000000002e-05 2024-09-16T01:49:46.609 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:82 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'52, mlcod=17'50)", 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.219972+0000", 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.382412539000001, 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006421459, 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219972+0000", 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219972+0000", 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219973+0000", 2024-09-16T01:49:46.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7999999999999995e-07 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219980+0000", 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.64e-06 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219990+0000", 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8989999999999996e-06 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.219995+0000", 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0220000000000004e-06 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220033+0000", 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8810999999999998e-05 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220047+0000", 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3876999999999999e-05 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226360+0000", 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063126700000000003 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226394+0000", 2024-09-16T01:49:46.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3859999999999998e-05 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:83 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'53, mlcod=17'50)", 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.220300+0000", 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.382084685000002, 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061152699999999999, 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.615 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220300+0000", 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220300+0000", 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220301+0000", 2024-09-16T01:49:46.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8400000000000004e-07 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220307+0000", 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8449999999999999e-06 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220314+0000", 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2760000000000003e-06 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220318+0000", 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9609999999999999e-06 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220362+0000", 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4365e-05 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220381+0000", 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9199999999999999e-05 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226398+0000", 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060167570000000002 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226415+0000", 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7282000000000002e-05 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.620 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:84 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'54, mlcod=17'50)", 2024-09-16T01:49:46.620 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.220645+0000", 2024-09-16T01:49:46.620 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.381739856999999, 2024-09-16T01:49:46.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057886719999999999, 2024-09-16T01:49:46.620 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.621 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.621 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.621 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220645+0000", 2024-09-16T01:49:46.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220645+0000", 2024-09-16T01:49:46.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220646+0000", 2024-09-16T01:49:46.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:49:46.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220707+0000", 2024-09-16T01:49:46.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1493000000000002e-05 2024-09-16T01:49:46.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220717+0000", 2024-09-16T01:49:46.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0030999999999999e-05 2024-09-16T01:49:46.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220721+0000", 2024-09-16T01:49:46.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1180000000000002e-06 2024-09-16T01:49:46.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220754+0000", 2024-09-16T01:49:46.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2499999999999997e-05 2024-09-16T01:49:46.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.220768+0000", 2024-09-16T01:49:46.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3824e-05 2024-09-16T01:49:46.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226417+0000", 2024-09-16T01:49:46.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056497759999999996 2024-09-16T01:49:46.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226434+0000", 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6339000000000001e-05 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.631 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:85 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'55, mlcod=17'50)", 2024-09-16T01:49:46.631 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.221055+0000", 2024-09-16T01:49:46.631 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.381329809999997, 2024-09-16T01:49:46.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005392188, 2024-09-16T01:49:46.631 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.632 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.632 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.632 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221055+0000", 2024-09-16T01:49:46.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221055+0000", 2024-09-16T01:49:46.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221056+0000", 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-07 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221061+0000", 2024-09-16T01:49:46.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7950000000000004e-06 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221069+0000", 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2869999999999999e-06 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221072+0000", 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7220000000000001e-06 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223322+0000", 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002249675 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223336+0000", 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3617e-05 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226435+0000", 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003099667 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226447+0000", 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1895e-05 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:86 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'56, mlcod=17'50)", 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.380957332000001, 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076983349999999997, 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221428+0000", 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.06e-07 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221436+0000", 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221446+0000", 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0903999999999999e-05 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221451+0000", 2024-09-16T01:49:46.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9130000000000002e-06 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223533+0000", 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020812259999999998 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223554+0000", 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1747999999999999e-05 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229094+0000", 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055398820000000003 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229126+0000", 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1642e-05 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:87 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'57, mlcod=17'50)", 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.221732+0000", 2024-09-16T01:49:46.644 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.380652451000003, 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074182919999999999, 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221732+0000", 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221732+0000", 2024-09-16T01:49:46.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221733+0000", 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2699999999999999e-07 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221772+0000", 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9201e-05 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221782+0000", 2024-09-16T01:49:46.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7450000000000007e-06 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.221786+0000", 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0369999999999996e-06 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223814+0000", 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020279600000000001 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.223827+0000", 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3002e-05 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229127+0000", 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00530045 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229151+0000", 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3370000000000002e-05 2024-09-16T01:49:46.648 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:88 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'58, mlcod=17'50)", 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.222408+0000", 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.379976999999997, 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0069243669999999998, 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222408+0000", 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222408+0000", 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222409+0000", 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1600000000000001e-07 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222417+0000", 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4840000000000004e-06 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222427+0000", 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0305e-05 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.222433+0000", 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7830000000000004e-06 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226475+0000", 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040419139999999998 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.226489+0000", 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3971999999999999e-05 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229315+0000", 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028261060000000001 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.229332+0000", 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7087000000000001e-05 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:116 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'78, mlcod=17'68)", 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.471060+0000", 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.131324954999997, 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0085754279999999995, 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.654 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471060+0000", 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471060+0000", 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471061+0000", 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8999999999999996e-07 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471068+0000", 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4240000000000004e-06 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471077+0000", 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:49:46.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471083+0000", 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.297e-06 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471421+0000", 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00033748300000000002 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471439+0000", 2024-09-16T01:49:46.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8131e-05 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479591+0000", 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0081523910000000002 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479635+0000", 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4156000000000002e-05 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:117 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'79, mlcod=17'68)", 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.130972155000002, 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0082421309999999998, 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471413+0000", 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471419+0000", 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2519999999999999e-06 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471426+0000", 2024-09-16T01:49:46.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4680000000000003e-06 2024-09-16T01:49:46.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471432+0000", 2024-09-16T01:49:46.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8449999999999999e-06 2024-09-16T01:49:46.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471667+0000", 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00023510999999999999 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.471679+0000", 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2302e-05 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479638+0000", 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0079587300000000007 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:48.479655+0000", 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6833e-05 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:139 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'97, mlcod=17'89)", 2024-09-16T01:49:46.664 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.041023+0000", 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.561361605000002, 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056538869999999998, 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041023+0000", 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041023+0000", 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041024+0000", 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5900000000000002e-07 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041028+0000", 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2710000000000003e-06 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041034+0000", 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041038+0000", 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7529999999999999e-06 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041072+0000", 2024-09-16T01:49:46.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4354999999999999e-05 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041085+0000", 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2466999999999999e-05 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046641+0000", 2024-09-16T01:49:46.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055564880000000001 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046677+0000", 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5799000000000001e-05 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.669 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:140 1.0 e17/16 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'98, mlcod=17'89)", 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.561033268999999, 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053458910000000002, 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041352+0000", 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5899999999999998e-07 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041412+0000", 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0520999999999999e-05 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041419+0000", 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7769999999999997e-06 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041422+0000", 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2229999999999999e-06 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041446+0000", 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4046999999999999e-05 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.041458+0000", 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.164e-05 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046679+0000", 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005221275 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.046697+0000", 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8148999999999999e-05 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:143 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'101, mlcod=17'98)", 2024-09-16T01:49:46.674 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.048358+0000", 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.554026882000002, 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065183130000000004, 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048358+0000", 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048358+0000", 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048359+0000", 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-07 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048422+0000", 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3003000000000006e-05 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048427+0000", 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.93e-06 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048431+0000", 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2789999999999999e-06 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048464+0000", 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2839999999999997e-05 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048479+0000", 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5664000000000001e-05 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054749+0000", 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062695320000000004 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054876+0000", 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012753499999999999 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:144 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'102, mlcod=17'98)", 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.048708+0000", 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.553676643999999, 2024-09-16T01:49:46.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061860739999999997, 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048708+0000", 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048708+0000", 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048709+0000", 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0699999999999998e-07 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048761+0000", 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2342000000000001e-05 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048768+0000", 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6259999999999997e-06 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048771+0000", 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.208e-06 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048797+0000", 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6361999999999999e-05 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.048809+0000", 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1697e-05 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054879+0000", 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060705220000000001 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.054894+0000", 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.491e-05 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4201.0:145 1.0 e17/16 1:30306672:devicehealth::main.db.0000000000000000:head v 17'103, mlcod=17'98)", 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:48:49.049177+0000", 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.553207794999999, 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059428930000000003, 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049177+0000", 2024-09-16T01:49:46.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049177+0000", 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049178+0000", 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.58e-07 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049267+0000", 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9957999999999995e-05 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049273+0000", 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6110000000000001e-06 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049277+0000", 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1509999999999997e-06 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049310+0000", 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2486e-05 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.049323+0000", 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3349999999999999e-05 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.055098+0000", 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057749769999999997 2024-09-16T01:49:46.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:48:49.055120+0000", 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1902000000000001e-05 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825295+0000", 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.777090268000002, 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073553409999999996, 2024-09-16T01:49:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825295+0000", 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825295+0000", 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9740000000000001e-06 2024-09-16T01:49:46.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825308+0000", 2024-09-16T01:49:46.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9129999999999992e-06 2024-09-16T01:49:46.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825328+0000", 2024-09-16T01:49:46.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.999e-05 2024-09-16T01:49:46.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825338+0000", 2024-09-16T01:49:46.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:49:46.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825911+0000", 2024-09-16T01:49:46.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00057331000000000005 2024-09-16T01:49:46.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826029+0000", 2024-09-16T01:49:46.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011776499999999999 2024-09-16T01:49:46.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:46.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826161+0000", 2024-09-16T01:49:46.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013176900000000001 2024-09-16T01:49:46.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:46.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827819+0000", 2024-09-16T01:49:46.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016579030000000001 2024-09-16T01:49:46.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:46.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832580+0000", 2024-09-16T01:49:46.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047607259999999998 2024-09-16T01:49:46.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832618+0000", 2024-09-16T01:49:46.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8398000000000001e-05 2024-09-16T01:49:46.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832650+0000", 2024-09-16T01:49:46.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1903000000000003e-05 2024-09-16T01:49:46.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.701 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.702 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:46.702 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.896865+0000", 2024-09-16T01:49:46.702 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 31.705519503000001, 2024-09-16T01:49:46.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063761060000000003, 2024-09-16T01:49:46.703 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.703 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.703 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:46.703 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896865+0000", 2024-09-16T01:49:46.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896865+0000", 2024-09-16T01:49:46.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896868+0000", 2024-09-16T01:49:46.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4119999999999999e-06 2024-09-16T01:49:46.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896874+0000", 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.596e-06 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896887+0000", 2024-09-16T01:49:46.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2351000000000001e-05 2024-09-16T01:49:46.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896895+0000", 2024-09-16T01:49:46.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4030000000000007e-06 2024-09-16T01:49:46.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896962+0000", 2024-09-16T01:49:46.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7176999999999997e-05 2024-09-16T01:49:46.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897082+0000", 2024-09-16T01:49:46.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119671 2024-09-16T01:49:46.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903241+0000", 2024-09-16T01:49:46.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061594960000000004 2024-09-16T01:49:46.711 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.711 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.711 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.712 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:5 2.0 e22/19 2:03625d61:::614.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:46.712 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:46.712 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 31.704975065999999, 2024-09-16T01:49:46.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005944024, 2024-09-16T01:49:46.713 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.713 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.713 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.713 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:46.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:46.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897410+0000", 2024-09-16T01:49:46.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8299999999999997e-07 2024-09-16T01:49:46.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897417+0000", 2024-09-16T01:49:46.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7789999999999998e-06 2024-09-16T01:49:46.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897427+0000", 2024-09-16T01:49:46.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0271e-05 2024-09-16T01:49:46.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897435+0000", 2024-09-16T01:49:46.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7230000000000004e-06 2024-09-16T01:49:46.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897513+0000", 2024-09-16T01:49:46.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7733999999999998e-05 2024-09-16T01:49:46.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897532+0000", 2024-09-16T01:49:46.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9072e-05 2024-09-16T01:49:46.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903278+0000", 2024-09-16T01:49:46.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057462040000000004 2024-09-16T01:49:46.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903354+0000", 2024-09-16T01:49:46.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5658e-05 2024-09-16T01:49:46.723 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.723 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.723 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.724 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:1 2.1 e22/19 2:85bbe569:::202.00000000:head v 22'4, mlcod=0'0)", 2024-09-16T01:49:46.724 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898125+0000", 2024-09-16T01:49:46.724 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 31.704259691000001, 2024-09-16T01:49:46.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0051968099999999996, 2024-09-16T01:49:46.725 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.725 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.725 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:46.725 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898125+0000", 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898125+0000", 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898126+0000", 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3699999999999999e-07 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898134+0000", 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9340000000000006e-06 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898151+0000", 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7206000000000001e-05 2024-09-16T01:49:46.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898160+0000", 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0489999999999993e-06 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898280+0000", 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011925399999999999 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898298+0000", 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610999999999999e-05 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903276+0000", 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049776760000000003 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903322+0000", 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6143000000000002e-05 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:29 2.d 2:b2057508:::204.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:46.731 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.936198+0000", 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.666187195999999, 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0091713439999999997, 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936198+0000", 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936198+0000", 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936200+0000", 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5610000000000001e-06 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936207+0000", 2024-09-16T01:49:46.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1350000000000003e-06 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936224+0000", 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6254000000000001e-05 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936235+0000", 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1094000000000001e-05 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:46.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936314+0000", 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9560000000000004e-05 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936461+0000", 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014680499999999999 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945369+0000", 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0089079350000000005 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.735 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.736 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.736 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:46.736 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:46.736 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:46.736 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:46.603+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:46.737 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:49:46.739 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:46.738+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:46.839 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:46.839+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:46.883 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:46.884 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:46.884 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:49:46.884 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:46.892 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:49:46.940 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:46.939+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:46.997 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:46.997 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:46.998 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:46.998 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:46.998 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:47.006 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:49:47.050 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:47.049+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:47.112 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 0 entries 22'1 (0'0) error 2:ddb4e4d7:::200.00000002:head by mds.0.4:33 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:47.113 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.283872322999997, 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001054845, 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.57e-06 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826088+0000", 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5670000000000001e-05 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826118+0000", 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9603000000000001e-05 2024-09-16T01:49:47.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826136+0000", 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.88e-05 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826229+0000", 2024-09-16T01:49:47.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2366000000000003e-05 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827126+0000", 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00089683600000000003 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.119 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.1.9:17 2.1b e22/19 2:dbae76b5:::mds1_inotable:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.249332049000003, 2024-09-16T01:49:47.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016283739999999999, 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:47.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860613+0000", 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.446e-06 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860625+0000", 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2549e-05 2024-09-16T01:49:47.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860645+0000", 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9545e-05 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860655+0000", 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9550000000000004e-06 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860902+0000", 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024690900000000002 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860935+0000", 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3788999999999998e-05 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862198+0000", 2024-09-16T01:49:47.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012625169999999999 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862239+0000", 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1664e-05 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 4 entries 22'3 (0'0) error 2:db457a8a:::201.00000001:head by mds.1.9:20 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.248016245000002, 2024-09-16T01:49:47.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011507889999999999, 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:47.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861928+0000", 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6099999999999999e-07 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861940+0000", 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1817999999999999e-05 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861962+0000", 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2078000000000001e-05 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861976+0000", 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4338000000000001e-05 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862056+0000", 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0253999999999994e-05 2024-09-16T01:49:47.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863078+0000", 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213399999999999 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 5 entries 22'4 (0'0) error 2:d80f8093:::201.00000002:head by mds.1.9:21 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.247688203000003, 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014125819999999999, 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.131 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862256+0000", 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5000000000000003e-07 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862264+0000", 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.789e-06 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862280+0000", 2024-09-16T01:49:47.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5633000000000001e-05 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862294+0000", 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4316e-05 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862755+0000", 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00046123399999999998 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863668+0000", 2024-09-16T01:49:47.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00091206000000000002 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.1.9:34 2.1b e22/19 2:db457a8a:::201.00000001:head v 22'5, mlcod=22'3)", 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.240460067000001, 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00078421100000000004, 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.135 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869484+0000", 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0699999999999999e-06 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869492+0000", 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0860000000000002e-06 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869505+0000", 2024-09-16T01:49:47.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.308e-05 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869512+0000", 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3010000000000003e-06 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869573+0000", 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0956999999999999e-05 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869595+0000", 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2311999999999999e-05 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.870232+0000", 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000637456 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.870267+0000", 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4949e-05 2024-09-16T01:49:47.139 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:35 2.a 2:54ba6923:::mds1_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.219604220999997, 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014025660000000001, 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890341+0000", 2024-09-16T01:49:47.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.607e-06 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890347+0000", 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9530000000000002e-06 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890358+0000", 2024-09-16T01:49:47.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0910000000000001e-05 2024-09-16T01:49:47.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890368+0000", 2024-09-16T01:49:47.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6330000000000008e-06 2024-09-16T01:49:47.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890418+0000", 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0290999999999997e-05 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890538+0000", 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011986599999999999 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890640+0000", 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101625 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891432+0000", 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00079213799999999996 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891662+0000", 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000230237 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891697+0000", 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4866999999999997e-05 2024-09-16T01:49:47.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891741+0000", 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4438999999999999e-05 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:11 2.c 2:36924187:::618.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.214226859999997, 2024-09-16T01:49:47.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055247409999999997, 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:47.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895717+0000", 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1289999999999999e-06 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895724+0000", 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0769999999999996e-06 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895733+0000", 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4210000000000001e-06 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895741+0000", 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0290000000000005e-06 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895816+0000", 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5190000000000003e-05 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895955+0000", 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138737 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896073+0000", 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011843799999999999 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897963+0000", 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001890061 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901132+0000", 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031683089999999998 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901196+0000", 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4132000000000002e-05 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901241+0000", 2024-09-16T01:49:47.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5218e-05 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:12 2.c 2:36924187:::618.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.214174565, 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056235809999999999, 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.153 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895782+0000", 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2383e-05 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0747999999999999e-05 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895799+0000", 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.849e-06 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896381+0000", 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058177900000000004 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896459+0000", 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8511999999999996e-05 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897012+0000", 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055262699999999998 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900008+0000", 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029960410000000001 2024-09-16T01:49:47.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901291+0000", 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00128261 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901351+0000", 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0603000000000002e-05 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901392+0000", 2024-09-16T01:49:47.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0799e-05 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.214129290000002, 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055047550000000001, 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895815+0000", 2024-09-16T01:49:47.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.85e-07 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895824+0000", 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4350000000000003e-06 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895842+0000", 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8264999999999999e-05 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895850+0000", 2024-09-16T01:49:47.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8339999999999999e-06 2024-09-16T01:49:47.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895962+0000", 2024-09-16T01:49:47.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111644 2024-09-16T01:49:47.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896088+0000", 2024-09-16T01:49:47.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012620200000000001 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896219+0000", 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000130782 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898012+0000", 2024-09-16T01:49:47.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001793474 2024-09-16T01:49:47.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901212+0000", 2024-09-16T01:49:47.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031999390000000002 2024-09-16T01:49:47.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:47.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9030999999999998e-05 2024-09-16T01:49:47.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901319+0000", 2024-09-16T01:49:47.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7364000000000003e-05 2024-09-16T01:49:47.169 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.169 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.169 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.170 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.170 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:47.170 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.214071103999999, 2024-09-16T01:49:47.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055742070000000003, 2024-09-16T01:49:47.171 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.171 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.171 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.171 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:47.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895873+0000", 2024-09-16T01:49:47.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8100000000000001e-07 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895877+0000", 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3710000000000002e-06 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895886+0000", 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0739999999999999e-06 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895893+0000", 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3550000000000001e-06 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896519+0000", 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00062635700000000002 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896612+0000", 2024-09-16T01:49:47.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3059000000000005e-05 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896737+0000", 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012455499999999999 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900030+0000", 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032929740000000002 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901357+0000", 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001327204 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901403+0000", 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6508000000000003e-05 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901446+0000", 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2868999999999999e-05 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.178 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.214023097999998, 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0070037900000000002, 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895921+0000", 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8899999999999999e-07 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895926+0000", 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.023e-06 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895934+0000", 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1939999999999998e-06 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895941+0000", 2024-09-16T01:49:47.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0770000000000002e-06 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897276+0000", 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001334614 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897409+0000", 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000133215 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897545+0000", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013621099999999999 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901441+0000", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003895459 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902804+0000", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013631120000000001 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902877+0000", 2024-09-16T01:49:47.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3647999999999997e-05 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902924+0000", 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6548000000000002e-05 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.2 epoch 22/19 rep_tid 20 entries 22'1 (0'0) error 2:40dbbc66:::202.00000003:head by mds.2.11:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.212813699000002, 2024-09-16T01:49:47.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029206929999999998, 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897130+0000", 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897142+0000", 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1095e-05 2024-09-16T01:49:47.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897165+0000", 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3278999999999999e-05 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897180+0000", 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5613000000000001e-05 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897269+0000", 2024-09-16T01:49:47.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8749e-05 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900050+0000", 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027809940000000002 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:28 2.c 2:35dcb8d2:::203.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.1981666, 2024-09-16T01:49:47.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047880470000000001, 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911779+0000", 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3609999999999999e-06 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911799+0000", 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9721e-05 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911817+0000", 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.802e-05 2024-09-16T01:49:47.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911851+0000", 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4289000000000003e-05 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911931+0000", 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0142000000000006e-05 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912062+0000", 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013077100000000001 2024-09-16T01:49:47.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916565+0000", 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004502743 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:33 2.a 2:556620e9:::203.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.198045951000001, 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026481180000000001, 2024-09-16T01:49:47.192 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911898+0000", 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2799999999999995e-07 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911912+0000", 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4003e-05 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911922+0000", 2024-09-16T01:49:47.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.045e-05 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911929+0000", 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6850000000000004e-06 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911993+0000", 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4196999999999995e-05 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912089+0000", 2024-09-16T01:49:47.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5383000000000005e-05 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914545+0000", 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024565720000000002 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:34 2.c 2:35dcb8d2:::203.00000001:head [write 0~588] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.192982222000001, 2024-09-16T01:49:47.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041374530000000001, 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:47.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916963+0000", 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.64e-06 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916972+0000", 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7659999999999993e-06 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916990+0000", 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7445e-05 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916999+0000", 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3859999999999996e-06 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917077+0000", 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8307999999999996e-05 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917172+0000", 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4617000000000006e-05 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917357+0000", 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018458799999999999 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918065+0000", 2024-09-16T01:49:47.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070841800000000005 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.920973+0000", 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029077769999999998 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.921071+0000", 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7918000000000007e-05 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.921098+0000", 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7589999999999998e-05 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:35 2.a 2:579bf162:::mds3_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.180079326000001, 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001587256, 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.202 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929867+0000", 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.858e-06 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929878+0000", 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1651000000000001e-05 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929898+0000", 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9828999999999999e-05 2024-09-16T01:49:47.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929907+0000", 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1239999999999994e-06 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929986+0000", 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8608000000000004e-05 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.930169+0000", 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018343600000000001 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.930319+0000", 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014941299999999999 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931104+0000", 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00078539600000000005 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931374+0000", 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027029000000000002 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931415+0000", 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0068999999999999e-05 2024-09-16T01:49:47.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931451+0000", 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6582000000000003e-05 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:24 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:47.208 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.173504024, 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048886399999999997, 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936441+0000", 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0600000000000002e-06 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936464+0000", 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3161999999999999e-05 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936480+0000", 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5537e-05 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936490+0000", 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6889999999999999e-06 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936578+0000", 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8596000000000004e-05 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936729+0000", 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015100000000000001 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936946+0000", 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000216586 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939283+0000", 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023372559999999998 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941230+0000", 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001946677 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941279+0000", 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9478999999999998e-05 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941328+0000", 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8597999999999998e-05 2024-09-16T01:49:47.214 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.4.15:11 2.2 e22/19 2:447438e6:::62c.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.172654664, 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002990131, 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937290+0000", 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.299e-06 2024-09-16T01:49:47.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937298+0000", 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4400000000000005e-06 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937309+0000", 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1049e-05 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3410000000000003e-06 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937382+0000", 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4307999999999995e-05 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937403+0000", 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0664000000000001e-05 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940230+0000", 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028275319999999998 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940279+0000", 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8498000000000002e-05 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.4.15:12 2.2 e22/19 2:447438e6:::62c.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.171833470999999, 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0022086779999999999, 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938111+0000", 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.95e-07 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938119+0000", 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9370000000000004e-06 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938137+0000", 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8360999999999999e-05 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938145+0000", 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4060000000000005e-06 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938218+0000", 2024-09-16T01:49:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2739999999999998e-05 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938238+0000", 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9738999999999999e-05 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940284+0000", 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020457850000000001 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940318+0000", 2024-09-16T01:49:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4715000000000002e-05 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.2 epoch 22/19 rep_tid 37 entries 22'4 (0'0) error 2:4182a243:::204.00000001:head by mds.4.15:28 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.171768316000001, 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002163842, 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5599999999999995e-07 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938191+0000", 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5967000000000001e-05 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938224+0000", 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2169000000000003e-05 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938239+0000", 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5063e-05 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938468+0000", 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000229082 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940339+0000", 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001871005 2024-09-16T01:49:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.228 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:47.229 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:49:47.230 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:47.228+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:47.330 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:47.329+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:47.374 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:47.374 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:47.374 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:49:47.375 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:47.383 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:49:47.431 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:47.430+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:47.487 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:47.487 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:47.487 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:47.487 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:47.487 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:47.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:49:47.540 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:47.539+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:47.602 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.775715069, 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055002239999999997, 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.603 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824857+0000", 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1457000000000001e-05 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824870+0000", 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2483999999999999e-05 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824885+0000", 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4758e-05 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824894+0000", 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5319999999999995e-06 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015500299999999999 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825242+0000", 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000192986 2024-09-16T01:49:47.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825387+0000", 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000144986 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828137+0000", 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027499 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830217+0000", 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020800369999999999 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830297+0000", 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0011000000000004e-05 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830346+0000", 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:47.610 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.775642718, 2024-09-16T01:49:47.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055792059999999997, 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824919+0000", 2024-09-16T01:49:47.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7000000000000005e-07 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824922+0000", 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824931+0000", 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824935+0000", 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:47.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825813+0000", 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00087810899999999996 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825889+0000", 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6157999999999998e-05 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826034+0000", 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014488599999999999 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828143+0000", 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002108408 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830415+0000", 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00227247 2024-09-16T01:49:47.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830467+0000", 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830498+0000", 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0956e-05 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.619 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.775597638999997, 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0072987340000000003, 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824964+0000", 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2800000000000002e-07 2024-09-16T01:49:47.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824974+0000", 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0627e-05 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824982+0000", 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3159999999999999e-06 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2280000000000001e-06 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825059+0000", 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9754999999999999e-05 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825232+0000", 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000172986 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825435+0000", 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00020313199999999999 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828256+0000", 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028206009999999998 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832187+0000", 2024-09-16T01:49:47.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039315920000000002 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832228+0000", 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0420999999999998e-05 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832262+0000", 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4647999999999998e-05 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.625 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.775550494999997, 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073664890000000004, 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.626 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:47.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825015+0000", 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7129999999999999e-06 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825024+0000", 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5629999999999992e-06 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825029+0000", 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9269999999999996e-06 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825830+0000", 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00080144400000000001 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825970+0000", 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014016899999999999 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826100+0000", 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012975500000000001 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828596+0000", 2024-09-16T01:49:47.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024965830000000001 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832306+0000", 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037092140000000002 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832349+0000", 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3210000000000001e-05 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832377+0000", 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8161e-05 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.775514469999997, 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059226180000000002, 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.632 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3399999999999999e-07 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825061+0000", 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4144999999999999e-05 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825069+0000", 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5209999999999997e-06 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825074+0000", 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8450000000000002e-06 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826274+0000", 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001200586 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826528+0000", 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000253446 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826691+0000", 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016368900000000001 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828551+0000", 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0018598 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830882+0000", 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023311180000000001 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830929+0000", 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6882999999999998e-05 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830969+0000", 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0051e-05 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.740026180999998, 2024-09-16T01:49:47.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:49:47.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:49:47.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:49:47.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.739927674, 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.645 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:49:47.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:7 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.704936519999997, 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057447150000000001, 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.651 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895627+0000", 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0480000000000001e-06 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895632+0000", 2024-09-16T01:49:47.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5879999999999997e-06 2024-09-16T01:49:47.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895650+0000", 2024-09-16T01:49:47.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7322000000000001e-05 2024-09-16T01:49:47.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895657+0000", 2024-09-16T01:49:47.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1949999999999997e-06 2024-09-16T01:49:47.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895729+0000", 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2120999999999996e-05 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895880+0000", 2024-09-16T01:49:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015082600000000001 2024-09-16T01:49:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896034+0000", 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015426799999999999 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897859+0000", 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001824736 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:47.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034126600000000001 2024-09-16T01:49:47.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901329+0000", 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7836e-05 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901369+0000", 2024-09-16T01:49:47.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114999999999998e-05 2024-09-16T01:49:47.661 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.661 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.661 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.661 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:8 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.704887337000002, 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058392840000000001, 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9900000000000001e-07 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895677+0000", 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1e-06 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895684+0000", 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5769999999999999e-06 2024-09-16T01:49:47.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895699+0000", 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.473e-05 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896845+0000", 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011460960000000001 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896914+0000", 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9115999999999994e-05 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897029+0000", 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114974 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900398+0000", 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033696749999999999 2024-09-16T01:49:47.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901420+0000", 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213030000000001 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901470+0000", 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0083999999999997e-05 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901513+0000", 2024-09-16T01:49:47.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3130000000000002e-05 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.704834222000002, 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048597969999999999, 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.668 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895737+0000", 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2070000000000001e-06 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895743+0000", 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3430000000000001e-06 2024-09-16T01:49:47.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895747+0000", 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0940000000000001e-06 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895803+0000", 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6339999999999999e-05 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895947+0000", 2024-09-16T01:49:47.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143199 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896084+0000", 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000137126 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900351+0000", 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042672250000000004 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900515+0000", 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016424500000000001 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900552+0000", 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6804999999999997e-05 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900587+0000", 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4582999999999997e-05 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.674 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.704803259000002, 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049417419999999998, 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895761+0000", 2024-09-16T01:49:47.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4760000000000001e-06 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895765+0000", 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2540000000000003e-06 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895772+0000", 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4679999999999997e-06 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896412+0000", 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064042899999999998 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896492+0000", 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0500000000000005e-05 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896615+0000", 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000122194 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900360+0000", 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037456830000000001 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900635+0000", 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027509799999999999 2024-09-16T01:49:47.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900673+0000", 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7265999999999999e-05 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900700+0000", 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.688e-05 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.680 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.704781646000001, 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0078497150000000002, 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895780+0000", 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1699999999999998e-07 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895783+0000", 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9239999999999999e-06 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895789+0000", 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8080000000000001e-06 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4089999999999999e-06 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897342+0000", 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001550287 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897499+0000", 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015634399999999999 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897630+0000", 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013139099999999999 2024-09-16T01:49:47.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902981+0000", 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053504950000000003 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903540+0000", 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055920099999999995 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903591+0000", 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1224999999999998e-05 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903629+0000", 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8114000000000001e-05 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.686 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1e epoch 22/19 rep_tid 7 entries 22'2 (0'0) error 2:7d92db23:::202.00000002:head by mds.2.11:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.703222697000001, 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056237750000000001, 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.687 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897339+0000", 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.029e-06 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897350+0000", 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0913e-05 2024-09-16T01:49:47.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898029+0000", 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00067867600000000002 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898053+0000", 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3799999999999999e-05 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898142+0000", 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9368000000000002e-05 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902962+0000", 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048199890000000002 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:17 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'11, mlcod=22'9)", 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.701968651999998, 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043821320000000004, 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898593+0000", 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.023e-06 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898603+0000", 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2490000000000007e-06 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898619+0000", 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6073e-05 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898627+0000", 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3950000000000001e-06 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898688+0000", 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0545999999999999e-05 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898704+0000", 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5903999999999999e-05 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902933+0000", 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042294389999999998 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902975+0000", 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1502999999999999e-05 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:18 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'12, mlcod=22'9)", 2024-09-16T01:49:47.696 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.701821058, 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042640050000000004, 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898741+0000", 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8999999999999995e-07 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898747+0000", 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3029999999999996e-06 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898756+0000", 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3969999999999994e-06 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898764+0000", 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898924+0000", 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016043599999999999 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898948+0000", 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4068999999999999e-05 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902977+0000", 2024-09-16T01:49:47.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040289449999999999 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903004+0000", 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6784e-05 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:26 2.10 e22/19 2:091e33df:::502.00000000:head v 22'14, mlcod=22'9)", 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:47.701 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.700864430999999, 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039915870000000004, 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899698+0000", 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9999999999999995e-07 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899706+0000", 2024-09-16T01:49:47.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.96e-06 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899791+0000", 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5702e-05 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899800+0000", 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6840000000000002e-06 2024-09-16T01:49:47.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899870+0000", 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9617999999999997e-05 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899888+0000", 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8046e-05 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903646+0000", 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037583310000000002 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903688+0000", 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2246000000000002e-05 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.706 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:26 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.688645777000001, 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050127030000000003, 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911918+0000", 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3479999999999998e-06 2024-09-16T01:49:47.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911934+0000", 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6659000000000001e-05 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911948+0000", 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3334e-05 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911957+0000", 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8390000000000002e-06 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912051+0000", 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3702999999999994e-05 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912182+0000", 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013119100000000001 2024-09-16T01:49:47.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000179147 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913761+0000", 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001399382 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916815+0000", 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030542350000000002 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916884+0000", 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8823000000000002e-05 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916928+0000", 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4042000000000003e-05 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:24 2.13 2:cbf94b35:::mds3_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.688565451999999, 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057070769999999996, 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.713 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912001+0000", 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2320000000000003e-06 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912015+0000", 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4562000000000001e-05 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912022+0000", 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.165e-06 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912138+0000", 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116098 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912260+0000", 2024-09-16T01:49:47.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121224 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101796 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913434+0000", 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010729610000000001 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917594+0000", 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041594010000000001 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917652+0000", 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8357999999999999e-05 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917703+0000", 2024-09-16T01:49:47.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0460000000000001e-05 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:13 2.13 2:c81eaac8:::62d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.664860923999999, 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004875094, 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935702+0000", 2024-09-16T01:49:47.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.277e-06 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935711+0000", 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6100000000000006e-06 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935729+0000", 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8224e-05 2024-09-16T01:49:47.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935742+0000", 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2405999999999999e-05 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935832+0000", 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0382000000000003e-05 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936059+0000", 2024-09-16T01:49:47.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022734599999999999 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936207+0000", 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014755500000000001 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938029+0000", 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001821694 2024-09-16T01:49:47.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940508+0000", 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024797479999999999 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8040000000000002e-05 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940575+0000", 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8812000000000001e-05 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:14 2.13 2:c81eaac8:::62d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.664698251000001, 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055486820000000001, 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:47.725 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935864+0000", 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935868+0000", 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7639999999999996e-06 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935877+0000", 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7560000000000005e-06 2024-09-16T01:49:47.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935888+0000", 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1184e-05 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937525+0000", 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001637023 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:47.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937611+0000", 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6114999999999998e-05 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937759+0000", 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014728699999999999 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938463+0000", 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070381299999999996 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941309+0000", 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002846374 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941366+0000", 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6848000000000002e-05 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941411+0000", 2024-09-16T01:49:47.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5698000000000002e-05 2024-09-16T01:49:47.731 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.731 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.731 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.731 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:47.731 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:47.731 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:47.733 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:47.731+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:47.733 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:49:47.833 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:47.832+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:47.897 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:47.897 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:47.897 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:47.898 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:47.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:49:47.934 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:47.933+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:48.036 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:48.035+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:48.107 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:48.107 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:48.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:49:48.166 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:48.165+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:48.237 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:48.237+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:48.279 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.451690560999999, 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:49:48.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:49:48.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.451584511999997, 2024-09-16T01:49:48.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:49:48.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:49:48.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:49:48.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:49:48.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.294 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.451529886000003, 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.295 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:49:48.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.451461553000001, 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:49:48.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:49:48.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:48.308 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.417206350000001, 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:48.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:49:48.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:49:48.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.417023151999999, 2024-09-16T01:49:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:49:48.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:49:48.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.320 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.416969143999999, 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:49:48.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:49:48.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:48.326 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.416936704000001, 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:48.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:49:48.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:49:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:49:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.413568703000003, 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:49:48.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:48.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:49:48.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.340 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.413498027000003, 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:49:48.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:49:48.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:49:48.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.413461652000002, 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:48.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:49:48.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:48.353 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.412833909, 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:49:48.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:49:48.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:49:48.359 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.412779301999997, 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:49:48.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:49:48.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340640894, 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.365 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:49:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:49:48.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340582582, 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:49:48.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340508506999999, 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:48.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:49:48.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:49:48.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:48.383 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340449885999998, 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:49:48.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:49:48.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340427951999999, 2024-09-16T01:49:48.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:49:48.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:49:48.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:49:48.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.395 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340397591999999, 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:49:48.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:49:48.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.340371012999999, 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:48.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:49:48.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:49:48.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:49:48.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.408 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:48.408 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:48.408 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:48.408 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:49:48.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:49:48.411 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:48.409+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:48.511 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:48.510+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:48.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:49:48.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.608+0000 7fa6b8800640 1 Processor -- start 2024-09-16T01:49:48.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.608+0000 7fa6b8800640 1 -- start start 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.609+0000 7fa6b8800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b00cc0c0 0x7fa6b015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.609+0000 7fa6b8800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa6b00cb2f0 0x7fa6b00cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.609+0000 7fa6b8800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6b015b1d0 con 0x7fa6b015e9d0 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.609+0000 7fa6b8800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa6b00cbd00 con 0x7fa6b00cb2f0 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.609+0000 7fa6b8800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b015ba70 con 0x7fa6b00cc0c0 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b00cc0c0 0x7fa6b015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b5400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa6b00cb2f0 0x7fa6b00cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b6800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa6b015e9d0 0x7fa6b00cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47114/0 (socket says 172.21.15.146:47114) 2024-09-16T01:49:48.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b6800640 1 -- 172.21.15.146:0/706325676 learned_addr learned my addr 172.21.15.146:0/706325676 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b5e00640 1 -- 172.21.15.146:0/706325676 >> v2:172.21.15.161:3300/0 conn(0x7fa6b00cb2f0 msgr2=0x7fa6b00cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b5e00640 1 --2- 172.21.15.146:0/706325676 >> v2:172.21.15.161:3300/0 conn(0x7fa6b00cb2f0 0x7fa6b00cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b5e00640 1 -- 172.21.15.146:0/706325676 >> v1:172.21.15.146:6789/0 conn(0x7fa6b015e9d0 legacy=0x7fa6b00cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.610+0000 7fa6b5e00640 1 -- 172.21.15.146:0/706325676 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b015b4b0 con 0x7fa6b00cc0c0 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.611+0000 7fa6b5e00640 1 --2- 172.21.15.146:0/706325676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b00cc0c0 0x7fa6b015aab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40097a0 tx=0x7fa6a402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56ac93a35d70b744 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.612+0000 7fa6b4a00640 1 -- 172.21.15.146:0/706325676 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4037070 con 0x7fa6b00cc0c0 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.612+0000 7fa6b4a00640 1 -- 172.21.15.146:0/706325676 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a4032440 con 0x7fa6b00cc0c0 2024-09-16T01:49:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.612+0000 7fa6b4a00640 1 -- 172.21.15.146:0/706325676 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4032810 con 0x7fa6b00cc0c0 2024-09-16T01:49:48.614 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:48.611+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:48.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.612+0000 7fa6b8800640 1 -- 172.21.15.146:0/706325676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b00cc0c0 msgr2=0x7fa6b015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.612+0000 7fa6b8800640 1 --2- 172.21.15.146:0/706325676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b00cc0c0 0x7fa6b015aab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40097a0 tx=0x7fa6a402a180 comp rx=0 tx=0).stop 2024-09-16T01:49:48.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.613+0000 7fa6b8800640 1 -- 172.21.15.146:0/706325676 shutdown_connections 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.613+0000 7fa6b8800640 1 --2- 172.21.15.146:0/706325676 >> v2:172.21.15.161:3300/0 conn(0x7fa6b00cb2f0 0x7fa6b00cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.613+0000 7fa6b8800640 1 --2- 172.21.15.146:0/706325676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b00cc0c0 0x7fa6b015aab0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.613+0000 7fa6b8800640 1 -- 172.21.15.146:0/706325676 >> 172.21.15.146:0/706325676 conn(0x7fa6b006d560 msgr2=0x7fa6b006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.613+0000 7fa6b8800640 1 -- 172.21.15.146:0/706325676 shutdown_connections 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.613+0000 7fa6b8800640 1 -- 172.21.15.146:0/706325676 wait complete. 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:48.615 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:48.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.614+0000 7fa6b8800640 1 Processor -- start 2024-09-16T01:49:48.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.614+0000 7fa6b8800640 1 -- start start 2024-09-16T01:49:48.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.615+0000 7fa6b8800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6b00cb2f0 0x7fa6b014e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.615+0000 7fa6b8800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 0x7fa6b00c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.615+0000 7fa6b8800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b015e9d0 0x7fa6b00c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.615+0000 7fa6b8800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b015e1d0 con 0x7fa6b00cc0c0 2024-09-16T01:49:48.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.615+0000 7fa6b8800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b015e540 con 0x7fa6b00cb2f0 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.615+0000 7fa6b8800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b00c5560 con 0x7fa6b015e9d0 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 0x7fa6b00c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 0x7fa6b00c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59772/0 (socket says 172.21.15.146:59772) 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b5400640 1 -- 172.21.15.146:0/3697513049 learned_addr learned my addr 172.21.15.146:0/3697513049 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b6800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b015e9d0 0x7fa6b00c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b5e00640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6b00cb2f0 0x7fa6b014e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b6800640 1 -- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6b00cb2f0 msgr2=0x7fa6b014e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b6800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6b00cb2f0 0x7fa6b014e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b6800640 1 -- 172.21.15.146:0/3697513049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 msgr2=0x7fa6b00c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b6800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 0x7fa6b00c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b6800640 1 -- 172.21.15.146:0/3697513049 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b00c6ad0 con 0x7fa6b015e9d0 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.616+0000 7fa6b5400640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 0x7fa6b00c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:48.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.617+0000 7fa6b6800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b015e9d0 0x7fa6b00c79c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00cad0 tx=0x7fa6ac00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.617+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac017070 con 0x7fa6b015e9d0 2024-09-16T01:49:48.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.617+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6ac00ab70 con 0x7fa6b015e9d0 2024-09-16T01:49:48.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.617+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac012400 con 0x7fa6b015e9d0 2024-09-16T01:49:48.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.617+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b00c6d90 con 0x7fa6b015e9d0 2024-09-16T01:49:48.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.617+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b01f04a0 con 0x7fa6b015e9d0 2024-09-16T01:49:48.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.618+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b00c84c0 con 0x7fa6b015e9d0 2024-09-16T01:49:48.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.619+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa6ac01a070 con 0x7fa6b015e9d0 2024-09-16T01:49:48.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.620+0000 7fa6aa000640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa69406c2b0 0x7fa69406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.620+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5767+0+0 (secure 0 0 0) 0x7fa6ac08af80 con 0x7fa6b015e9d0 2024-09-16T01:49:48.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.620+0000 7fa6b5e00640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa69406c2b0 0x7fa69406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.621+0000 7fa6b5e00640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa69406c2b0 0x7fa69406e770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4034000 tx=0x7fa6a402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.625+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa6ac059050 con 0x7fa6b015e9d0 2024-09-16T01:49:48.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:49:48.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.696+0000 7f4a5fa00640 1 Processor -- start 2024-09-16T01:49:48.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.696+0000 7f4a5fa00640 1 -- start start 2024-09-16T01:49:48.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5fa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a58154c10 0x7f4a5815b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5fa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a580cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5fa00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5815e0b0 con 0x7f4a580cca70 2024-09-16T01:49:48.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5fa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4a5815de60 con 0x7f4a58154c10 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5fa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5815e390 con 0x7f4a580cafa0 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a58154c10 0x7f4a5815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a4fe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4a580cca70 0x7f4a58073340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:48876/0 (socket says 172.21.15.146:48876) 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a4fe00640 1 -- 172.21.15.146:0/2768247151 learned_addr learned my addr 172.21.15.146:0/2768247151 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5da00640 1 --2- 172.21.15.146:0/2768247151 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a580cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a57e00640 1 -- 172.21.15.146:0/2768247151 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2892433522 0 0) 0x7f4a5815e0b0 con 0x7f4a580cca70 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a57e00640 1 -- 172.21.15.146:0/2768247151 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a3c003520 con 0x7f4a580cca70 2024-09-16T01:49:48.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5d000640 1 -- 172.21.15.146:0/2768247151 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 msgr2=0x7f4a580cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:48.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5d000640 1 --2- 172.21.15.146:0/2768247151 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a580cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5d000640 1 -- 172.21.15.146:0/2768247151 >> v1:172.21.15.161:6789/0 conn(0x7f4a580cca70 legacy=0x7f4a58073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.698+0000 7f4a5d000640 1 -- 172.21.15.146:0/2768247151 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5815e670 con 0x7f4a58154c10 2024-09-16T01:49:48.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.699+0000 7f4a5d000640 1 --2- 172.21.15.146:0/2768247151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a58154c10 0x7f4a5815b370 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a480097a0 tx=0x7f4a4802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574b063c9fe22320 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a57e00640 1 -- 172.21.15.146:0/2768247151 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48037070 con 0x7f4a58154c10 2024-09-16T01:49:48.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a57e00640 1 -- 172.21.15.146:0/2768247151 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a48032440 con 0x7f4a58154c10 2024-09-16T01:49:48.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a57e00640 1 -- 172.21.15.146:0/2768247151 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48032810 con 0x7f4a58154c10 2024-09-16T01:49:48.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a5fa00640 1 -- 172.21.15.146:0/2768247151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a58154c10 msgr2=0x7f4a5815b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/2768247151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a58154c10 0x7f4a5815b370 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a480097a0 tx=0x7f4a4802a180 comp rx=0 tx=0).stop 2024-09-16T01:49:48.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a5fa00640 1 -- 172.21.15.146:0/2768247151 shutdown_connections 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/2768247151 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a580cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/2768247151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a58154c10 0x7f4a5815b370 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.700+0000 7f4a5fa00640 1 -- 172.21.15.146:0/2768247151 >> 172.21.15.146:0/2768247151 conn(0x7f4a5806ce80 msgr2=0x7f4a5806d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.701+0000 7f4a5fa00640 1 -- 172.21.15.146:0/2768247151 shutdown_connections 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.701+0000 7f4a5fa00640 1 -- 172.21.15.146:0/2768247151 wait complete. 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.702+0000 7f4a5fa00640 1 Processor -- start 2024-09-16T01:49:48.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.702+0000 7f4a5fa00640 1 -- start start 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5fa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a58150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5fa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a580cca70 0x7f4a580b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5fa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a58154c10 0x7f4a580c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5fa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4a58154ff0 con 0x7f4a580cca70 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5fa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5815ec30 con 0x7f4a58154c10 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5fa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4a58073a50 con 0x7f4a580cafa0 2024-09-16T01:49:48.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a580cca70 0x7f4a580b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a580cca70 0x7f4a580b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59800/0 (socket says 172.21.15.146:59800) 2024-09-16T01:49:48.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a4fe00640 1 -- 172.21.15.146:0/386593943 learned_addr learned my addr 172.21.15.146:0/386593943 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:48.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a58150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.703+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a58154c10 0x7f4a580c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5d000640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a58154c10 msgr2=0x7f4a580c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a58154c10 0x7f4a580c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5d000640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a580cca70 msgr2=0x7f4a580b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a580cca70 0x7f4a580b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5d000640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a580b6080 con 0x7f4a580cafa0 2024-09-16T01:49:48.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a580cca70 0x7f4a580b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:48.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a58150e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a48031000 tx=0x7f4a480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48041030 con 0x7f4a580cafa0 2024-09-16T01:49:48.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a4803d4e0 con 0x7f4a580cafa0 2024-09-16T01:49:48.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a581509b0 con 0x7f4a580cafa0 2024-09-16T01:49:48.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.704+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48037040 con 0x7f4a580cafa0 2024-09-16T01:49:48.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.705+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a581f3a30 con 0x7f4a580cafa0 2024-09-16T01:49:48.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.706+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4a5815b370 con 0x7f4a580cafa0 2024-09-16T01:49:48.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.707+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4a4803d8b0 con 0x7f4a580cafa0 2024-09-16T01:49:48.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a55600640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3406c2e0 0x7f4a3406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 5767+0+0 (secure 0 0 0) 0x7f4a480ab600 con 0x7f4a580cafa0 2024-09-16T01:49:48.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3406c2e0 0x7f4a3406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v28) ==== 54+0+16 (secure 0 0 0) 0x7f4a4804b020 con 0x7f4a580cafa0 2024-09-16T01:49:48.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4a580ca2f0 0x7f4a581f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a581f3500 con 0x7f4a580ca2f0 2024-09-16T01:49:48.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.708+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3406c2e0 0x7f4a3406e7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4a5815b7f0 tx=0x7f4a40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.709+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4a580ca2f0 0x7f4a581f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.709+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4a580ca2f0 0x7f4a581f3c70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.710+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a581f3500 con 0x7f4a580ca2f0 2024-09-16T01:49:48.720 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:48.719+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:48.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.725+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a580cde90 con 0x7f4a580ca2f0 2024-09-16T01:49:48.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.726+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a580cde90 con 0x7f4a580ca2f0 2024-09-16T01:49:48.728 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:48.728 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:48.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.727+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4a5814d2e0 0x7f4a5814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.727+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4a580cc610 con 0x7f4a5814d2e0 2024-09-16T01:49:48.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.727+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4a5814d2e0 0x7f4a5814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.727+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4a5814d2e0 0x7f4a5814d6c0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.728+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a580cc610 con 0x7f4a5814d2e0 2024-09-16T01:49:48.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.743+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a580b55b0 con 0x7f4a5814d2e0 2024-09-16T01:49:48.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.745+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a580b55b0 con 0x7f4a5814d2e0 2024-09-16T01:49:48.746 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:48.747 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:48.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.745+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4a5814f3c0 0x7f4a5814f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.745+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4a5815ca60 con 0x7f4a5814f3c0 2024-09-16T01:49:48.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.745+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4a5814f3c0 0x7f4a5814f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.746+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4a5814f3c0 0x7f4a5814f7e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.747+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a5815ca60 con 0x7f4a5814f3c0 2024-09-16T01:49:48.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.760+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a5815cf70 con 0x7f4a5814f3c0 2024-09-16T01:49:48.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.762+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a5815cf70 con 0x7f4a5814f3c0 2024-09-16T01:49:48.763 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:48.764 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:48.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.762+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4a58062630 0x7f4a58062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.762+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4a58062fb0 con 0x7f4a58062630 2024-09-16T01:49:48.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.763+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4a58062630 0x7f4a58062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.763+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4a58062630 0x7f4a58062a70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.764+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a58062fb0 con 0x7f4a58062630 2024-09-16T01:49:48.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.778+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a58062fb0 con 0x7f4a58062630 2024-09-16T01:49:48.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.780+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a58062fb0 con 0x7f4a58062630 2024-09-16T01:49:48.782 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:48.783 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:48.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.780+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4a5802cb50 0x7f4a5802cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.780+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4a58062fb0 con 0x7f4a5802cb50 2024-09-16T01:49:48.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.781+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4a5802cb50 0x7f4a5802cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.781+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4a5802cb50 0x7f4a5802cf50 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.782+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a58062fb0 con 0x7f4a5802cb50 2024-09-16T01:49:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:48.796 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:48.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.796+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a58062fb0 con 0x7f4a5802cb50 2024-09-16T01:49:48.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.798+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a58062fb0 con 0x7f4a5802cb50 2024-09-16T01:49:48.800 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:48.801 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:48.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.798+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4a58204880 0x7f4a58206c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.798+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4a5802d9a0 con 0x7f4a58204880 2024-09-16T01:49:48.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.798+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4a58204880 0x7f4a58206c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.799+0000 7f4a4fe00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4a58204880 0x7f4a58206c70 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.800+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a5802d9a0 con 0x7f4a58204880 2024-09-16T01:49:48.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:49:48.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.831+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a5802d9a0 con 0x7f4a58204880 2024-09-16T01:49:48.833 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:48.832+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:48.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.834+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a5802d9a0 con 0x7f4a58204880 2024-09-16T01:49:48.835 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:48.835 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:48.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.834+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4a582076c0 0x7f4a58209ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.834+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4a5820c880 con 0x7f4a582076c0 2024-09-16T01:49:48.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.834+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4a582076c0 0x7f4a58209ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.835+0000 7f4a5da00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4a582076c0 0x7f4a58209ab0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.836+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a5820c880 con 0x7f4a582076c0 2024-09-16T01:49:48.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.857+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fa6b00cbea0 con 0x7fa6b015e9d0 2024-09-16T01:49:48.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.862+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a58209ff0 con 0x7f4a582076c0 2024-09-16T01:49:48.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.864+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a58209ff0 con 0x7f4a582076c0 2024-09-16T01:49:48.865 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:48.865 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:48.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.864+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a5820be70 0x7f4a5820c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:48.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.864+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4a5820fc20 con 0x7f4a5820be70 2024-09-16T01:49:48.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.864+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a5820be70 0x7f4a5820c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:48.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.865+0000 7f4a5d000640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a5820be70 0x7f4a5820c2b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:48.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.866+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a5820fc20 con 0x7f4a5820be70 2024-09-16T01:49:48.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.879+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4a5820a200 con 0x7f4a5820be70 2024-09-16T01:49:48.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a55600640 1 -- 172.21.15.146:0/386593943 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4a5820a200 con 0x7f4a5820be70 2024-09-16T01:49:48.883 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:48.884 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:48.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4a580ca2f0 msgr2=0x7f4a581f3c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4a580ca2f0 0x7f4a581f3c70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4a5814d2e0 msgr2=0x7f4a5814d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4a5814d2e0 0x7f4a5814d6c0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4a5814f3c0 msgr2=0x7f4a5814f7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4a5814f3c0 0x7f4a5814f7e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4a58062630 msgr2=0x7f4a58062a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4a58062630 0x7f4a58062a70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4a5802cb50 msgr2=0x7f4a5802cf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4a5802cb50 0x7f4a5802cf50 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4a58204880 msgr2=0x7f4a58206c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4a58204880 0x7f4a58206c70 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4a582076c0 msgr2=0x7f4a58209ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4a582076c0 0x7f4a58209ab0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a5820be70 msgr2=0x7f4a5820c2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a5820be70 0x7f4a5820c2b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5da00640 1 -- 172.21.15.146:0/386593943 reap_dead start 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5da00640 1 -- 172.21.15.146:0/386593943 reap_dead start 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3406c2e0 msgr2=0x7f4a3406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3406c2e0 0x7f4a3406e7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4a5815b7f0 tx=0x7f4a40009040 comp rx=0 tx=0).stop 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 msgr2=0x7f4a58150e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:48.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 --2- 172.21.15.146:0/386593943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a580cafa0 0x7f4a58150e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a48031000 tx=0x7f4a480046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5da00640 1 -- 172.21.15.146:0/386593943 reap_dead start 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5da00640 1 -- 172.21.15.146:0/386593943 reap_dead start 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5da00640 1 -- 172.21.15.146:0/386593943 reap_dead start 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 shutdown_connections 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.882+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 >> 172.21.15.146:0/386593943 conn(0x7f4a5806ce80 msgr2=0x7f4a58073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.883+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 shutdown_connections 2024-09-16T01:49:48.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:48.883+0000 7f4a5fa00640 1 -- 172.21.15.146:0/386593943 wait complete. 2024-09-16T01:49:48.942 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:48.941+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:48.973 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:48.973 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:48.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:49:49.022 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:49.021+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:49.123 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:49.122+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:49.181 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:49.181 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:49.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:49:49.235 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:49.234+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:49.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.295+0000 7fa6aa000640 1 -- 172.21.15.146:0/3697513049 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v29) ==== 101+0+0 (secure 0 0 0) 0x7fa6ac05d350 con 0x7fa6b015e9d0 2024-09-16T01:49:49.297 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:49:49.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa69406c2b0 msgr2=0x7fa69406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:49.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa69406c2b0 0x7fa69406e770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4034000 tx=0x7fa6a402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:49.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b015e9d0 msgr2=0x7fa6b00c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:49.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b015e9d0 0x7fa6b00c79c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00cad0 tx=0x7fa6ac00cf90 comp rx=0 tx=0).stop 2024-09-16T01:49:49.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 shutdown_connections 2024-09-16T01:49:49.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa69406c2b0 0x7fa69406e770 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:49.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6b015e9d0 0x7fa6b00c79c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:49.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b00cc0c0 0x7fa6b00c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:49.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 --2- 172.21.15.146:0/3697513049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6b00cb2f0 0x7fa6b014e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:49.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 >> 172.21.15.146:0/3697513049 conn(0x7fa6b006d560 msgr2=0x7fa6b00b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:49.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 shutdown_connections 2024-09-16T01:49:49.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:49.297+0000 7fa6b8800640 1 -- 172.21.15.146:0/3697513049 wait complete. 2024-09-16T01:49:49.325 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:49.324+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:49.393 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:49.394 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:49.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:49:49.452 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:49.451+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:49.528 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:49.527+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:49.564 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:49.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:49:49.628 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:49.628+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:49.729 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:49.728+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:49.774 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:49.774 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:49.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:49:49.832 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:49.830+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:49.931 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:49.930+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:49.984 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:49.984 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:49.996 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:49:50.032 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:50.031+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:50.133 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:50.132+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:50.153 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.153 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:49:50.233 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:50.233+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:50.282 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.282 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.291 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:49:50.335 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:50.334+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:50.395 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.395 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:49:50.445 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:50.444+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:50.537 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:50.536+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:50.548 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.548 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.557 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:49:50.637 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:50.636+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:50.661 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.661 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.669 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:49:50.753 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:50.752+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:50.813 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.813 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.822 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:49:50.863 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:50.862+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:50.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:49:50.948 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:50.947+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:50.966 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:50.966 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:50.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:49:51.041 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:51.040+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.055+0000 7ff547200640 1 Processor -- start 2024-09-16T01:49:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.056+0000 7ff547200640 1 -- start start 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff547200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff548154c70 0x7ff54815e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff547200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff548155050 con 0x7ff5480cb990 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff547200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54815ad10 con 0x7ff5480ca0e0 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff547200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff5480cb060 con 0x7ff548154c70 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff545400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff5480cb990 0x7ff54806f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47124/0 (socket says 172.21.15.146:47124) 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff545400640 1 -- 172.21.15.146:0/2336580419 learned_addr learned my addr 172.21.15.146:0/2336580419 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.057+0000 7ff545e00640 1 --2- 172.21.15.146:0/2336580419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff548154c70 0x7ff54815e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.058+0000 7ff545e00640 1 -- 172.21.15.146:0/2336580419 >> v1:172.21.15.161:6789/0 conn(0x7ff5480ca0e0 legacy=0x7ff548165230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.058+0000 7ff545e00640 1 -- 172.21.15.146:0/2336580419 >> v1:172.21.15.146:6789/0 conn(0x7ff5480cb990 legacy=0x7ff54806f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.058+0000 7ff545e00640 1 -- 172.21.15.146:0/2336580419 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54815aff0 con 0x7ff548154c70 2024-09-16T01:49:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.058+0000 7ff544a00640 1 -- 172.21.15.146:0/2336580419 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194528279 0 0) 0x7ff548155050 con 0x7ff5480cb990 2024-09-16T01:49:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff545e00640 1 --2- 172.21.15.146:0/2336580419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff548154c70 0x7ff54815e8b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5380097d0 tx=0x7ff53802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3692ae17d57d5637 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff544a00640 1 -- 172.21.15.146:0/2336580419 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538037070 con 0x7ff548154c70 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff544a00640 1 -- 172.21.15.146:0/2336580419 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff538032440 con 0x7ff548154c70 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff544a00640 1 -- 172.21.15.146:0/2336580419 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538032810 con 0x7ff548154c70 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff547200640 1 -- 172.21.15.146:0/2336580419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff548154c70 msgr2=0x7ff54815e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff547200640 1 --2- 172.21.15.146:0/2336580419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff548154c70 0x7ff54815e8b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5380097d0 tx=0x7ff53802a210 comp rx=0 tx=0).stop 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff547200640 1 -- 172.21.15.146:0/2336580419 shutdown_connections 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff547200640 1 --2- 172.21.15.146:0/2336580419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff548154c70 0x7ff54815e8b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff547200640 1 -- 172.21.15.146:0/2336580419 >> 172.21.15.146:0/2336580419 conn(0x7ff548071630 msgr2=0x7ff548071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.059+0000 7ff547200640 1 -- 172.21.15.146:0/2336580419 shutdown_connections 2024-09-16T01:49:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.060+0000 7ff547200640 1 -- 172.21.15.146:0/2336580419 wait complete. 2024-09-16T01:49:51.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.061+0000 7ff547200640 1 Processor -- start 2024-09-16T01:49:51.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.061+0000 7ff547200640 1 -- start start 2024-09-16T01:49:51.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff547200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 0x7ff5480af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff547200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5480cb990 0x7ff54815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff547200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff548154c70 0x7ff5481f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff547200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff54815b5b0 con 0x7ff548154c70 2024-09-16T01:49:51.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff547200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff54815b890 con 0x7ff5480cb990 2024-09-16T01:49:51.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff547200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff5480c9bc0 con 0x7ff5480ca0e0 2024-09-16T01:49:51.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff545e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 0x7ff5480af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff546800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff548154c70 0x7ff5481f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff545400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5480cb990 0x7ff54815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff545e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 0x7ff5480af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:33116/0 (socket says 172.21.15.146:33116) 2024-09-16T01:49:51.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.062+0000 7ff545e00640 1 -- 172.21.15.146:0/3864329835 learned_addr learned my addr 172.21.15.146:0/3864329835 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff545e00640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5480cb990 msgr2=0x7ff54815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5480cb990 0x7ff54815cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff545e00640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff548154c70 msgr2=0x7ff5481f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff548154c70 0x7ff5481f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff545e00640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5481e6050 con 0x7ff5480ca0e0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 0x7ff5480af470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff5380097d0 tx=0x7ff5380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538037040 con 0x7ff5480ca0e0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff538030770 con 0x7ff5480ca0e0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff548150b80 con 0x7ff5480ca0e0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538030b40 con 0x7ff5480ca0e0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.063+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5481f4c30 con 0x7ff5480ca0e0 2024-09-16T01:49:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.064+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff54806eb30 con 0x7ff5480ca0e0 2024-09-16T01:49:51.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.065+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff538044020 con 0x7ff5480ca0e0 2024-09-16T01:49:51.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.066+0000 7ff53e000640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff52806c2b0 0x7ff52806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.066+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 5767+0+0 (secure 0 0 0) 0x7ff5380ab760 con 0x7ff5480ca0e0 2024-09-16T01:49:51.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.066+0000 7ff545400640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff52806c2b0 0x7ff52806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.066+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v29) ==== 54+0+16 (secure 0 0 0) 0x7ff54806eb30 con 0x7ff5480ca0e0 2024-09-16T01:49:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.067+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff548150e10 0x7ff5480aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.067+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5481e41d0 con 0x7ff548150e10 2024-09-16T01:49:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.067+0000 7ff545400640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff52806c2b0 0x7ff52806e770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff534007f70 tx=0x7ff534007f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.067+0000 7ff546800640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff548150e10 0x7ff5480aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.067+0000 7ff546800640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff548150e10 0x7ff5480aeab0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.068+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5481e41d0 con 0x7ff548150e10 2024-09-16T01:49:51.079 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:51.080 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.083+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff5481e3ef0 con 0x7ff548150e10 2024-09-16T01:49:51.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.084+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5481e3ef0 con 0x7ff548150e10 2024-09-16T01:49:51.085 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:51.086 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:51.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.084+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff5481511f0 0x7ff5480b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.084+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff5481e6b20 con 0x7ff5481511f0 2024-09-16T01:49:51.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.085+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff5481511f0 0x7ff5480b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.085+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff5481511f0 0x7ff5480b3c10 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.086+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5481e6b20 con 0x7ff5481511f0 2024-09-16T01:49:51.088 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:49:51.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.101+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff5480aeff0 con 0x7ff5481511f0 2024-09-16T01:49:51.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.102+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5480aeff0 con 0x7ff5481511f0 2024-09-16T01:49:51.103 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:51.105 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:51.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.102+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff5481e7310 0x7ff5481e76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.102+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff5481e7c30 con 0x7ff5481e7310 2024-09-16T01:49:51.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.103+0000 7ff545400640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff5481e7310 0x7ff5481e76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.103+0000 7ff545400640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff5481e7310 0x7ff5481e76f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.104+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5481e7c30 con 0x7ff5481e7310 2024-09-16T01:49:51.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.117+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff548062630 con 0x7ff5481e7310 2024-09-16T01:49:51.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.119+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff548062630 con 0x7ff5481e7310 2024-09-16T01:49:51.120 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.119+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff548063110 0x7ff54814ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.119+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff548063550 con 0x7ff548063110 2024-09-16T01:49:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.119+0000 7ff546800640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff548063110 0x7ff54814ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.120+0000 7ff546800640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff548063110 0x7ff54814ed10 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.120+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff548063550 con 0x7ff548063110 2024-09-16T01:49:51.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.134+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff548063550 con 0x7ff548063110 2024-09-16T01:49:51.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.136+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff548063550 con 0x7ff548063110 2024-09-16T01:49:51.137 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.136+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff54814fa90 0x7ff54802c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.136+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff548063550 con 0x7ff54814fa90 2024-09-16T01:49:51.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.136+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff54814fa90 0x7ff54802c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.137+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff54814fa90 0x7ff54802c640 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.138+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff548063550 con 0x7ff54814fa90 2024-09-16T01:49:51.142 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:51.141+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:51.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.151+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff548063550 con 0x7ff54814fa90 2024-09-16T01:49:51.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.154+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff548063550 con 0x7ff54814fa90 2024-09-16T01:49:51.155 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:51.156 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:51.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.154+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff54802d090 0x7ff54802d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.154+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff54802da30 con 0x7ff54802d090 2024-09-16T01:49:51.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.155+0000 7ff545400640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff54802d090 0x7ff54802d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.155+0000 7ff545400640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff54802d090 0x7ff54802d4f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.156+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff54802da30 con 0x7ff54802d090 2024-09-16T01:49:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.169+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff54802da30 con 0x7ff54802d090 2024-09-16T01:49:51.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.171+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff54802da30 con 0x7ff54802d090 2024-09-16T01:49:51.172 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.171+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff548207740 0x7ff548209b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.171+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff54820c980 con 0x7ff548207740 2024-09-16T01:49:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.172+0000 7ff546800640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff548207740 0x7ff548209b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.172+0000 7ff546800640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff548207740 0x7ff548209b30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.173+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff54820c980 con 0x7ff548207740 2024-09-16T01:49:51.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.187+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff54820a0f0 con 0x7ff548207740 2024-09-16T01:49:51.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.189+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff54820a0f0 con 0x7ff548207740 2024-09-16T01:49:51.190 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.190+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff54820a810 0x7ff54820ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.190+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff54820fda0 con 0x7ff54820a810 2024-09-16T01:49:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.190+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff54820a810 0x7ff54820ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.190+0000 7ff545e00640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff54820a810 0x7ff54820ac70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.191+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff54820fda0 con 0x7ff54820a810 2024-09-16T01:49:51.195 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:51.195 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:51.195 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:51.196 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:51.196 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:51.205 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:49:51.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.204+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff54820b2c0 con 0x7ff54820a810 2024-09-16T01:49:51.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff53e000640 1 -- 172.21.15.146:0/3864329835 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff54820b2c0 con 0x7ff54820a810 2024-09-16T01:49:51.208 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff548150e10 msgr2=0x7ff5480aeab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff548150e10 0x7ff5480aeab0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff5481511f0 msgr2=0x7ff5480b3c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff5481511f0 0x7ff5480b3c10 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff5481e7310 msgr2=0x7ff5481e76f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.206+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff5481e7310 0x7ff5481e76f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff548063110 msgr2=0x7ff54814ed10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff548063110 0x7ff54814ed10 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff54814fa90 msgr2=0x7ff54802c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff54814fa90 0x7ff54802c640 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff54802d090 msgr2=0x7ff54802d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff54802d090 0x7ff54802d4f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff548207740 msgr2=0x7ff548209b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff548207740 0x7ff548209b30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff546800640 1 -- 172.21.15.146:0/3864329835 reap_dead start 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff54820a810 msgr2=0x7ff54820ac70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff54820a810 0x7ff54820ac70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff546800640 1 -- 172.21.15.146:0/3864329835 reap_dead start 2024-09-16T01:49:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff546800640 1 -- 172.21.15.146:0/3864329835 reap_dead start 2024-09-16T01:49:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff546800640 1 -- 172.21.15.146:0/3864329835 reap_dead start 2024-09-16T01:49:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff52806c2b0 msgr2=0x7ff52806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff52806c2b0 0x7ff52806e770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff534007f70 tx=0x7ff534007f00 comp rx=0 tx=0).stop 2024-09-16T01:49:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 msgr2=0x7ff5480af470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 0x7ff5480af470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff5380097d0 tx=0x7ff5380046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 shutdown_connections 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff52806c2b0 0x7ff52806e770 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 --2- 172.21.15.146:0/3864329835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5480ca0e0 0x7ff5480af470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 >> 172.21.15.146:0/3864329835 conn(0x7ff548071630 msgr2=0x7ff54806f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 shutdown_connections 2024-09-16T01:49:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.207+0000 7ff547200640 1 -- 172.21.15.146:0/3864329835 wait complete. 2024-09-16T01:49:51.265 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:51.264+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:51.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 0 entries 22'1 (0'0) error 2:ddb4e4d7:::200.00000002:head by mds.0.4:33 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 47.526032004000001, 2024-09-16T01:49:51.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001054845, 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826071+0000", 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.57e-06 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826088+0000", 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5670000000000001e-05 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826118+0000", 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9603000000000001e-05 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826136+0000", 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.88e-05 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826229+0000", 2024-09-16T01:49:51.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2366000000000003e-05 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827126+0000", 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00089683600000000003 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.1.9:17 2.1b e22/19 2:dbae76b5:::mds1_inotable:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:51.358 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.49149173, 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016283739999999999, 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860611+0000", 2024-09-16T01:49:51.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860613+0000", 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.446e-06 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860625+0000", 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2549e-05 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860645+0000", 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9545e-05 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860655+0000", 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9550000000000004e-06 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860902+0000", 2024-09-16T01:49:51.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024690900000000002 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860935+0000", 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3788999999999998e-05 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862198+0000", 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012625169999999999 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.362 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862239+0000", 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1664e-05 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 4 entries 22'3 (0'0) error 2:db457a8a:::201.00000001:head by mds.1.9:20 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.490175925999999, 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011507889999999999, 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.363 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861927+0000", 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861928+0000", 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6099999999999999e-07 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861940+0000", 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1817999999999999e-05 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861962+0000", 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2078000000000001e-05 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861976+0000", 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4338000000000001e-05 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862056+0000", 2024-09-16T01:49:51.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0253999999999994e-05 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863078+0000", 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213399999999999 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1b epoch 22/19 rep_tid 5 entries 22'4 (0'0) error 2:d80f8093:::201.00000002:head by mds.1.9:21 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:51.367 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.489847884, 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014125819999999999, 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862255+0000", 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862256+0000", 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5000000000000003e-07 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862264+0000", 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.789e-06 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862280+0000", 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5633000000000001e-05 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862294+0000", 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4316e-05 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862755+0000", 2024-09-16T01:49:51.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00046123399999999998 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863668+0000", 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00091206000000000002 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.1.9:34 2.1b e22/19 2:db457a8a:::201.00000001:head v 22'5, mlcod=22'3)", 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:51.371 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.482619747999998, 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00078421100000000004, 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869483+0000", 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869484+0000", 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0699999999999999e-06 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869492+0000", 2024-09-16T01:49:51.375 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0860000000000002e-06 2024-09-16T01:49:51.375 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869505+0000", 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.308e-05 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869512+0000", 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3010000000000003e-06 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.376 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869573+0000", 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0956999999999999e-05 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.869595+0000", 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2311999999999999e-05 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.870232+0000", 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000637456 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.870267+0000", 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4949e-05 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:35 2.a 2:54ba6923:::mds1_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.461763902000001, 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014025660000000001, 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.378 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890339+0000", 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890341+0000", 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.607e-06 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890347+0000", 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9530000000000002e-06 2024-09-16T01:49:51.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890358+0000", 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0910000000000001e-05 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890368+0000", 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6330000000000008e-06 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890418+0000", 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0290999999999997e-05 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890538+0000", 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011986599999999999 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.890640+0000", 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101625 2024-09-16T01:49:51.382 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891432+0000", 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00079213799999999996 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891662+0000", 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000230237 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.383 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891697+0000", 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4866999999999997e-05 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.891741+0000", 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4438999999999999e-05 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.384 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:11 2.c 2:36924187:::618.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.456386541000001, 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055247409999999997, 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895716+0000", 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895717+0000", 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1289999999999999e-06 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895724+0000", 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0769999999999996e-06 2024-09-16T01:49:51.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895733+0000", 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4210000000000001e-06 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895741+0000", 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0290000000000005e-06 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895816+0000", 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5190000000000003e-05 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895955+0000", 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138737 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896073+0000", 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011843799999999999 2024-09-16T01:49:51.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897963+0000", 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001890061 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901132+0000", 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031683089999999998 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901196+0000", 2024-09-16T01:49:51.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4132000000000002e-05 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901241+0000", 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5218e-05 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:12 2.c 2:36924187:::618.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:51.390 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.456334245999997, 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056235809999999999, 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895769+0000", 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895782+0000", 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2383e-05 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0747999999999999e-05 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895799+0000", 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.849e-06 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896381+0000", 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058177900000000004 2024-09-16T01:49:51.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896459+0000", 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8511999999999996e-05 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897012+0000", 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055262699999999998 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900008+0000", 2024-09-16T01:49:51.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029960410000000001 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901291+0000", 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00128261 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901351+0000", 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0603000000000002e-05 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.395 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901392+0000", 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0799e-05 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.456288970999999, 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055047550000000001, 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.396 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895814+0000", 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.397 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895815+0000", 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.85e-07 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895824+0000", 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4350000000000003e-06 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895842+0000", 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8264999999999999e-05 2024-09-16T01:49:51.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895850+0000", 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8339999999999999e-06 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895962+0000", 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111644 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896088+0000", 2024-09-16T01:49:51.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012620200000000001 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896219+0000", 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000130782 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898012+0000", 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001793474 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.400 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901212+0000", 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031999390000000002 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9030999999999998e-05 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901319+0000", 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7364000000000003e-05 2024-09-16T01:49:51.401 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.456230785000002, 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055742070000000003, 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.402 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895872+0000", 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895873+0000", 2024-09-16T01:49:51.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8100000000000001e-07 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895877+0000", 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3710000000000002e-06 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895886+0000", 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0739999999999999e-06 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895893+0000", 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3550000000000001e-06 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896519+0000", 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00062635700000000002 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896612+0000", 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3059000000000005e-05 2024-09-16T01:49:51.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896737+0000", 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012455499999999999 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900030+0000", 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032929740000000002 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901357+0000", 2024-09-16T01:49:51.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001327204 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901403+0000", 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6508000000000003e-05 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901446+0000", 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2868999999999999e-05 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.456182779000002, 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0070037900000000002, 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895920+0000", 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895921+0000", 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8899999999999999e-07 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895926+0000", 2024-09-16T01:49:51.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.023e-06 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895934+0000", 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1939999999999998e-06 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895941+0000", 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0770000000000002e-06 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897276+0000", 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001334614 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897409+0000", 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000133215 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897545+0000", 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013621099999999999 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901441+0000", 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003895459 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902804+0000", 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013631120000000001 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902877+0000", 2024-09-16T01:49:51.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3647999999999997e-05 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902924+0000", 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6548000000000002e-05 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.2 epoch 22/19 rep_tid 20 entries 22'1 (0'0) error 2:40dbbc66:::202.00000003:head by mds.2.11:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:51.413 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.454973379999998, 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029206929999999998, 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897129+0000", 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897130+0000", 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897142+0000", 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1095e-05 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897165+0000", 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3278999999999999e-05 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897180+0000", 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5613000000000001e-05 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897269+0000", 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8749e-05 2024-09-16T01:49:51.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900050+0000", 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027809940000000002 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:28 2.c 2:35dcb8d2:::203.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.440326281000001, 2024-09-16T01:49:51.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047880470000000001, 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911777+0000", 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911779+0000", 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3609999999999999e-06 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911799+0000", 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9721e-05 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911817+0000", 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.802e-05 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911851+0000", 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4289000000000003e-05 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911931+0000", 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0142000000000006e-05 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912062+0000", 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013077100000000001 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916565+0000", 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004502743 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:33 2.a 2:556620e9:::203.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.440205632000001, 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026481180000000001, 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.422 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911897+0000", 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911898+0000", 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2799999999999995e-07 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911912+0000", 2024-09-16T01:49:51.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4003e-05 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911922+0000", 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.045e-05 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911929+0000", 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6850000000000004e-06 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911993+0000", 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4196999999999995e-05 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912089+0000", 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5383000000000005e-05 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914545+0000", 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024565720000000002 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.425 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:34 2.c 2:35dcb8d2:::203.00000001:head [write 0~588] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.435141903000002, 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041374530000000001, 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916961+0000", 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916963+0000", 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.64e-06 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.427 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916972+0000", 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7659999999999993e-06 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916990+0000", 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7445e-05 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916999+0000", 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3859999999999996e-06 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917077+0000", 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8307999999999996e-05 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917172+0000", 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4617000000000006e-05 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917357+0000", 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018458799999999999 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918065+0000", 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070841800000000005 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.920973+0000", 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029077769999999998 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.921071+0000", 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7918000000000007e-05 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.921098+0000", 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7589999999999998e-05 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.431 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:35 2.a 2:579bf162:::mds3_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.422239007000002, 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001587256, 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929864+0000", 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929867+0000", 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.858e-06 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929878+0000", 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1651000000000001e-05 2024-09-16T01:49:51.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929898+0000", 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9828999999999999e-05 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929907+0000", 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1239999999999994e-06 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.929986+0000", 2024-09-16T01:49:51.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8608000000000004e-05 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.930169+0000", 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018343600000000001 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.930319+0000", 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014941299999999999 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931104+0000", 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00078539600000000005 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931374+0000", 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027029000000000002 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931415+0000", 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0068999999999999e-05 2024-09-16T01:49:51.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.931451+0000", 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6582000000000003e-05 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:24 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.415663705, 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048886399999999997, 2024-09-16T01:49:51.437 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936439+0000", 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936441+0000", 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0600000000000002e-06 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936464+0000", 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3161999999999999e-05 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.439 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936480+0000", 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5537e-05 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936490+0000", 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6889999999999999e-06 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936578+0000", 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8596000000000004e-05 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936729+0000", 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015100000000000001 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936946+0000", 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000216586 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939283+0000", 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023372559999999998 2024-09-16T01:49:51.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941230+0000", 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001946677 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941279+0000", 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9478999999999998e-05 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941328+0000", 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8597999999999998e-05 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.4.15:11 2.2 e22/19 2:447438e6:::62c.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.414814345, 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002990131, 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.443 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937288+0000", 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937290+0000", 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.299e-06 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937298+0000", 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4400000000000005e-06 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937309+0000", 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1049e-05 2024-09-16T01:49:51.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3410000000000003e-06 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937382+0000", 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4307999999999995e-05 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.446 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937403+0000", 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0664000000000001e-05 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940230+0000", 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028275319999999998 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940279+0000", 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8498000000000002e-05 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.447 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.4.15:12 2.2 e22/19 2:447438e6:::62c.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.413993152, 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0022086779999999999, 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938110+0000", 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938111+0000", 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.95e-07 2024-09-16T01:49:51.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938119+0000", 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9370000000000004e-06 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938137+0000", 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8360999999999999e-05 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938145+0000", 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4060000000000005e-06 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938218+0000", 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2739999999999998e-05 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938238+0000", 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9738999999999999e-05 2024-09-16T01:49:51.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940284+0000", 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020457850000000001 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940318+0000", 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4715000000000002e-05 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.2 epoch 22/19 rep_tid 37 entries 22'4 (0'0) error 2:4182a243:::204.00000001:head by mds.4.15:28 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.413927996999998, 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002163842, 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:51.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938175+0000", 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5599999999999995e-07 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938191+0000", 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5967000000000001e-05 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938224+0000", 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2169000000000003e-05 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938239+0000", 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5063e-05 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938468+0000", 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000229082 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940339+0000", 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001871005 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.456 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:51.457 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:49:51.457 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:49:51.457 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:49:51.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-09-16T01:49:51.458 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:51.362+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:51.459 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:49:51.460 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:51.458+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:51.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.470+0000 7fb0c4600640 1 Processor -- start 2024-09-16T01:49:51.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.470+0000 7fb0c4600640 1 -- start start 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c4600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc15aab0 0x7fb0bc0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c4600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0bc15af00 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c4600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0bc15b1d0 con 0x7fb0bc0cc0c0 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c4600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb0bc0cbd00 con 0x7fb0bc15aab0 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c1c00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb0bc0cc0c0 0x7fb0bc15e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:48886/0 (socket says 172.21.15.146:48886) 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c1c00640 1 -- 172.21.15.146:0/4044425211 learned_addr learned my addr 172.21.15.146:0/4044425211 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:51.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c2600640 1 --2- 172.21.15.146:0/4044425211 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc15aab0 0x7fb0bc0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0c1200640 1 --1- 172.21.15.146:0/4044425211 >> v1:172.21.15.146:6789/0 conn(0x7fb0bc0cb2f0 0x7fb0bc163420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47140/0 (socket says 172.21.15.146:47140) 2024-09-16T01:49:51.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.471+0000 7fb0bbe00640 1 -- 172.21.15.146:0/4044425211 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 263820721 0 0) 0x7fb0bc15b1d0 con 0x7fb0bc0cc0c0 2024-09-16T01:49:51.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.472+0000 7fb0bbe00640 1 -- 172.21.15.146:0/4044425211 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a4003520 con 0x7fb0bc0cc0c0 2024-09-16T01:49:51.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.472+0000 7fb0c2600640 1 -- 172.21.15.146:0/4044425211 >> v1:172.21.15.161:6789/0 conn(0x7fb0bc0cc0c0 legacy=0x7fb0bc15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.472+0000 7fb0c2600640 1 -- 172.21.15.146:0/4044425211 >> v1:172.21.15.146:6789/0 conn(0x7fb0bc0cb2f0 legacy=0x7fb0bc163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.472+0000 7fb0bbe00640 1 -- 172.21.15.146:0/4044425211 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187213854 0 0) 0x7fb0bc15af00 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.472+0000 7fb0c2600640 1 -- 172.21.15.146:0/4044425211 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc15b4b0 con 0x7fb0bc15aab0 2024-09-16T01:49:51.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.473+0000 7fb0c2600640 1 --2- 172.21.15.146:0/4044425211 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc15aab0 0x7fb0bc0cc4a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009dc0 tx=0x7fb0b402a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f53d2ce731d0a2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.473+0000 7fb0bbe00640 1 -- 172.21.15.146:0/4044425211 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4007740 con 0x7fb0bc15aab0 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.473+0000 7fb0bbe00640 1 -- 172.21.15.146:0/4044425211 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b40078e0 con 0x7fb0bc15aab0 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.473+0000 7fb0bbe00640 1 -- 172.21.15.146:0/4044425211 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4007cb0 con 0x7fb0bc15aab0 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.473+0000 7fb0c4600640 1 -- 172.21.15.146:0/4044425211 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc15aab0 msgr2=0x7fb0bc0cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.473+0000 7fb0c4600640 1 --2- 172.21.15.146:0/4044425211 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc15aab0 0x7fb0bc0cc4a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009dc0 tx=0x7fb0b402a6d0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.474+0000 7fb0c4600640 1 -- 172.21.15.146:0/4044425211 shutdown_connections 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.474+0000 7fb0c4600640 1 --2- 172.21.15.146:0/4044425211 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc15aab0 0x7fb0bc0cc4a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.474+0000 7fb0c4600640 1 -- 172.21.15.146:0/4044425211 >> 172.21.15.146:0/4044425211 conn(0x7fb0bc06d560 msgr2=0x7fb0bc06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:51.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.474+0000 7fb0c4600640 1 -- 172.21.15.146:0/4044425211 shutdown_connections 2024-09-16T01:49:51.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.474+0000 7fb0c4600640 1 -- 172.21.15.146:0/4044425211 wait complete. 2024-09-16T01:49:51.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.475+0000 7fb0c4600640 1 Processor -- start 2024-09-16T01:49:51.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.475+0000 7fb0c4600640 1 -- start start 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.476+0000 7fb0c4600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 0x7fb0bc1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.476+0000 7fb0c4600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0bc0cc0c0 0x7fb0bc14a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.476+0000 7fb0c4600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0bc15aab0 0x7fb0bc148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.476+0000 7fb0c4600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc15e540 con 0x7fb0bc0cc0c0 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.476+0000 7fb0c4600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc15ba70 con 0x7fb0bc15aab0 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.476+0000 7fb0c4600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb0bc15e1d0 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0bc0cc0c0 0x7fb0bc14a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 0x7fb0bc1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 0x7fb0bc1567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:33132/0 (socket says 172.21.15.146:33132) 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 -- 172.21.15.146:0/2925158322 learned_addr learned my addr 172.21.15.146:0/2925158322 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c2600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0bc15aab0 0x7fb0bc148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 -- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0bc15aab0 msgr2=0x7fb0bc148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0bc15aab0 0x7fb0bc148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 -- 172.21.15.146:0/2925158322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0bc0cc0c0 msgr2=0x7fb0bc14a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0bc0cc0c0 0x7fb0bc14a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 -- 172.21.15.146:0/2925158322 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc156d10 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.477+0000 7fb0c1c00640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 0x7fb0bc1567d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c460 tx=0x7fb0b000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.478+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0017070 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.478+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b0005ce0 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.478+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0007290 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.478+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc15ce10 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.478+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc15be10 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.479+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0bc0c5560 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.480+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb0b001a070 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.481+0000 7fb0b9600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb09c06c2b0 0x7fb09c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.481+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 5767+0+0 (secure 0 0 0) 0x7fb0b008a7c0 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.481+0000 7fb0c1200640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb09c06c2b0 0x7fb09c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.482+0000 7fb0c1200640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb09c06c2b0 0x7fb09c06e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc14f000 tx=0x7fb0a8007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.485+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb0b0058800 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.560 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:51.559+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:51.603 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:51.604 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:51.604 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:49:51.604 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:51.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:49:51.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.616+0000 7fd4d3200640 1 Processor -- start 2024-09-16T01:49:51.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.616+0000 7fd4d3200640 1 -- start start 2024-09-16T01:49:51.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d3200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4d40c6b70 0x7fd4d415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40c75a0 0x7fd4d40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4d40cbab0 0x7fd4d40cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d415b880 con 0x7fd4d40cbab0 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d3200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d40cab20 con 0x7fd4d40c6b70 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d406d140 con 0x7fd4d40c75a0 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d1e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4d40c6b70 0x7fd4d415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4cbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40c75a0 0x7fd4d40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d1e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4d40c6b70 0x7fd4d415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:58102/0 (socket says 172.21.15.146:58102) 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d1e00640 1 -- 172.21.15.146:0/1369841173 learned_addr learned my addr 172.21.15.146:0/1369841173 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.617+0000 7fd4d2800640 1 --2- 172.21.15.146:0/1369841173 >> v2:172.21.15.146:3300/0 conn(0x7fd4d40cbab0 0x7fd4d40cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.618+0000 7fd4d1e00640 1 -- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40c75a0 msgr2=0x7fd4d40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:51.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.618+0000 7fd4d1e00640 1 --2- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40c75a0 0x7fd4d40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.618+0000 7fd4d1e00640 1 -- 172.21.15.146:0/1369841173 >> v2:172.21.15.146:3300/0 conn(0x7fd4d40cbab0 msgr2=0x7fd4d40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.618+0000 7fd4d1e00640 1 --2- 172.21.15.146:0/1369841173 >> v2:172.21.15.146:3300/0 conn(0x7fd4d40cbab0 0x7fd4d40cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.618+0000 7fd4d1e00640 1 -- 172.21.15.146:0/1369841173 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d406d420 con 0x7fd4d40c6b70 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d1e00640 1 --2- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d415b340 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4075ec0 tx=0x7fd4c402a0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51f801ff1ab76b63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d1400640 1 -- 172.21.15.146:0/1369841173 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4037070 con 0x7fd4d40c6b70 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d1400640 1 -- 172.21.15.146:0/1369841173 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c402ae00 con 0x7fd4d40c6b70 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d1400640 1 -- 172.21.15.146:0/1369841173 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4032660 con 0x7fd4d40c6b70 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 -- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 msgr2=0x7fd4d415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 --2- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d415b340 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4075ec0 tx=0x7fd4c402a0b0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 -- 172.21.15.146:0/1369841173 shutdown_connections 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 --2- 172.21.15.146:0/1369841173 >> v2:172.21.15.146:3300/0 conn(0x7fd4d40cbab0 0x7fd4d40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 --2- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40c75a0 0x7fd4d40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 --2- 172.21.15.146:0/1369841173 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d415b340 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.619+0000 7fd4d3200640 1 -- 172.21.15.146:0/1369841173 >> 172.21.15.146:0/1369841173 conn(0x7fd4d40ccc20 msgr2=0x7fd4d40cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.620+0000 7fd4d3200640 1 -- 172.21.15.146:0/1369841173 shutdown_connections 2024-09-16T01:49:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.620+0000 7fd4d3200640 1 -- 172.21.15.146:0/1369841173 wait complete. 2024-09-16T01:49:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.621+0000 7fd4d3200640 1 Processor -- start 2024-09-16T01:49:51.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.621+0000 7fd4d3200640 1 -- start start 2024-09-16T01:49:51.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d41519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4d40c75a0 0x7fd4d41f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40cbab0 0x7fd4d415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d40cb050 con 0x7fd4d40c75a0 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d40cad70 con 0x7fd4d40c6b70 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d406cd20 con 0x7fd4d40cbab0 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d41519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40cbab0 0x7fd4d415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d41519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:58118/0 (socket says 172.21.15.146:58118) 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.622+0000 7fd4d1e00640 1 -- 172.21.15.146:0/2796480571 learned_addr learned my addr 172.21.15.146:0/2796480571 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4cbe00640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4d40c75a0 0x7fd4d41f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d2800640 1 -- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 msgr2=0x7fd4d41519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d2800640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d41519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d2800640 1 -- 172.21.15.146:0/2796480571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4d40c75a0 msgr2=0x7fd4d41f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d2800640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4d40c75a0 0x7fd4d41f4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:51.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d2800640 1 -- 172.21.15.146:0/2796480571 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d4151f30 con 0x7fd4d40cbab0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d1e00640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d41519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d2800640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40cbab0 0x7fd4d415a530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00efc0 tx=0x7fd4cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc018020 con 0x7fd4d40cbab0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4cc014480 con 0x7fd4d40cbab0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc014870 con 0x7fd4d40cbab0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d41563b0 con 0x7fd4d40cbab0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.623+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d40c25e0 con 0x7fd4d40cbab0 2024-09-16T01:49:51.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.624+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d40c2920 con 0x7fd4d40cbab0 2024-09-16T01:49:51.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.625+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd4cc005d10 con 0x7fd4d40cbab0 2024-09-16T01:49:51.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.626+0000 7fd4ca000640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4b006c2b0 0x7fd4b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:51.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.626+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 5767+0+0 (secure 0 0 0) 0x7fd4cc08c1f0 con 0x7fd4d40cbab0 2024-09-16T01:49:51.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.626+0000 7fd4d1e00640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4b006c2b0 0x7fd4b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:51.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.627+0000 7fd4d1e00640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4b006c2b0 0x7fd4b006e770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4035040 tx=0x7fd4c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:51.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.630+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd4d40c2920 con 0x7fd4d40cbab0 2024-09-16T01:49:51.660 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:51.660+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:51.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.677+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fb0bc159930 con 0x7fb0bc0cb2f0 2024-09-16T01:49:51.715 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:51.715 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:51.715 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:51.716 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:51.716 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:51.724 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:49:51.762 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:51.761+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:51.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:51.814+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd out", "ids": ["7"]} v 0) -- 0x7fd4d4158240 con 0x7fd4d40cbab0 2024-09-16T01:49:51.815 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:49:51.815+0000 7f49bee00640 -1 mon.a@0(leader).osd e29 definitely_dead 0 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.003039483000002, 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055002239999999997, 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.830 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824857+0000", 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1457000000000001e-05 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824870+0000", 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2483999999999999e-05 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824885+0000", 2024-09-16T01:49:51.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4758e-05 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824894+0000", 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5319999999999995e-06 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015500299999999999 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825242+0000", 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000192986 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825387+0000", 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000144986 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828137+0000", 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027499 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830217+0000", 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020800369999999999 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830297+0000", 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0011000000000004e-05 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830346+0000", 2024-09-16T01:49:51.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.002967132000002, 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055792059999999997, 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.836 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824919+0000", 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7000000000000005e-07 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824922+0000", 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824931+0000", 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824935+0000", 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825813+0000", 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00087810899999999996 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825889+0000", 2024-09-16T01:49:51.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6157999999999998e-05 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826034+0000", 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014488599999999999 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828143+0000", 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002108408 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830415+0000", 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00227247 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830467+0000", 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830498+0000", 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0956e-05 2024-09-16T01:49:51.841 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.002922052999999, 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0072987340000000003, 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.842 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824964+0000", 2024-09-16T01:49:51.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2800000000000002e-07 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824974+0000", 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0627e-05 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824982+0000", 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3159999999999999e-06 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2280000000000001e-06 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825059+0000", 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9754999999999999e-05 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825232+0000", 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000172986 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825435+0000", 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00020313199999999999 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828256+0000", 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028206009999999998 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832187+0000", 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039315920000000002 2024-09-16T01:49:51.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832228+0000", 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0420999999999998e-05 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832262+0000", 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4647999999999998e-05 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.002874908999999, 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073664890000000004, 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825015+0000", 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7129999999999999e-06 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825024+0000", 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5629999999999992e-06 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825029+0000", 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9269999999999996e-06 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825830+0000", 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00080144400000000001 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825970+0000", 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014016899999999999 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826100+0000", 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012975500000000001 2024-09-16T01:49:51.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828596+0000", 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024965830000000001 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832306+0000", 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037092140000000002 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832349+0000", 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3210000000000001e-05 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832377+0000", 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8161e-05 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:51.853 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.002838883999999, 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059226180000000002, 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3399999999999999e-07 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825061+0000", 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4144999999999999e-05 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825069+0000", 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5209999999999997e-06 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825074+0000", 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8450000000000002e-06 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826274+0000", 2024-09-16T01:49:51.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001200586 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826528+0000", 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000253446 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826691+0000", 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016368900000000001 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828551+0000", 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0018598 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830882+0000", 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023311180000000001 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830929+0000", 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6882999999999998e-05 2024-09-16T01:49:51.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830969+0000", 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0051e-05 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.967350594999999, 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:49:51.859 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:51.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:49:51.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.967252088000002, 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:49:51.865 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:49:51.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:49:51.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:7 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.932260933999999, 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057447150000000001, 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.871 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895627+0000", 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0480000000000001e-06 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895632+0000", 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5879999999999997e-06 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895650+0000", 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7322000000000001e-05 2024-09-16T01:49:51.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895657+0000", 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1949999999999997e-06 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895729+0000", 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2120999999999996e-05 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895880+0000", 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015082600000000001 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896034+0000", 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015426799999999999 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897859+0000", 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001824736 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034126600000000001 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901329+0000", 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7836e-05 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901369+0000", 2024-09-16T01:49:51.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114999999999998e-05 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:8 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.932211750999997, 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058392840000000001, 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.877 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9900000000000001e-07 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895677+0000", 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1e-06 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895684+0000", 2024-09-16T01:49:51.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5769999999999999e-06 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895699+0000", 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.473e-05 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896845+0000", 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011460960000000001 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896914+0000", 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9115999999999994e-05 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897029+0000", 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114974 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900398+0000", 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033696749999999999 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901420+0000", 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213030000000001 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901470+0000", 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0083999999999997e-05 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901513+0000", 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3130000000000002e-05 2024-09-16T01:49:51.882 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.932158635999997, 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048597969999999999, 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895737+0000", 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2070000000000001e-06 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895743+0000", 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3430000000000001e-06 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895747+0000", 2024-09-16T01:49:51.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0940000000000001e-06 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895803+0000", 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6339999999999999e-05 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895947+0000", 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143199 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:51.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896084+0000", 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000137126 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900351+0000", 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042672250000000004 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900515+0000", 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016424500000000001 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900552+0000", 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6804999999999997e-05 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900587+0000", 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4582999999999997e-05 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.932127672999997, 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049417419999999998, 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895761+0000", 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4760000000000001e-06 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895765+0000", 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2540000000000003e-06 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895772+0000", 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4679999999999997e-06 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896412+0000", 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064042899999999998 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896492+0000", 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0500000000000005e-05 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896615+0000", 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000122194 2024-09-16T01:49:51.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900360+0000", 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037456830000000001 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900635+0000", 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027509799999999999 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900673+0000", 2024-09-16T01:49:51.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7265999999999999e-05 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900700+0000", 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.688e-05 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:51.894 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.932106060000002, 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0078497150000000002, 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:51.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895780+0000", 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1699999999999998e-07 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895783+0000", 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9239999999999999e-06 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895789+0000", 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8080000000000001e-06 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4089999999999999e-06 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897342+0000", 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001550287 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897499+0000", 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015634399999999999 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897630+0000", 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013139099999999999 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902981+0000", 2024-09-16T01:49:51.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053504950000000003 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903540+0000", 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055920099999999995 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903591+0000", 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1224999999999998e-05 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903629+0000", 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8114000000000001e-05 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1e epoch 22/19 rep_tid 7 entries 22'2 (0'0) error 2:7d92db23:::202.00000002:head by mds.2.11:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.930547111000003, 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056237750000000001, 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.900 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897339+0000", 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.029e-06 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897350+0000", 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0913e-05 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898029+0000", 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00067867600000000002 2024-09-16T01:49:51.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898053+0000", 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3799999999999999e-05 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898142+0000", 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9368000000000002e-05 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902962+0000", 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048199890000000002 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:17 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'11, mlcod=22'9)", 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.929293066, 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043821320000000004, 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.904 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898593+0000", 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.023e-06 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898603+0000", 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2490000000000007e-06 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898619+0000", 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6073e-05 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898627+0000", 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3950000000000001e-06 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898688+0000", 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0545999999999999e-05 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898704+0000", 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5903999999999999e-05 2024-09-16T01:49:51.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902933+0000", 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042294389999999998 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902975+0000", 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1502999999999999e-05 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:18 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'12, mlcod=22'9)", 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.929145472000002, 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042640050000000004, 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:51.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898741+0000", 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8999999999999995e-07 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898747+0000", 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3029999999999996e-06 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898756+0000", 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3969999999999994e-06 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898764+0000", 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898924+0000", 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016043599999999999 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898948+0000", 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4068999999999999e-05 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902977+0000", 2024-09-16T01:49:51.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040289449999999999 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903004+0000", 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6784e-05 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:26 2.10 e22/19 2:091e33df:::502.00000000:head v 22'14, mlcod=22'9)", 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.928188845000001, 2024-09-16T01:49:51.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039915870000000004, 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:51.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899698+0000", 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9999999999999995e-07 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899706+0000", 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.96e-06 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899791+0000", 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5702e-05 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899800+0000", 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6840000000000002e-06 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899870+0000", 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9617999999999997e-05 2024-09-16T01:49:51.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899888+0000", 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8046e-05 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903646+0000", 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037583310000000002 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903688+0000", 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2246000000000002e-05 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:26 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.915970191, 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050127030000000003, 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.918 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911918+0000", 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3479999999999998e-06 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911934+0000", 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6659000000000001e-05 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911948+0000", 2024-09-16T01:49:51.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3334e-05 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911957+0000", 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8390000000000002e-06 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912051+0000", 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3702999999999994e-05 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912182+0000", 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013119100000000001 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000179147 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913761+0000", 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001399382 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916815+0000", 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030542350000000002 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916884+0000", 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8823000000000002e-05 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916928+0000", 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4042000000000003e-05 2024-09-16T01:49:51.923 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:24 2.13 2:cbf94b35:::mds3_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.915889866000001, 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057070769999999996, 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:51.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912001+0000", 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2320000000000003e-06 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912015+0000", 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4562000000000001e-05 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912022+0000", 2024-09-16T01:49:51.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.165e-06 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912138+0000", 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116098 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912260+0000", 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121224 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101796 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913434+0000", 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010729610000000001 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917594+0000", 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041594010000000001 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917652+0000", 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8357999999999999e-05 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917703+0000", 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0460000000000001e-05 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.929 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:13 2.13 2:c81eaac8:::62d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.892185338000001, 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004875094, 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935702+0000", 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.277e-06 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935711+0000", 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6100000000000006e-06 2024-09-16T01:49:51.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935729+0000", 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8224e-05 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935742+0000", 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2405999999999999e-05 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935832+0000", 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0382000000000003e-05 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936059+0000", 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022734599999999999 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936207+0000", 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014755500000000001 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938029+0000", 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001821694 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940508+0000", 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024797479999999999 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8040000000000002e-05 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940575+0000", 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8812000000000001e-05 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:14 2.13 2:c81eaac8:::62d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.892022664999999, 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055486820000000001, 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:51.935 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935864+0000", 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935868+0000", 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7639999999999996e-06 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935877+0000", 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7560000000000005e-06 2024-09-16T01:49:51.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935888+0000", 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1184e-05 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937525+0000", 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001637023 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937611+0000", 2024-09-16T01:49:51.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6114999999999998e-05 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937759+0000", 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014728699999999999 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938463+0000", 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070381299999999996 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941309+0000", 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002846374 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941366+0000", 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6848000000000002e-05 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941411+0000", 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5698000000000002e-05 2024-09-16T01:49:51.940 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.941 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:51.941 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:51.941 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:51.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:49:51.943 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:51.941+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:52.043 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:52.042+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:52.100 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:52.100 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:52.100 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:52.100 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:52.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:49:52.159 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:52.158+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:52.246 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:52.245+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:52.270 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:52.271 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:52.271 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:52.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:49:52.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.307+0000 7fd4ca000640 1 -- 172.21.15.146:0/2796480571 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["7"]}]=0 marked out osd.7. v30) ==== 87+0+0 (secure 0 0 0) 0x7fd4cc022030 con 0x7fd4d40cbab0 2024-09-16T01:49:52.309 INFO:teuthology.orchestra.run.smithi146.stderr:marked out osd.7. 2024-09-16T01:49:52.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.308+0000 7fb0b9600640 1 -- 172.21.15.146:0/2925158322 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v30) ==== 87+0+0 (secure 0 0 0) 0x7fb0b005cb00 con 0x7fb0bc0cb2f0 2024-09-16T01:49:52.309 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:49:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb09c06c2b0 msgr2=0x7fb09c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb09c06c2b0 0x7fb09c06e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc14f000 tx=0x7fb0a8007de0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 msgr2=0x7fb0bc1567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 0x7fb0bc1567d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c460 tx=0x7fb0b000c920 comp rx=0 tx=0).stop 2024-09-16T01:49:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 shutdown_connections 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb09c06c2b0 0x7fb09c06e770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0bc15aab0 0x7fb0bc148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0bc0cc0c0 0x7fb0bc14a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 --2- 172.21.15.146:0/2925158322 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb0bc0cb2f0 0x7fb0bc1567d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 >> 172.21.15.146:0/2925158322 conn(0x7fb0bc06d560 msgr2=0x7fb0bc0cf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.314+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 shutdown_connections 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.314+0000 7fb0c4600640 1 -- 172.21.15.146:0/2925158322 wait complete. 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4b006c2b0 msgr2=0x7fd4b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4b006c2b0 0x7fd4b006e770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4035040 tx=0x7fd4c402e000 comp rx=0 tx=0).stop 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40cbab0 msgr2=0x7fd4d415a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40cbab0 0x7fd4d415a530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00efc0 tx=0x7fd4cc00c490 comp rx=0 tx=0).stop 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 shutdown_connections 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4b006c2b0 0x7fd4b006e770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4d40cbab0 0x7fd4d415a530 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4d40c75a0 0x7fd4d41f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 --2- 172.21.15.146:0/2796480571 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4d40c6b70 0x7fd4d41519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.312+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 >> 172.21.15.146:0/2796480571 conn(0x7fd4d40ccc20 msgr2=0x7fd4d40c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.313+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 shutdown_connections 2024-09-16T01:49:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:52.313+0000 7fd4d3200640 1 -- 172.21.15.146:0/2796480571 wait complete. 2024-09-16T01:49:52.347 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:52.346+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:52.448 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:52.447+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.646410945, 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:49:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:49:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.646304895999997, 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:49:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:49:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:49:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.646250270000003, 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:49:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:49:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:49:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.646181937000001, 2024-09-16T01:49:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:49:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:49:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.611926734000001, 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:49:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:49:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:49:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.611743535999999, 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:49:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:49:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.611689527999999, 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:49:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:49:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:49:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.611657088000001, 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:49:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.608289087000003, 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:49:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.608218411000003, 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:49:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.608182036000002, 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:49:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.607554293, 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:49:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:49:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.541 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.607499685999997, 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:49:52.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:49:52.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:52.546 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535361278, 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:52.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:49:52.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535302966, 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.553 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:49:52.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535228890999999, 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:52.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:49:52.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535170269999998, 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:49:52.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:49:52.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535148335999999, 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.571 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:52.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:49:52.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:49:52.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535117975999999, 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.831 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:49:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:49:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.535091396999999, 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:52.842 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:52.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:49:52.844 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:52.843+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:52.944 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:52.944+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:53.042 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:53.043 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:53.045 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:53.044+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:53.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:49:53.146 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:53.145+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:53.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:49:53.255 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:53.255 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:53.255 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:53.254+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:53.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:49:53.354 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:53.353+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:53.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.395+0000 7fc9fe800640 1 Processor -- start 2024-09-16T01:49:53.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.396+0000 7fc9fe800640 1 -- start start 2024-09-16T01:49:53.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9fe800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80c68f0 0x7fc9f815e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9fe800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f815eee0 con 0x7fc9f80cb3a0 2024-09-16T01:49:53.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9fe800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f815e640 con 0x7fc9f80ca270 2024-09-16T01:49:53.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9fe800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f815e920 con 0x7fc9f80c68f0 2024-09-16T01:49:53.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f7e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc9f80cb3a0 0x7fc9f80cc000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42892/0 (socket says 172.21.15.146:42892) 2024-09-16T01:49:53.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f7e00640 1 -- 172.21.15.146:0/1685420397 learned_addr learned my addr 172.21.15.146:0/1685420397 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:53.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f7400640 1 --2- 172.21.15.146:0/1685420397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80c68f0 0x7fc9f815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2199823746 0 0) 0x7fc9f815eee0 con 0x7fc9f80cb3a0 2024-09-16T01:49:53.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9dc003520 con 0x7fc9f80cb3a0 2024-09-16T01:49:53.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 564461447 0 0) 0x7fc9f815e640 con 0x7fc9f80ca270 2024-09-16T01:49:53.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.397+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9f815eee0 con 0x7fc9f80ca270 2024-09-16T01:49:53.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.398+0000 7fc9f7400640 1 -- 172.21.15.146:0/1685420397 >> v1:172.21.15.161:6789/0 conn(0x7fc9f80ca270 legacy=0x7fc9f815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.398+0000 7fc9f7400640 1 -- 172.21.15.146:0/1685420397 >> v1:172.21.15.146:6789/0 conn(0x7fc9f80cb3a0 legacy=0x7fc9f80cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.398+0000 7fc9f7400640 1 -- 172.21.15.146:0/1685420397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f815ec00 con 0x7fc9f80c68f0 2024-09-16T01:49:53.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9f7400640 1 --2- 172.21.15.146:0/1685420397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80c68f0 0x7fc9f815e000 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80097d0 tx=0x7fc9e802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279070115919c3cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8037070 con 0x7fc9f80c68f0 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e8032440 con 0x7fc9f80c68f0 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9f6a00640 1 -- 172.21.15.146:0/1685420397 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8032810 con 0x7fc9f80c68f0 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9fe800640 1 -- 172.21.15.146:0/1685420397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80c68f0 msgr2=0x7fc9f815e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9fe800640 1 --2- 172.21.15.146:0/1685420397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80c68f0 0x7fc9f815e000 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80097d0 tx=0x7fc9e802a180 comp rx=0 tx=0).stop 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9fe800640 1 -- 172.21.15.146:0/1685420397 shutdown_connections 2024-09-16T01:49:53.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.399+0000 7fc9fe800640 1 --2- 172.21.15.146:0/1685420397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80c68f0 0x7fc9f815e000 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.400+0000 7fc9fe800640 1 -- 172.21.15.146:0/1685420397 >> 172.21.15.146:0/1685420397 conn(0x7fc9f806cb70 msgr2=0x7fc9f806e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:53.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.400+0000 7fc9fe800640 1 -- 172.21.15.146:0/1685420397 shutdown_connections 2024-09-16T01:49:53.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.400+0000 7fc9fe800640 1 -- 172.21.15.146:0/1685420397 wait complete. 2024-09-16T01:49:53.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.401+0000 7fc9fe800640 1 Processor -- start 2024-09-16T01:49:53.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.401+0000 7fc9fe800640 1 -- start start 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9fe800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 0x7fc9f81f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9fe800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9f80ca270 0x7fc9f81528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9fe800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80cb3a0 0x7fc9f80cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9fe800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f80c7770 con 0x7fc9f80c68f0 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9fe800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f80cbad0 con 0x7fc9f80ca270 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9fe800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f80c78f0 con 0x7fc9f80cb3a0 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9f7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 0x7fc9f81f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.402+0000 7fc9efe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9f80ca270 0x7fc9f81528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 0x7fc9f81f3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33114/0 (socket says 172.21.15.146:33114) 2024-09-16T01:49:53.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 -- 172.21.15.146:0/2554452833 learned_addr learned my addr 172.21.15.146:0/2554452833 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:53.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80cb3a0 0x7fc9f80cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80cb3a0 msgr2=0x7fc9f80cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9f80cb3a0 0x7fc9f80cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9f80ca270 msgr2=0x7fc9f81528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9f80ca270 0x7fc9f81528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f80c4eb0 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.403+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 0x7fc9f81f3c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80098f0 tx=0x7fc9e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.404+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8041020 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.404+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e803d6e0 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.404+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8037040 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.404+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f81f0700 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.404+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f80cffa0 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.405+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc9f815e360 con 0x7fc9f80c68f0 2024-09-16T01:49:53.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.406+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc9e8044020 con 0x7fc9f80c68f0 2024-09-16T01:49:53.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9ef400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9d406c2b0 0x7fc9d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5738+0+0 (secure 0 0 0) 0x7fc9e80aba90 con 0x7fc9f80c68f0 2024-09-16T01:49:53.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9efe00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9d406c2b0 0x7fc9d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v31) ==== 54+0+16 (secure 0 0 0) 0x7fc9f815e360 con 0x7fc9f80c68f0 2024-09-16T01:49:53.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc9f81585e0 0x7fc9f80cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9f80c5d50 con 0x7fc9f81585e0 2024-09-16T01:49:53.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9efe00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9d406c2b0 0x7fc9d406e770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0007fd0 tx=0x7fc9e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.407+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc9f81585e0 0x7fc9f80cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.408+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc9f81585e0 0x7fc9f80cf6e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.409+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f80c5d50 con 0x7fc9f81585e0 2024-09-16T01:49:53.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.423+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f80cbde0 con 0x7fc9f81585e0 2024-09-16T01:49:53.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.425+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f80cbde0 con 0x7fc9f81585e0 2024-09-16T01:49:53.426 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:53.427 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:53.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.425+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc9f80ccd00 0x7fc9f8062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.425+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc9f81f0de0 con 0x7fc9f80ccd00 2024-09-16T01:49:53.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.426+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc9f80ccd00 0x7fc9f8062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.426+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc9f80ccd00 0x7fc9f8062630 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.427+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f81f0de0 con 0x7fc9f80ccd00 2024-09-16T01:49:53.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.442+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f81f1010 con 0x7fc9f80ccd00 2024-09-16T01:49:53.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.443+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f81f1010 con 0x7fc9f80ccd00 2024-09-16T01:49:53.445 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:53.445 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:53.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.444+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc9f80cd0e0 0x7fc9f8063080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.444+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc9f80635c0 con 0x7fc9f80cd0e0 2024-09-16T01:49:53.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.444+0000 7fc9efe00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc9f80cd0e0 0x7fc9f8063080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.445+0000 7fc9efe00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc9f80cd0e0 0x7fc9f8063080 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.446+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f80635c0 con 0x7fc9f80cd0e0 2024-09-16T01:49:53.449 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:53.448+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:53.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.460+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f80635c0 con 0x7fc9f80cd0e0 2024-09-16T01:49:53.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.462+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f80635c0 con 0x7fc9f80cd0e0 2024-09-16T01:49:53.463 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:53.464 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:53.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.463+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc9f8159480 0x7fc9f81e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.463+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc9f80635c0 con 0x7fc9f8159480 2024-09-16T01:49:53.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.463+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc9f8159480 0x7fc9f81e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.463+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc9f8159480 0x7fc9f81e5ca0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.464+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f80635c0 con 0x7fc9f8159480 2024-09-16T01:49:53.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.478+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f8159b40 con 0x7fc9f8159480 2024-09-16T01:49:53.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.480+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f8159b40 con 0x7fc9f8159480 2024-09-16T01:49:53.481 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:53.482 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:53.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.480+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc9f81e6ab0 0x7fc9f802c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.480+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc9f80635c0 con 0x7fc9f81e6ab0 2024-09-16T01:49:53.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.481+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc9f81e6ab0 0x7fc9f802c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.481+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc9f81e6ab0 0x7fc9f802c640 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.482+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f80635c0 con 0x7fc9f81e6ab0 2024-09-16T01:49:53.489 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:53.489 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:53.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.495+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f802ce10 con 0x7fc9f81e6ab0 2024-09-16T01:49:53.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.497+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f802ce10 con 0x7fc9f81e6ab0 2024-09-16T01:49:53.499 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:53.499 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:53.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.498+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9f802d530 0x7fc9f8207060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.498+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc9f802d970 con 0x7fc9f802d530 2024-09-16T01:49:53.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.498+0000 7fc9efe00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9f802d530 0x7fc9f8207060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.499+0000 7fc9efe00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9f802d530 0x7fc9f8207060 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.499+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f802d970 con 0x7fc9f802d530 2024-09-16T01:49:53.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:49:53.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.513+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f802db50 con 0x7fc9f802d530 2024-09-16T01:49:53.518 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.515+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f802db50 con 0x7fc9f802d530 2024-09-16T01:49:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.515+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc9f8207ab0 0x7fc9f8209ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.515+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc9f820cc70 con 0x7fc9f8207ab0 2024-09-16T01:49:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.515+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc9f8207ab0 0x7fc9f8209ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.516+0000 7fc9f7e00640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc9f8207ab0 0x7fc9f8209ea0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.517+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f820cc70 con 0x7fc9f8207ab0 2024-09-16T01:49:53.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.530+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f820a460 con 0x7fc9f8207ab0 2024-09-16T01:49:53.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.532+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f820a460 con 0x7fc9f8207ab0 2024-09-16T01:49:53.533 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:53.534 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:53.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.533+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc9f820aa60 0x7fc9f820ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:53.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.533+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc9f820fef0 con 0x7fc9f820aa60 2024-09-16T01:49:53.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.533+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc9f820aa60 0x7fc9f820ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:53.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.533+0000 7fc9f7400640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc9f820aa60 0x7fc9f820ce50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:53.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.534+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9f820fef0 con 0x7fc9f820aa60 2024-09-16T01:49:53.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.548+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc9f80009b0 con 0x7fc9f820aa60 2024-09-16T01:49:53.550 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:53.549+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:53.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9ef400640 1 -- 172.21.15.146:0/2554452833 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc9f80009b0 con 0x7fc9f820aa60 2024-09-16T01:49:53.551 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:53.552 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:53.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc9f81585e0 msgr2=0x7fc9f80cf6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fc9f81585e0 0x7fc9f80cf6e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc9f80ccd00 msgr2=0x7fc9f8062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fc9f80ccd00 0x7fc9f8062630 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc9f80cd0e0 msgr2=0x7fc9f8063080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fc9f80cd0e0 0x7fc9f8063080 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc9f8159480 msgr2=0x7fc9f81e5ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fc9f8159480 0x7fc9f81e5ca0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc9f81e6ab0 msgr2=0x7fc9f802c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fc9f81e6ab0 0x7fc9f802c640 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9f802d530 msgr2=0x7fc9f8207060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9f802d530 0x7fc9f8207060 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc9f8207ab0 msgr2=0x7fc9f8209ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc9f8207ab0 0x7fc9f8209ea0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc9f820aa60 msgr2=0x7fc9f820ce50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fc9f820aa60 0x7fc9f820ce50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9f7e00640 1 -- 172.21.15.146:0/2554452833 reap_dead start 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9f7e00640 1 -- 172.21.15.146:0/2554452833 reap_dead start 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9f7e00640 1 -- 172.21.15.146:0/2554452833 reap_dead start 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9f7e00640 1 -- 172.21.15.146:0/2554452833 reap_dead start 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9f7e00640 1 -- 172.21.15.146:0/2554452833 reap_dead start 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9d406c2b0 msgr2=0x7fc9d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9d406c2b0 0x7fc9d406e770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0007fd0 tx=0x7fc9e0009040 comp rx=0 tx=0).stop 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 msgr2=0x7fc9f81f3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.550+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 0x7fc9f81f3c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80098f0 tx=0x7fc9e80046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.551+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 shutdown_connections 2024-09-16T01:49:53.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.551+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9d406c2b0 0x7fc9d406e770 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.551+0000 7fc9fe800640 1 --2- 172.21.15.146:0/2554452833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80c68f0 0x7fc9f81f3c70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:53.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.551+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 >> 172.21.15.146:0/2554452833 conn(0x7fc9f806cb70 msgr2=0x7fc9f8154760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:53.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.551+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 shutdown_connections 2024-09-16T01:49:53.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:53.551+0000 7fc9fe800640 1 -- 172.21.15.146:0/2554452833 wait complete. 2024-09-16T01:49:53.651 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:53.650+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:53.671 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:53.671 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:53.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:49:53.752 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:53.751+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:53.840 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:53.841 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:53.852 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:53.851+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:53.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:49:53.953 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:53.952+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:54.010 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:54.010 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:54.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:49:54.068 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:54.067+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:54.156 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:54.155+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:54.178 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:54.179 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:54.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:49:54.257 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:54.256+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:54.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:49:54.358 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:54.357+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:54.393 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:54.393 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:54.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:49:54.459 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:54.458+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:54.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.498+0000 7ff176800640 1 Processor -- start 2024-09-16T01:49:54.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.498+0000 7ff176800640 1 -- start start 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.499+0000 7ff176800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff17006eaa0 0x7ff1700ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff176800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1700c9440 con 0x7ff170158140 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff176800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1700b0d40 con 0x7ff17015aa00 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff176800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff1700c8dc0 con 0x7ff17006eaa0 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff170158140 0x7ff17006cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42896/0 (socket says 172.21.15.146:42896) 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff17006eaa0 0x7ff1700ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:54.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16f400640 1 -- 172.21.15.146:0/3323183108 learned_addr learned my addr 172.21.15.146:0/3323183108 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:54.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16ea00640 1 -- 172.21.15.146:0/3323183108 >> v1:172.21.15.161:6789/0 conn(0x7ff17015aa00 legacy=0x7ff1700ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:54.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16ea00640 1 -- 172.21.15.146:0/3323183108 >> v1:172.21.15.146:6789/0 conn(0x7ff170158140 legacy=0x7ff17006cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:54.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16ea00640 1 -- 172.21.15.146:0/3323183108 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1700c96e0 con 0x7ff17006eaa0 2024-09-16T01:49:54.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.500+0000 7ff16e000640 1 -- 172.21.15.146:0/3323183108 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463731227 0 0) 0x7ff1700c9440 con 0x7ff170158140 2024-09-16T01:49:54.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.501+0000 7ff16ea00640 1 --2- 172.21.15.146:0/3323183108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17006eaa0 0x7ff1700ce190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0097a0 tx=0x7ff15c02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6418432b59641a58 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:54.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff16e000640 1 -- 172.21.15.146:0/3323183108 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15c037070 con 0x7ff17006eaa0 2024-09-16T01:49:54.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff16e000640 1 -- 172.21.15.146:0/3323183108 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff15c032640 con 0x7ff17006eaa0 2024-09-16T01:49:54.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff16e000640 1 -- 172.21.15.146:0/3323183108 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15c032a10 con 0x7ff17006eaa0 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff176800640 1 -- 172.21.15.146:0/3323183108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17006eaa0 msgr2=0x7ff1700ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff176800640 1 --2- 172.21.15.146:0/3323183108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17006eaa0 0x7ff1700ce190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0097a0 tx=0x7ff15c02a2c0 comp rx=0 tx=0).stop 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff176800640 1 -- 172.21.15.146:0/3323183108 shutdown_connections 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff176800640 1 --2- 172.21.15.146:0/3323183108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17006eaa0 0x7ff1700ce190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.502+0000 7ff176800640 1 -- 172.21.15.146:0/3323183108 >> 172.21.15.146:0/3323183108 conn(0x7ff170067950 msgr2=0x7ff170067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.503+0000 7ff176800640 1 -- 172.21.15.146:0/3323183108 shutdown_connections 2024-09-16T01:49:54.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.503+0000 7ff176800640 1 -- 172.21.15.146:0/3323183108 wait complete. 2024-09-16T01:49:54.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.504+0000 7ff176800640 1 Processor -- start 2024-09-16T01:49:54.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.504+0000 7ff176800640 1 -- start start 2024-09-16T01:49:54.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff176800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 0x7ff1700b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:54.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff176800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 0x7ff17015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:54.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff176800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 0x7ff1700b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff176800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff1700b1020 con 0x7ff17015aa00 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff176800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff170158520 con 0x7ff17006eaa0 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff176800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff1700cede0 con 0x7ff170158140 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 0x7ff1700b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 0x7ff1700b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16ea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 0x7ff17015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 0x7ff1700b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47966/0 (socket says 172.21.15.146:47966) 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 0x7ff1700b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33142/0 (socket says 172.21.15.146:33142) 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16ea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 0x7ff17015ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56072/0 (socket says 172.21.15.146:56072) 2024-09-16T01:49:54.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.505+0000 7ff16f400640 1 -- 172.21.15.146:0/2717951246 learned_addr learned my addr 172.21.15.146:0/2717951246 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:54.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16fe00640 1 -- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 msgr2=0x7ff17015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:54.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16fe00640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 0x7ff17015ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:54.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16fe00640 1 -- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 msgr2=0x7ff1700b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:54.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16fe00640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 0x7ff1700b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16fe00640 1 -- 172.21.15.146:0/2717951246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1701f06a0 con 0x7ff17015aa00 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16ea00640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 0x7ff17015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16f400640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 0x7ff1700b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff16fe00640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 0x7ff1700b0150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff16800c730 tx=0x7ff16800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168003b70 con 0x7ff17015aa00 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff168003d10 con 0x7ff17015aa00 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.506+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168005310 con 0x7ff17015aa00 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.507+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1701f0960 con 0x7ff17015aa00 2024-09-16T01:49:54.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.507+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1701f0f20 con 0x7ff17015aa00 2024-09-16T01:49:54.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.508+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17015e980 con 0x7ff17015aa00 2024-09-16T01:49:54.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.509+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff1680056c0 con 0x7ff17015aa00 2024-09-16T01:49:54.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.509+0000 7ff167400640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff15006c2b0 0x7ff15006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:54.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.510+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5738+0+0 (secure 0 0 0) 0x7ff16808b290 con 0x7ff17015aa00 2024-09-16T01:49:54.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.510+0000 7ff16f400640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff15006c2b0 0x7ff15006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:54.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.510+0000 7ff16f400640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff15006c2b0 0x7ff15006e770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff160007b10 tx=0x7ff160007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:54.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.519+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff168059380 con 0x7ff17015aa00 2024-09-16T01:49:54.560 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:54.559+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:54.577 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:54.577 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:54.590 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:49:54.661 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:54.660+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:54.706 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:54.706 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:54.706+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff170157960 con 0x7ff17015aa00 2024-09-16T01:49:54.717 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:49:54.802 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:54.801+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:54.862 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:54.861+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:54.863 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:54.863 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:54.872 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:49:54.964 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:54.963+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:55.016 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.016 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.025 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:49:55.070 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:55.069+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:55.129 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.129 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.138 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:49:55.166 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:55.165+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:55.267 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:55.266+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:55.281 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.282 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.290 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:49:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.322+0000 7ff167400640 1 -- 172.21.15.146:0/2717951246 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v32) ==== 97+0+0 (secure 0 0 0) 0x7ff16805d680 con 0x7ff17015aa00 2024-09-16T01:49:55.323 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:49:55.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.324+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff15006c2b0 msgr2=0x7ff15006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.324+0000 7ff176800640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff15006c2b0 0x7ff15006e770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff160007b10 tx=0x7ff160007a60 comp rx=0 tx=0).stop 2024-09-16T01:49:55.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.324+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 msgr2=0x7ff1700b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.324+0000 7ff176800640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 0x7ff1700b0150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff16800c730 tx=0x7ff16800cbf0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 shutdown_connections 2024-09-16T01:49:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff15006c2b0 0x7ff15006e770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff17015aa00 0x7ff1700b0150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff170158140 0x7ff17015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 --2- 172.21.15.146:0/2717951246 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff17006eaa0 0x7ff1700b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 >> 172.21.15.146:0/2717951246 conn(0x7ff170067950 msgr2=0x7ff1700ce680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:55.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 shutdown_connections 2024-09-16T01:49:55.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.325+0000 7ff176800640 1 -- 172.21.15.146:0/2717951246 wait complete. 2024-09-16T01:49:55.386 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:55.385+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:55.396 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.396 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.405 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:49:55.469 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:55.468+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:55.509 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.509 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.518 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:49:55.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:49:55.549 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:49:55.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:49:55.591 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:55.590+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:55.624 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.624 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.633 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:49:55.676 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:55.673+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:55.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.751+0000 7fe9a3200640 1 Processor -- start 2024-09-16T01:49:55.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.752+0000 7fe9a3200640 1 -- start start 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0870 0x7fe9a4154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe9a40c6d00 0x7fe9a4154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a415e220 con 0x7fe9a40d0490 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe9a40c6160 con 0x7fe9a40c6d00 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a415e500 con 0x7fe9a40d0870 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a2800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe9a40c6d00 0x7fe9a4154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a1e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe9a40d0490 0x7fe9a40c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42906/0 (socket says 172.21.15.146:42906) 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a1e00640 1 -- 172.21.15.146:0/1296322070 learned_addr learned my addr 172.21.15.146:0/1296322070 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:55.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.753+0000 7fe9a1400640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0870 0x7fe9a4154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.754+0000 7fe9a2800640 1 -- 172.21.15.146:0/1296322070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0870 msgr2=0x7fe9a4154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:55.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.754+0000 7fe9a2800640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0870 0x7fe9a4154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.754+0000 7fe9a2800640 1 -- 172.21.15.146:0/1296322070 >> v1:172.21.15.146:6789/0 conn(0x7fe9a40d0490 legacy=0x7fe9a40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.754+0000 7fe9a0a00640 1 -- 172.21.15.146:0/1296322070 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3852430588 0 0) 0x7fe9a415e220 con 0x7fe9a40d0490 2024-09-16T01:49:55.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.754+0000 7fe9a2800640 1 -- 172.21.15.146:0/1296322070 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a415e7e0 con 0x7fe9a40c6d00 2024-09-16T01:49:55.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.754+0000 7fe9a1400640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0870 0x7fe9a4154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:55.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.755+0000 7fe9a2800640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4154bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009d90 tx=0x7fe99c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f710f9629a894d89 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.755+0000 7fe9a0a00640 1 -- 172.21.15.146:0/1296322070 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c007740 con 0x7fe9a40c6d00 2024-09-16T01:49:55.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.755+0000 7fe9a0a00640 1 -- 172.21.15.146:0/1296322070 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe99c0078e0 con 0x7fe9a40c6d00 2024-09-16T01:49:55.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.755+0000 7fe9a0a00640 1 -- 172.21.15.146:0/1296322070 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c007cb0 con 0x7fe9a40c6d00 2024-09-16T01:49:55.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 -- 172.21.15.146:0/1296322070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 msgr2=0x7fe9a4154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4154bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009d90 tx=0x7fe99c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 -- 172.21.15.146:0/1296322070 shutdown_connections 2024-09-16T01:49:55.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4154bd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 --2- 172.21.15.146:0/1296322070 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0870 0x7fe9a4154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 -- 172.21.15.146:0/1296322070 >> 172.21.15.146:0/1296322070 conn(0x7fe9a406d6f0 msgr2=0x7fe9a406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:55.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 -- 172.21.15.146:0/1296322070 shutdown_connections 2024-09-16T01:49:55.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.756+0000 7fe9a3200640 1 -- 172.21.15.146:0/1296322070 wait complete. 2024-09-16T01:49:55.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.758+0000 7fe9a3200640 1 Processor -- start 2024-09-16T01:49:55.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.758+0000 7fe9a3200640 1 -- start start 2024-09-16T01:49:55.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 0x7fe9a41557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a40d0870 0x7fe9a41512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a415dec0 con 0x7fe9a40c6d00 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a415eda0 con 0x7fe9a40d0870 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a40c51f0 con 0x7fe9a40d0490 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 0x7fe9a41557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 0x7fe9a41557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56090/0 (socket says 172.21.15.146:56090) 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 -- 172.21.15.146:0/3374949489 learned_addr learned my addr 172.21.15.146:0/3374949489 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a40d0870 0x7fe9a41512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a40d0870 msgr2=0x7fe9a41512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a40d0870 0x7fe9a41512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 msgr2=0x7fe9a4155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1400640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a41f0750 con 0x7fe9a40d0490 2024-09-16T01:49:55.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.759+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40c6d00 0x7fe9a4155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:55.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.760+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 0x7fe9a41557c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe99000e990 tx=0x7fe99000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.760+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990017070 con 0x7fe9a40d0490 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.760+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe99000c9e0 con 0x7fe9a40d0490 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.760+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99000cdb0 con 0x7fe9a40d0490 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.760+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a40c9b30 con 0x7fe9a40d0490 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.760+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a4156de0 con 0x7fe9a40d0490 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.761+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe9a40b6180 con 0x7fe9a40d0490 2024-09-16T01:49:55.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.762+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe99001a070 con 0x7fe9a40d0490 2024-09-16T01:49:55.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.763+0000 7fe99a000640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe98406c2b0 0x7fe98406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.763+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 5738+0+0 (secure 0 0 0) 0x7fe99008b340 con 0x7fe9a40d0490 2024-09-16T01:49:55.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.763+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe98406c2b0 0x7fe98406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.763+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v32) ==== 54+0+16 (secure 0 0 0) 0x7fe99008d8e0 con 0x7fe9a40d0490 2024-09-16T01:49:55.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.764+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe9a41e40d0 0x7fe9a4156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.764+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9a40cca90 con 0x7fe9a41e40d0 2024-09-16T01:49:55.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.764+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe98406c2b0 0x7fe98406e770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe994000f60 tx=0x7fe994007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.764+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe9a41e40d0 0x7fe9a4156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.765+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe9a41e40d0 0x7fe9a4156fe0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.766+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a40cca90 con 0x7fe9a41e40d0 2024-09-16T01:49:55.772 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:55.771+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:55.779 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.779 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.784+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a40cc7b0 con 0x7fe9a41e40d0 2024-09-16T01:49:55.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.787+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a40cc7b0 con 0x7fe9a41e40d0 2024-09-16T01:49:55.788 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:55.790 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:49:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.787+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe9a414fde0 0x7fe9a41501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.787+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe9a40c8d40 con 0x7fe9a414fde0 2024-09-16T01:49:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.787+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe9a414fde0 0x7fe9a41501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.788+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe9a414fde0 0x7fe9a41501c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.789+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a40c8d40 con 0x7fe9a414fde0 2024-09-16T01:49:55.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.807+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a40c8d40 con 0x7fe9a414fde0 2024-09-16T01:49:55.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.809+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a40c8d40 con 0x7fe9a414fde0 2024-09-16T01:49:55.810 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:55.810 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.809+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe9a41f3ea0 0x7fe9a41f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.809+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe9a41f47e0 con 0x7fe9a41f3ea0 2024-09-16T01:49:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.810+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe9a41f3ea0 0x7fe9a41f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.810+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe9a41f3ea0 0x7fe9a41f42a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.811+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9900593a0 con 0x7fe9a41f3ea0 2024-09-16T01:49:55.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.840+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a4062880 con 0x7fe9a41f3ea0 2024-09-16T01:49:55.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.841+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a4062880 con 0x7fe9a41f3ea0 2024-09-16T01:49:55.842 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:55.843 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:55.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.841+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe9a4063360 0x7fe9a40b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.841+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe9a4062880 con 0x7fe9a4063360 2024-09-16T01:49:55.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.842+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe9a4063360 0x7fe9a40b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.842+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe9a4063360 0x7fe9a40b3150 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.843+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a4062880 con 0x7fe9a4063360 2024-09-16T01:49:55.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.860+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a4062880 con 0x7fe9a4063360 2024-09-16T01:49:55.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.862+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a4062880 con 0x7fe9a4063360 2024-09-16T01:49:55.863 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:55.864 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:55.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.862+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe9a40b3ed0 0x7fe9a402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.862+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe9a4206420 con 0x7fe9a40b3ed0 2024-09-16T01:49:55.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.862+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe9a40b3ed0 0x7fe9a402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.863+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe9a40b3ed0 0x7fe9a402c640 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.864+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a4206420 con 0x7fe9a40b3ed0 2024-09-16T01:49:55.873 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:55.872+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:55.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.879+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a402cb80 con 0x7fe9a40b3ed0 2024-09-16T01:49:55.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.881+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a402cb80 con 0x7fe9a40b3ed0 2024-09-16T01:49:55.882 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:55.883 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:55.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.881+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe9a402d2a0 0x7fe9a402d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.881+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe9a42095b0 con 0x7fe9a402d2a0 2024-09-16T01:49:55.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.882+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe9a402d2a0 0x7fe9a402d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.882+0000 7fe9a1400640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe9a402d2a0 0x7fe9a402d700 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.883+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a42095b0 con 0x7fe9a402d2a0 2024-09-16T01:49:55.895 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:55.896 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:55.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.900+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a4206d20 con 0x7fe9a402d2a0 2024-09-16T01:49:55.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.901+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a4206d20 con 0x7fe9a402d2a0 2024-09-16T01:49:55.902 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:55.903 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:55.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.902+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe9a42073a0 0x7fe9a4209790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.902+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe9a420c560 con 0x7fe9a42073a0 2024-09-16T01:49:55.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.902+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe9a42073a0 0x7fe9a4209790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.903+0000 7fe9a2800640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe9a42073a0 0x7fe9a4209790 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.905 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:49:55.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.904+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a420c560 con 0x7fe9a42073a0 2024-09-16T01:49:55.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.917+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a4209cd0 con 0x7fe9a42073a0 2024-09-16T01:49:55.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.919+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a4209cd0 con 0x7fe9a42073a0 2024-09-16T01:49:55.920 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:55.921 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:55.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.919+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe9a420a350 0x7fe9a420c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:55.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.919+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe9a420f510 con 0x7fe9a420a350 2024-09-16T01:49:55.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.920+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe9a420a350 0x7fe9a420c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:55.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.920+0000 7fe9a1e00640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe9a420a350 0x7fe9a420c740 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:55.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.921+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9a420f510 con 0x7fe9a420a350 2024-09-16T01:49:55.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.934+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe9a420cc80 con 0x7fe9a420a350 2024-09-16T01:49:55.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe99a000640 1 -- 172.21.15.146:0/3374949489 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe9a420cc80 con 0x7fe9a420a350 2024-09-16T01:49:55.937 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:55.938 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:55.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe9a41e40d0 msgr2=0x7fe9a4156fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe9a41e40d0 0x7fe9a4156fe0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe9a414fde0 msgr2=0x7fe9a41501c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe9a414fde0 0x7fe9a41501c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe9a41f3ea0 msgr2=0x7fe9a41f42a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe9a41f3ea0 0x7fe9a41f42a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe9a4063360 msgr2=0x7fe9a40b3150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe9a4063360 0x7fe9a40b3150 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe9a40b3ed0 msgr2=0x7fe9a402c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe9a40b3ed0 0x7fe9a402c640 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe9a402d2a0 msgr2=0x7fe9a402d700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe9a402d2a0 0x7fe9a402d700 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe9a42073a0 msgr2=0x7fe9a4209790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe9a42073a0 0x7fe9a4209790 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe9a420a350 msgr2=0x7fe9a420c740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe9a420a350 0x7fe9a420c740 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a2800640 1 -- 172.21.15.146:0/3374949489 reap_dead start 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a2800640 1 -- 172.21.15.146:0/3374949489 reap_dead start 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a2800640 1 -- 172.21.15.146:0/3374949489 reap_dead start 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a2800640 1 -- 172.21.15.146:0/3374949489 reap_dead start 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a2800640 1 -- 172.21.15.146:0/3374949489 reap_dead start 2024-09-16T01:49:55.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe98406c2b0 msgr2=0x7fe98406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe98406c2b0 0x7fe98406e770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe994000f60 tx=0x7fe994007d50 comp rx=0 tx=0).stop 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 msgr2=0x7fe9a41557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.936+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 0x7fe9a41557c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe99000e990 tx=0x7fe99000ee50 comp rx=0 tx=0).stop 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.937+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 shutdown_connections 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.937+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe98406c2b0 0x7fe98406e770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.937+0000 7fe9a3200640 1 --2- 172.21.15.146:0/3374949489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe9a40d0490 0x7fe9a41557c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:55.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.937+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 >> 172.21.15.146:0/3374949489 conn(0x7fe9a406d6f0 msgr2=0x7fe9a4154460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:55.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.937+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 shutdown_connections 2024-09-16T01:49:55.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:55.937+0000 7fe9a3200640 1 -- 172.21.15.146:0/3374949489 wait complete. 2024-09-16T01:49:55.989 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:55.988+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:56.049 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:56.049 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:49:56.049 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:49:56.049 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:49:56.049 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:56.057 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:49:56.099 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:56.098+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:56.176 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:56.175+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.377066292000002, 2024-09-16T01:49:56.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055002239999999997, 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824846+0000", 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824857+0000", 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1457000000000001e-05 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824870+0000", 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2483999999999999e-05 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824885+0000", 2024-09-16T01:49:56.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4758e-05 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824894+0000", 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5319999999999995e-06 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015500299999999999 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825242+0000", 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000192986 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825387+0000", 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000144986 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828137+0000", 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027499 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830217+0000", 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020800369999999999 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830297+0000", 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0011000000000004e-05 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830346+0000", 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.209 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.376993941000002, 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055792059999999997, 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824918+0000", 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824919+0000", 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7000000000000005e-07 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824922+0000", 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824931+0000", 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824935+0000", 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825813+0000", 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00087810899999999996 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825889+0000", 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6157999999999998e-05 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826034+0000", 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014488599999999999 2024-09-16T01:49:56.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828143+0000", 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002108408 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830415+0000", 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00227247 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830467+0000", 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:49:56.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830498+0000", 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0956e-05 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:28 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.376948861999999, 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0072987340000000003, 2024-09-16T01:49:56.215 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824963+0000", 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824964+0000", 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2800000000000002e-07 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824974+0000", 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0627e-05 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824982+0000", 2024-09-16T01:49:56.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3159999999999999e-06 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2280000000000001e-06 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825059+0000", 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9754999999999999e-05 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825232+0000", 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000172986 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825435+0000", 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00020313199999999999 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828256+0000", 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028206009999999998 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832187+0000", 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039315920000000002 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832228+0000", 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0420999999999998e-05 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832262+0000", 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4647999999999998e-05 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:27 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.376901717999999, 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073664890000000004, 2024-09-16T01:49:56.221 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825011+0000", 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825015+0000", 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7129999999999999e-06 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825024+0000", 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5629999999999992e-06 2024-09-16T01:49:56.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825029+0000", 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9269999999999996e-06 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825830+0000", 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00080144400000000001 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825970+0000", 2024-09-16T01:49:56.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014016899999999999 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826100+0000", 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012975500000000001 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828596+0000", 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024965830000000001 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832306+0000", 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037092140000000002 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832349+0000", 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3210000000000001e-05 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832377+0000", 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8161e-05 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.226 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.376865692999999, 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059226180000000002, 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:56.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825047+0000", 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3399999999999999e-07 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825061+0000", 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4144999999999999e-05 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825069+0000", 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5209999999999997e-06 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825074+0000", 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8450000000000002e-06 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826274+0000", 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001200586 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826528+0000", 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000253446 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826691+0000", 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016368900000000001 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828551+0000", 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0018598 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830882+0000", 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023311180000000001 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830929+0000", 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6882999999999998e-05 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830969+0000", 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0051e-05 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 47.341377403999999, 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.232 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:49:56.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:49:56.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:49:56.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 47.341278897000002, 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:49:56.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:7 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.306287742999999, 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057447150000000001, 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.243 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895625+0000", 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895627+0000", 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0480000000000001e-06 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895632+0000", 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5879999999999997e-06 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895650+0000", 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7322000000000001e-05 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895657+0000", 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1949999999999997e-06 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895729+0000", 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2120999999999996e-05 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895880+0000", 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015082600000000001 2024-09-16T01:49:56.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896034+0000", 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015426799999999999 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897859+0000", 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001824736 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901271+0000", 2024-09-16T01:49:56.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034126600000000001 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901329+0000", 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7836e-05 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901369+0000", 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114999999999998e-05 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:8 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.306238559999997, 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058392840000000001, 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895674+0000", 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9900000000000001e-07 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895677+0000", 2024-09-16T01:49:56.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1e-06 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895684+0000", 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5769999999999999e-06 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895699+0000", 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.473e-05 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896845+0000", 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011460960000000001 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896914+0000", 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9115999999999994e-05 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897029+0000", 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114974 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900398+0000", 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033696749999999999 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901420+0000", 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010213030000000001 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901470+0000", 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0083999999999997e-05 2024-09-16T01:49:56.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901513+0000", 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3130000000000002e-05 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.306185444999997, 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048597969999999999, 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.254 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895727+0000", 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3e-07 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895737+0000", 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2070000000000001e-06 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895743+0000", 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3430000000000001e-06 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895747+0000", 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0940000000000001e-06 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895803+0000", 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6339999999999999e-05 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895947+0000", 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143199 2024-09-16T01:49:56.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896084+0000", 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000137126 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900351+0000", 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042672250000000004 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900515+0000", 2024-09-16T01:49:56.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016424500000000001 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900552+0000", 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6804999999999997e-05 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900587+0000", 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4582999999999997e-05 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.306154481999997, 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049417419999999998, 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895758+0000", 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895761+0000", 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4760000000000001e-06 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895765+0000", 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2540000000000003e-06 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895772+0000", 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4679999999999997e-06 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896412+0000", 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064042899999999998 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896492+0000", 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0500000000000005e-05 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896615+0000", 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000122194 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900360+0000", 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037456830000000001 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900635+0000", 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00027509799999999999 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900673+0000", 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7265999999999999e-05 2024-09-16T01:49:56.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900700+0000", 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.688e-05 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.306132869000002, 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0078497150000000002, 2024-09-16T01:49:56.265 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895779+0000", 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895780+0000", 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1699999999999998e-07 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895783+0000", 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9239999999999999e-06 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895789+0000", 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8080000000000001e-06 2024-09-16T01:49:56.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.895792+0000", 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4089999999999999e-06 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897342+0000", 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001550287 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897499+0000", 2024-09-16T01:49:56.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015634399999999999 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897630+0000", 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013139099999999999 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902981+0000", 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053504950000000003 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903540+0000", 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00055920099999999995 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903591+0000", 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1224999999999998e-05 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903629+0000", 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8114000000000001e-05 2024-09-16T01:49:56.270 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "pg_update_log_missing(2.1e epoch 22/19 rep_tid 7 entries 22'2 (0'0) error 2:7d92db23:::202.00000002:head by mds.2.11:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.304573920000003, 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056237750000000001, 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "reached pg", 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897338+0000", 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897339+0000", 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.029e-06 2024-09-16T01:49:56.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897350+0000", 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0913e-05 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898029+0000", 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00067867600000000002 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898053+0000", 2024-09-16T01:49:56.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3799999999999999e-05 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898142+0000", 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9368000000000002e-05 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902962+0000", 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048199890000000002 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:17 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'11, mlcod=22'9)", 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.303319875, 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043821320000000004, 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898592+0000", 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898593+0000", 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.023e-06 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898603+0000", 2024-09-16T01:49:56.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2490000000000007e-06 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898619+0000", 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6073e-05 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898627+0000", 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3950000000000001e-06 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898688+0000", 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0545999999999999e-05 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898704+0000", 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5903999999999999e-05 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902933+0000", 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042294389999999998 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902975+0000", 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1502999999999999e-05 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:18 2.10 e22/19 2:0e9da45e:::61b.00000000:head v 22'12, mlcod=22'9)", 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.303172281000002, 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042640050000000004, 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.279 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898740+0000", 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898741+0000", 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8999999999999995e-07 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898747+0000", 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3029999999999996e-06 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898756+0000", 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3969999999999994e-06 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898764+0000", 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898924+0000", 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016043599999999999 2024-09-16T01:49:56.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898948+0000", 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4068999999999999e-05 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902977+0000", 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040289449999999999 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903004+0000", 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6784e-05 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(mds.2.11:26 2.10 e22/19 2:091e33df:::502.00000000:head v 22'14, mlcod=22'9)", 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.302215654000001, 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039915870000000004, 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.284 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899697+0000", 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899698+0000", 2024-09-16T01:49:56.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9999999999999995e-07 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899706+0000", 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.96e-06 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899791+0000", 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5702e-05 2024-09-16T01:49:56.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899800+0000", 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6840000000000002e-06 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899870+0000", 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9617999999999997e-05 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.899888+0000", 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8046e-05 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903646+0000", 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037583310000000002 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903688+0000", 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2246000000000002e-05 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:26 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.289997, 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050127030000000003, 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.290 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911915+0000", 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911918+0000", 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3479999999999998e-06 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911934+0000", 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6659000000000001e-05 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911948+0000", 2024-09-16T01:49:56.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3334e-05 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911957+0000", 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8390000000000002e-06 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912051+0000", 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3702999999999994e-05 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912182+0000", 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013119100000000001 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000179147 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913761+0000", 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001399382 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916815+0000", 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030542350000000002 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916884+0000", 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8823000000000002e-05 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916928+0000", 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4042000000000003e-05 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.295 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:24 2.13 2:cbf94b35:::mds3_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.289916675000001, 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057070769999999996, 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911996+0000", 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912001+0000", 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2320000000000003e-06 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912015+0000", 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4562000000000001e-05 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912022+0000", 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.165e-06 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912138+0000", 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116098 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912260+0000", 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121224 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912362+0000", 2024-09-16T01:49:56.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101796 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913434+0000", 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0010729610000000001 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917594+0000", 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0041594010000000001 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917652+0000", 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8357999999999999e-05 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917703+0000", 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0460000000000001e-05 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:13 2.13 2:c81eaac8:::62d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:56.301 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.266212147000001, 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004875094, 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935700+0000", 2024-09-16T01:49:56.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935702+0000", 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.277e-06 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935711+0000", 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6100000000000006e-06 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935729+0000", 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8224e-05 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935742+0000", 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2405999999999999e-05 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935832+0000", 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0382000000000003e-05 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936059+0000", 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022734599999999999 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.936207+0000", 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014755500000000001 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938029+0000", 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001821694 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940508+0000", 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024797479999999999 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8040000000000002e-05 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940575+0000", 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8812000000000001e-05 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:14 2.13 2:c81eaac8:::62d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.266049473999999, 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055486820000000001, 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:49:56.307 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935863+0000", 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935864+0000", 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1999999999999998e-07 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935868+0000", 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7639999999999996e-06 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935877+0000", 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7560000000000005e-06 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.935888+0000", 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1184e-05 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937525+0000", 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001637023 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937611+0000", 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6114999999999998e-05 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937759+0000", 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014728699999999999 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.938463+0000", 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00070381299999999996 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941309+0000", 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002846374 2024-09-16T01:49:56.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941366+0000", 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6848000000000002e-05 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.941411+0000", 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5698000000000002e-05 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:49:56.312 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:49:56.313 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:49:56.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:49:56.315 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:56.313+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:56.414 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:56.413+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:56.476 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:56.477 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:56.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:49:56.515 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:56.514+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:56.616 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:56.615+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:56.662 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:56.662 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:56.662 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:56.662 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:56.662 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:56.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:49:56.716 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:56.715+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:56.817 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:56.817+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:56.881 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:49:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:49:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:49:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.053499596999998, 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:49:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:49:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:49:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.053393548000003, 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:49:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.053338922000002, 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:49:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:49:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:49:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.053270589, 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:49:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:49:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.019015386, 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:49:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.018832187999998, 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.018778179999998, 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:49:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:49:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.01874574, 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:49:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:49:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.015377739000002, 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:49:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.015307063000002, 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.015270688000001, 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:49:56.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.942 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.014642944999999, 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:56.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.014588338000003, 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942449929999999, 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:49:56.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942391617999998, 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:56.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:49:56.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942317543000001, 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:49:56.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:49:56.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:49:56.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942258922000001, 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:56.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:56.974 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942236988000001, 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:49:56.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942206628000001, 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.980 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:49:56.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.985 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.942180049000001, 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:56.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:49:56.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:49:56.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:56.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:56.991 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:56.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:49:56.993 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:56.992+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:57.093 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:57.093+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:57.156 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:57.156 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:57.156 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:57.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:49:57.194 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:57.193+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:57.295 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:57.295+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:57.332 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:57.332 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:57.332 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:57.332 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:57.332 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:57.336 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:49:57.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:49:57.337 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T01:49:57.337 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-16T01:49:57.337 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T01:49:57.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:49:57.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:49:57.433 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:57.432+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:57.497 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:57.496+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:57.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.506+0000 7fea9b200640 1 Processor -- start 2024-09-16T01:49:57.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.506+0000 7fea9b200640 1 -- start start 2024-09-16T01:49:57.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea9b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea9c154ac0 0x7fea9c15b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea9b200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fea9c06d820 0x7fea9c0d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea9b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c0d0d00 0x7fea9c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea9b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c15b6d0 con 0x7fea9c154ac0 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea9b200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c154ea0 con 0x7fea9c06d820 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea9b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c0c5340 con 0x7fea9c0d0d00 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea99e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea9c154ac0 0x7fea9c15b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea99e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea9c154ac0 0x7fea9c15b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33170/0 (socket says 172.21.15.146:33170) 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea99e00640 1 -- 172.21.15.146:0/2264004254 learned_addr learned my addr 172.21.15.146:0/2264004254 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.507+0000 7fea99400640 1 --2- 172.21.15.146:0/2264004254 >> v2:172.21.15.161:3300/0 conn(0x7fea9c06d820 0x7fea9c0d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.508+0000 7fea9a800640 1 --2- 172.21.15.146:0/2264004254 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c0d0d00 0x7fea9c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.508+0000 7fea99e00640 1 -- 172.21.15.146:0/2264004254 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c0d0d00 msgr2=0x7fea9c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:57.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.508+0000 7fea99e00640 1 --2- 172.21.15.146:0/2264004254 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c0d0d00 0x7fea9c15bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.508+0000 7fea99e00640 1 -- 172.21.15.146:0/2264004254 >> v2:172.21.15.161:3300/0 conn(0x7fea9c06d820 msgr2=0x7fea9c0d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:57.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.508+0000 7fea99e00640 1 --2- 172.21.15.146:0/2264004254 >> v2:172.21.15.161:3300/0 conn(0x7fea9c06d820 0x7fea9c0d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.508+0000 7fea99e00640 1 -- 172.21.15.146:0/2264004254 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0c5620 con 0x7fea9c154ac0 2024-09-16T01:49:57.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.509+0000 7fea99e00640 1 --2- 172.21.15.146:0/2264004254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c154ac0 0x7fea9c15b190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0b2e60 tx=0x7fea8c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4968d38ddc90230 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:57.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.510+0000 7fea98a00640 1 -- 172.21.15.146:0/2264004254 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c037070 con 0x7fea9c154ac0 2024-09-16T01:49:57.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.510+0000 7fea98a00640 1 -- 172.21.15.146:0/2264004254 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c032440 con 0x7fea9c154ac0 2024-09-16T01:49:57.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.510+0000 7fea98a00640 1 -- 172.21.15.146:0/2264004254 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c032810 con 0x7fea9c154ac0 2024-09-16T01:49:57.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 -- 172.21.15.146:0/2264004254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c154ac0 msgr2=0x7fea9c15b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 --2- 172.21.15.146:0/2264004254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c154ac0 0x7fea9c15b190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0b2e60 tx=0x7fea8c02a2a0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 -- 172.21.15.146:0/2264004254 shutdown_connections 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 --2- 172.21.15.146:0/2264004254 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c0d0d00 0x7fea9c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 --2- 172.21.15.146:0/2264004254 >> v2:172.21.15.161:3300/0 conn(0x7fea9c06d820 0x7fea9c0d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 --2- 172.21.15.146:0/2264004254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c154ac0 0x7fea9c15b190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 -- 172.21.15.146:0/2264004254 >> 172.21.15.146:0/2264004254 conn(0x7fea9c06e400 msgr2=0x7fea9c06e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 -- 172.21.15.146:0/2264004254 shutdown_connections 2024-09-16T01:49:57.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.511+0000 7fea9b200640 1 -- 172.21.15.146:0/2264004254 wait complete. 2024-09-16T01:49:57.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.514+0000 7fea9b200640 1 Processor -- start 2024-09-16T01:49:57.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.514+0000 7fea9b200640 1 -- start start 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.515+0000 7fea9b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 0x7fea9c158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.515+0000 7fea9b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c0d0d00 0x7fea9c159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.515+0000 7fea9b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c154ac0 0x7fea9c159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.515+0000 7fea9b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c0c5ec0 con 0x7fea9c0d0d00 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.515+0000 7fea9b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c0c5be0 con 0x7fea9c06d820 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.515+0000 7fea9b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c0c61a0 con 0x7fea9c154ac0 2024-09-16T01:49:57.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c0d0d00 0x7fea9c159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 0x7fea9c158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c0d0d00 0x7fea9c159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33174/0 (socket says 172.21.15.146:33174) 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 0x7fea9c158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47986/0 (socket says 172.21.15.146:47986) 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99400640 1 -- 172.21.15.146:0/69877953 learned_addr learned my addr 172.21.15.146:0/69877953 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea9a800640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c154ac0 0x7fea9c159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 -- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c154ac0 msgr2=0x7fea9c159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c154ac0 0x7fea9c159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 -- 172.21.15.146:0/69877953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c0d0d00 msgr2=0x7fea9c159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c0d0d00 0x7fea9c159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.516+0000 7fea99e00640 1 -- 172.21.15.146:0/69877953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0cb3b0 con 0x7fea9c06d820 2024-09-16T01:49:57.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.517+0000 7fea99e00640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 0x7fea9c158dd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea8c031000 tx=0x7fea8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.517+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c041020 con 0x7fea9c06d820 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.517+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c03d640 con 0x7fea9c06d820 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.517+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c037040 con 0x7fea9c06d820 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.517+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9c0cb690 con 0x7fea9c06d820 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.517+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9c0ccce0 con 0x7fea9c06d820 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.519+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9c0cad60 con 0x7fea9c06d820 2024-09-16T01:49:57.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.520+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fea8c044020 con 0x7fea9c06d820 2024-09-16T01:49:57.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.522+0000 7fea92000640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fea7c06c2b0 0x7fea7c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.522+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5738+0+0 (secure 0 0 0) 0x7fea8c0ab920 con 0x7fea9c06d820 2024-09-16T01:49:57.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.522+0000 7fea99400640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fea7c06c2b0 0x7fea7c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.523+0000 7fea99400640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fea7c06c2b0 0x7fea7c06e770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0cbb40 tx=0x7fea88009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:57.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.526+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fea8c04b030 con 0x7fea9c06d820 2024-09-16T01:49:57.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.560+0000 7fd6a2600640 1 Processor -- start 2024-09-16T01:49:57.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.560+0000 7fd6a2600640 1 -- start start 2024-09-16T01:49:57.562 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:57.562 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:49:57.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:49:57.562 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.791334436, 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:49:57.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:49:57.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:49:57.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.569 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.359780704000002, 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.570 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.577 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.339856482000002, 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.578 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:49:57.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.339836563999995, 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:49:57.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:49:57.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:49:57.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.339775979999999, 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:49:57.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.093490459999998, 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:49:57.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:49:57.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:49:57.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.093339946, 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:49:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:49:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:49:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.093320378000001, 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:49:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:49:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:49:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.523164163999994, 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:49:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522996345999999, 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:49:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:49:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522977363999999, 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:49:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:49:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:49:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:49:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522957732999998, 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:49:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522823826999996, 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:49:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:49:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:49:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:49:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522812289000001, 2024-09-16T01:49:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:49:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:49:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522793878000002, 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:49:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:49:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.522679208, 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:49:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:49:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:49:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.511897480000002, 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:49:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:49:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.511780822000006, 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:49:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.696247583000002, 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:49:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:49:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:49:57.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.561+0000 7fd6a2600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd69c15aa70 0x7fd69c0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.561+0000 7fd6a2600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c06c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:49:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:49:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.696186245, 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:49:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:49:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:49:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:49:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:49:57.694 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.561+0000 7fd6a2600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69c15af00 con 0x7fd69c0c5eb0 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.561+0000 7fd6a2600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd69c06dbd0 con 0x7fd69c15aa70 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.561+0000 7fd6a2600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd69c15b1e0 con 0x7fd69c0c6290 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd69be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd69c15aa70 0x7fd69c0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c06c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd6a1200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd69c0c5eb0 0x7fd69c154910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:51240/0 (socket says 172.21.15.146:51240) 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd6a1200640 1 -- 172.21.15.146:0/1963286497 learned_addr learned my addr 172.21.15.146:0/1963286497 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd69be00640 1 -- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 msgr2=0x7fd69c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd69be00640 1 --2- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c06c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd69be00640 1 -- 172.21.15.146:0/1963286497 >> v1:172.21.15.161:6789/0 conn(0x7fd69c0c5eb0 legacy=0x7fd69c154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd69b400640 1 -- 172.21.15.146:0/1963286497 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2668745662 0 0) 0x7fd69c15af00 con 0x7fd69c0c5eb0 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.562+0000 7fd69be00640 1 -- 172.21.15.146:0/1963286497 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69c15b4c0 con 0x7fd69c15aa70 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.564+0000 7fd69be00640 1 --2- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c15aa70 0x7fd69c0cac30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd6880097a0 tx=0x7fd68802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b1e58b71e2d5214 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.564+0000 7fd69b400640 1 -- 172.21.15.146:0/1963286497 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688037070 con 0x7fd69c15aa70 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.564+0000 7fd69b400640 1 -- 172.21.15.146:0/1963286497 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd688032440 con 0x7fd69c15aa70 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.564+0000 7fd69b400640 1 -- 172.21.15.146:0/1963286497 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688032810 con 0x7fd69c15aa70 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 -- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c15aa70 msgr2=0x7fd69c0cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c15aa70 0x7fd69c0cac30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd6880097a0 tx=0x7fd68802a210 comp rx=0 tx=0).stop 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 -- 172.21.15.146:0/1963286497 shutdown_connections 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c06c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1963286497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c15aa70 0x7fd69c0cac30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 -- 172.21.15.146:0/1963286497 >> 172.21.15.146:0/1963286497 conn(0x7fd69c067b30 msgr2=0x7fd69c067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:57.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 -- 172.21.15.146:0/1963286497 shutdown_connections 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.565+0000 7fd6a2600640 1 -- 172.21.15.146:0/1963286497 wait complete. 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.567+0000 7fd6a2600640 1 Processor -- start 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.567+0000 7fd6a2600640 1 -- start start 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a2600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd69c0c5eb0 0x7fd69c1e6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a2600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a2600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 0x7fd69c1e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a2600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd69c155020 con 0x7fd69c0c5eb0 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a2600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd69c06c370 con 0x7fd69c1e4db0 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a2600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd69c1551a0 con 0x7fd69c0c6290 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd69be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd69c0c5eb0 0x7fd69c1e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 0x7fd69c1e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd69be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56112/0 (socket says 172.21.15.146:56112) 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd69c0c5eb0 0x7fd69c1e6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33186/0 (socket says 172.21.15.146:33186) 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 0x7fd69c1e5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48014/0 (socket says 172.21.15.146:48014) 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.568+0000 7fd69be00640 1 -- 172.21.15.146:0/1750595585 learned_addr learned my addr 172.21.15.146:0/1750595585 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd69be00640 1 -- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 msgr2=0x7fd69c1e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd69be00640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 0x7fd69c1e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd69be00640 1 -- 172.21.15.146:0/1750595585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd69c0c5eb0 msgr2=0x7fd69c1e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd69be00640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd69c0c5eb0 0x7fd69c1e6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd69be00640 1 -- 172.21.15.146:0/1750595585 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69c14f440 con 0x7fd69c0c6290 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd6a1c00640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 0x7fd69c1e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.569+0000 7fd69be00640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c0ca9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd688000c00 tx=0x7fd6880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.570+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688037040 con 0x7fd69c0c6290 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.570+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd688042070 con 0x7fd69c0c6290 2024-09-16T01:49:57.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.570+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688030940 con 0x7fd69c0c6290 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.570+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69c14de90 con 0x7fd69c0c6290 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.570+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69c14ee60 con 0x7fd69c0c6290 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.571+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69c0d0de0 con 0x7fd69c0c6290 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.574+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd688030ae0 con 0x7fd69c0c6290 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.575+0000 7fd698c00640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd67c06c2b0 0x7fd67c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.575+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 5738+0+0 (secure 0 0 0) 0x7fd6880abde0 con 0x7fd69c0c6290 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.575+0000 7fd6a1200640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd67c06c2b0 0x7fd67c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:57.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.576+0000 7fd6a1200640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd67c06c2b0 0x7fd67c06e770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd690007fd0 tx=0x7fd690007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:57.703 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:57.695+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:57.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:49:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.577+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd69c0d0de0 con 0x7fd69c0c6290 2024-09-16T01:49:57.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.746+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7fea9c1594b0 con 0x7fea9c06d820 2024-09-16T01:49:57.798 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:57.797+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:57.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:57.828+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd69c0c56f0 con 0x7fd69c0c6290 2024-09-16T01:49:57.899 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:57.898+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:57.910 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:57.910 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:57.910 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:49:57.910 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:57.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:49:57.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:49:58.000 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:57.999+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:58.101 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:58.100+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:58.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.123+0000 7f7067e00640 1 Processor -- start 2024-09-16T01:49:58.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.123+0000 7f7067e00640 1 -- start start 2024-09-16T01:49:58.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.124+0000 7f7067e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70680d0490 0x7f70680c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.124+0000 7f7067e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7068154da0 0x7f70680c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.124+0000 7f7067e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f706815e220 con 0x7f70680d0870 2024-09-16T01:49:58.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.124+0000 7f7067e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f70680c6160 con 0x7f7068154da0 2024-09-16T01:49:58.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.124+0000 7f7067e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f706815e500 con 0x7f70680d0490 2024-09-16T01:49:58.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70680d0490 0x7f70680c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f70680d0870 0x7f7068154690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42920/0 (socket says 172.21.15.146:42920) 2024-09-16T01:49:58.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066000640 1 -- 172.21.15.146:0/861066512 learned_addr learned my addr 172.21.15.146:0/861066512 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7067400640 1 --2- 172.21.15.146:0/861066512 >> v2:172.21.15.146:3300/0 conn(0x7f7068154da0 0x7f70680c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066a00640 1 -- 172.21.15.146:0/861066512 >> v2:172.21.15.146:3300/0 conn(0x7f7068154da0 msgr2=0x7f70680c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066a00640 1 --2- 172.21.15.146:0/861066512 >> v2:172.21.15.146:3300/0 conn(0x7f7068154da0 0x7f70680c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066a00640 1 -- 172.21.15.146:0/861066512 >> v1:172.21.15.146:6789/0 conn(0x7f70680d0870 legacy=0x7f7068154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7066a00640 1 -- 172.21.15.146:0/861066512 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706815e7e0 con 0x7f70680d0490 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7065600640 1 -- 172.21.15.146:0/861066512 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845485430 0 0) 0x7f706815e220 con 0x7f70680d0870 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.125+0000 7f7067400640 1 --2- 172.21.15.146:0/861066512 >> v2:172.21.15.146:3300/0 conn(0x7f7068154da0 0x7f70680c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:49:58.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.126+0000 7f7066a00640 1 --2- 172.21.15.146:0/861066512 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70680d0490 0x7f70680c65f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f70580097a0 tx=0x7f705802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36edc4708c4115a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.126+0000 7f7065600640 1 -- 172.21.15.146:0/861066512 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058037070 con 0x7f70680d0490 2024-09-16T01:49:58.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.126+0000 7f7065600640 1 -- 172.21.15.146:0/861066512 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7058032440 con 0x7f70680d0490 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.126+0000 7f7065600640 1 -- 172.21.15.146:0/861066512 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058032810 con 0x7f70680d0490 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 -- 172.21.15.146:0/861066512 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70680d0490 msgr2=0x7f70680c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 --2- 172.21.15.146:0/861066512 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70680d0490 0x7f70680c65f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f70580097a0 tx=0x7f705802a230 comp rx=0 tx=0).stop 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 -- 172.21.15.146:0/861066512 shutdown_connections 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 --2- 172.21.15.146:0/861066512 >> v2:172.21.15.146:3300/0 conn(0x7f7068154da0 0x7f70680c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 --2- 172.21.15.146:0/861066512 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70680d0490 0x7f70680c65f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 -- 172.21.15.146:0/861066512 >> 172.21.15.146:0/861066512 conn(0x7f706806d6f0 msgr2=0x7f706806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 -- 172.21.15.146:0/861066512 shutdown_connections 2024-09-16T01:49:58.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.127+0000 7f7067e00640 1 -- 172.21.15.146:0/861066512 wait complete. 2024-09-16T01:49:58.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.129+0000 7f7067e00640 1 Processor -- start 2024-09-16T01:49:58.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.129+0000 7f7067e00640 1 -- start start 2024-09-16T01:49:58.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70680d0490 0x7f70681f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 0x7f70681f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7068154da0 0x7f70681f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f706815dec0 con 0x7f70680d0870 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f706815eda0 con 0x7f70680d0490 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70680c51f0 con 0x7f7068154da0 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7066000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 0x7f70681f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7066000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 0x7f70681f40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33196/0 (socket says 172.21.15.146:33196) 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7066000640 1 -- 172.21.15.146:0/2496527640 learned_addr learned my addr 172.21.15.146:0/2496527640 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70680d0490 0x7f70681f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.130+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7068154da0 0x7f70681f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066000640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7068154da0 msgr2=0x7f70681f19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7068154da0 0x7f70681f19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066000640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70680d0490 msgr2=0x7f70681f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70680d0490 0x7f70681f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066000640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70681e40f0 con 0x7f70680d0870 2024-09-16T01:49:58.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70680d0490 0x7f70681f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 0x7f70681f40d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f705400c4d0 tx=0x7f705400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054018070 con 0x7f70680d0870 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7054007500 con 0x7f70680d0870 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.131+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70680b5f10 con 0x7f70680d0870 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.132+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054004a20 con 0x7f70680d0870 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.132+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70681e5f80 con 0x7f70680d0870 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:49:58.134 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:49:58.135 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:49:58.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.133+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7068154690 con 0x7f70680d0870 2024-09-16T01:49:58.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.134+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f705401b070 con 0x7f70680d0870 2024-09-16T01:49:58.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.135+0000 7f705ea00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f704806c2b0 0x7f704806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.135+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5738+0+0 (secure 0 0 0) 0x7f705408a6b0 con 0x7f70680d0870 2024-09-16T01:49:58.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.135+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v32) ==== 54+0+16 (secure 0 0 0) 0x7f7054090170 con 0x7f70680d0870 2024-09-16T01:49:58.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.135+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f704806c2b0 0x7f704806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.135+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f70680b2ac0 0x7f70680ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.135+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70680cc3f0 con 0x7f70680b2ac0 2024-09-16T01:49:58.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.136+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f70680b2ac0 0x7f70680ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.136+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f704806c2b0 0x7f704806e770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7058034000 tx=0x7f705802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.136+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f70680b2ac0 0x7f70680ccf40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.137+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70680cc3f0 con 0x7f70680b2ac0 2024-09-16T01:49:58.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:49:58.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.154+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f70681e4790 con 0x7f70680b2ac0 2024-09-16T01:49:58.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.156+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70681e4790 con 0x7f70680b2ac0 2024-09-16T01:49:58.157 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:49:58.158 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:49:58.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.156+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f70681e66e0 0x7f706814df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.156+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f70680cc060 con 0x7f70681e66e0 2024-09-16T01:49:58.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.157+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f70681e66e0 0x7f706814df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.157+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f70681e66e0 0x7f706814df10 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.158+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70680cc060 con 0x7f70681e66e0 2024-09-16T01:49:58.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.179+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f70680cbd80 con 0x7f70681e66e0 2024-09-16T01:49:58.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.181+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70680cbd80 con 0x7f70681e66e0 2024-09-16T01:49:58.182 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:49:58.183 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:49:58.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.181+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f70681f08c0 0x7f70681f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.181+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f706814e810 con 0x7f70681f08c0 2024-09-16T01:49:58.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.182+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f70681f08c0 0x7f70681f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.182+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f70681f08c0 0x7f70681f0ce0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.183+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706814e810 con 0x7f70681f08c0 2024-09-16T01:49:58.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.197+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7068062900 con 0x7f70681f08c0 2024-09-16T01:49:58.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.199+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7068062900 con 0x7f70681f08c0 2024-09-16T01:49:58.200 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:49:58.201 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:49:58.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.199+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7068063380 0x7f706815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.199+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f70681f1220 con 0x7f7068063380 2024-09-16T01:49:58.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.199+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7068063380 0x7f706815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.200+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7068063380 0x7f706815ba90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.201+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70681f1220 con 0x7f7068063380 2024-09-16T01:49:58.202 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:58.202+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:58.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.217+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f70681f1220 con 0x7f7068063380 2024-09-16T01:49:58.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.219+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70681f1220 con 0x7f7068063380 2024-09-16T01:49:58.220 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:49:58.221 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:49:58.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.219+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f706802c640 0x7f706802caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.219+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f70681f1220 con 0x7f706802c640 2024-09-16T01:49:58.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.220+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f706802c640 0x7f706802caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.220+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f706802c640 0x7f706802caa0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.221+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70681f1220 con 0x7f706802c640 2024-09-16T01:49:58.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.235+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f70681f1220 con 0x7f706802c640 2024-09-16T01:49:58.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.238+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70681f1220 con 0x7f706802c640 2024-09-16T01:49:58.239 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:49:58.239 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:49:58.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.238+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f706802d4f0 0x7f7068206ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.238+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f706815c8a0 con 0x7f706802d4f0 2024-09-16T01:49:58.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.238+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f706802d4f0 0x7f7068206ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.239+0000 7f7066000640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f706802d4f0 0x7f7068206ec0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.239+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706815c8a0 con 0x7f706802d4f0 2024-09-16T01:49:58.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.257+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f70681f1220 con 0x7f706802d4f0 2024-09-16T01:49:58.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.259+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70681f1220 con 0x7f706802d4f0 2024-09-16T01:49:58.260 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:49:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:49:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.259+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f706802d950 0x7f7068209920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.259+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f706820c6f0 con 0x7f706802d950 2024-09-16T01:49:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.260+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f706802d950 0x7f7068209920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.260+0000 7f7067400640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f706802d950 0x7f7068209920 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.261+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706820c6f0 con 0x7f706802d950 2024-09-16T01:49:58.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.280+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7068209e60 con 0x7f706802d950 2024-09-16T01:49:58.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.282+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7068209e60 con 0x7f706802d950 2024-09-16T01:49:58.283 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:49:58.283 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:49:58.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.282+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f706820bce0 0x7f706820c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:49:58.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.282+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f706820fa60 con 0x7f706820bce0 2024-09-16T01:49:58.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.282+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f706820bce0 0x7f706820c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:49:58.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.283+0000 7f7066a00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f706820bce0 0x7f706820c100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:49:58.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.284+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706820fa60 con 0x7f706820bce0 2024-09-16T01:49:58.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.297+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f706820e8e0 con 0x7f706820bce0 2024-09-16T01:49:58.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f705ea00640 1 -- 172.21.15.146:0/2496527640 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706820e8e0 con 0x7f706820bce0 2024-09-16T01:49:58.301 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:49:58.303 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:49:58.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f70680b2ac0 msgr2=0x7f70680ccf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f70680b2ac0 0x7f70680ccf40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f70681e66e0 msgr2=0x7f706814df10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f70681e66e0 0x7f706814df10 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f70681f08c0 msgr2=0x7f70681f0ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f70681f08c0 0x7f70681f0ce0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7068063380 msgr2=0x7f706815ba90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7068063380 0x7f706815ba90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f706802c640 msgr2=0x7f706802caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f706802c640 0x7f706802caa0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f706802d4f0 msgr2=0x7f7068206ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f706802d4f0 0x7f7068206ec0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f706802d950 msgr2=0x7f7068209920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f706802d950 0x7f7068209920 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f706820bce0 msgr2=0x7f706820c100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f706820bce0 0x7f706820c100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067400640 1 -- 172.21.15.146:0/2496527640 reap_dead start 2024-09-16T01:49:58.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067400640 1 -- 172.21.15.146:0/2496527640 reap_dead start 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067400640 1 -- 172.21.15.146:0/2496527640 reap_dead start 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067400640 1 -- 172.21.15.146:0/2496527640 reap_dead start 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.300+0000 7f7067400640 1 -- 172.21.15.146:0/2496527640 reap_dead start 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f704806c2b0 msgr2=0x7f704806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f704806c2b0 0x7f704806e770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7058034000 tx=0x7f705802e000 comp rx=0 tx=0).stop 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 msgr2=0x7f70681f40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 0x7f70681f40d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f705400c4d0 tx=0x7f705400c990 comp rx=0 tx=0).stop 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 shutdown_connections 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f704806c2b0 0x7f704806e770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 --2- 172.21.15.146:0/2496527640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70680d0870 0x7f70681f40d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 >> 172.21.15.146:0/2496527640 conn(0x7f706806d6f0 msgr2=0x7f70680ce1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:58.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 shutdown_connections 2024-09-16T01:49:58.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.301+0000 7f7067e00640 1 -- 172.21.15.146:0/2496527640 wait complete. 2024-09-16T01:49:58.307 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:58.306+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:58.332 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:58.332 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:58.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.332+0000 7fd698c00640 1 -- 172.21.15.146:0/1750595585 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v33) ==== 91+0+0 (secure 0 0 0) 0x7fd68807a050 con 0x7fd69c0c6290 2024-09-16T01:49:58.333 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd67c06c2b0 msgr2=0x7fd67c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd67c06c2b0 0x7fd67c06e770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd690007fd0 tx=0x7fd690007de0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 msgr2=0x7fd69c0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c0ca9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd688000c00 tx=0x7fd6880046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 shutdown_connections 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd67c06c2b0 0x7fd67c06e770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd69c1e4db0 0x7fd69c1e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd69c0c6290 0x7fd69c0ca9d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 --2- 172.21.15.146:0/1750595585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd69c0c5eb0 0x7fd69c1e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.334+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 >> 172.21.15.146:0/1750595585 conn(0x7fd69c067b30 msgr2=0x7fd69c0ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:58.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.335+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 shutdown_connections 2024-09-16T01:49:58.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.335+0000 7fd6a2600640 1 -- 172.21.15.146:0/1750595585 wait complete. 2024-09-16T01:49:58.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.335+0000 7fea92000640 1 -- 172.21.15.146:0/69877953 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v33) ==== 133+0+0 (secure 0 0 0) 0x7fea8c0ab020 con 0x7fea9c06d820 2024-09-16T01:49:58.337 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T01:49:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.338+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fea7c06c2b0 msgr2=0x7fea7c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.338+0000 7fea9b200640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fea7c06c2b0 0x7fea7c06e770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0cbb40 tx=0x7fea88009190 comp rx=0 tx=0).stop 2024-09-16T01:49:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.338+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 msgr2=0x7fea9c158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:49:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.338+0000 7fea9b200640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 0x7fea9c158dd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fea8c031000 tx=0x7fea8c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 shutdown_connections 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fea7c06c2b0 0x7fea7c06e770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fea9c154ac0 0x7fea9c159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea9c0d0d00 0x7fea9c159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 --2- 172.21.15.146:0/69877953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea9c06d820 0x7fea9c158dd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 >> 172.21.15.146:0/69877953 conn(0x7fea9c06e400 msgr2=0x7fea9c14ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 shutdown_connections 2024-09-16T01:49:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:49:58.340+0000 7fea9b200640 1 -- 172.21.15.146:0/69877953 wait complete. 2024-09-16T01:49:58.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:49:58.408 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:58.407+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:58.509 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:58.508+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:58.528 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:58.528 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:58.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:49:58.610 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:58.609+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:58.704 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:58.705 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:58.711 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:58.711+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:58.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:49:58.812 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:58.811+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:58.913 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:58.912+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:58.930 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:58.930 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:58.944 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:49:59.015 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:59.014+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:59.062 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.062 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.071 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:49:59.116 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:49:59.115+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:49:59.175 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.175 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.184 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:49:59.216 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:49:59.216+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:49:59.318 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:59.317+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:59.328 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.329 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.338 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:49:59.418 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:49:59.418+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:49:59.442 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.442 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.451 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:49:59.520 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:49:59.519+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:49:59.554 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.554 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:49:59.621 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:49:59.620+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:49:59.665 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.665 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.673 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:49:59.722 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:49:59.721+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:49:59.778 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.779 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.787 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:49:59.828 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:49:59.827+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:49:59.923 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:49:59.922+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:49:59.931 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:49:59.931 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:49:59.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:00.024 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:00.023+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:00.090 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:00.090 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:00.100 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:00.125 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:00.125+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:00.204 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:00.205 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:00.214 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:00.226 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:00.226+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:00.316 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:00.317 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:00.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:00.325 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:00.327 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:00.326+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:00.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:50:00.428 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:00.427+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:00.432 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:00.432 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:00.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:00.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.493+0000 7f4c32e00640 1 Processor -- start 2024-09-16T01:50:00.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.494+0000 7f4c32e00640 1 -- start start 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c32e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4c2c154c10 0x7f4c2c15b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c32e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c2c0cca70 0x7f4c2c073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c32e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2c15e0b0 con 0x7f4c2c0cafa0 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c32e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2c15de60 con 0x7f4c2c0cca70 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c32e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2c15e390 con 0x7f4c2c154c10 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c2b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c2c0cca70 0x7f4c2c073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c2b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c2c0cca70 0x7f4c2c073340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33200/0 (socket says 172.21.15.146:33200) 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c2b400640 1 -- 172.21.15.146:0/2549504112 learned_addr learned my addr 172.21.15.146:0/2549504112 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c30e00640 1 --1- 172.21.15.146:0/2549504112 >> v1:172.21.15.161:6789/0 conn(0x7f4c2c0cafa0 0x7f4c2c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:51254/0 (socket says 172.21.15.146:51254) 2024-09-16T01:50:00.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.496+0000 7f4c2be00640 1 --2- 172.21.15.146:0/2549504112 >> v2:172.21.15.161:3300/0 conn(0x7f4c2c154c10 0x7f4c2c15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236417447 0 0) 0x7f4c2c15e0b0 con 0x7f4c2c0cafa0 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c10003520 con 0x7f4c2c0cafa0 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3950886845 0 0) 0x7f4c10003520 con 0x7f4c2c0cafa0 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 >> v2:172.21.15.161:3300/0 conn(0x7f4c2c154c10 msgr2=0x7f4c2c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 --2- 172.21.15.146:0/2549504112 >> v2:172.21.15.161:3300/0 conn(0x7f4c2c154c10 0x7f4c2c15b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 >> v2:172.21.15.146:3300/0 conn(0x7f4c2c0cca70 msgr2=0x7f4c2c073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 --2- 172.21.15.146:0/2549504112 >> v2:172.21.15.146:3300/0 conn(0x7f4c2c0cca70 0x7f4c2c073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c15e670 con 0x7f4c2c0cafa0 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f4c240019f0 con 0x7f4c2c0cafa0 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 >> v1:172.21.15.161:6789/0 conn(0x7f4c2c0cafa0 legacy=0x7f4c2c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c10003a00 0x7f4c10003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c100051c0 0x7f4c1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c15de60 con 0x7f4c10003a00 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c15e390 con 0x7f4c10004800 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4c1000c070 con 0x7f4c100051c0 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c2b400640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.497+0000 7f4c30e00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c100051c0 0x7f4c1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2be00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c10003a00 0x7f4c10003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2b400640 1 -- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c100051c0 msgr2=0x7f4c1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2b400640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c100051c0 0x7f4c1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2b400640 1 -- 172.21.15.146:0/2549504112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c10003a00 msgr2=0x7f4c10003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2b400640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c10003a00 0x7f4c10003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2b400640 1 -- 172.21.15.146:0/2549504112 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c10003520 con 0x7f4c10004800 2024-09-16T01:50:00.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c2be00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c10003a00 0x7f4c10003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:00.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.498+0000 7f4c30e00640 1 -- 172.21.15.146:0/2549504112 reap_dead start 2024-09-16T01:50:00.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.499+0000 7f4c2b400640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c10004c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009880 tx=0x7f4c10005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1e110b996a18c5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.499+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18037070 con 0x7f4c10004800 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.499+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1802abf0 con 0x7f4c10004800 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.499+0000 7f4c2aa00640 1 -- 172.21.15.146:0/2549504112 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800f040 con 0x7f4c10004800 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 -- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 msgr2=0x7f4c10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c10004c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009880 tx=0x7f4c10005830 comp rx=0 tx=0).stop 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 -- 172.21.15.146:0/2549504112 shutdown_connections 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 --2- 172.21.15.146:0/2549504112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c10004c80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 -- 172.21.15.146:0/2549504112 >> 172.21.15.146:0/2549504112 conn(0x7f4c2c06ce80 msgr2=0x7f4c2c06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 -- 172.21.15.146:0/2549504112 shutdown_connections 2024-09-16T01:50:00.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.500+0000 7f4c32e00640 1 -- 172.21.15.146:0/2549504112 wait complete. 2024-09-16T01:50:00.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.502+0000 7f4c32e00640 1 Processor -- start 2024-09-16T01:50:00.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.502+0000 7f4c32e00640 1 -- start start 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c32e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c2c0c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c32e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c2c0c8810 0x7f4c2c151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c32e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c2c15a2f0 0x7f4c2c0cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c32e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c154ff0 con 0x7f4c2c15a2f0 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c32e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c15ec30 con 0x7f4c10004800 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c32e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2c15b370 con 0x7f4c2c0c8810 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c2b400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c2c0c8810 0x7f4c2c151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c2be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c2c0c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c30e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c2c15a2f0 0x7f4c2c0cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c2be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c2c0c8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48052/0 (socket says 172.21.15.146:48052) 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.504+0000 7f4c2be00640 1 -- 172.21.15.146:0/1446681438 learned_addr learned my addr 172.21.15.146:0/1446681438 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:00.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2b400640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 msgr2=0x7f4c2c0c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c2c0c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2b400640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c2c15a2f0 msgr2=0x7f4c2c0cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c2c15a2f0 0x7f4c2c0cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2b400640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c1f41c0 con 0x7f4c2c0c8810 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c2c15a2f0 0x7f4c2c0cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c10004800 0x7f4c2c0c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.505+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c2c0c8810 0x7f4c2c151930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000c040 tx=0x7f4c18004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.506+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18037050 con 0x7f4c2c0c8810 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.506+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c18043070 con 0x7f4c2c0c8810 2024-09-16T01:50:00.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.506+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180044e0 con 0x7f4c2c0c8810 2024-09-16T01:50:00.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.507+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2c1efef0 con 0x7f4c2c0c8810 2024-09-16T01:50:00.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.507+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c2c1f0190 con 0x7f4c2c0c8810 2024-09-16T01:50:00.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.508+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4c2c15b6a0 con 0x7f4c2c0c8810 2024-09-16T01:50:00.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.510+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4c180655d0 con 0x7f4c2c0c8810 2024-09-16T01:50:00.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.510+0000 7f4c23e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4c0c06c2e0 0x7f4c0c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.511+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 5770+0+0 (secure 0 0 0) 0x7f4c180ac140 con 0x7f4c2c0c8810 2024-09-16T01:50:00.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.511+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v33) ==== 54+0+16 (secure 0 0 0) 0x7f4c180ac5d0 con 0x7f4c2c0c8810 2024-09-16T01:50:00.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.511+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4c0c06c2e0 0x7f4c0c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.511+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4c2c1f0970 0x7f4c2c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.511+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c1f3c80 con 0x7f4c2c1f0970 2024-09-16T01:50:00.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.511+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4c2c1f0970 0x7f4c2c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.512+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4c0c06c2e0 0x7f4c0c06e7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009e40 tx=0x7f4c1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.512+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4c2c1f0970 0x7f4c2c0cfa00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.513+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c240143a0 con 0x7f4c2c1f0970 2024-09-16T01:50:00.529 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:00.528+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:00.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.530+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c0c6ac0 con 0x7f4c2c1f0970 2024-09-16T01:50:00.532 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:00.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.531+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c0c6ac0 con 0x7f4c2c1f0970 2024-09-16T01:50:00.533 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:00.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.532+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4c2c158550 0x7f4c2c1589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.532+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c0cf210 con 0x7f4c2c158550 2024-09-16T01:50:00.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.532+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4c2c158550 0x7f4c2c1589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.533+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4c2c158550 0x7f4c2c1589b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.534+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c0cf210 con 0x7f4c2c158550 2024-09-16T01:50:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.545+0000 7fe435a00640 1 Processor -- start 2024-09-16T01:50:00.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.545+0000 7fe435a00640 1 -- start start 2024-09-16T01:50:00.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.546+0000 7fe435a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4300c75a0 0x7fe43015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.546+0000 7fe435a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe43006d140 con 0x7fe4300cc040 2024-09-16T01:50:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.546+0000 7fe435a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe43006d420 con 0x7fe4300c6b70 2024-09-16T01:50:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.546+0000 7fe435a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe43015b880 con 0x7fe4300c75a0 2024-09-16T01:50:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42fe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe4300c6b70 0x7fe4300cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:51256/0 (socket says 172.21.15.146:51256) 2024-09-16T01:50:00.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42fe00640 1 -- 172.21.15.146:0/1276249804 learned_addr learned my addr 172.21.15.146:0/1276249804 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42f400640 1 --2- 172.21.15.146:0/1276249804 >> v2:172.21.15.146:3300/0 conn(0x7fe4300c75a0 0x7fe43015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951372641 0 0) 0x7fe43006d420 con 0x7fe4300c6b70 2024-09-16T01:50:00.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe414003520 con 0x7fe4300c6b70 2024-09-16T01:50:00.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4010886659 0 0) 0x7fe414003520 con 0x7fe4300c6b70 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 >> v2:172.21.15.146:3300/0 conn(0x7fe4300c75a0 msgr2=0x7fe43015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 --2- 172.21.15.146:0/1276249804 >> v2:172.21.15.146:3300/0 conn(0x7fe4300c75a0 0x7fe43015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 >> v1:172.21.15.146:6789/0 conn(0x7fe4300cc040 legacy=0x7fe430165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.547+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43006d700 con 0x7fe4300c6b70 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fe420001670 con 0x7fe4300c6b70 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 >> v1:172.21.15.161:6789/0 conn(0x7fe4300c6b70 legacy=0x7fe4300cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4140039b0 0x7fe414003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe414004800 0x7fe414004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4140051c0 0x7fe41400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe43015b880 con 0x7fe414004800 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe41400c0f0 con 0x7fe4140039b0 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe41400c2f0 con 0x7fe4140051c0 2024-09-16T01:50:00.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42fe00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4140039b0 0x7fe414003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe435000640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4140051c0 0x7fe41400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42f400640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe414004800 0x7fe414004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42fe00640 1 -- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4140051c0 msgr2=0x7fe41400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42fe00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4140051c0 0x7fe41400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42fe00640 1 -- 172.21.15.146:0/1276249804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe414004800 msgr2=0x7fe414004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42fe00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe414004800 0x7fe414004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.548+0000 7fe42fe00640 1 -- 172.21.15.146:0/1276249804 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe414003520 con 0x7fe4140039b0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.549+0000 7fe435000640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4140051c0 0x7fe41400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.549+0000 7fe435000640 1 -- 172.21.15.146:0/1276249804 reap_dead start 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.550+0000 7fe42fe00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4140039b0 0x7fe414003db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe420014970 tx=0x7fe414005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f876de37a616b530 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.550+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420007c80 con 0x7fe4140039b0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.550+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe420037070 con 0x7fe4140039b0 2024-09-16T01:50:00.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.550+0000 7fe42ea00640 1 -- 172.21.15.146:0/1276249804 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420005d50 con 0x7fe4140039b0 2024-09-16T01:50:00.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.550+0000 7fe435a00640 1 -- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4140039b0 msgr2=0x7fe414003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.550+0000 7fe435a00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4140039b0 0x7fe414003db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe420014970 tx=0x7fe414005830 comp rx=0 tx=0).stop 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.551+0000 7fe435a00640 1 -- 172.21.15.146:0/1276249804 shutdown_connections 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.551+0000 7fe435a00640 1 --2- 172.21.15.146:0/1276249804 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4140039b0 0x7fe414003db0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.551+0000 7fe435a00640 1 -- 172.21.15.146:0/1276249804 >> 172.21.15.146:0/1276249804 conn(0x7fe4300ccc20 msgr2=0x7fe4300cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.551+0000 7fe435a00640 1 -- 172.21.15.146:0/1276249804 shutdown_connections 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.551+0000 7fe435a00640 1 -- 172.21.15.146:0/1276249804 wait complete. 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.552+0000 7fe435a00640 1 Processor -- start 2024-09-16T01:50:00.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.553+0000 7fe435a00640 1 -- start start 2024-09-16T01:50:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 0x7fe4301f0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe43015e600 0x7fe4300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4301ec4d0 0x7fe43015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4300cb050 con 0x7fe4140039b0 2024-09-16T01:50:00.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4300cab20 con 0x7fe43015e600 2024-09-16T01:50:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe43006cd20 con 0x7fe4301ec4d0 2024-09-16T01:50:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 0x7fe4301f0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe435000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4301ec4d0 0x7fe43015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 0x7fe4301f0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33244/0 (socket says 172.21.15.146:33244) 2024-09-16T01:50:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe42fe00640 1 -- 172.21.15.146:0/175791503 learned_addr learned my addr 172.21.15.146:0/175791503 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:00.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.554+0000 7fe42f400640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe43015e600 0x7fe4300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435000640 1 -- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe43015e600 msgr2=0x7fe4300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435000640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe43015e600 0x7fe4300cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435000640 1 -- 172.21.15.146:0/175791503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 msgr2=0x7fe4301f0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435000640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 0x7fe4301f0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435000640 1 -- 172.21.15.146:0/175791503 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43015de00 con 0x7fe4301ec4d0 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe42fe00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 0x7fe4301f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435000640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4301ec4d0 0x7fe43015cea0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe430075ec0 tx=0x7fe42800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428001a50 con 0x7fe4301ec4d0 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe428006b50 con 0x7fe4301ec4d0 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428006d00 con 0x7fe4301ec4d0 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.555+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe43006f690 con 0x7fe4301ec4d0 2024-09-16T01:50:00.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.556+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43006f0d0 con 0x7fe4301ec4d0 2024-09-16T01:50:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.557+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe43015dfd0 con 0x7fe4301ec4d0 2024-09-16T01:50:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.558+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe42801c020 con 0x7fe4301ec4d0 2024-09-16T01:50:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.558+0000 7fe427e00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe41006c2b0 0x7fe41006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.559+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 5770+0+0 (secure 0 0 0) 0x7fe42808afa0 con 0x7fe4301ec4d0 2024-09-16T01:50:00.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.559+0000 7fe42fe00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe41006c2b0 0x7fe41006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.559+0000 7fe42fe00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe41006c2b0 0x7fe41006e770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe420007c10 tx=0x7fe420009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.563+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe428058fe0 con 0x7fe4301ec4d0 2024-09-16T01:50:00.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.565+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c158ef0 con 0x7f4c2c158550 2024-09-16T01:50:00.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.567+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c158ef0 con 0x7f4c2c158550 2024-09-16T01:50:00.568 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:00.568 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:00.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.567+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4c2c062b40 0x7f4c2c062f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.567+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c158ef0 con 0x7f4c2c062b40 2024-09-16T01:50:00.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.568+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4c2c062b40 0x7f4c2c062f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.569+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4c2c062b40 0x7f4c2c062f70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.570+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c158ef0 con 0x7f4c2c062b40 2024-09-16T01:50:00.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.588+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c158ef0 con 0x7f4c2c062b40 2024-09-16T01:50:00.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.591+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c158ef0 con 0x7f4c2c062b40 2024-09-16T01:50:00.592 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:00.592 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:00.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.591+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4c2c15c320 0x7f4c2c02c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.591+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c158ef0 con 0x7f4c2c15c320 2024-09-16T01:50:00.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.592+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4c2c15c320 0x7f4c2c02c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.592+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4c2c15c320 0x7f4c2c02c640 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.594+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c158ef0 con 0x7f4c2c15c320 2024-09-16T01:50:00.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.621+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c158ef0 con 0x7f4c2c15c320 2024-09-16T01:50:00.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.624+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c158ef0 con 0x7f4c2c15c320 2024-09-16T01:50:00.625 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:00.626 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:00.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.625+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4c2c02d090 0x7f4c2c02d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.625+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c02da10 con 0x7f4c2c02d090 2024-09-16T01:50:00.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.625+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4c2c02d090 0x7f4c2c02d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.626+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4c2c02d090 0x7f4c2c02d4d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.626+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c02da10 con 0x7f4c2c02d090 2024-09-16T01:50:00.629 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:00.629+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:00.641 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:00.641 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:00.641 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:00.641 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:00.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.640+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c02da10 con 0x7f4c2c02d090 2024-09-16T01:50:00.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.642+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c02da10 con 0x7f4c2c02d090 2024-09-16T01:50:00.643 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:00.644 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:00.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.643+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4c2c204b50 0x7f4c2c206f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.643+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c02da10 con 0x7f4c2c204b50 2024-09-16T01:50:00.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.643+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4c2c204b50 0x7f4c2c206f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.644+0000 7f4c2b400640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4c2c204b50 0x7f4c2c206f40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.644+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c02da10 con 0x7f4c2c204b50 2024-09-16T01:50:00.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:00.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.661+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c02da10 con 0x7f4c2c204b50 2024-09-16T01:50:00.666 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.663+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c02da10 con 0x7f4c2c204b50 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.663+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4c2c207990 0x7f4c2c209d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.663+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c20cbd0 con 0x7f4c2c207990 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.664+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4c2c207990 0x7f4c2c209d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.664+0000 7f4c30e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4c2c207990 0x7f4c2c209d80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.665+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c20cbd0 con 0x7f4c2c207990 2024-09-16T01:50:00.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.679+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c20a2c0 con 0x7f4c2c207990 2024-09-16T01:50:00.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.681+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c20a2c0 con 0x7f4c2c207990 2024-09-16T01:50:00.682 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:00.682 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:00.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.681+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4c2c20a9e0 0x7f4c2c20ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:00.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.681+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4c2c20cc30 con 0x7f4c2c20a9e0 2024-09-16T01:50:00.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.681+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4c2c20a9e0 0x7f4c2c20ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:00.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.682+0000 7f4c2be00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4c2c20a9e0 0x7f4c2c20ae00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:00.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.683+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c2c20cc30 con 0x7f4c2c20a9e0 2024-09-16T01:50:00.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.699+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f4c2c20cc30 con 0x7f4c2c20a9e0 2024-09-16T01:50:00.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.700+0000 7f4c23e00640 1 -- 172.21.15.146:0/1446681438 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4c2c20cc30 con 0x7f4c2c20a9e0 2024-09-16T01:50:00.702 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:00.702 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4c2c1f0970 msgr2=0x7f4c2c0cfa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4c2c1f0970 0x7f4c2c0cfa00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4c2c158550 msgr2=0x7f4c2c1589b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4c2c158550 0x7f4c2c1589b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4c2c062b40 msgr2=0x7f4c2c062f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4c2c062b40 0x7f4c2c062f70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4c2c15c320 msgr2=0x7f4c2c02c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4c2c15c320 0x7f4c2c02c640 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4c2c02d090 msgr2=0x7f4c2c02d4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4c2c02d090 0x7f4c2c02d4d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4c2c204b50 msgr2=0x7f4c2c206f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4c2c204b50 0x7f4c2c206f40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4c2c207990 msgr2=0x7f4c2c209d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4c2c207990 0x7f4c2c209d80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4c2c20a9e0 msgr2=0x7f4c2c20ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4c2c20a9e0 0x7f4c2c20ae00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c30e00640 1 -- 172.21.15.146:0/1446681438 reap_dead start 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c30e00640 1 -- 172.21.15.146:0/1446681438 reap_dead start 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c30e00640 1 -- 172.21.15.146:0/1446681438 reap_dead start 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c30e00640 1 -- 172.21.15.146:0/1446681438 reap_dead start 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c30e00640 1 -- 172.21.15.146:0/1446681438 reap_dead start 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4c0c06c2e0 msgr2=0x7f4c0c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4c0c06c2e0 0x7f4c0c06e7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009e40 tx=0x7f4c1c008040 comp rx=0 tx=0).stop 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c2c0c8810 msgr2=0x7f4c2c151930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c2c0c8810 0x7f4c2c151930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000c040 tx=0x7f4c18004730 comp rx=0 tx=0).stop 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 shutdown_connections 2024-09-16T01:50:00.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4c0c06c2e0 0x7f4c0c06e7a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:00.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 --2- 172.21.15.146:0/1446681438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c2c0c8810 0x7f4c2c151930 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000c040 tx=0x7f4c18004730 comp rx=0 tx=0).stop 2024-09-16T01:50:00.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.701+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 >> 172.21.15.146:0/1446681438 conn(0x7f4c2c06ce80 msgr2=0x7f4c2c073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:00.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.702+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 shutdown_connections 2024-09-16T01:50:00.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.702+0000 7f4c32e00640 1 -- 172.21.15.146:0/1446681438 wait complete. 2024-09-16T01:50:00.730 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:00.729+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:00.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:00.802+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fe4300ca280 con 0x7fe4301ec4d0 2024-09-16T01:50:00.830 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:00.830+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:00.849 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:00.849 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:00.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:00.947 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:00.946+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:01.032 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:01.031+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.242466575999998, 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.070 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:50:01.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:50:01.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.242360527000002, 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.076 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:50:01.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:50:01.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.242305901000002, 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:01.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:50:01.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.242237568, 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:50:01.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:50:01.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:01.093 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.207982364999999, 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.099 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.207799166999997, 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:50:01.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.106 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.207745158999998, 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:50:01.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:50:01.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.207712719, 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:50:01.112 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:50:01.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.117 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.204344718000002, 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:50:01.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.204274042000002, 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:50:01.122 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:50:01.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:50:01.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.204237667000001, 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:01.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:50:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:01.131 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.203609923999998, 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:01.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:50:01.135 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.203555317000003, 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:50:01.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:50:01.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:50:01.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:01.140 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131416909000002, 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:01.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:50:01.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:50:01.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131358597000002, 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.146 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:01.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.151 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131284522000001, 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:50:01.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131225901000001, 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:50:01.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:50:01.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.162 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131203966999998, 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:50:01.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:50:01.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131173607000001, 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:50:01.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:50:01.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.131147028000001, 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:50:01.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:50:01.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.180 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:01.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:01.182 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:01.181+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:01.283 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:01.281+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:01.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.349+0000 7fe427e00640 1 -- 172.21.15.146:0/175791503 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v34) ==== 101+0+0 (secure 0 0 0) 0x7fe42805d2e0 con 0x7fe4301ec4d0 2024-09-16T01:50:01.350 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.351+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe41006c2b0 msgr2=0x7fe41006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.351+0000 7fe435a00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe41006c2b0 0x7fe41006e770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe420007c10 tx=0x7fe420009030 comp rx=0 tx=0).stop 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.351+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4301ec4d0 msgr2=0x7fe43015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.351+0000 7fe435a00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4301ec4d0 0x7fe43015cea0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe430075ec0 tx=0x7fe42800ff70 comp rx=0 tx=0).stop 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 shutdown_connections 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe41006c2b0 0x7fe41006e770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4301ec4d0 0x7fe43015cea0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe43015e600 0x7fe4300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 --2- 172.21.15.146:0/175791503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4140039b0 0x7fe4301f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 >> 172.21.15.146:0/175791503 conn(0x7fe4300ccc20 msgr2=0x7fe43015a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:01.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 shutdown_connections 2024-09-16T01:50:01.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:01.352+0000 7fe435a00640 1 -- 172.21.15.146:0/175791503 wait complete. 2024-09-16T01:50:01.383 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:01.382+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:01.404 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:01.405 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:01.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:01.501 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:01.500+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:01.585 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:01.583+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:01.642 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:01.643 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:01.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:01.697 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:01.696+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:01.786 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:01.786+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:01.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.096867158999999, 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:01.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:01.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.876 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.665313427000001, 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:01.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.645389205000001, 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:01.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.645369286999994, 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.891 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:01.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:01.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.645308702999998, 2024-09-16T01:50:01.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:01.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:01.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:01.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.399023182999997, 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.906 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:01.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.398872668999999, 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:01.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:01.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:01.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:01.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.398853101, 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.920 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:01.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:01.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:01.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828696887000007, 2024-09-16T01:50:01.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:01.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:01.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:01.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828529068999998, 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:01.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:01.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:01.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:01.942 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828510086999998, 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:01.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:01.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:01.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.950 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828490455999997, 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:01.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:01.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:01.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828356549999995, 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:01.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:01.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.966 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828345012, 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.967 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828326601000001, 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:01.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:01.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:01.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:01.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.828211930999998, 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.981 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:01.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:01.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.817430203000001, 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:01.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:01.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:01.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.817313545000005, 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:01.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:01.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:02.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.002 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.001780306000001, 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:02.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:02.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.007 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.001718967999999, 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.008 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:02.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:02.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.013 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:02.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:02.018 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:02.016+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:02.118 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:02.116+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:02.219 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:02.219 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:02.219 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:02.219 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:02.220 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:02.218+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:02.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:02.320 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:02.319+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:02.421 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:02.420+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:02.437 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:02.438 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:02.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:02.522 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:02.521+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:02.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.789699642999999, 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:02.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:02.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.789584822000002, 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.789542951999998, 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:02.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:02.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:02.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.789498672999997, 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:02.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.789469795999999, 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:02.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.646 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.785762589000001, 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:02.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:02.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:02.651 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.785708608, 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:02.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:02.655 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.750293384000003, 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:02.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:02.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.747233235000003, 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.660 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:02.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:02.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.746373787000003, 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.665 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:02.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:02.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.746319894000003, 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.671 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:02.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:02.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.740295920999998, 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:02.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.681 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.740245793, 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:02.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:02.686 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.740169459999997, 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:02.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.740138105, 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.691 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.702052649999999, 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:02.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.701261119000002, 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.700 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:02.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:02.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.677179293999998, 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:02.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:02.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:02.710 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.677107382000003, 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:02.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.346464982000001, 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:02.716 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:02.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:02.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:02.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:02.722 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:02.723 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:02.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:02.724 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:02.722+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:02.725 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:02.825 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:02.823+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:02.925 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:02.924+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:02.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.934+0000 7f2605a00640 1 Processor -- start 2024-09-16T01:50:02.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.934+0000 7f2605a00640 1 -- start start 2024-09-16T01:50:02.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.935+0000 7f2605a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2600152c30 0x7f26000c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.935+0000 7f2605a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f26000c6ab0 0x7f26000c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.935+0000 7f2605a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26000c4240 con 0x7f26000c6e90 2024-09-16T01:50:02.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.935+0000 7f2605a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f26000c4da0 con 0x7f26000c6ab0 2024-09-16T01:50:02.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.935+0000 7f2605a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f26000c4500 con 0x7f2600152c30 2024-09-16T01:50:02.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25ff400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f26000c6e90 0x7f26000c7ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52338/0 (socket says 172.21.15.146:52338) 2024-09-16T01:50:02.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25ff400640 1 -- 172.21.15.146:0/2280041119 learned_addr learned my addr 172.21.15.146:0/2280041119 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25fe000640 1 --2- 172.21.15.146:0/2280041119 >> v2:172.21.15.146:3300/0 conn(0x7f26000c6ab0 0x7f26000c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25fea00640 1 --2- 172.21.15.146:0/2280041119 >> v2:172.21.15.161:3300/0 conn(0x7f2600152c30 0x7f26000c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25fea00640 1 -- 172.21.15.146:0/2280041119 >> v2:172.21.15.146:3300/0 conn(0x7f26000c6ab0 msgr2=0x7f26000c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25fea00640 1 --2- 172.21.15.146:0/2280041119 >> v2:172.21.15.146:3300/0 conn(0x7f26000c6ab0 0x7f26000c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25fea00640 1 -- 172.21.15.146:0/2280041119 >> v1:172.21.15.146:6789/0 conn(0x7f26000c6e90 legacy=0x7f26000c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.936+0000 7f25fea00640 1 -- 172.21.15.146:0/2280041119 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26000c47e0 con 0x7f2600152c30 2024-09-16T01:50:02.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.937+0000 7f25fea00640 1 --2- 172.21.15.146:0/2280041119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26000c36b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2600071690 tx=0x7f25f002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ced941ccf77c50ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:02.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.938+0000 7f25fd600640 1 -- 172.21.15.146:0/2280041119 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0007740 con 0x7f2600152c30 2024-09-16T01:50:02.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.938+0000 7f25fd600640 1 -- 172.21.15.146:0/2280041119 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f00078e0 con 0x7f2600152c30 2024-09-16T01:50:02.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.938+0000 7f25fd600640 1 -- 172.21.15.146:0/2280041119 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0007cb0 con 0x7f2600152c30 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.938+0000 7f2605a00640 1 -- 172.21.15.146:0/2280041119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 msgr2=0x7f26000c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.938+0000 7f2605a00640 1 --2- 172.21.15.146:0/2280041119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26000c36b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2600071690 tx=0x7f25f002a680 comp rx=0 tx=0).stop 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.939+0000 7f2605a00640 1 -- 172.21.15.146:0/2280041119 shutdown_connections 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.939+0000 7f2605a00640 1 --2- 172.21.15.146:0/2280041119 >> v2:172.21.15.146:3300/0 conn(0x7f26000c6ab0 0x7f26000c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.939+0000 7f2605a00640 1 --2- 172.21.15.146:0/2280041119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26000c36b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.939+0000 7f2605a00640 1 -- 172.21.15.146:0/2280041119 >> 172.21.15.146:0/2280041119 conn(0x7f260006bc10 msgr2=0x7f260006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.939+0000 7f2605a00640 1 -- 172.21.15.146:0/2280041119 shutdown_connections 2024-09-16T01:50:02.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.939+0000 7f2605a00640 1 -- 172.21.15.146:0/2280041119 wait complete. 2024-09-16T01:50:02.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.940+0000 7f2605a00640 1 Processor -- start 2024-09-16T01:50:02.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.940+0000 7f2605a00640 1 -- start start 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.941+0000 7f2605a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26000c6ab0 0x7f26001f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.941+0000 7f2605a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26000c6e90 0x7f260014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.941+0000 7f2605a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26001e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.941+0000 7f2605a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f26000c8700 con 0x7f26000c6e90 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.941+0000 7f2605a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26000c6630 con 0x7f2600152c30 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.941+0000 7f2605a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f26000c8880 con 0x7f26000c6ab0 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26000c6e90 0x7f260014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26000c6ab0 0x7f26001f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26000c6e90 0x7f260014f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55728/0 (socket says 172.21.15.146:55728) 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 -- 172.21.15.146:0/3878972151 learned_addr learned my addr 172.21.15.146:0/3878972151 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26001e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26000c6ab0 msgr2=0x7f26001f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26000c6ab0 0x7f26001f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 msgr2=0x7f26001e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26001e64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25fe000640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26001f4e90 con 0x7f26000c6e90 2024-09-16T01:50:02.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.942+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2600152c30 0x7f26001e64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.943+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26000c6e90 0x7f260014f920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00e840 tx=0x7f25ec00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.943+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec017070 con 0x7f26000c6e90 2024-09-16T01:50:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.943+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ec00c940 con 0x7f26000c6e90 2024-09-16T01:50:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.943+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec00cd10 con 0x7f26000c6e90 2024-09-16T01:50:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.943+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2600071c60 con 0x7f26000c6e90 2024-09-16T01:50:02.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.943+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26001f4150 con 0x7f26000c6e90 2024-09-16T01:50:02.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.945+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f26000c36b0 con 0x7f26000c6e90 2024-09-16T01:50:02.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.946+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f25ec01a070 con 0x7f26000c6e90 2024-09-16T01:50:02.947 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:02.947 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:02.947 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:02.947 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:02.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.947+0000 7f25f6a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f25e006c2e0 0x7f25e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.947+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5770+0+0 (secure 0 0 0) 0x7f25ec08b270 con 0x7f26000c6e90 2024-09-16T01:50:02.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.947+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v34) ==== 54+0+16 (secure 0 0 0) 0x7f25ec08d810 con 0x7f26000c6e90 2024-09-16T01:50:02.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.947+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f25e006c2e0 0x7f25e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.947+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f26001f4350 0x7f2600148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.947+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2600071a10 con 0x7f26001f4350 2024-09-16T01:50:02.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.948+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f26001f4350 0x7f2600148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.948+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f25e006c2e0 0x7f25e006e7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f25f0034000 tx=0x7f25f002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:02.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.948+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f26001f4350 0x7f2600148920 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:02.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.949+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2600071a10 con 0x7f26001f4350 2024-09-16T01:50:02.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:02.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.966+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f26000c7530 con 0x7f26001f4350 2024-09-16T01:50:02.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.968+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f26000c7530 con 0x7f26001f4350 2024-09-16T01:50:02.969 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:02.970 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:02.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.968+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f26000724b0 0x7f26001e3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.968+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2600072180 con 0x7f26000724b0 2024-09-16T01:50:02.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.969+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f26000724b0 0x7f26001e3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.969+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f26000724b0 0x7f26001e3c80 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:02.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.971+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2600072180 con 0x7f26000724b0 2024-09-16T01:50:02.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.988+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f260014fe60 con 0x7f26000724b0 2024-09-16T01:50:02.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.990+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f260014fe60 con 0x7f26000724b0 2024-09-16T01:50:02.991 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:02.991 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:02.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.990+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f26000ca490 0x7f26001e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:02.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.990+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f260006ec10 con 0x7f26000ca490 2024-09-16T01:50:02.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.990+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f26000ca490 0x7f26001e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:02.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.991+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f26000ca490 0x7f26001e46d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:02.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:02.992+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f260006ec10 con 0x7f26000ca490 2024-09-16T01:50:03.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.010+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2600062880 con 0x7f26000ca490 2024-09-16T01:50:03.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.012+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2600062880 con 0x7f26000ca490 2024-09-16T01:50:03.013 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:03.014 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:03.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.012+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2600063360 0x7f26001e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.012+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f260006c2d0 con 0x7f2600063360 2024-09-16T01:50:03.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.013+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2600063360 0x7f26001e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.013+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2600063360 0x7f26001e50f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.014+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f260006c2d0 con 0x7f2600063360 2024-09-16T01:50:03.026 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:03.024+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:03.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.032+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f26001e5900 con 0x7f2600063360 2024-09-16T01:50:03.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.034+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f26001e5900 con 0x7f2600063360 2024-09-16T01:50:03.035 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:03.036 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:03.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.035+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f260002c640 0x7f260002caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.035+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f26001e6020 con 0x7f260002c640 2024-09-16T01:50:03.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.035+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f260002c640 0x7f260002caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.036+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f260002c640 0x7f260002caa0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.036+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f26001e6020 con 0x7f260002c640 2024-09-16T01:50:03.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.050+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f26001e6020 con 0x7f260002c640 2024-09-16T01:50:03.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.053+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f26001e6020 con 0x7f260002c640 2024-09-16T01:50:03.054 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:03.054 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:03.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.053+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f260002d660 0x7f2600207340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.053+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f260002dae0 con 0x7f260002d660 2024-09-16T01:50:03.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.053+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f260002d660 0x7f2600207340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.054+0000 7f25fe000640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f260002d660 0x7f2600207340 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.055+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f260002dae0 con 0x7f260002d660 2024-09-16T01:50:03.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.073+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f260002dae0 con 0x7f260002d660 2024-09-16T01:50:03.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.075+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f260002dae0 con 0x7f260002d660 2024-09-16T01:50:03.076 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:03.076 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:03.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.075+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2600207d90 0x7f260020a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.075+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f260020cfd0 con 0x7f2600207d90 2024-09-16T01:50:03.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.076+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2600207d90 0x7f260020a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.076+0000 7f25ff400640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2600207d90 0x7f260020a180 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.077+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f260020cfd0 con 0x7f2600207d90 2024-09-16T01:50:03.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.096+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f260020a740 con 0x7f2600207d90 2024-09-16T01:50:03.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.098+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f260020a740 con 0x7f2600207d90 2024-09-16T01:50:03.099 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:03.100 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:03.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.098+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f260020c5c0 0x7f260020ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.098+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2600210370 con 0x7f260020c5c0 2024-09-16T01:50:03.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.099+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f260020c5c0 0x7f260020ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.099+0000 7f25fea00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f260020c5c0 0x7f260020ca00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.100+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2600210370 con 0x7f260020c5c0 2024-09-16T01:50:03.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.114+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f260020f1f0 con 0x7f260020c5c0 2024-09-16T01:50:03.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f25f6a00640 1 -- 172.21.15.146:0/3878972151 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f260020f1f0 con 0x7f260020c5c0 2024-09-16T01:50:03.117 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f26001f4350 msgr2=0x7f2600148920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f26001f4350 0x7f2600148920 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f26000724b0 msgr2=0x7f26001e3c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f26000724b0 0x7f26001e3c80 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f26000ca490 msgr2=0x7f26001e46d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f26000ca490 0x7f26001e46d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2600063360 msgr2=0x7f26001e50f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2600063360 0x7f26001e50f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f260002c640 msgr2=0x7f260002caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f260002c640 0x7f260002caa0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f260002d660 msgr2=0x7f2600207340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f260002d660 0x7f2600207340 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2600207d90 msgr2=0x7f260020a180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2600207d90 0x7f260020a180 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f260020c5c0 msgr2=0x7f260020ca00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f260020c5c0 0x7f260020ca00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f25e006c2e0 msgr2=0x7f25e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f25e006c2e0 0x7f25e006e7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f25f0034000 tx=0x7f25f002f000 comp rx=0 tx=0).stop 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.116+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26000c6e90 msgr2=0x7f260014f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f2605a00640 1 --2- 172.21.15.146:0/3878972151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26000c6e90 0x7f260014f920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00e840 tx=0x7f25ec00ed00 comp rx=0 tx=0).stop 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f25ff400640 1 -- 172.21.15.146:0/3878972151 reap_dead start 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f25ff400640 1 -- 172.21.15.146:0/3878972151 reap_dead start 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f25ff400640 1 -- 172.21.15.146:0/3878972151 reap_dead start 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f25ff400640 1 -- 172.21.15.146:0/3878972151 reap_dead start 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f25ff400640 1 -- 172.21.15.146:0/3878972151 reap_dead start 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 shutdown_connections 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 >> 172.21.15.146:0/3878972151 conn(0x7f260006bc10 msgr2=0x7f260014d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 shutdown_connections 2024-09-16T01:50:03.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.117+0000 7f2605a00640 1 -- 172.21.15.146:0/3878972151 wait complete. 2024-09-16T01:50:03.127 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:03.126+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:03.147 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:03.147 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:03.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:03.227 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:03.225+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:03.328 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:03.327+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:03.365 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:03.365 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:03.365 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-16T01:50:03.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-09-16T01:50:03.366 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:03.366 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:03.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:03.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:50:03.428 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:03.426+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:03.494 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:03.494 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:03.503 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:03.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.529+0000 7f855b400640 1 Processor -- start 2024-09-16T01:50:03.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.529+0000 7f855b400640 1 -- start start 2024-09-16T01:50:03.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.531+0000 7f855b400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85540cb2f0 0x7f855415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.531+0000 7f855b400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855415b1d0 con 0x7f85540cc0c0 2024-09-16T01:50:03.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f855b400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855415b4b0 con 0x7f85540cc4a0 2024-09-16T01:50:03.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f855b400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f85540cbd00 con 0x7f85540cb2f0 2024-09-16T01:50:03.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8558a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f85540cc0c0 0x7f855415e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52342/0 (socket says 172.21.15.146:52342) 2024-09-16T01:50:03.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8558a00640 1 -- 172.21.15.146:0/2428898972 learned_addr learned my addr 172.21.15.146:0/2428898972 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553e00640 1 --2- 172.21.15.146:0/2428898972 >> v2:172.21.15.161:3300/0 conn(0x7f85540cb2f0 0x7f855415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1783489583 0 0) 0x7f855415b1d0 con 0x7f85540cc0c0 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f853c003520 con 0x7f85540cc0c0 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876959285 0 0) 0x7f855415b4b0 con 0x7f85540cc4a0 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855415b1d0 con 0x7f85540cc4a0 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553e00640 1 -- 172.21.15.146:0/2428898972 >> v1:172.21.15.161:6789/0 conn(0x7f85540cc4a0 legacy=0x7f8554165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553e00640 1 -- 172.21.15.146:0/2428898972 >> v1:172.21.15.146:6789/0 conn(0x7f85540cc0c0 legacy=0x7f855415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.532+0000 7f8553e00640 1 -- 172.21.15.146:0/2428898972 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855415b790 con 0x7f85540cb2f0 2024-09-16T01:50:03.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.533+0000 7f8553e00640 1 --2- 172.21.15.146:0/2428898972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cb2f0 0x7f855415aab0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85400097a0 tx=0x7f854002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9b56b70e480906 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.534+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540037070 con 0x7f85540cb2f0 2024-09-16T01:50:03.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.534+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8540032440 con 0x7f85540cb2f0 2024-09-16T01:50:03.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.534+0000 7f8553400640 1 -- 172.21.15.146:0/2428898972 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540032810 con 0x7f85540cb2f0 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 -- 172.21.15.146:0/2428898972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cb2f0 msgr2=0x7f855415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 --2- 172.21.15.146:0/2428898972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cb2f0 0x7f855415aab0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85400097a0 tx=0x7f854002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 -- 172.21.15.146:0/2428898972 shutdown_connections 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 --2- 172.21.15.146:0/2428898972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cb2f0 0x7f855415aab0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 -- 172.21.15.146:0/2428898972 >> 172.21.15.146:0/2428898972 conn(0x7f855406d560 msgr2=0x7f855406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 -- 172.21.15.146:0/2428898972 shutdown_connections 2024-09-16T01:50:03.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.535+0000 7f855b400640 1 -- 172.21.15.146:0/2428898972 wait complete. 2024-09-16T01:50:03.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.538+0000 7f855b400640 1 Processor -- start 2024-09-16T01:50:03.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.538+0000 7f855b400640 1 -- start start 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.539+0000 7f855b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540cb2f0 0x7f85540b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.539+0000 7f855b400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85540cc0c0 0x7f85540c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.539+0000 7f855b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cc4a0 0x7f85540c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.539+0000 7f855b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f855415e540 con 0x7f85540cb2f0 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.539+0000 7f855b400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f855415ba70 con 0x7f85540cc4a0 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.539+0000 7f855b400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f85540cc8b0 con 0x7f85540cc0c0 2024-09-16T01:50:03.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8558a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540cb2f0 0x7f85540b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8553e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85540cc0c0 0x7f85540c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8558a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540cb2f0 0x7f85540b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55742/0 (socket says 172.21.15.146:55742) 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8558a00640 1 -- 172.21.15.146:0/1634646883 learned_addr learned my addr 172.21.15.146:0/1634646883 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8559400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cc4a0 0x7f85540c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8553e00640 1 -- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cc4a0 msgr2=0x7f85540c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8553e00640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cc4a0 0x7f85540c9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8553e00640 1 -- 172.21.15.146:0/1634646883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540cb2f0 msgr2=0x7f85540b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8553e00640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540cb2f0 0x7f85540b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.540+0000 7f8553e00640 1 -- 172.21.15.146:0/1634646883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85541f0b40 con 0x7f85540cc0c0 2024-09-16T01:50:03.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.541+0000 7f8553e00640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85540cc0c0 0x7f85540c8ef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8540031000 tx=0x7f85400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.541+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540041030 con 0x7f85540cc0c0 2024-09-16T01:50:03.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.541+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f854003d4e0 con 0x7f85540cc0c0 2024-09-16T01:50:03.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.541+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85541f0e00 con 0x7f85540cc0c0 2024-09-16T01:50:03.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.541+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540037040 con 0x7f85540cc0c0 2024-09-16T01:50:03.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.541+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85541f13c0 con 0x7f85540cc0c0 2024-09-16T01:50:03.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.542+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85540cbb20 con 0x7f85540cc0c0 2024-09-16T01:50:03.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.543+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8540044070 con 0x7f85540cc0c0 2024-09-16T01:50:03.544 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:03.542+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:03.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.544+0000 7f8550c00640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f853406c2b0 0x7f853406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.544+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 5770+0+0 (secure 0 0 0) 0x7f85400ab670 con 0x7f85540cc0c0 2024-09-16T01:50:03.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.544+0000 7f8558a00640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f853406c2b0 0x7f853406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.545+0000 7f8558a00640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f853406c2b0 0x7f853406e770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f85480011e0 tx=0x7f8548009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.548+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85540cbb20 con 0x7f85540cc0c0 2024-09-16T01:50:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.599+0000 7f8327200640 1 Processor -- start 2024-09-16T01:50:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.599+0000 7f8327200640 1 -- start start 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.600+0000 7f8327200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c75a0 0x7f832815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8327200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83280cc040 0x7f83280cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8327200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832806d140 con 0x7f83280c6b70 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8327200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f832815b880 con 0x7f83280cc040 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8327200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f83280cab20 con 0x7f83280c75a0 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8325e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f83280c6b70 0x7f83280cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:54594/0 (socket says 172.21.15.146:54594) 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8325e00640 1 -- 172.21.15.146:0/645071186 learned_addr learned my addr 172.21.15.146:0/645071186 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8326800640 1 --2- 172.21.15.146:0/645071186 >> v2:172.21.15.161:3300/0 conn(0x7f83280cc040 0x7f83280cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8325400640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c75a0 0x7f832815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8324a00640 1 -- 172.21.15.146:0/645071186 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261557005 0 0) 0x7f832806d140 con 0x7f83280c6b70 2024-09-16T01:50:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8324a00640 1 -- 172.21.15.146:0/645071186 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f830c003520 con 0x7f83280c6b70 2024-09-16T01:50:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8326800640 1 -- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c75a0 msgr2=0x7f832815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8326800640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c75a0 0x7f832815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8326800640 1 -- 172.21.15.146:0/645071186 >> v1:172.21.15.161:6789/0 conn(0x7f83280c6b70 legacy=0x7f83280cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8326800640 1 -- 172.21.15.146:0/645071186 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832806d420 con 0x7f83280cc040 2024-09-16T01:50:03.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.601+0000 7f8325400640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c75a0 0x7f832815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.603+0000 7f8326800640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f83280cc4c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8320009d90 tx=0x7f832002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=784550debc6d4113 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.603+0000 7f8324a00640 1 -- 172.21.15.146:0/645071186 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320031600 con 0x7f83280cc040 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.603+0000 7f8324a00640 1 -- 172.21.15.146:0/645071186 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83200317a0 con 0x7f83280cc040 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.603+0000 7f8324a00640 1 -- 172.21.15.146:0/645071186 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320031b70 con 0x7f83280cc040 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 -- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 msgr2=0x7f83280cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f83280cc4c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8320009d90 tx=0x7f832002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 -- 172.21.15.146:0/645071186 shutdown_connections 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f83280cc4c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 --2- 172.21.15.146:0/645071186 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c75a0 0x7f832815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 -- 172.21.15.146:0/645071186 >> 172.21.15.146:0/645071186 conn(0x7f83280ccc20 msgr2=0x7f83280cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 -- 172.21.15.146:0/645071186 shutdown_connections 2024-09-16T01:50:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.604+0000 7f8327200640 1 -- 172.21.15.146:0/645071186 wait complete. 2024-09-16T01:50:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.606+0000 7f8327200640 1 Processor -- start 2024-09-16T01:50:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.606+0000 7f8327200640 1 -- start start 2024-09-16T01:50:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8327200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 0x7f83281e6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8327200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83280c75a0 0x7f83281f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8327200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f8328150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8327200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f83280cb050 con 0x7f83280c75a0 2024-09-16T01:50:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8327200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83280cad70 con 0x7f83280cc040 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8327200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f83280cbc80 con 0x7f83280c6b70 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8325e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 0x7f83281e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8325e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 0x7f83281e6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43474/0 (socket says 172.21.15.146:43474) 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8325e00640 1 -- 172.21.15.146:0/3822969699 learned_addr learned my addr 172.21.15.146:0/3822969699 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8326800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f8328150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.607+0000 7f8325400640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83280c75a0 0x7f83281f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8325e00640 1 -- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 msgr2=0x7f8328150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8325e00640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f8328150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8325e00640 1 -- 172.21.15.146:0/3822969699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83280c75a0 msgr2=0x7f83281f0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8325e00640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83280c75a0 0x7f83281f0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8325e00640 1 -- 172.21.15.146:0/3822969699 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8328157ba0 con 0x7f83280c6b70 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8325e00640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 0x7f83281e6490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f831800c390 tx=0x7f831800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8318017070 con 0x7f83280c6b70 2024-09-16T01:50:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8318005ce0 con 0x7f83280c6b70 2024-09-16T01:50:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180072d0 con 0x7f83280c6b70 2024-09-16T01:50:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8328157e60 con 0x7f83280c6b70 2024-09-16T01:50:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.608+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281f3e80 con 0x7f83280c6b70 2024-09-16T01:50:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.610+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f832806cd20 con 0x7f83280c6b70 2024-09-16T01:50:03.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.611+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f831801a070 con 0x7f83280c6b70 2024-09-16T01:50:03.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.612+0000 7f831e000640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f830806c2e0 0x7f830806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:03.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.612+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 5770+0+0 (secure 0 0 0) 0x7f831808a6e0 con 0x7f83280c6b70 2024-09-16T01:50:03.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.612+0000 7f8325400640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f830806c2e0 0x7f830806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:03.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.613+0000 7f8325400640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f830806c2e0 0x7f830806e7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8314009560 tx=0x7f8314008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:03.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.616+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f83180587c0 con 0x7f83280c6b70 2024-09-16T01:50:03.630 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:03.628+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:03.649 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:03.650 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:03.661 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:03.745 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:03.743+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:03.805 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:03.805 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:03.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.811+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd in", "ids": ["7"]} v 0) -- 0x7f85540c5560 con 0x7f85540cc0c0 2024-09-16T01:50:03.813 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:50:03.812+0000 7f49bee00640 -1 mon.a@0(leader).osd e34 definitely_dead 0 2024-09-16T01:50:03.814 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:03.855 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:03.853+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:03.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:03.871+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f83280ca280 con 0x7f83280c6b70 2024-09-16T01:50:03.933 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:03.932+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:03.958 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:03.967 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:04.034 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:04.033+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:04.070 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.070 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.078 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:04.135 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:04.134+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:04.181 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.181 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.190 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:04.235 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:04.234+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:04.293 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.293 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.302 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:04.336 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:04.335+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.369+0000 7f8550c00640 1 -- 172.21.15.146:0/1634646883 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["7"]}]=0 marked in osd.7. v35) ==== 85+0+0 (secure 0 0 0) 0x7f854004b070 con 0x7f85540cc0c0 2024-09-16T01:50:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:marked in osd.7. 2024-09-16T01:50:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.369+0000 7f831e000640 1 -- 172.21.15.146:0/3822969699 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v35) ==== 87+0+0 (secure 0 0 0) 0x7f831805cac0 con 0x7f83280c6b70 2024-09-16T01:50:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:50:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.373+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f853406c2b0 msgr2=0x7f853406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.373+0000 7f855b400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f853406c2b0 0x7f853406e770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f85480011e0 tx=0x7f8548009040 comp rx=0 tx=0).stop 2024-09-16T01:50:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.373+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85540cc0c0 msgr2=0x7f85540c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.373+0000 7f855b400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85540cc0c0 0x7f85540c8ef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8540031000 tx=0x7f85400046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.375+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f830806c2e0 msgr2=0x7f830806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.375+0000 7f8327200640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f830806c2e0 0x7f830806e7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8314009560 tx=0x7f8314008040 comp rx=0 tx=0).stop 2024-09-16T01:50:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.375+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 msgr2=0x7f83281e6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.375+0000 7f8327200640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 0x7f83281e6490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f831800c390 tx=0x7f831800c850 comp rx=0 tx=0).stop 2024-09-16T01:50:04.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 shutdown_connections 2024-09-16T01:50:04.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f8327200640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f830806c2e0 0x7f830806e7a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f8327200640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83280cc040 0x7f8328150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f8327200640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83280c75a0 0x7f83281f0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f8327200640 1 --2- 172.21.15.146:0/3822969699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83280c6b70 0x7f83281e6490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 >> 172.21.15.146:0/3822969699 conn(0x7f83280ccc20 msgr2=0x7f83280c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.378+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 shutdown_connections 2024-09-16T01:50:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.379+0000 7f8327200640 1 -- 172.21.15.146:0/3822969699 wait complete. 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 shutdown_connections 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f855b400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f853406c2b0 0x7f853406e770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f855b400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85540cc4a0 0x7f85540c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f855b400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85540cc0c0 0x7f85540c8ef0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f855b400640 1 --2- 172.21.15.146:0/1634646883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540cb2f0 0x7f85540b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.377+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 >> 172.21.15.146:0/1634646883 conn(0x7f855406d560 msgr2=0x7f85541527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.380+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 shutdown_connections 2024-09-16T01:50:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:04.380+0000 7f855b400640 1 -- 172.21.15.146:0/1634646883 wait complete. 2024-09-16T01:50:04.399 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-16T01:50:04.411 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.411 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.420 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:04.440 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:04.439+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:04.524 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.524 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.533 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:04.577 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:04.575+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:04.637 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.637 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.639 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:04.638+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:04.647 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:04.740 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:04.739+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:04.791 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.791 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.800 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:04.844 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:04.842+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:04.904 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:04.905 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:04.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:04.960 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:04.959+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:05.042 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:05.040+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:05.078 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:05.078 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:05.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:05.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:05.179 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:05.178+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:05.244 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:05.244+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:05.308 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:05.308 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:05.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:05.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.342+0000 7ff8df200640 1 Processor -- start 2024-09-16T01:50:05.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.342+0000 7ff8df200640 1 -- start start 2024-09-16T01:50:05.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.343+0000 7ff8df200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff8d80c6cd0 0x7ff8d806ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.343+0000 7ff8df200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8d80c8530 0x7ff8d806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.343+0000 7ff8df200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d815a9b0 con 0x7ff8d8154f10 2024-09-16T01:50:05.345 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:05.342+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.343+0000 7ff8df200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff8d80c7f20 con 0x7ff8d80c8530 2024-09-16T01:50:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.343+0000 7ff8df200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff8d806ea70 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8dd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8d80c8530 0x7ff8d806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d7400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff8d80c6cd0 0x7ff8d806ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8dd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8d80c8530 0x7ff8d806f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55754/0 (socket says 172.21.15.146:55754) 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8dd200640 1 -- 172.21.15.146:0/3544089521 learned_addr learned my addr 172.21.15.146:0/3544089521 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d7e00640 1 --1- 172.21.15.146:0/3544089521 >> v1:172.21.15.161:6789/0 conn(0x7ff8d8154f10 0x7ff8d80c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:54604/0 (socket says 172.21.15.146:54604) 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d7400640 1 -- 172.21.15.146:0/3544089521 >> v2:172.21.15.146:3300/0 conn(0x7ff8d80c8530 msgr2=0x7ff8d806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d7400640 1 --2- 172.21.15.146:0/3544089521 >> v2:172.21.15.146:3300/0 conn(0x7ff8d80c8530 0x7ff8d806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d7400640 1 -- 172.21.15.146:0/3544089521 >> v1:172.21.15.161:6789/0 conn(0x7ff8d8154f10 legacy=0x7ff8d80c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d6a00640 1 -- 172.21.15.146:0/3544089521 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028504997 0 0) 0x7ff8d815a9b0 con 0x7ff8d8154f10 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.344+0000 7ff8d7400640 1 -- 172.21.15.146:0/3544089521 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d815ac70 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.345+0000 7ff8d7400640 1 --2- 172.21.15.146:0/3544089521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d806ef00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80bfae0 tx=0x7ff8c402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5c41841e31c974f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.345+0000 7ff8d6a00640 1 -- 172.21.15.146:0/3544089521 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4037070 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8d6a00640 1 -- 172.21.15.146:0/3544089521 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c4032440 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8d6a00640 1 -- 172.21.15.146:0/3544089521 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4032810 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 -- 172.21.15.146:0/3544089521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 msgr2=0x7ff8d806ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 --2- 172.21.15.146:0/3544089521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d806ef00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80bfae0 tx=0x7ff8c402a230 comp rx=0 tx=0).stop 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 -- 172.21.15.146:0/3544089521 shutdown_connections 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 --2- 172.21.15.146:0/3544089521 >> v2:172.21.15.146:3300/0 conn(0x7ff8d80c8530 0x7ff8d806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 --2- 172.21.15.146:0/3544089521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d806ef00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 -- 172.21.15.146:0/3544089521 >> 172.21.15.146:0/3544089521 conn(0x7ff8d806bc10 msgr2=0x7ff8d806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 -- 172.21.15.146:0/3544089521 shutdown_connections 2024-09-16T01:50:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.346+0000 7ff8df200640 1 -- 172.21.15.146:0/3544089521 wait complete. 2024-09-16T01:50:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.348+0000 7ff8df200640 1 Processor -- start 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.348+0000 7ff8df200640 1 -- start start 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.349+0000 7ff8df200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d80cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.349+0000 7ff8df200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d80c8530 0x7ff8d8156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.349+0000 7ff8df200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8d8154f10 0x7ff8d8155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.349+0000 7ff8df200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d815b510 con 0x7ff8d80c8530 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.349+0000 7ff8df200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d815b230 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.349+0000 7ff8df200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d815b7f0 con 0x7ff8d8154f10 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d80cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d80cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:53400/0 (socket says 172.21.15.146:53400) 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d80c8530 0x7ff8d8156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 -- 172.21.15.146:0/148533172 learned_addr learned my addr 172.21.15.146:0/148533172 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8d8154f10 0x7ff8d8155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8d8154f10 msgr2=0x7ff8d8155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8d8154f10 0x7ff8d8155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d80c8530 msgr2=0x7ff8d8156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d80c8530 0x7ff8d8156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d80cab60 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d80c8530 0x7ff8d8156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.350+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d80cb8e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800c430 tx=0x7ff8c800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.351+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c80074e0 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.351+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c80047e0 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.351+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8004bd0 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.351+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d80ca480 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.351+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d80cb0e0 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.352+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff8d80c8130 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.353+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff8c8011070 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.354+0000 7ff8cfe00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff8b806c2e0 0x7ff8b806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.355+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5770+0+0 (secure 0 0 0) 0x7ff8c808b0b0 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.355+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff8b806c2e0 0x7ff8b806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.355+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v35) ==== 54+0+16 (secure 0 0 0) 0x7ff8c808d650 con 0x7ff8d80c6cd0 2024-09-16T01:50:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.355+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff8d80cb320 0x7ff8d80c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.355+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8d8072270 con 0x7ff8d80cb320 2024-09-16T01:50:05.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.355+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff8d80cb320 0x7ff8d80c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.356+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff8b806c2e0 0x7ff8b806e7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4034000 tx=0x7ff8c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.356+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff8d80cb320 0x7ff8d80c7130 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.357+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d8072270 con 0x7ff8d80cb320 2024-09-16T01:50:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.376+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d8071f90 con 0x7ff8d80cb320 2024-09-16T01:50:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.380+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d8071f90 con 0x7ff8d80cb320 2024-09-16T01:50:05.382 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.382+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff8d80c7670 0x7ff8d81f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.382+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff8d806c400 con 0x7ff8d80c7670 2024-09-16T01:50:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.384+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff8d80c7670 0x7ff8d81f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.384+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff8d80c7670 0x7ff8d81f44a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.386+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d806c400 con 0x7ff8d80c7670 2024-09-16T01:50:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.411+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d8151dc0 con 0x7ff8d80c7670 2024-09-16T01:50:05.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.413+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d8151dc0 con 0x7ff8d80c7670 2024-09-16T01:50:05.415 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.414+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff8d8148890 0x7ff8d8148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.414+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff8d81491d0 con 0x7ff8d8148890 2024-09-16T01:50:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.414+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff8d8148890 0x7ff8d8148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.415+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff8d8148890 0x7ff8d8148c90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.416+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d81491d0 con 0x7ff8d8148890 2024-09-16T01:50:05.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.430+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d81496e0 con 0x7ff8d8148890 2024-09-16T01:50:05.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.432+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8c4032d60 con 0x7ff8d8148890 2024-09-16T01:50:05.433 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:05.433 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:05.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.432+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff8d8062f00 0x7ff8d81504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.432+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff8d8063360 con 0x7ff8d8062f00 2024-09-16T01:50:05.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.432+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff8d8062f00 0x7ff8d81504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.433+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff8d8062f00 0x7ff8d81504c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.434+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d8063360 con 0x7ff8d8062f00 2024-09-16T01:50:05.445 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:05.443+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:05.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.448+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d8063360 con 0x7ff8d8062f00 2024-09-16T01:50:05.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.450+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d8063360 con 0x7ff8d8062f00 2024-09-16T01:50:05.451 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:05.451 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:05.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.450+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff8d802c640 0x7ff8d802ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.450+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff8d8063360 con 0x7ff8d802c640 2024-09-16T01:50:05.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.450+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff8d802c640 0x7ff8d802ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.451+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff8d802c640 0x7ff8d802ca60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.452+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d8063360 con 0x7ff8d802c640 2024-09-16T01:50:05.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.466+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d8063360 con 0x7ff8d802c640 2024-09-16T01:50:05.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.467+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d8063360 con 0x7ff8d802c640 2024-09-16T01:50:05.469 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:05.469 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:05.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.468+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff8d802d4b0 0x7ff8d82075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.468+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff8d8151f70 con 0x7ff8d802d4b0 2024-09-16T01:50:05.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.468+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff8d802d4b0 0x7ff8d82075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.468+0000 7ff8d7400640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff8d802d4b0 0x7ff8d82075a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.469+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8c403c700 con 0x7ff8d802d4b0 2024-09-16T01:50:05.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.483+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d80635a0 con 0x7ff8d802d4b0 2024-09-16T01:50:05.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.485+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d80635a0 con 0x7ff8d802d4b0 2024-09-16T01:50:05.486 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.486+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff8d802d910 0x7ff8d820a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.486+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff8d820cdd0 con 0x7ff8d802d910 2024-09-16T01:50:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.486+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff8d802d910 0x7ff8d820a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.486+0000 7ff8dd200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff8d802d910 0x7ff8d820a000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.487+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d820cdd0 con 0x7ff8d802d910 2024-09-16T01:50:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.503+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d820a540 con 0x7ff8d802d910 2024-09-16T01:50:05.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.504+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d820a540 con 0x7ff8d802d910 2024-09-16T01:50:05.506 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:05.506 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:05.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.505+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff8d820abc0 0x7ff8d820cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:05.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.505+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff8d820ff10 con 0x7ff8d820abc0 2024-09-16T01:50:05.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.505+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff8d820abc0 0x7ff8d820cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:05.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.506+0000 7ff8d7e00640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff8d820abc0 0x7ff8d820cfb0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:05.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.506+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d820ff10 con 0x7ff8d820abc0 2024-09-16T01:50:05.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.520+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff8d820ec30 con 0x7ff8d820abc0 2024-09-16T01:50:05.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8cfe00640 1 -- 172.21.15.146:0/148533172 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff8d820ec30 con 0x7ff8d820abc0 2024-09-16T01:50:05.523 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff8d80cb320 msgr2=0x7ff8d80c7130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff8d80cb320 0x7ff8d80c7130 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff8d80c7670 msgr2=0x7ff8d81f44a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff8d80c7670 0x7ff8d81f44a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff8d8148890 msgr2=0x7ff8d8148c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff8d8148890 0x7ff8d8148c90 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff8d8062f00 msgr2=0x7ff8d81504c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff8d8062f00 0x7ff8d81504c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff8d802c640 msgr2=0x7ff8d802ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff8d802c640 0x7ff8d802ca60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff8d802d4b0 msgr2=0x7ff8d82075a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff8d802d4b0 0x7ff8d82075a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff8d802d910 msgr2=0x7ff8d820a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff8d802d910 0x7ff8d820a000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.522+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff8d820abc0 msgr2=0x7ff8d820cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff8d820abc0 0x7ff8d820cfb0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff8b806c2e0 msgr2=0x7ff8b806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff8b806c2e0 0x7ff8b806e7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4034000 tx=0x7ff8c402e000 comp rx=0 tx=0).stop 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8dd200640 1 -- 172.21.15.146:0/148533172 reap_dead start 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8dd200640 1 -- 172.21.15.146:0/148533172 reap_dead start 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8dd200640 1 -- 172.21.15.146:0/148533172 reap_dead start 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8dd200640 1 -- 172.21.15.146:0/148533172 reap_dead start 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8dd200640 1 -- 172.21.15.146:0/148533172 reap_dead start 2024-09-16T01:50:05.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 msgr2=0x7ff8d80cb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d80cb8e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800c430 tx=0x7ff8c800c8f0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 shutdown_connections 2024-09-16T01:50:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 --2- 172.21.15.146:0/148533172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8d80c6cd0 0x7ff8d80cb8e0 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800c430 tx=0x7ff8c800c8f0 comp rx=0 tx=0).stop 2024-09-16T01:50:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 >> 172.21.15.146:0/148533172 conn(0x7ff8d806bc10 msgr2=0x7ff8d8153200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 shutdown_connections 2024-09-16T01:50:05.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:05.523+0000 7ff8df200640 1 -- 172.21.15.146:0/148533172 wait complete. 2024-09-16T01:50:05.535 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:05.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.707849209999999, 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:50:05.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.707743161000003, 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:50:05.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:50:05.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:05.547 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.707688535000003, 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:05.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:50:05.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.707620202000001, 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.553 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:50:05.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:50:05.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.673364999, 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:50:05.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:50:05.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.564 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.673181800999998, 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:50:05.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:50:05.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.673127792999999, 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:50:05.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:50:05.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:50:05.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.576 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.673095353000001, 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:50:05.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:05.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.669727352000002, 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:50:05.582 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:50:05.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.669656676000002, 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.669620301000002, 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:05.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.668992557999999, 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.668937950999997, 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:50:05.600 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:05.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:50:05.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596799543000003, 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:50:05.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:50:05.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596741231000003, 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:05.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.616 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596667156000002, 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:50:05.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596608535000001, 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.622 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:50:05.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:50:05.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596586600999998, 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:05.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:50:05.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:50:05.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596556241000002, 2024-09-16T01:50:05.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:50:05.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.638 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.596529662000002, 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:50:05.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:05.644 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:05.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:05.647 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:05.643+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:05.746 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:05.745+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:05.847 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:05.846+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:05.863 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:05.863 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:05.863 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:05.863 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:05.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:05.948 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:05.947+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:06.049 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:06.048+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:06.083 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:06.084 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:06.097 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:06.150 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:06.149+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:06.251 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:06.249+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.502533396000004, 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:06.274 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:06.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:06.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.070979664000006, 2024-09-16T01:50:06.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:06.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:06.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:06.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.051055442000006, 2024-09-16T01:50:06.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:06.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:06.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.051035524, 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:06.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.050974940000003, 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:06.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:06.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.804689420000003, 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:06.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:06.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.804538906000005, 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.313 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:06.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:06.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.804519338000006, 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.319 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:06.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:06.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.234363123999998, 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.328 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:06.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:06.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.234195306000004, 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.334 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:06.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.234176324000003, 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.340 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:06.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.234156693000003, 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:06.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.234022787000001, 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:06.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:06.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.234011249000005, 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:06.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:06.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.233992838000006, 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.364 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:06.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.369 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.233878168000004, 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:06.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:06.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.223096440000006, 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.376 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:06.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:06.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.222979781999996, 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:06.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:06.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:06.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.407446542999999, 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:06.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:06.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.407385204999997, 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:06.392 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:06.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:06.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:06.396 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:06.397 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:06.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:06.398 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:06.396+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:06.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:50:06.499 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:06.498+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:06.600 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:06.599+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:06.613 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:06.613 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:06.613 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:06.613 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:06.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.616+0000 7fcf1aa00640 1 Processor -- start 2024-09-16T01:50:06.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.616+0000 7fcf1aa00640 1 -- start start 2024-09-16T01:50:06.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c6b70 0x7fcf1415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1aa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcf140c75a0 0x7fcf140cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1aa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1406d140 con 0x7fcf140cbab0 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1aa00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcf1415b880 con 0x7fcf140c75a0 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1aa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcf140cab20 con 0x7fcf140c6b70 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1a000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcf140cbab0 0x7fcf140cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52348/0 (socket says 172.21.15.146:52348) 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf1a000640 1 -- 172.21.15.146:0/3271338495 learned_addr learned my addr 172.21.15.146:0/3271338495 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf18c00640 1 --2- 172.21.15.146:0/3271338495 >> v2:172.21.15.161:3300/0 conn(0x7fcf140c75a0 0x7fcf140cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:06.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.617+0000 7fcf19600640 1 --2- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c6b70 0x7fcf1415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.618+0000 7fcf13e00640 1 -- 172.21.15.146:0/3271338495 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117293680 0 0) 0x7fcf1406d140 con 0x7fcf140cbab0 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.618+0000 7fcf13e00640 1 -- 172.21.15.146:0/3271338495 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcefc003520 con 0x7fcf140cbab0 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.618+0000 7fcf18c00640 1 -- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c6b70 msgr2=0x7fcf1415b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.618+0000 7fcf18c00640 1 --2- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c6b70 0x7fcf1415b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.618+0000 7fcf18c00640 1 -- 172.21.15.146:0/3271338495 >> v1:172.21.15.146:6789/0 conn(0x7fcf140cbab0 legacy=0x7fcf140cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.618+0000 7fcf18c00640 1 -- 172.21.15.146:0/3271338495 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1406d420 con 0x7fcf140c75a0 2024-09-16T01:50:06.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.619+0000 7fcf18c00640 1 --2- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c75a0 0x7fcf140cb570 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcf000097a0 tx=0x7fcf0002a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0de85b43ec16a6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.619+0000 7fcf13e00640 1 -- 172.21.15.146:0/3271338495 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf00037070 con 0x7fcf140c75a0 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf13e00640 1 -- 172.21.15.146:0/3271338495 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0002adb0 con 0x7fcf140c75a0 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf13e00640 1 -- 172.21.15.146:0/3271338495 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf00032660 con 0x7fcf140c75a0 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf1aa00640 1 -- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c75a0 msgr2=0x7fcf140cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c75a0 0x7fcf140cb570 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcf000097a0 tx=0x7fcf0002a060 comp rx=0 tx=0).stop 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf1aa00640 1 -- 172.21.15.146:0/3271338495 shutdown_connections 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c75a0 0x7fcf140cb570 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/3271338495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c6b70 0x7fcf1415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.620+0000 7fcf1aa00640 1 -- 172.21.15.146:0/3271338495 >> 172.21.15.146:0/3271338495 conn(0x7fcf140ccc20 msgr2=0x7fcf140cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.621+0000 7fcf1aa00640 1 -- 172.21.15.146:0/3271338495 shutdown_connections 2024-09-16T01:50:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.621+0000 7fcf1aa00640 1 -- 172.21.15.146:0/3271338495 wait complete. 2024-09-16T01:50:06.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.622+0000 7fcf1aa00640 1 Processor -- start 2024-09-16T01:50:06.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.622+0000 7fcf1aa00640 1 -- start start 2024-09-16T01:50:06.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.623+0000 7fcf1aa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 0x7fcf140aba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:06.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.623+0000 7fcf1aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c75a0 0x7fcf140c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:06.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.623+0000 7fcf1aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf140cbab0 0x7fcf141563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:06.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.623+0000 7fcf1aa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcf140cb050 con 0x7fcf140cbab0 2024-09-16T01:50:06.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.623+0000 7fcf1aa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcf140cad70 con 0x7fcf140c6b70 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.623+0000 7fcf1aa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1406cd20 con 0x7fcf140c75a0 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 0x7fcf140aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 0x7fcf140aba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:53420/0 (socket says 172.21.15.146:53420) 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 -- 172.21.15.146:0/2602388447 learned_addr learned my addr 172.21.15.146:0/2602388447 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf18c00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c75a0 0x7fcf140c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf1a000640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf140cbab0 0x7fcf141563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 -- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c75a0 msgr2=0x7fcf140c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c75a0 0x7fcf140c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 -- 172.21.15.146:0/2602388447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf140cbab0 msgr2=0x7fcf141563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf140cbab0 0x7fcf141563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 -- 172.21.15.146:0/2602388447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1415cc00 con 0x7fcf140c6b70 2024-09-16T01:50:06.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.624+0000 7fcf19600640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 0x7fcf140aba90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800e8d0 tx=0x7fcf0800ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:06.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.625+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08017070 con 0x7fcf140c6b70 2024-09-16T01:50:06.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.625+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0800c940 con 0x7fcf140c6b70 2024-09-16T01:50:06.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.625+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf141f0c60 con 0x7fcf140c6b70 2024-09-16T01:50:06.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.625+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0800cd30 con 0x7fcf140c6b70 2024-09-16T01:50:06.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.625+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf141f1200 con 0x7fcf140c6b70 2024-09-16T01:50:06.629 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:06.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.626+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1415ae20 con 0x7fcf140c6b70 2024-09-16T01:50:06.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.627+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcf0801a070 con 0x7fcf140c6b70 2024-09-16T01:50:06.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.628+0000 7fcf11600640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcef406c2b0 0x7fcef406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:06.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.628+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5770+0+0 (secure 0 0 0) 0x7fcf0808b260 con 0x7fcf140c6b70 2024-09-16T01:50:06.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.628+0000 7fcf18c00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcef406c2b0 0x7fcef406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:06.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.629+0000 7fcf18c00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcef406c2b0 0x7fcef406e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcf00034000 tx=0x7fcf0002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:06.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.632+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcf1415ae20 con 0x7fcf140c6b70 2024-09-16T01:50:06.701 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:06.699+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:06.797 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:06.797 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:06.797 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:06.797 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:06.798 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:06.802 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:06.801+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:06.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:06.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:06.841+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fcf1406da90 con 0x7fcf140c6b70 2024-09-16T01:50:06.903 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:06.901+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:07.004 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:07.003+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.192274748000003, 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:07.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:07.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.192159926999999, 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:07.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:07.031 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.192118057000002, 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:07.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:07.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.192073778000001, 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:07.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:07.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.192044901000003, 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:07.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.188337693999998, 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:07.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:07.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.188283712999997, 2024-09-16T01:50:07.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:07.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.152868488999999, 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.055 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:07.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:07.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:07.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.14980834, 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:07.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:07.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.064 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.148948892, 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:07.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:07.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.148894998999999, 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.070 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:07.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.075 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.142871026000002, 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:07.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:07.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.142820897999997, 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:07.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.084 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.142744565000001, 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:07.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.142713209999997, 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:07.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.104627755000003, 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:07.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:07.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:07.098 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.103836223999998, 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:07.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.079754399000002, 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:07.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.079682486999999, 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:07.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.749040087000001, 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:07.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:07.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:07.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:07.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.121 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:07.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:07.123 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:07.121+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:07.223 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:07.222+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:07.324 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:07.323+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:07.361 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:07.361 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:07.361 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:07.361 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:07.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.386+0000 7fcf11600640 1 -- 172.21.15.146:0/2602388447 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v37) ==== 97+0+0 (secure 0 0 0) 0x7fcf080593f0 con 0x7fcf140c6b70 2024-09-16T01:50:07.387 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:50:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcef406c2b0 msgr2=0x7fcef406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcef406c2b0 0x7fcef406e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcf00034000 tx=0x7fcf0002e000 comp rx=0 tx=0).stop 2024-09-16T01:50:07.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 msgr2=0x7fcf140aba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 0x7fcf140aba90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800e8d0 tx=0x7fcf0800ed90 comp rx=0 tx=0).stop 2024-09-16T01:50:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 shutdown_connections 2024-09-16T01:50:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcef406c2b0 0x7fcef406e770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf140cbab0 0x7fcf141563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcf140c75a0 0x7fcf140c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 --2- 172.21.15.146:0/2602388447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf140c6b70 0x7fcf140aba90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 >> 172.21.15.146:0/2602388447 conn(0x7fcf140ccc20 msgr2=0x7fcf140cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 shutdown_connections 2024-09-16T01:50:07.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.389+0000 7fcf1aa00640 1 -- 172.21.15.146:0/2602388447 wait complete. 2024-09-16T01:50:07.449 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:07.448+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:07.525 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:07.524+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:07.550 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:07.550 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:07.551 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:07.551 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:07.551 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:07.563 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:07.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:07.649 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:07.648+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:07.727 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:07.726+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.783 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.953388394999998, 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063070110000000004, 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825667+0000", 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825672+0000", 2024-09-16T01:50:07.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6920000000000001e-06 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825689+0000", 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7039999999999999e-05 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825696+0000", 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5909999999999998e-06 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826617+0000", 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00092117800000000004 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826697+0000", 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9598999999999995e-05 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826832+0000", 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013548 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829938+0000", 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031059059999999999 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831894+0000", 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019552240000000002 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831942+0000", 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8389e-05 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831973+0000", 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1043000000000001e-05 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:29 2.b 2:d0630e4c:::mds0_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.953346846999999, 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087271810000000005, 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:07.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-09-16T01:50:07.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825734+0000", 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5638000000000001e-05 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825743+0000", 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7350000000000002e-06 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825750+0000", 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827333+0000", 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015834689999999999 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827472+0000", 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138796 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827572+0000", 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9910999999999999e-05 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832328+0000", 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047559079999999997 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834353+0000", 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002025654 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834392+0000", 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8099000000000002e-05 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834435+0000", 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3169e-05 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:34 2.b 2:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.953278586000003, 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088227679999999999, 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825777+0000", 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825781+0000", 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0350000000000003e-06 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825788+0000", 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825794+0000", 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4240000000000001e-06 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827917+0000", 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021233570000000002 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828052+0000", 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013528300000000001 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834599+0000", 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065463650000000002 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912780720999997, 2024-09-16T01:50:07.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012518068, 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866276+0000", 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866303+0000", 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7229e-05 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866329+0000", 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.563e-05 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866343+0000", 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4287999999999999e-05 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867782+0000", 2024-09-16T01:50:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014384510000000001 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867911+0000", 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012957999999999999 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868028+0000", 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117025 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877878+0000", 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098499209999999993 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878708+0000", 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00082986900000000001 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878752+0000", 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3735000000000001e-05 2024-09-16T01:50:07.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878792+0000", 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0025999999999999e-05 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912680440999999, 2024-09-16T01:50:07.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014487969, 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866375+0000", 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9899999999999999e-07 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866379+0000", 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6519999999999997e-06 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866388+0000", 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9579999999999996e-06 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866395+0000", 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7750000000000004e-06 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868287+0000", 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001892344 2024-09-16T01:50:07.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868360+0000", 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3023999999999997e-05 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868473+0000", 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112321 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878461+0000", 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099884610000000006 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880760+0000", 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022984479999999998 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880823+0000", 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3276999999999997e-05 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880862+0000", 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:7 2.b 2:d70a01c6:::60c.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:07.812 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912650403000001, 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020806099000000002, 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.82e-07 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866408+0000", 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0110000000000002e-06 2024-09-16T01:50:07.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866415+0000", 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5830000000000003e-06 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866421+0000", 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6659999999999996e-06 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022638100000000002 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868842+0000", 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015727699999999999 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868942+0000", 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100182 2024-09-16T01:50:07.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878519+0000", 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095771399999999996 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887083+0000", 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085641750000000003 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887155+0000", 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2117000000000001e-05 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887210+0000", 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4656e-05 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.818 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:8 2.b 2:d70a01c6:::60c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912612011999997, 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020917458, 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866443+0000", 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866447+0000", 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8340000000000001e-06 2024-09-16T01:50:07.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866455+0000", 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866460+0000", 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5629999999999999e-06 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869260+0000", 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027996039999999998 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869328+0000", 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8730000000000001e-05 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869461+0000", 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264099999999999 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878525+0000", 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090635630000000002 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887259+0000", 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087339310000000003 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887305+0000", 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6538000000000001e-05 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887360+0000", 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4706000000000001e-05 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:10 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.91258543, 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01479957, 2024-09-16T01:50:07.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8100000000000003e-07 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866473+0000", 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1089999999999998e-06 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866481+0000", 2024-09-16T01:50:07.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866486+0000", 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.566e-06 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870818+0000", 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004332519 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870937+0000", 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011880899999999999 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871093+0000", 2024-09-16T01:50:07.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015636400000000001 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877793+0000", 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066997560000000003 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881167+0000", 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033735029999999999 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881242+0000", 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5139e-05 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881269+0000", 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6794000000000001e-05 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.830 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:27 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912547656999998, 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014854865, 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866511+0000", 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7479999999999999e-06 2024-09-16T01:50:07.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866518+0000", 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866525+0000", 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2750000000000001e-06 2024-09-16T01:50:07.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875004+0000", 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084799100000000002 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875082+0000", 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7698e-05 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:07.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875190+0000", 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010785000000000001 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877830+0000", 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026396480000000001 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881273+0000", 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034436839999999998 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881341+0000", 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7359000000000003e-05 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881362+0000", 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0928e-05 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:33 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912522219000003, 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020942512, 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866533+0000", 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.27e-07 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866552+0000", 2024-09-16T01:50:07.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8921999999999999e-05 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866559+0000", 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2570000000000001e-06 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866564+0000", 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869714+0000", 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031497920000000002 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869828+0000", 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114331 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869968+0000", 2024-09-16T01:50:07.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013971799999999999 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880969+0000", 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011001031 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887401+0000", 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064320319999999999 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887441+0000", 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9937000000000001e-05 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887475+0000", 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4010999999999997e-05 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:2 2.5 2:a6e36b3f:::401.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.842 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912483235000003, 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014499972999999999, 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866572+0000", 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8400000000000005e-07 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866575+0000", 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0929999999999999e-06 2024-09-16T01:50:07.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866592+0000", 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7765999999999999e-05 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866598+0000", 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4219999999999999e-06 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871180+0000", 2024-09-16T01:50:07.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045822939999999998 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871272+0000", 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1308000000000001e-05 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871419+0000", 2024-09-16T01:50:07.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014714 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875993+0000", 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004574486 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881013+0000", 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050202240000000002 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881046+0000", 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2944000000000002e-05 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881071+0000", 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4811999999999999e-05 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:18 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912435518999999, 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020977030000000001, 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1799999999999995e-07 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866623+0000", 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4709999999999999e-06 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866630+0000", 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866635+0000", 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3079999999999998e-06 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870303+0000", 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036679210000000002 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870398+0000", 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4464999999999998e-05 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870501+0000", 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010291800000000001 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880977+0000", 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010476014000000001 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887510+0000", 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065331759999999999 2024-09-16T01:50:07.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887553+0000", 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2737000000000001e-05 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887596+0000", 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3343000000000002e-05 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:22 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912411139, 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01481146, 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866644+0000", 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866646+0000", 2024-09-16T01:50:07.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2139999999999999e-06 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866652+0000", 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2160000000000003e-06 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866677+0000", 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4663000000000001e-05 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876491+0000", 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098146110000000009 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876616+0000", 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012430699999999999 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881455+0000", 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048389840000000002 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:24 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912369851000001, 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016598702999999999, 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:07.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:07.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5400000000000002e-07 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866688+0000", 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866695+0000", 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.883e-06 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866700+0000", 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7280000000000003e-06 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868221+0000", 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015209620000000001 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868334+0000", 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113584 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883283+0000", 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014949100999999999 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:25 2.0 2:0250a5a6:::201.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.912335225, 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011987815000000001, 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.863 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866720+0000", 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6300000000000001e-07 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866738+0000", 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8218000000000001e-05 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866745+0000", 2024-09-16T01:50:07.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7279999999999995e-06 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866751+0000", 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868589+0000", 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018382730000000001 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868709+0000", 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119571 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878707+0000", 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099980020000000006 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.2.11:5 2.0 2:03625d61:::614.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:07.867 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.883625735999999, 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081102819999999999, 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895434+0000", 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4909999999999999e-06 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895444+0000", 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0560000000000001e-05 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895451+0000", 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5590000000000002e-06 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896385+0000", 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093368200000000002 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896454+0000", 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9567000000000007e-05 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896559+0000", 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104609 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.899222+0000", 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026632000000000001 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903485+0000", 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042627999999999997 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903517+0000", 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2345999999999998e-05 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903539+0000", 2024-09-16T01:50:07.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1838e-05 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:14 2.1a 2:594e21d0:::623.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.867219497000001, 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068542239999999999, 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:07.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:07.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911838+0000", 2024-09-16T01:50:07.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.435e-06 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911844+0000", 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.122e-06 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911849+0000", 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6979999999999997e-06 2024-09-16T01:50:07.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914481+0000", 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026327249999999998 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914555+0000", 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3886000000000004e-05 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:07.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914697+0000", 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014193800000000001 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918088+0000", 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033908559999999998 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918610+0000", 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052154100000000004 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918656+0000", 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-05 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918689+0000", 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3052000000000001e-05 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.880 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:23 2.18 2:1c1db442:::103.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.867085951, 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067043830000000004, 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911969+0000", 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911980+0000", 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1090999999999999e-05 2024-09-16T01:50:07.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911989+0000", 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6270000000000006e-06 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911994+0000", 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3639999999999998e-06 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912099+0000", 2024-09-16T01:50:07.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010514900000000001 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912210+0000", 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111346 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912342+0000", 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131197 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915846+0000", 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035043269999999998 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918573+0000", 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027272289999999999 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918632+0000", 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8854000000000003e-05 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918673+0000", 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0728e-05 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.842486620999999, 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084942400000000001, 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.886 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000004e-07 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936594+0000", 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5137999999999999e-05 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936602+0000", 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8999999999999995e-06 2024-09-16T01:50:07.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936609+0000", 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9480000000000002e-06 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938373+0000", 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001763857 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938479+0000", 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105816 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938622+0000", 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014314699999999999 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941905+0000", 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032832 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944943+0000", 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030378670000000001 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945019+0000", 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5532999999999997e-05 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945062+0000", 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180999999999999e-05 2024-09-16T01:50:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.842421121999998, 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085927369999999996, 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936634+0000", 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936637+0000", 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.287e-06 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936647+0000", 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7189999999999996e-06 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936653+0000", 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.319e-06 2024-09-16T01:50:07.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938867+0000", 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022140079999999999 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938935+0000", 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7644999999999994e-05 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939072+0000", 2024-09-16T01:50:07.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013753600000000001 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941912+0000", 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028393350000000001 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945119+0000", 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032077160000000002 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:07.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945183+0000", 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3723e-05 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945226+0000", 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2805000000000001e-05 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:07.897 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:07.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:07.904 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:07.897+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:07.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.824+0000 7fe827200640 1 Processor -- start 2024-09-16T01:50:07.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.824+0000 7fe827200640 1 -- start start 2024-09-16T01:50:07.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.825+0000 7fe827200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe8281529d0 0x7fe828159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe827200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe82806f470 0x7fe8280cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe827200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8280c5770 con 0x7fe8280ce4d0 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe827200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe828159780 con 0x7fe82806f470 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe827200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe8280c5520 con 0x7fe8281529d0 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe826800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe8280ce4d0 0x7fe8280c75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52364/0 (socket says 172.21.15.146:52364) 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe825400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe82806f470 0x7fe8280cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe826800640 1 -- 172.21.15.146:0/650389228 learned_addr learned my addr 172.21.15.146:0/650389228 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe825e00640 1 --2- 172.21.15.146:0/650389228 >> v2:172.21.15.161:3300/0 conn(0x7fe8281529d0 0x7fe828159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe824a00640 1 -- 172.21.15.146:0/650389228 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4226345091 0 0) 0x7fe8280c5770 con 0x7fe8280ce4d0 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe825400640 1 -- 172.21.15.146:0/650389228 >> v2:172.21.15.161:3300/0 conn(0x7fe8281529d0 msgr2=0x7fe828159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe825400640 1 --2- 172.21.15.146:0/650389228 >> v2:172.21.15.161:3300/0 conn(0x7fe8281529d0 0x7fe828159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe825400640 1 -- 172.21.15.146:0/650389228 >> v1:172.21.15.146:6789/0 conn(0x7fe8280ce4d0 legacy=0x7fe8280c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.826+0000 7fe825400640 1 -- 172.21.15.146:0/650389228 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8280c5a50 con 0x7fe82806f470 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.828+0000 7fe825400640 1 --2- 172.21.15.146:0/650389228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe82806f470 0x7fe8280cdf90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8280686b0 tx=0x7fe81402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc268341584fe7be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.828+0000 7fe824a00640 1 -- 172.21.15.146:0/650389228 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814037070 con 0x7fe82806f470 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.828+0000 7fe824a00640 1 -- 172.21.15.146:0/650389228 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe814032440 con 0x7fe82806f470 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.828+0000 7fe824a00640 1 -- 172.21.15.146:0/650389228 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814032810 con 0x7fe82806f470 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 -- 172.21.15.146:0/650389228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe82806f470 msgr2=0x7fe8280cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 --2- 172.21.15.146:0/650389228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe82806f470 0x7fe8280cdf90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe8280686b0 tx=0x7fe81402a250 comp rx=0 tx=0).stop 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 -- 172.21.15.146:0/650389228 shutdown_connections 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 --2- 172.21.15.146:0/650389228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe82806f470 0x7fe8280cdf90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 --2- 172.21.15.146:0/650389228 >> v2:172.21.15.161:3300/0 conn(0x7fe8281529d0 0x7fe828159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 -- 172.21.15.146:0/650389228 >> 172.21.15.146:0/650389228 conn(0x7fe82806c000 msgr2=0x7fe828067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 -- 172.21.15.146:0/650389228 shutdown_connections 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.829+0000 7fe827200640 1 -- 172.21.15.146:0/650389228 wait complete. 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.831+0000 7fe827200640 1 Processor -- start 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.831+0000 7fe827200640 1 -- start start 2024-09-16T01:50:07.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe827200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe82806f470 0x7fe8281f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe827200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 0x7fe8281f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe827200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8281529d0 0x7fe8280c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe827200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe8280681a0 con 0x7fe8280ce4d0 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe827200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe828070000 con 0x7fe8281529d0 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe827200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe828152db0 con 0x7fe82806f470 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe825400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 0x7fe8281f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe825400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 0x7fe8281f03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55792/0 (socket says 172.21.15.146:55792) 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe825400640 1 -- 172.21.15.146:0/4115697692 learned_addr learned my addr 172.21.15.146:0/4115697692 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe82806f470 0x7fe8281f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.832+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8281529d0 0x7fe8280c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe825400640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe82806f470 msgr2=0x7fe8281f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe82806f470 0x7fe8281f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe825400640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8281529d0 msgr2=0x7fe8280c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8281529d0 0x7fe8280c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe825400640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8280c4520 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe8281529d0 0x7fe8280c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 0x7fe8281f03f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe814000c00 tx=0x7fe8140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814037040 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe814042070 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814030960 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8280c4800 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.833+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8281f3ce0 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.834+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe8280cdf90 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.835+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe814030b00 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.836+0000 7fe81e000640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe80806c2b0 0x7fe80806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.836+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5770+0+0 (secure 0 0 0) 0x7fe8140abd80 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.837+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v37) ==== 54+0+16 (secure 0 0 0) 0x7fe8140ae320 con 0x7fe8280ce4d0 2024-09-16T01:50:07.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.837+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe80806c2b0 0x7fe80806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.837+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe8281f3f20 0x7fe82806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.837+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8280c7f30 con 0x7fe8281f3f20 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.837+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe8281f3f20 0x7fe82806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.837+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe80806c2b0 0x7fe80806e770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe818009aa0 tx=0x7fe818009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.838+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe8281f3f20 0x7fe82806e110 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.839+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8280c7f30 con 0x7fe8281f3f20 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.853+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8280c1810 con 0x7fe8281f3f20 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.855+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8280c1810 con 0x7fe8281f3f20 2024-09-16T01:50:07.909 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:07.910 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:07.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.855+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe8281e2e20 0x7fe82814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.855+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe82814ec20 con 0x7fe8281e2e20 2024-09-16T01:50:07.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.855+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe8281e2e20 0x7fe82814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.856+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe8281e2e20 0x7fe82814e6e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.857+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe82814ec20 con 0x7fe8281e2e20 2024-09-16T01:50:07.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.874+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe828150e10 con 0x7fe8281e2e20 2024-09-16T01:50:07.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.875+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe828150e10 con 0x7fe8281e2e20 2024-09-16T01:50:07.911 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:07.912 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:07.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.876+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe8280ca360 0x7fe8280ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.876+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe8280cac80 con 0x7fe8280ca360 2024-09-16T01:50:07.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.876+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe8280ca360 0x7fe8280ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.876+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe8280ca360 0x7fe8280ca740 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.877+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8280cac80 con 0x7fe8280ca360 2024-09-16T01:50:07.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.894+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe828062630 con 0x7fe8280ca360 2024-09-16T01:50:07.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.896+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe828062630 con 0x7fe8280ca360 2024-09-16T01:50:07.913 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:07.914 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:07.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.896+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe828063110 0x7fe82806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.896+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe8280634f0 con 0x7fe828063110 2024-09-16T01:50:07.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.896+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe828063110 0x7fe82806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.897+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe828063110 0x7fe82806cd20 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.898+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8280634f0 con 0x7fe828063110 2024-09-16T01:50:07.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.918+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8280634f0 con 0x7fe828063110 2024-09-16T01:50:07.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.920+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8280634f0 con 0x7fe828063110 2024-09-16T01:50:07.921 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:07.921 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:07.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.920+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe82802c640 0x7fe82802ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.920+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe8280634f0 con 0x7fe82802c640 2024-09-16T01:50:07.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.920+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe82802c640 0x7fe82802ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.921+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe82802c640 0x7fe82802ca60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.922+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8280634f0 con 0x7fe82802c640 2024-09-16T01:50:07.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.935+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe8280634f0 con 0x7fe82802c640 2024-09-16T01:50:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.937+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe8280634f0 con 0x7fe82802c640 2024-09-16T01:50:07.938 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.937+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe82802d4b0 0x7fe828206cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.937+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe82802d910 con 0x7fe82802d4b0 2024-09-16T01:50:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.938+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe82802d4b0 0x7fe828206cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.938+0000 7fe825400640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe82802d4b0 0x7fe828206cf0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.939+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe82802d910 con 0x7fe82802d4b0 2024-09-16T01:50:07.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.955+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe82802d910 con 0x7fe82802d4b0 2024-09-16T01:50:07.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.957+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe82802d910 con 0x7fe82802d4b0 2024-09-16T01:50:07.958 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:07.958 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:07.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.957+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe828207740 0x7fe828209b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.957+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe82820ca80 con 0x7fe828207740 2024-09-16T01:50:07.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.957+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe828207740 0x7fe828209b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.958+0000 7fe826800640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe828207740 0x7fe828209b30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.958+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe82820ca80 con 0x7fe828207740 2024-09-16T01:50:07.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.973+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe828000950 con 0x7fe828207740 2024-09-16T01:50:07.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.975+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe828000950 con 0x7fe828207740 2024-09-16T01:50:07.976 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:07.977 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:07.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.975+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe82820a5d0 0x7fe82820aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:07.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.975+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe82820fe50 con 0x7fe82820a5d0 2024-09-16T01:50:07.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.976+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe82820a5d0 0x7fe82820aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:07.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.976+0000 7fe825e00640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe82820a5d0 0x7fe82820aa30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:07.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.977+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe82820fe50 con 0x7fe82820a5d0 2024-09-16T01:50:07.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.993+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe82820afe0 con 0x7fe82820a5d0 2024-09-16T01:50:07.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.995+0000 7fe81e000640 1 -- 172.21.15.146:0/4115697692 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe82820afe0 con 0x7fe82820a5d0 2024-09-16T01:50:07.996 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.995+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe8281f3f20 msgr2=0x7fe82806e110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.995+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe8281f3f20 0x7fe82806e110 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.995+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe8281e2e20 msgr2=0x7fe82814e6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.995+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe8281e2e20 0x7fe82814e6e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.995+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe8280ca360 msgr2=0x7fe8280ca740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe8280ca360 0x7fe8280ca740 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe828063110 msgr2=0x7fe82806cd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe828063110 0x7fe82806cd20 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe82802c640 msgr2=0x7fe82802ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe82802c640 0x7fe82802ca60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe82802d4b0 msgr2=0x7fe828206cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe82802d4b0 0x7fe828206cf0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe828207740 msgr2=0x7fe828209b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe828207740 0x7fe828209b30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe82820a5d0 msgr2=0x7fe82820aa30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe826800640 1 -- 172.21.15.146:0/4115697692 reap_dead start 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe82820a5d0 0x7fe82820aa30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe826800640 1 -- 172.21.15.146:0/4115697692 reap_dead start 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe826800640 1 -- 172.21.15.146:0/4115697692 reap_dead start 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe826800640 1 -- 172.21.15.146:0/4115697692 reap_dead start 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe80806c2b0 msgr2=0x7fe80806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe80806c2b0 0x7fe80806e770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe818009aa0 tx=0x7fe818009190 comp rx=0 tx=0).stop 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 msgr2=0x7fe8281f03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 0x7fe8281f03f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe814000c00 tx=0x7fe8140046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 shutdown_connections 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe80806c2b0 0x7fe80806e770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 --2- 172.21.15.146:0/4115697692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8280ce4d0 0x7fe8281f03f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 >> 172.21.15.146:0/4115697692 conn(0x7fe82806c000 msgr2=0x7fe8280713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 shutdown_connections 2024-09-16T01:50:08.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:07.996+0000 7fe827200640 1 -- 172.21.15.146:0/4115697692 wait complete. 2024-09-16T01:50:08.001 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:07.999+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:08.021 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.021 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.031 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:08.102 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:08.101+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:08.175 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.175 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.184 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:08.203 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:08.202+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:08.287 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.287 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.296 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:08.304 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:08.303+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:08.401 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.402 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.405 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:08.404+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:08.411 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:08.506 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:08.505+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:08.515 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.515 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:08.606 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:08.604+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:08.629 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.629 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.638 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:08.708 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:08.707+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:08.783 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.783 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.791 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:08.808 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:08.807+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:08.894 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:08.894 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:08.903 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:08.947 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:08.945+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:09.007 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:09.007 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:09.010 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:09.008+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:09.016 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:09.111 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:09.110+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:09.161 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:09.161 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:09.170 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:09.212 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:09.211+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:09.273 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:09.273 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:09.282 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:09.312 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:09.311+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:09.388 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:09.388 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:09.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:09.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:09.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:09.400 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.8634274518061451 2024-09-16T01:50:09.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 0.8634274518061451 2024-09-16T01:50:09.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:50:09.446 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:09.445+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:09.514 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:09.512+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:09.582 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:09.582 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:09.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.587+0000 7efcade00640 1 Processor -- start 2024-09-16T01:50:09.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.587+0000 7efcade00640 1 -- start start 2024-09-16T01:50:09.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efcade00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efca8154ac0 0x7efca815b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efcade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 0x7efca80d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efcade00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efca8154ea0 con 0x7efca80d0d00 2024-09-16T01:50:09.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efcade00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efca815b6d0 con 0x7efca8154ac0 2024-09-16T01:50:09.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efcade00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efca80c5340 con 0x7efca806d820 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efca7e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 0x7efca80d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efcaca00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efca8154ac0 0x7efca815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efca7e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 0x7efca80d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43506/0 (socket says 172.21.15.146:43506) 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.589+0000 7efca7e00640 1 -- 172.21.15.146:0/3245856257 learned_addr learned my addr 172.21.15.146:0/3245856257 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.590+0000 7efca7e00640 1 -- 172.21.15.146:0/3245856257 >> v2:172.21.15.161:3300/0 conn(0x7efca8154ac0 msgr2=0x7efca815b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.590+0000 7efca7e00640 1 --2- 172.21.15.146:0/3245856257 >> v2:172.21.15.161:3300/0 conn(0x7efca8154ac0 0x7efca815b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.590+0000 7efca7e00640 1 -- 172.21.15.146:0/3245856257 >> v1:172.21.15.146:6789/0 conn(0x7efca80d0d00 legacy=0x7efca815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.590+0000 7efca7e00640 1 -- 172.21.15.146:0/3245856257 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca80c5620 con 0x7efca806d820 2024-09-16T01:50:09.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.591+0000 7efca7e00640 1 --2- 172.21.15.146:0/3245856257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 0x7efca80d07c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc94009870 tx=0x7efc9402a310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69e8b082af97821a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:09.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.591+0000 7efca7400640 1 -- 172.21.15.146:0/3245856257 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94037070 con 0x7efca806d820 2024-09-16T01:50:09.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.591+0000 7efca7400640 1 -- 172.21.15.146:0/3245856257 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc94032640 con 0x7efca806d820 2024-09-16T01:50:09.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.591+0000 7efca7400640 1 -- 172.21.15.146:0/3245856257 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9402ae60 con 0x7efca806d820 2024-09-16T01:50:09.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 -- 172.21.15.146:0/3245856257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 msgr2=0x7efca80d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 --2- 172.21.15.146:0/3245856257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 0x7efca80d07c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efc94009870 tx=0x7efc9402a310 comp rx=0 tx=0).stop 2024-09-16T01:50:09.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 -- 172.21.15.146:0/3245856257 shutdown_connections 2024-09-16T01:50:09.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 --2- 172.21.15.146:0/3245856257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca806d820 0x7efca80d07c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 --2- 172.21.15.146:0/3245856257 >> v2:172.21.15.161:3300/0 conn(0x7efca8154ac0 0x7efca815b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 -- 172.21.15.146:0/3245856257 >> 172.21.15.146:0/3245856257 conn(0x7efca806e400 msgr2=0x7efca806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:09.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 -- 172.21.15.146:0/3245856257 shutdown_connections 2024-09-16T01:50:09.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.592+0000 7efcade00640 1 -- 172.21.15.146:0/3245856257 wait complete. 2024-09-16T01:50:09.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.594+0000 7efcade00640 1 Processor -- start 2024-09-16T01:50:09.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.594+0000 7efcade00640 1 -- start start 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.595+0000 7efcade00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efca806d820 0x7efca814a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcade00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca80d0d00 0x7efca81f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca8154ac0 0x7efca80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcade00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efca80c5ec0 con 0x7efca80d0d00 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcade00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efca80c5be0 con 0x7efca806d820 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcade00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efca80c61a0 con 0x7efca8154ac0 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcaca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efca806d820 0x7efca814a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcaca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efca806d820 0x7efca814a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:53460/0 (socket says 172.21.15.146:53460) 2024-09-16T01:50:09.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcad400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca8154ac0 0x7efca80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcaca00640 1 -- 172.21.15.146:0/3393050630 learned_addr learned my addr 172.21.15.146:0/3393050630 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:09.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efca7e00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca80d0d00 0x7efca81f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcad400640 1 -- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efca806d820 msgr2=0x7efca814a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcad400640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efca806d820 0x7efca814a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcad400640 1 -- 172.21.15.146:0/3393050630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca80d0d00 msgr2=0x7efca81f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcad400640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca80d0d00 0x7efca81f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.596+0000 7efcad400640 1 -- 172.21.15.146:0/3393050630 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca80c9ed0 con 0x7efca8154ac0 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.597+0000 7efcad400640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca8154ac0 0x7efca80ccbf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efca000c3f0 tx=0x7efca000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.597+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca00052d0 con 0x7efca8154ac0 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.597+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca0005470 con 0x7efca8154ac0 2024-09-16T01:50:09.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.597+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca80ca1b0 con 0x7efca8154ac0 2024-09-16T01:50:09.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.597+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0007260 con 0x7efca8154ac0 2024-09-16T01:50:09.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.597+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efca81f0420 con 0x7efca8154ac0 2024-09-16T01:50:09.601 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:09.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.599+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca80cad60 con 0x7efca8154ac0 2024-09-16T01:50:09.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.599+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efca0007400 con 0x7efca8154ac0 2024-09-16T01:50:09.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.600+0000 7efca4c00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efc8806c2e0 0x7efc8806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.600+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 5770+0+0 (secure 0 0 0) 0x7efca008a870 con 0x7efca8154ac0 2024-09-16T01:50:09.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.600+0000 7efcaca00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efc8806c2e0 0x7efc8806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.601+0000 7efcaca00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efc8806c2e0 0x7efc8806e7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009a40 tx=0x7efc9c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:09.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.604+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efca00588b0 con 0x7efca8154ac0 2024-09-16T01:50:09.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.640+0000 7f894d200640 1 Processor -- start 2024-09-16T01:50:09.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.640+0000 7f894d200640 1 -- start start 2024-09-16T01:50:09.642 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:09.641+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:09.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f894d200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8948158ea0 0x7f89480c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f894d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480c1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f894d200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f894815fb60 con 0x7f8948073e90 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f894d200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f89480c6b60 con 0x7f8948158ea0 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f894d200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f894806cda0 con 0x7f894815f670 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8948158ea0 0x7f89480c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8946a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8948073e90 0x7f8948162160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:54610/0 (socket says 172.21.15.146:54610) 2024-09-16T01:50:09.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947e00640 1 -- 172.21.15.146:0/263637357 learned_addr learned my addr 172.21.15.146:0/263637357 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:09.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947400640 1 -- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 msgr2=0x7f89480c1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:09.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947400640 1 --2- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480c1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8946000640 1 -- 172.21.15.146:0/263637357 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338926414 0 0) 0x7f894815fb60 con 0x7f8948073e90 2024-09-16T01:50:09.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947400640 1 -- 172.21.15.146:0/263637357 >> v1:172.21.15.161:6789/0 conn(0x7f8948073e90 legacy=0x7f8948162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.642+0000 7f8947400640 1 -- 172.21.15.146:0/263637357 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894806c820 con 0x7f8948158ea0 2024-09-16T01:50:09.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.643+0000 7f8946a00640 1 --2- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480c1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:09.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.644+0000 7f8947400640 1 --2- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948158ea0 0x7f89480c1040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f89380097a0 tx=0x7f893802a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41ded41c10f4ee42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:09.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.644+0000 7f8946000640 1 -- 172.21.15.146:0/263637357 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8938037070 con 0x7f8948158ea0 2024-09-16T01:50:09.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.644+0000 7f8946000640 1 -- 172.21.15.146:0/263637357 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893802ae40 con 0x7f8948158ea0 2024-09-16T01:50:09.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.644+0000 7f8946000640 1 -- 172.21.15.146:0/263637357 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8938032660 con 0x7f8948158ea0 2024-09-16T01:50:09.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 -- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948158ea0 msgr2=0x7f89480c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 --2- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948158ea0 0x7f89480c1040 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f89380097a0 tx=0x7f893802a0f0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 -- 172.21.15.146:0/263637357 shutdown_connections 2024-09-16T01:50:09.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 --2- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480c1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 --2- 172.21.15.146:0/263637357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948158ea0 0x7f89480c1040 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 -- 172.21.15.146:0/263637357 >> 172.21.15.146:0/263637357 conn(0x7f8948067950 msgr2=0x7f8948067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:09.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 -- 172.21.15.146:0/263637357 shutdown_connections 2024-09-16T01:50:09.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.646+0000 7f894d200640 1 -- 172.21.15.146:0/263637357 wait complete. 2024-09-16T01:50:09.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.649+0000 7f894d200640 1 Processor -- start 2024-09-16T01:50:09.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.649+0000 7f894d200640 1 -- start start 2024-09-16T01:50:09.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.650+0000 7f894d200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948073e90 0x7f8948157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.650+0000 7f894d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 0x7f89480ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.650+0000 7f894d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.650+0000 7f894d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89480c7360 con 0x7f8948158ea0 2024-09-16T01:50:09.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.650+0000 7f894d200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f89480c7080 con 0x7f8948073e90 2024-09-16T01:50:09.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.650+0000 7f894d200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8948159280 con 0x7f894815f670 2024-09-16T01:50:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8946a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 0x7f89480ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8946a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 0x7f89480ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55804/0 (socket says 172.21.15.146:55804) 2024-09-16T01:50:09.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8946a00640 1 -- 172.21.15.146:0/3317862378 learned_addr learned my addr 172.21.15.146:0/3317862378 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947e00640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947400640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948073e90 0x7f8948157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947e00640 1 -- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948073e90 msgr2=0x7f8948157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947e00640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948073e90 0x7f8948157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947e00640 1 -- 172.21.15.146:0/3317862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 msgr2=0x7f89480ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947e00640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 0x7f89480ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947e00640 1 -- 172.21.15.146:0/3317862378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89481f5c10 con 0x7f894815f670 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8946a00640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 0x7f89480ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.651+0000 7f8947400640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948073e90 0x7f8948157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.652+0000 7f8947e00640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480ca940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f894000c780 tx=0x7f894000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.652+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940018070 con 0x7f894815f670 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.652+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f894000ce30 con 0x7f894815f670 2024-09-16T01:50:09.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.652+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894000a860 con 0x7f894815f670 2024-09-16T01:50:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.652+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8948155530 con 0x7f894815f670 2024-09-16T01:50:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.652+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89481557d0 con 0x7f894815f670 2024-09-16T01:50:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.653+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f894815fd50 con 0x7f894815f670 2024-09-16T01:50:09.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.656+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f894000aa00 con 0x7f894815f670 2024-09-16T01:50:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.656+0000 7f893f400640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f892806c2b0 0x7f892806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:09.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.656+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 5770+0+0 (secure 0 0 0) 0x7f894008a7c0 con 0x7f894815f670 2024-09-16T01:50:09.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.657+0000 7f8947400640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f892806c2b0 0x7f892806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:09.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.657+0000 7f8947400640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f892806c2b0 0x7f892806e770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8938034000 tx=0x7f893802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:09.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.660+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f894815fd50 con 0x7f894815f670 2024-09-16T01:50:09.716 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:09.714+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:09.784 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:09.784 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:09.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:09.816 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:09.815+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:09.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.843+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 0.8634274518061451} v 0) -- 0x7efca814b490 con 0x7efca8154ac0 2024-09-16T01:50:09.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:09.874+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f894815e540 con 0x7f894815f670 2024-09-16T01:50:09.917 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:09.916+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:10.011 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:10.012 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:10.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:10.019 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:10.018+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:10.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:10.120 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:10.119+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:10.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.177+0000 7f16ae600640 1 Processor -- start 2024-09-16T01:50:10.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.177+0000 7f16ae600640 1 -- start start 2024-09-16T01:50:10.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.178+0000 7f16ae600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16a80d0870 0x7f16a8154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.178+0000 7f16ae600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 0x7f16a8154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.178+0000 7f16ae600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a815e220 con 0x7f16a80d0490 2024-09-16T01:50:10.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.178+0000 7f16ae600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f16a80c6160 con 0x7f16a80d0870 2024-09-16T01:50:10.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.178+0000 7f16ae600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f16a815e500 con 0x7f16a80c6d00 2024-09-16T01:50:10.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 0x7f16a8154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16a7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16a80d0870 0x7f16a8154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 0x7f16a8154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43558/0 (socket says 172.21.15.146:43558) 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 -- 172.21.15.146:0/4080617467 learned_addr learned my addr 172.21.15.146:0/4080617467 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16ad200640 1 --1- 172.21.15.146:0/4080617467 >> v1:172.21.15.146:6789/0 conn(0x7f16a80d0490 0x7f16a80c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52376/0 (socket says 172.21.15.146:52376) 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 -- 172.21.15.146:0/4080617467 >> v2:172.21.15.146:3300/0 conn(0x7f16a80d0870 msgr2=0x7f16a8154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 --2- 172.21.15.146:0/4080617467 >> v2:172.21.15.146:3300/0 conn(0x7f16a80d0870 0x7f16a8154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 -- 172.21.15.146:0/4080617467 >> v1:172.21.15.146:6789/0 conn(0x7f16a80d0490 legacy=0x7f16a80c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.179+0000 7f16adc00640 1 -- 172.21.15.146:0/4080617467 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a815e7e0 con 0x7f16a80c6d00 2024-09-16T01:50:10.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.180+0000 7f16adc00640 1 --2- 172.21.15.146:0/4080617467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 0x7f16a8154bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009e50 tx=0x7f16a002a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=473c2db862cbaeda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.181+0000 7f16a7400640 1 -- 172.21.15.146:0/4080617467 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0036070 con 0x7f16a80c6d00 2024-09-16T01:50:10.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.181+0000 7f16a7400640 1 -- 172.21.15.146:0/4080617467 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a0031600 con 0x7f16a80c6d00 2024-09-16T01:50:10.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.181+0000 7f16a7400640 1 -- 172.21.15.146:0/4080617467 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00319d0 con 0x7f16a80c6d00 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.181+0000 7f16ae600640 1 -- 172.21.15.146:0/4080617467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 msgr2=0x7f16a8154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.181+0000 7f16ae600640 1 --2- 172.21.15.146:0/4080617467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 0x7f16a8154bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009e50 tx=0x7f16a002a7a0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.182+0000 7f16ae600640 1 -- 172.21.15.146:0/4080617467 shutdown_connections 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.182+0000 7f16ae600640 1 --2- 172.21.15.146:0/4080617467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80c6d00 0x7f16a8154bd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.182+0000 7f16ae600640 1 --2- 172.21.15.146:0/4080617467 >> v2:172.21.15.146:3300/0 conn(0x7f16a80d0870 0x7f16a8154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.182+0000 7f16ae600640 1 -- 172.21.15.146:0/4080617467 >> 172.21.15.146:0/4080617467 conn(0x7f16a806d6f0 msgr2=0x7f16a806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.182+0000 7f16ae600640 1 -- 172.21.15.146:0/4080617467 shutdown_connections 2024-09-16T01:50:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.182+0000 7f16ae600640 1 -- 172.21.15.146:0/4080617467 wait complete. 2024-09-16T01:50:10.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.184+0000 7f16ae600640 1 Processor -- start 2024-09-16T01:50:10.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.184+0000 7f16ae600640 1 -- start start 2024-09-16T01:50:10.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16ae600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a80c6d00 0x7f16a81f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16ae600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80d0490 0x7f16a81533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16ae600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a80d0870 0x7f16a8151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16ae600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f16a815dec0 con 0x7f16a80d0870 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16ae600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f16a815eda0 con 0x7f16a80c6d00 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16ae600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f16a80c51f0 con 0x7f16a80d0490 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.185+0000 7f16adc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a80d0870 0x7f16a8151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16adc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a80d0870 0x7f16a8151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55824/0 (socket says 172.21.15.146:55824) 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16adc00640 1 -- 172.21.15.146:0/705965625 learned_addr learned my addr 172.21.15.146:0/705965625 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80d0490 0x7f16a81533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a80c6d00 0x7f16a81f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a80c6d00 msgr2=0x7f16a81f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a80c6d00 0x7f16a81f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a80d0870 msgr2=0x7f16a8151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a80d0870 0x7f16a8151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a80b5120 con 0x7f16a80d0490 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a80d0870 0x7f16a8151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.186+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80d0490 0x7f16a81533e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f169400e840 tx=0x7f169400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.187+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694017070 con 0x7f16a80d0490 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.187+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f169400c730 con 0x7f16a80d0490 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.187+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a81f1530 con 0x7f16a80d0490 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.187+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169400cb00 con 0x7f16a80d0490 2024-09-16T01:50:10.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.187+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a80b49b0 con 0x7f16a80d0490 2024-09-16T01:50:10.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.188+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f16a80c65f0 con 0x7f16a80d0490 2024-09-16T01:50:10.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.189+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f169401a070 con 0x7f16a80d0490 2024-09-16T01:50:10.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.190+0000 7f16a4c00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f168806c2b0 0x7f168806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.190+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 5770+0+0 (secure 0 0 0) 0x7f169408b1b0 con 0x7f16a80d0490 2024-09-16T01:50:10.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.190+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f168806c2b0 0x7f168806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.191+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v37) ==== 54+0+16 (secure 0 0 0) 0x7f169408d750 con 0x7f16a80d0490 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.191+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f16a814eba0 0x7f16a81f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.191+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16a80cc070 con 0x7f16a814eba0 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.191+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f168806c2b0 0x7f168806e770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f169c007fd0 tx=0x7f169c007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.191+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f16a814eba0 0x7f16a81f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.192+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f16a814eba0 0x7f16a81f0640 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.193+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a80cc070 con 0x7f16a814eba0 2024-09-16T01:50:10.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.215+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a80cbd90 con 0x7f16a814eba0 2024-09-16T01:50:10.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.217+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a80cbd90 con 0x7f16a814eba0 2024-09-16T01:50:10.218 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:10.219 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:10.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.217+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16a8151770 0x7f16a80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.217+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f16a81e4ac0 con 0x7f16a8151770 2024-09-16T01:50:10.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.218+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16a8151770 0x7f16a80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.218+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16a8151770 0x7f16a80b2200 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.220 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:10.219+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:10.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.219+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a81e4ac0 con 0x7f16a8151770 2024-09-16T01:50:10.238 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:10.238 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:10.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.241+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a80b3120 con 0x7f16a8151770 2024-09-16T01:50:10.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.243+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a80b3120 con 0x7f16a8151770 2024-09-16T01:50:10.244 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:10.245 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:10.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.243+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f16a81f4060 0x7f16a81f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.243+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f16a81f4980 con 0x7f16a81f4060 2024-09-16T01:50:10.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.244+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f16a81f4060 0x7f16a81f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.244+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f16a81f4060 0x7f16a81f4440 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.245+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a81f4980 con 0x7f16a81f4060 2024-09-16T01:50:10.252 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:10.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.262+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a8062880 con 0x7f16a81f4060 2024-09-16T01:50:10.267 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.264+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a8062880 con 0x7f16a81f4060 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.264+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f16a8063360 0x7f16a81e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.264+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f16a81e7500 con 0x7f16a8063360 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.264+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f16a8063360 0x7f16a81e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.265+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f16a8063360 0x7f16a81e6fc0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.266+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a81e7500 con 0x7f16a8063360 2024-09-16T01:50:10.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.280+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a81e7980 con 0x7f16a8063360 2024-09-16T01:50:10.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.282+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a81e7980 con 0x7f16a8063360 2024-09-16T01:50:10.284 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:10.285 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:10.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.283+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f16a802cb50 0x7f16a802cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.283+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f16a81e7f50 con 0x7f16a802cb50 2024-09-16T01:50:10.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.283+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f16a802cb50 0x7f16a802cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.284+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f16a802cb50 0x7f16a802cfb0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.285+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a81e7f50 con 0x7f16a802cb50 2024-09-16T01:50:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.300+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a802d7c0 con 0x7f16a802cb50 2024-09-16T01:50:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.302+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a802d7c0 con 0x7f16a802cb50 2024-09-16T01:50:10.304 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.303+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f16a8204ed0 0x7f16a82072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.303+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f16a802da00 con 0x7f16a8204ed0 2024-09-16T01:50:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.303+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f16a8204ed0 0x7f16a82072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.304+0000 7f16a7e00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f16a8204ed0 0x7f16a82072c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.304+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a802da00 con 0x7f16a8204ed0 2024-09-16T01:50:10.321 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:10.320+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:10.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.320+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a802da00 con 0x7f16a8204ed0 2024-09-16T01:50:10.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.322+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a802da00 con 0x7f16a8204ed0 2024-09-16T01:50:10.323 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:10.325 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:10.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.322+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16a8207d10 0x7f16a820a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.322+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f16a820cf50 con 0x7f16a8207d10 2024-09-16T01:50:10.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.323+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16a8207d10 0x7f16a820a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.323+0000 7f16adc00640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16a8207d10 0x7f16a820a100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.324+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a820cf50 con 0x7f16a8207d10 2024-09-16T01:50:10.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.340+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a820a6c0 con 0x7f16a8207d10 2024-09-16T01:50:10.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.342+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a820a6c0 con 0x7f16a8207d10 2024-09-16T01:50:10.343 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.342+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f16a820ade0 0x7f16a820b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.342+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f16a82102c0 con 0x7f16a820ade0 2024-09-16T01:50:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.342+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f16a820ade0 0x7f16a820b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.343+0000 7f16ad200640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f16a820ade0 0x7f16a820b220 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.344+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16a82102c0 con 0x7f16a820ade0 2024-09-16T01:50:10.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.361+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f16a820b7d0 con 0x7f16a820ade0 2024-09-16T01:50:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.363+0000 7f16a4c00640 1 -- 172.21.15.146:0/705965625 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16a820b7d0 con 0x7f16a820ade0 2024-09-16T01:50:10.365 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f16a814eba0 msgr2=0x7f16a81f0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f16a814eba0 0x7f16a81f0640 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16a8151770 msgr2=0x7f16a80b2200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16a8151770 0x7f16a80b2200 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f16a81f4060 msgr2=0x7f16a81f4440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f16a81f4060 0x7f16a81f4440 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f16a8063360 msgr2=0x7f16a81e6fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f16a8063360 0x7f16a81e6fc0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f16a802cb50 msgr2=0x7f16a802cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f16a802cb50 0x7f16a802cfb0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f16a8204ed0 msgr2=0x7f16a82072c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f16a8204ed0 0x7f16a82072c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16a8207d10 msgr2=0x7f16a820a100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16a8207d10 0x7f16a820a100 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f16a820ade0 msgr2=0x7f16a820b220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f16a820ade0 0x7f16a820b220 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16adc00640 1 -- 172.21.15.146:0/705965625 reap_dead start 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16adc00640 1 -- 172.21.15.146:0/705965625 reap_dead start 2024-09-16T01:50:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16adc00640 1 -- 172.21.15.146:0/705965625 reap_dead start 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16adc00640 1 -- 172.21.15.146:0/705965625 reap_dead start 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16adc00640 1 -- 172.21.15.146:0/705965625 reap_dead start 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f168806c2b0 msgr2=0x7f168806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f168806c2b0 0x7f168806e770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f169c007fd0 tx=0x7f169c007e80 comp rx=0 tx=0).stop 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80d0490 msgr2=0x7f16a81533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80d0490 0x7f16a81533e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f169400e840 tx=0x7f169400ed00 comp rx=0 tx=0).stop 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 shutdown_connections 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f168806c2b0 0x7f168806e770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 --2- 172.21.15.146:0/705965625 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16a80d0490 0x7f16a81533e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 >> 172.21.15.146:0/705965625 conn(0x7f16a806d6f0 msgr2=0x7f16a80ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.364+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 shutdown_connections 2024-09-16T01:50:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.365+0000 7f16ae600640 1 -- 172.21.15.146:0/705965625 wait complete. 2024-09-16T01:50:10.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.390+0000 7efca4c00640 1 -- 172.21.15.146:0/3393050630 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 0.8634274518061451}]=0 reweighted osd.4 to 0.863427 (dd09) v38) ==== 134+0+0 (secure 0 0 0) 0x7efca005cbb0 con 0x7efca8154ac0 2024-09-16T01:50:10.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.390+0000 7f893f400640 1 -- 172.21.15.146:0/3317862378 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v38) ==== 91+0+0 (secure 0 0 0) 0x7f894815e540 con 0x7f894815f670 2024-09-16T01:50:10.392 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:50:10.392 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.4 to 0.863427 (dd09) 2024-09-16T01:50:10.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.394+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f892806c2b0 msgr2=0x7f892806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.394+0000 7f894d200640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f892806c2b0 0x7f892806e770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8938034000 tx=0x7f893802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:10.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.394+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 msgr2=0x7f89480ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480ca940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f894000c780 tx=0x7f894000cc40 comp rx=0 tx=0).stop 2024-09-16T01:50:10.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efc8806c2e0 msgr2=0x7efc8806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efc8806c2e0 0x7efc8806e7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009a40 tx=0x7efc9c00b040 comp rx=0 tx=0).stop 2024-09-16T01:50:10.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca8154ac0 msgr2=0x7efca80ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca8154ac0 0x7efca80ccbf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efca000c3f0 tx=0x7efca000c8b0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 shutdown_connections 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efc8806c2e0 0x7efc8806e7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efca8154ac0 0x7efca80ccbf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca80d0d00 0x7efca81f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 --2- 172.21.15.146:0/3393050630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efca806d820 0x7efca814a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 >> 172.21.15.146:0/3393050630 conn(0x7efca806e400 msgr2=0x7efca814f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:10.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 shutdown_connections 2024-09-16T01:50:10.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f892806c2b0 0x7f892806e770 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8938034000 tx=0x7f893802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:10.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f894815f670 0x7f89480ca940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8948158ea0 0x7f89480ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 --2- 172.21.15.146:0/3317862378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8948073e90 0x7f8948157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:10.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.395+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 >> 172.21.15.146:0/3317862378 conn(0x7f8948067950 msgr2=0x7f894806bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:10.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.396+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 shutdown_connections 2024-09-16T01:50:10.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.396+0000 7efcade00640 1 -- 172.21.15.146:0/3393050630 wait complete. 2024-09-16T01:50:10.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.397+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 shutdown_connections 2024-09-16T01:50:10.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:10.397+0000 7f894d200640 1 -- 172.21.15.146:0/3317862378 wait complete. 2024-09-16T01:50:10.422 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:10.421+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:10.444 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:10.445 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:10.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:10.522 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:10.521+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:10.623 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:10.622+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:10.671 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.899277744000003, 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:10.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.467724012000005, 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:10.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.688 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.447799790000005, 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:10.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:10.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:10.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.695 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.447779871999998, 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.696 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.447719288000002, 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.702 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:10.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.201433768000001, 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:10.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.201283254000003, 2024-09-16T01:50:10.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:10.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:10.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:10.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.201263686000004, 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:10.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:10.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.631107471999997, 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:10.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:10.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630939654000002, 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:10.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:10.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630920672000002, 2024-09-16T01:50:10.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:10.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:10.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630901041000001, 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:10.745 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:10.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:10.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.751 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630767134999999, 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:10.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.757 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630755597000004, 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:10.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:10.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:10.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:10.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.767 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630737186000005, 2024-09-16T01:50:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.768 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:10.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.630622516000003, 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:10.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:10.782 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.619840788000005, 2024-09-16T01:50:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:10.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:10.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.619724129999994, 2024-09-16T01:50:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:10.795 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.796 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:10.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:10.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:10.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:10.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:10.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.806 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.804190890999998, 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:10.807 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.811 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.804129553000003, 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:10.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:10.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:10.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:10.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:10.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:10.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:10.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:10.823 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:10.825 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:10.823+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:10.826 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:10.926 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:10.924+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:11.026 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:11.025+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:11.033 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:11.034 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:11.034 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:11.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:11.132 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:11.131+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:11.228 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:11.228+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:11.251 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:11.251 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:11.263 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:11.351 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:11.350+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:11.431 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:11.430+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:11.516 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:11.516 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:11.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:11.516 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.688094652999993, 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:11.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:11.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:11.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.687979831999996, 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.524 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:11.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.687937962000007, 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:11.531 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:11.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:11.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.687893682999999, 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:11.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.687864805999993, 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.545 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:11.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.549 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.684157599000002, 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:11.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:11.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:11.554 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.684103617999995, 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:11.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.648688393999997, 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:11.562 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.645628244999997, 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.563 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:11.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:11.567 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.644768796999998, 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.644714903999997, 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.573 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:11.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:11.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:11.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:11.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.638690930999999, 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:11.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:11.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.638640803000001, 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.583 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:11.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:11.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.638564469999999, 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:11.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.638533115000001, 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:11.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.60044766, 2024-09-16T01:50:11.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:11.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.599656129000003, 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.602 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:11.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:11.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.575574304, 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.607 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:11.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:11.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.575502391999997, 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:11.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:11.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.244859992000002, 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:11.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:11.625 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:11.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:11.627 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:11.625+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:11.727 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:11.726+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:11.828 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:11.827+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:11.888 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:11.888 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:11.889 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:11.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:11.955 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:11.954+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:12.030 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:12.029+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:12.089 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:12.089 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:12.089 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:12.089 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:12.089 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:12.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:12.130 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:12.129+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:12.231 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:12.230+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:12.332 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:12.331+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.506256187000005, 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063070110000000004, 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825667+0000", 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825672+0000", 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6920000000000001e-06 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825689+0000", 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7039999999999999e-05 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825696+0000", 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5909999999999998e-06 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826617+0000", 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00092117800000000004 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826697+0000", 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9598999999999995e-05 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826832+0000", 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013548 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829938+0000", 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031059059999999999 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831894+0000", 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019552240000000002 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831942+0000", 2024-09-16T01:50:12.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8389e-05 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831973+0000", 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1043000000000001e-05 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:29 2.b 2:d0630e4c:::mds0_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:12.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.506214639000007, 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087271810000000005, 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825734+0000", 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5638000000000001e-05 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825743+0000", 2024-09-16T01:50:12.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7350000000000002e-06 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825750+0000", 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827333+0000", 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015834689999999999 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827472+0000", 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138796 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827572+0000", 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9910999999999999e-05 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832328+0000", 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047559079999999997 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834353+0000", 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002025654 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834392+0000", 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8099000000000002e-05 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834435+0000", 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3169e-05 2024-09-16T01:50:12.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:34 2.b 2:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.506146377999997, 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088227679999999999, 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825777+0000", 2024-09-16T01:50:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825781+0000", 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0350000000000003e-06 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825788+0000", 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825794+0000", 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4240000000000001e-06 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827917+0000", 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021233570000000002 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828052+0000", 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013528300000000001 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834599+0000", 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065463650000000002 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465648512999998, 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012518068, 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866276+0000", 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866303+0000", 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7229e-05 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866329+0000", 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.563e-05 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866343+0000", 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4287999999999999e-05 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867782+0000", 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014384510000000001 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867911+0000", 2024-09-16T01:50:12.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012957999999999999 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868028+0000", 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117025 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877878+0000", 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098499209999999993 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878708+0000", 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00082986900000000001 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878752+0000", 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3735000000000001e-05 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878792+0000", 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0025999999999999e-05 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.356 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465548233, 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014487969, 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866375+0000", 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9899999999999999e-07 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866379+0000", 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6519999999999997e-06 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866388+0000", 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9579999999999996e-06 2024-09-16T01:50:12.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866395+0000", 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7750000000000004e-06 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868287+0000", 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001892344 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868360+0000", 2024-09-16T01:50:12.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3023999999999997e-05 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868473+0000", 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112321 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878461+0000", 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099884610000000006 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880760+0000", 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022984479999999998 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880823+0000", 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3276999999999997e-05 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880862+0000", 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:7 2.b 2:d70a01c6:::60c.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465518195000001, 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020806099000000002, 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:12.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.82e-07 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866408+0000", 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0110000000000002e-06 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866415+0000", 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5830000000000003e-06 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866421+0000", 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6659999999999996e-06 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022638100000000002 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868842+0000", 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015727699999999999 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868942+0000", 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100182 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878519+0000", 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095771399999999996 2024-09-16T01:50:12.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887083+0000", 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085641750000000003 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887155+0000", 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2117000000000001e-05 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887210+0000", 2024-09-16T01:50:12.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4656e-05 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:8 2.b 2:d70a01c6:::60c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465479803999997, 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020917458, 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.369 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866443+0000", 2024-09-16T01:50:12.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866447+0000", 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8340000000000001e-06 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866455+0000", 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866460+0000", 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5629999999999999e-06 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869260+0000", 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027996039999999998 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869328+0000", 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8730000000000001e-05 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869461+0000", 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264099999999999 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878525+0000", 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090635630000000002 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887259+0000", 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087339310000000003 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887305+0000", 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6538000000000001e-05 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887360+0000", 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4706000000000001e-05 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:10 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.374 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465453222000001, 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01479957, 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8100000000000003e-07 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866473+0000", 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1089999999999998e-06 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866481+0000", 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866486+0000", 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.566e-06 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870818+0000", 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004332519 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870937+0000", 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011880899999999999 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871093+0000", 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015636400000000001 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877793+0000", 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066997560000000003 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881167+0000", 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033735029999999999 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881242+0000", 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5139e-05 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881269+0000", 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6794000000000001e-05 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:27 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465415448999998, 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014854865, 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866511+0000", 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7479999999999999e-06 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866518+0000", 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866525+0000", 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2750000000000001e-06 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875004+0000", 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084799100000000002 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875082+0000", 2024-09-16T01:50:12.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7698e-05 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875190+0000", 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010785000000000001 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877830+0000", 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026396480000000001 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881273+0000", 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034436839999999998 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881341+0000", 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7359000000000003e-05 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881362+0000", 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0928e-05 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.385 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:33 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465390010999997, 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020942512, 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866533+0000", 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.27e-07 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866552+0000", 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8921999999999999e-05 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866559+0000", 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2570000000000001e-06 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866564+0000", 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869714+0000", 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031497920000000002 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869828+0000", 2024-09-16T01:50:12.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114331 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869968+0000", 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013971799999999999 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880969+0000", 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011001031 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887401+0000", 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064320319999999999 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887441+0000", 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9937000000000001e-05 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887475+0000", 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4010999999999997e-05 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:2 2.5 2:a6e36b3f:::401.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465351026999997, 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014499972999999999, 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.392 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866572+0000", 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8400000000000005e-07 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866575+0000", 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0929999999999999e-06 2024-09-16T01:50:12.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866592+0000", 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7765999999999999e-05 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866598+0000", 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4219999999999999e-06 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871180+0000", 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045822939999999998 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871272+0000", 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1308000000000001e-05 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:12.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871419+0000", 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014714 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875993+0000", 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004574486 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881013+0000", 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050202240000000002 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881046+0000", 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2944000000000002e-05 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881071+0000", 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4811999999999999e-05 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.398 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:18 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465303311, 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020977030000000001, 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1799999999999995e-07 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866623+0000", 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4709999999999999e-06 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866630+0000", 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866635+0000", 2024-09-16T01:50:12.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3079999999999998e-06 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870303+0000", 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036679210000000002 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870398+0000", 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4464999999999998e-05 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:12.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870501+0000", 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010291800000000001 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880977+0000", 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010476014000000001 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887510+0000", 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065331759999999999 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887553+0000", 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2737000000000001e-05 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887596+0000", 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3343000000000002e-05 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:22 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465278931, 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01481146, 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866644+0000", 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866646+0000", 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2139999999999999e-06 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866652+0000", 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2160000000000003e-06 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866677+0000", 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4663000000000001e-05 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876491+0000", 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098146110000000009 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876616+0000", 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012430699999999999 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881455+0000", 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048389840000000002 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.408 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:24 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465237643000002, 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016598702999999999, 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5400000000000002e-07 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866688+0000", 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-09-16T01:50:12.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866695+0000", 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.883e-06 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866700+0000", 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7280000000000003e-06 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868221+0000", 2024-09-16T01:50:12.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015209620000000001 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868334+0000", 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113584 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883283+0000", 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014949100999999999 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:25 2.0 2:0250a5a6:::201.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.465203017, 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011987815000000001, 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866720+0000", 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6300000000000001e-07 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866738+0000", 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8218000000000001e-05 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866745+0000", 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7279999999999995e-06 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866751+0000", 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868589+0000", 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018382730000000001 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868709+0000", 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119571 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878707+0000", 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099980020000000006 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.2.11:5 2.0 2:03625d61:::614.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.436493528, 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081102819999999999, 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895434+0000", 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4909999999999999e-06 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895444+0000", 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0560000000000001e-05 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895451+0000", 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5590000000000002e-06 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896385+0000", 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093368200000000002 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896454+0000", 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9567000000000007e-05 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896559+0000", 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104609 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.899222+0000", 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026632000000000001 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903485+0000", 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042627999999999997 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903517+0000", 2024-09-16T01:50:12.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2345999999999998e-05 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903539+0000", 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1838e-05 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:14 2.1a 2:594e21d0:::623.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.420087289000001, 2024-09-16T01:50:12.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068542239999999999, 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911838+0000", 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.435e-06 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911844+0000", 2024-09-16T01:50:12.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.122e-06 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911849+0000", 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6979999999999997e-06 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914481+0000", 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026327249999999998 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914555+0000", 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3886000000000004e-05 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914697+0000", 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014193800000000001 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918088+0000", 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033908559999999998 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918610+0000", 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052154100000000004 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918656+0000", 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-05 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918689+0000", 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3052000000000001e-05 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:23 2.18 2:1c1db442:::103.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.419953743000001, 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067043830000000004, 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911969+0000", 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911980+0000", 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1090999999999999e-05 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911989+0000", 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6270000000000006e-06 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911994+0000", 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3639999999999998e-06 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912099+0000", 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010514900000000001 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912210+0000", 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111346 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912342+0000", 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131197 2024-09-16T01:50:12.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915846+0000", 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035043269999999998 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918573+0000", 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027272289999999999 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918632+0000", 2024-09-16T01:50:12.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8854000000000003e-05 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918673+0000", 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0728e-05 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.395354413, 2024-09-16T01:50:12.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084942400000000001, 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000004e-07 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936594+0000", 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5137999999999999e-05 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936602+0000", 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8999999999999995e-06 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936609+0000", 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9480000000000002e-06 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938373+0000", 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001763857 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938479+0000", 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105816 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938622+0000", 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014314699999999999 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941905+0000", 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032832 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944943+0000", 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030378670000000001 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945019+0000", 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5532999999999997e-05 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945062+0000", 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180999999999999e-05 2024-09-16T01:50:12.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.395288913999998, 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085927369999999996, 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936634+0000", 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936637+0000", 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.287e-06 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936647+0000", 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7189999999999996e-06 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936653+0000", 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.319e-06 2024-09-16T01:50:12.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938867+0000", 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022140079999999999 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938935+0000", 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7644999999999994e-05 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939072+0000", 2024-09-16T01:50:12.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013753600000000001 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941912+0000", 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028393350000000001 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945119+0000", 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032077160000000002 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945183+0000", 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3723e-05 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945226+0000", 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2805000000000001e-05 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:12.444 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:12.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:12.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:50:12.447 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:12.447 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:12.446+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:12.547 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:12.546+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:12.564 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:12.564 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:12.564 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:12.564 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:12.573 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:12.648 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:12.647+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:12.678 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:12.678 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:12.678 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:12.678 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:12.678 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:12.686 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.699+0000 7fda72000640 1 Processor -- start 2024-09-16T01:50:12.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.700+0000 7fda72000640 1 -- start start 2024-09-16T01:50:12.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.701+0000 7fda72000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0490 0x7fda6c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.701+0000 7fda72000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda6c0d0870 0x7fda6c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.701+0000 7fda72000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda6c15e220 con 0x7fda6c154690 2024-09-16T01:50:12.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.701+0000 7fda72000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fda6c0c6160 con 0x7fda6c0d0870 2024-09-16T01:50:12.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.701+0000 7fda72000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c15e500 con 0x7fda6c0d0490 2024-09-16T01:50:12.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.701+0000 7fda6be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda6c0d0870 0x7fda6c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda71600640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fda6c154690 0x7fda6c154ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:54626/0 (socket says 172.21.15.146:54626) 2024-09-16T01:50:12.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda71600640 1 -- 172.21.15.146:0/1186059500 learned_addr learned my addr 172.21.15.146:0/1186059500 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:12.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda70c00640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0490 0x7fda6c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda6be00640 1 -- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0490 msgr2=0x7fda6c0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:12.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda6be00640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0490 0x7fda6c0c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda6be00640 1 -- 172.21.15.146:0/1186059500 >> v1:172.21.15.161:6789/0 conn(0x7fda6c154690 legacy=0x7fda6c154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda6be00640 1 -- 172.21.15.146:0/1186059500 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c15e7e0 con 0x7fda6c0d0870 2024-09-16T01:50:12.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.702+0000 7fda70c00640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0490 0x7fda6c0c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:12.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.703+0000 7fda6be00640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0870 0x7fda6c0c6b30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fda580097a0 tx=0x7fda5802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e254ab92c3926a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.703+0000 7fda6b400640 1 -- 172.21.15.146:0/1186059500 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58037070 con 0x7fda6c0d0870 2024-09-16T01:50:12.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.703+0000 7fda6b400640 1 -- 172.21.15.146:0/1186059500 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda58032440 con 0x7fda6c0d0870 2024-09-16T01:50:12.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.703+0000 7fda6b400640 1 -- 172.21.15.146:0/1186059500 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58032810 con 0x7fda6c0d0870 2024-09-16T01:50:12.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.704+0000 7fda72000640 1 -- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0870 msgr2=0x7fda6c0c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.704+0000 7fda72000640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0870 0x7fda6c0c6b30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fda580097a0 tx=0x7fda5802a230 comp rx=0 tx=0).stop 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.704+0000 7fda72000640 1 -- 172.21.15.146:0/1186059500 shutdown_connections 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.704+0000 7fda72000640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0870 0x7fda6c0c6b30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.704+0000 7fda72000640 1 --2- 172.21.15.146:0/1186059500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0490 0x7fda6c0c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.704+0000 7fda72000640 1 -- 172.21.15.146:0/1186059500 >> 172.21.15.146:0/1186059500 conn(0x7fda6c06d6f0 msgr2=0x7fda6c06e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.705+0000 7fda72000640 1 -- 172.21.15.146:0/1186059500 shutdown_connections 2024-09-16T01:50:12.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.705+0000 7fda72000640 1 -- 172.21.15.146:0/1186059500 wait complete. 2024-09-16T01:50:12.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.707+0000 7fda72000640 1 Processor -- start 2024-09-16T01:50:12.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.707+0000 7fda72000640 1 -- start start 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda72000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 0x7fda6c0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda72000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0870 0x7fda6c0cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda72000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda6c154690 0x7fda6c1503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda72000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c15dec0 con 0x7fda6c154690 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda72000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c15eda0 con 0x7fda6c0d0490 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda72000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c0c51f0 con 0x7fda6c0d0870 2024-09-16T01:50:12.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda70c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 0x7fda6c0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda70c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 0x7fda6c0b5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:53510/0 (socket says 172.21.15.146:53510) 2024-09-16T01:50:12.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda70c00640 1 -- 172.21.15.146:0/3998514486 learned_addr learned my addr 172.21.15.146:0/3998514486 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:12.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0870 0x7fda6c0cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda6c154690 0x7fda6c1503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda70c00640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0870 msgr2=0x7fda6c0cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda6c0d0870 0x7fda6c0cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.709+0000 7fda70c00640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda6c154690 msgr2=0x7fda6c1503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda6c154690 0x7fda6c1503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda70c00640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c1f0b80 con 0x7fda6c0d0490 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda6c154690 0x7fda6c1503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 0x7fda6c0b5fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fda6000ec10 tx=0x7fda6000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60017070 con 0x7fda6c0d0490 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda6000cc30 con 0x7fda6c0d0490 2024-09-16T01:50:12.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.710+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60013470 con 0x7fda6c0d0490 2024-09-16T01:50:12.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.711+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6c153660 con 0x7fda6c0d0490 2024-09-16T01:50:12.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.711+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6c1f0e20 con 0x7fda6c0d0490 2024-09-16T01:50:12.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.712+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fda6c154ac0 con 0x7fda6c0d0490 2024-09-16T01:50:12.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.713+0000 7fa527200640 1 Processor -- start 2024-09-16T01:50:12.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.713+0000 7fa527200640 1 -- start start 2024-09-16T01:50:12.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.714+0000 7fa527200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa52015aa70 0x7fa5200cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.714+0000 7fa527200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52015af00 con 0x7fa5200c6290 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.714+0000 7fa527200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52015b1e0 con 0x7fa5200c5eb0 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.714+0000 7fa527200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa52006dbd0 con 0x7fa52015aa70 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa525e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa5200c5eb0 0x7fa520154910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:54630/0 (socket says 172.21.15.146:54630) 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa525e00640 1 -- 172.21.15.146:0/987749312 learned_addr learned my addr 172.21.15.146:0/987749312 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa525400640 1 --2- 172.21.15.146:0/987749312 >> v2:172.21.15.161:3300/0 conn(0x7fa52015aa70 0x7fa5200cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa524a00640 1 -- 172.21.15.146:0/987749312 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576664499 0 0) 0x7fa52015b1e0 con 0x7fa5200c5eb0 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa524a00640 1 -- 172.21.15.146:0/987749312 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa508003520 con 0x7fa5200c5eb0 2024-09-16T01:50:12.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa525400640 1 -- 172.21.15.146:0/987749312 >> v1:172.21.15.161:6789/0 conn(0x7fa5200c5eb0 legacy=0x7fa520154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa525400640 1 -- 172.21.15.146:0/987749312 >> v1:172.21.15.146:6789/0 conn(0x7fa5200c6290 legacy=0x7fa52006c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:12.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fa525400640 1 -- 172.21.15.146:0/987749312 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52015b4c0 con 0x7fa52015aa70 2024-09-16T01:50:12.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.714+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fda6001a070 con 0x7fda6c0d0490 2024-09-16T01:50:12.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fda68c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fda4c06c2b0 0x7fda4c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.715+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5770+0+0 (secure 0 0 0) 0x7fda6008b540 con 0x7fda6c0d0490 2024-09-16T01:50:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.716+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v39) ==== 54+0+16 (secure 0 0 0) 0x7fda6008b930 con 0x7fda6c0d0490 2024-09-16T01:50:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.716+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fda4c06c2b0 0x7fda4c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.716+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fda6c150930 0x7fda6c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.716+0000 7fa525400640 1 --2- 172.21.15.146:0/987749312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52015aa70 0x7fa5200cac30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa510013db0 tx=0x7fa510032960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9884b5fc7f7f387a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.716+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda6c1e3f70 con 0x7fda6c150930 2024-09-16T01:50:12.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.717+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fda6c150930 0x7fda6c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.717+0000 7fa524a00640 1 -- 172.21.15.146:0/987749312 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51003c020 con 0x7fa52015aa70 2024-09-16T01:50:12.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.717+0000 7fa524a00640 1 -- 172.21.15.146:0/987749312 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa510038820 con 0x7fa52015aa70 2024-09-16T01:50:12.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.717+0000 7fa524a00640 1 -- 172.21.15.146:0/987749312 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa510038bf0 con 0x7fa52015aa70 2024-09-16T01:50:12.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.717+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fda6c150930 0x7fda6c0c65f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 -- 172.21.15.146:0/987749312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52015aa70 msgr2=0x7fa5200cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 --2- 172.21.15.146:0/987749312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52015aa70 0x7fa5200cac30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa510013db0 tx=0x7fa510032960 comp rx=0 tx=0).stop 2024-09-16T01:50:12.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fda4c06c2b0 0x7fda4c06e770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda58034000 tx=0x7fda5802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 -- 172.21.15.146:0/987749312 shutdown_connections 2024-09-16T01:50:12.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 --2- 172.21.15.146:0/987749312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa52015aa70 0x7fa5200cac30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 -- 172.21.15.146:0/987749312 >> 172.21.15.146:0/987749312 conn(0x7fa520067b30 msgr2=0x7fa520067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:12.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 -- 172.21.15.146:0/987749312 shutdown_connections 2024-09-16T01:50:12.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fa527200640 1 -- 172.21.15.146:0/987749312 wait complete. 2024-09-16T01:50:12.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.718+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c1e3f70 con 0x7fda6c150930 2024-09-16T01:50:12.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.720+0000 7fa527200640 1 Processor -- start 2024-09-16T01:50:12.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.720+0000 7fa527200640 1 -- start start 2024-09-16T01:50:12.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa527200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 0x7fa520159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa527200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5200c6290 0x7fa520149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa527200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5201544c0 0x7fa52014a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa527200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa520155020 con 0x7fa5200c6290 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa527200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa52006c370 con 0x7fa5201544c0 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa527200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa5201551a0 con 0x7fa5200c5eb0 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa525e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 0x7fa520159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa525e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 0x7fa520159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43590/0 (socket says 172.21.15.146:43590) 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa525e00640 1 -- 172.21.15.146:0/2928428522 learned_addr learned my addr 172.21.15.146:0/2928428522 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa525400640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5200c6290 0x7fa520149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa525e00640 1 -- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5201544c0 msgr2=0x7fa52014a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.721+0000 7fa526800640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5201544c0 0x7fa52014a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa525e00640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5201544c0 0x7fa52014a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa525e00640 1 -- 172.21.15.146:0/2928428522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5200c6290 msgr2=0x7fa520149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa525e00640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5200c6290 0x7fa520149de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa525e00640 1 -- 172.21.15.146:0/2928428522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52014bf20 con 0x7fa5200c5eb0 2024-09-16T01:50:12.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa525400640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5200c6290 0x7fa520149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa525e00640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 0x7fa520159ac0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa51400cad0 tx=0x7fa51400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa526800640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5201544c0 0x7fa52014a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.722+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514017070 con 0x7fa5200c5eb0 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.723+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa51400ac30 con 0x7fa5200c5eb0 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.723+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514012400 con 0x7fa5200c5eb0 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.723+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5201fdf80 con 0x7fa5200c5eb0 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.723+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52015a000 con 0x7fa5200c5eb0 2024-09-16T01:50:12.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.724+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5200c56f0 con 0x7fa5200c5eb0 2024-09-16T01:50:12.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.725+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa51401a070 con 0x7fa5200c5eb0 2024-09-16T01:50:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.726+0000 7fa51a000640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa50406c2e0 0x7fa50406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.726+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 5770+0+0 (secure 0 0 0) 0x7fa51408b1a0 con 0x7fa5200c5eb0 2024-09-16T01:50:12.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.727+0000 7fa525400640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa50406c2e0 0x7fa50406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.727+0000 7fa525400640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa50406c2e0 0x7fa50406e7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa51003d000 tx=0x7fa510008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.731+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa5140591e0 con 0x7fa5200c5eb0 2024-09-16T01:50:12.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.735+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c150d10 con 0x7fda6c150930 2024-09-16T01:50:12.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.738+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c150d10 con 0x7fda6c150930 2024-09-16T01:50:12.739 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:12.740 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:12.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.738+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fda6c1f3a50 0x7fda6c1f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.738+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fda6c1f4370 con 0x7fda6c1f3a50 2024-09-16T01:50:12.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.738+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fda6c1f3a50 0x7fda6c1f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.739+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fda6c1f3a50 0x7fda6c1f3e30 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.740+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c1f4370 con 0x7fda6c1f3a50 2024-09-16T01:50:12.749 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:12.748+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:12.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.768+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c14fbf0 con 0x7fda6c1f3a50 2024-09-16T01:50:12.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.769+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c14fbf0 con 0x7fda6c1f3a50 2024-09-16T01:50:12.771 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:12.771 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:12.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.770+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fda6c14d520 0x7fda6c14d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.770+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fda6c0ce090 con 0x7fda6c14d520 2024-09-16T01:50:12.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.770+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fda6c14d520 0x7fda6c14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.771+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fda6c14d520 0x7fda6c14d920 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.772+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c0ce090 con 0x7fda6c14d520 2024-09-16T01:50:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.798+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c0ce490 con 0x7fda6c14d520 2024-09-16T01:50:12.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.800+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c0ce490 con 0x7fda6c14d520 2024-09-16T01:50:12.801 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.800+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fda6c062630 0x7fda6c062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.800+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fda6c0ce490 con 0x7fda6c062630 2024-09-16T01:50:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.801+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fda6c062630 0x7fda6c062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.801+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fda6c062630 0x7fda6c062a70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.802+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c0ce490 con 0x7fda6c062630 2024-09-16T01:50:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.816+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c0ce490 con 0x7fda6c062630 2024-09-16T01:50:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.818+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c0ce490 con 0x7fda6c062630 2024-09-16T01:50:12.819 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.818+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fda6c063370 0x7fda6c02cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.818+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fda6c0ce490 con 0x7fda6c063370 2024-09-16T01:50:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.818+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fda6c063370 0x7fda6c02cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.819+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fda6c063370 0x7fda6c02cb50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.819+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c0ce490 con 0x7fda6c063370 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.003688109999999, 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073505619999999997, 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825023+0000", 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1150000000000004e-06 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825036+0000", 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3022e-05 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3779e-05 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3620000000000006e-06 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825161+0000", 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010298600000000001 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825279+0000", 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118087 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825418+0000", 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013899299999999999 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829035+0000", 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036175130000000002 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832264+0000", 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003228908 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832322+0000", 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7751000000000003e-05 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832369+0000", 2024-09-16T01:50:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7046000000000003e-05 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.003601105000001, 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0091091320000000007, 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4900000000000005e-07 2024-09-16T01:50:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825113+0000", 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.229e-06 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825125+0000", 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2493e-05 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825133+0000", 2024-09-16T01:50:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6340000000000001e-06 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825778+0000", 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064556499999999996 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825887+0000", 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108416 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826017+0000", 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013044 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829740+0000", 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037227200000000001 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834119+0000", 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043795930000000002 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834175+0000", 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5492999999999999e-05 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834215+0000", 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9700000000000003e-05 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.003406893000005, 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061249809999999998, 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.34e-07 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825303+0000", 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.96e-06 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825311+0000", 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6000000000000001e-06 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825317+0000", 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6139999999999997e-06 2024-09-16T01:50:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828779+0000", 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034614210000000001 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829036+0000", 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00025713999999999999 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829173+0000", 2024-09-16T01:50:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000136797 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831180+0000", 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020068859999999998 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831366+0000", 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018597399999999999 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831398+0000", 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2518999999999998e-05 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831425+0000", 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6636e-05 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.003381099999999, 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006225062, 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2e-07 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825335+0000", 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4339999999999998e-06 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825342+0000", 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6900000000000003e-06 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825348+0000", 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5470000000000004e-06 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829429+0000", 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040815720000000003 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829541+0000", 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112285 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829647+0000", 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105592 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831435+0000", 2024-09-16T01:50:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017880330000000001 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831474+0000", 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831518+0000", 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3516999999999997e-05 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831551+0000", 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3062000000000003e-05 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.003351385000002, 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062900040000000001, 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825356+0000", 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825359+0000", 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0620000000000001e-06 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825366+0000", 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6899999999999992e-06 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825380+0000", 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3193000000000001e-05 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829978+0000", 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045982879999999999 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830056+0000", 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8466000000000004e-05 2024-09-16T01:50:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830226+0000", 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000169944 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831440+0000", 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012132790000000001 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831590+0000", 2024-09-16T01:50:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015076400000000001 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831621+0000", 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0440999999999999e-05 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831645+0000", 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4383e-05 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.002397907000002, 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080128709999999995, 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826310+0000", 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3200000000000003e-07 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826315+0000", 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6980000000000003e-06 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826328+0000", 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2568999999999999e-05 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826336+0000", 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9670000000000001e-06 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826916+0000", 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058005899999999996 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827058+0000", 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000142533 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827196+0000", 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013736299999999999 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829940+0000", 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027445600000000001 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834254+0000", 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043136210000000001 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834292+0000", 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8266000000000003e-05 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834322+0000", 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9303e-05 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.002351997999995, 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080982150000000006, 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5799999999999999e-07 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826368+0000", 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3152999999999999e-05 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826378+0000", 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826385+0000", 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5440000000000004e-06 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827629+0000", 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001243886 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827855+0000", 2024-09-16T01:50:12.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000226615 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834453+0000", 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006597548 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:14 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 63.968154910999999, 2024-09-16T01:50:12.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013360132, 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6600000000000002e-07 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860560+0000", 2024-09-16T01:50:12.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7919999999999993e-06 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860569+0000", 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3999999999999998e-06 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860576+0000", 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8589999999999999e-06 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860646+0000", 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9676000000000002e-05 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143559 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860917+0000", 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012731900000000001 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862638+0000", 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017212670000000001 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873802+0000", 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011164074 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873866+0000", 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4381999999999994e-05 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873912+0000", 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5337999999999999e-05 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:31 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 63.967966689000001, 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013335217, 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.875 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860741+0000", 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9899999999999999e-07 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860748+0000", 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3250000000000002e-06 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860761+0000", 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2958e-05 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860769+0000", 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2230000000000007e-06 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861199+0000", 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000429178 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861286+0000", 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7436999999999996e-05 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861415+0000", 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012907000000000001 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863136+0000", 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001720627 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873957+0000", 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010821809999999999 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874029+0000", 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1641000000000001e-05 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874075+0000", 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6048999999999999e-05 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:19 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.880 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 63.967922518000002, 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013415967000000001, 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860785+0000", 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6300000000000005e-07 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5279999999999997e-06 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860797+0000", 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.418e-06 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860802+0000", 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.344e-06 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861655+0000", 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085269000000000002 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861767+0000", 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011184900000000001 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861914+0000", 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014756700000000001 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863730+0000", 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001815564 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874127+0000", 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010396924 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874163+0000", 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6285999999999998e-05 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874200+0000", 2024-09-16T01:50:12.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7134000000000001e-05 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:6 2.1f 2:fa28013d:::615.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.932068065999999, 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006133548, 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896640+0000", 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:50:12.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896645+0000", 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6540000000000004e-06 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896655+0000", 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0316000000000001e-05 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896664+0000", 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2090000000000005e-06 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897019+0000", 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035541199999999999 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897088+0000", 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9388999999999996e-05 2024-09-16T01:50:12.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897195+0000", 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010608299999999999 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900354+0000", 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031591919999999999 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902602+0000", 2024-09-16T01:50:12.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002248468 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902659+0000", 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6889999999999999e-05 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902772+0000", 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112972 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.891 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:9 2.6 2:62978f0a:::617.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.931276836000002, 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063594389999999997, 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:12.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897431+0000", 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8899999999999998e-07 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897437+0000", 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3060000000000003e-06 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897447+0000", 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9920000000000002e-06 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897455+0000", 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.627e-06 2024-09-16T01:50:12.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897523+0000", 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7912999999999998e-05 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897652+0000", 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000129259 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897777+0000", 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012468599999999999 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901149+0000", 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033725389999999999 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903683+0000", 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025335539999999999 2024-09-16T01:50:12.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903730+0000", 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7339999999999997e-05 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903789+0000", 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9234000000000002e-05 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:10 2.6 2:62978f0a:::617.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.930983804999997, 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061735380000000001, 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897724+0000", 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0139999999999999e-06 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897730+0000", 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7629999999999997e-06 2024-09-16T01:50:12.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897740+0000", 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0499999999999999e-05 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897748+0000", 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1519999999999992e-06 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898076+0000", 2024-09-16T01:50:12.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00032781800000000001 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898143+0000", 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6905999999999999e-05 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898267+0000", 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123979 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901274+0000", 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00300716 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903796+0000", 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025223210000000001 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903859+0000", 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2638000000000006e-05 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903896+0000", 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7286999999999997e-05 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.93089209, 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060644100000000001, 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.903 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:12.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5099999999999999e-07 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897831+0000", 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5534000000000001e-05 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897850+0000", 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8825999999999999e-05 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897858+0000", 2024-09-16T01:50:12.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3939999999999996e-06 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898448+0000", 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059040600000000003 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898526+0000", 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7600000000000002e-05 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898646+0000", 2024-09-16T01:50:12.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011967400000000001 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903214+0000", 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045682589999999999 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903787+0000", 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00057291199999999997 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903842+0000", 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4747999999999998e-05 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903879+0000", 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7305999999999998e-05 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.908 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:8 2.6 2:6642f7a8:::620.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.916891421000003, 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0071307610000000002, 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911816+0000", 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2299999999999999e-07 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911822+0000", 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0460000000000002e-06 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911846+0000", 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4193999999999999e-05 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911857+0000", 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.042e-05 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912714+0000", 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085704200000000003 2024-09-16T01:50:12.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912790+0000", 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6019999999999994e-05 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912934+0000", 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014454 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916495+0000", 2024-09-16T01:50:12.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035605770000000001 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918865+0000", 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002370451 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918906+0000", 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0123999999999997e-05 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918946+0000", 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:12 2.1f 2:ffe2d39d:::622.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.916448973999998, 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060451970000000004, 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.914 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912259+0000", 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0729999999999999e-06 2024-09-16T01:50:12.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912265+0000", 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4409999999999998e-06 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912276+0000", 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0798000000000001e-05 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912286+0000", 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0176999999999999e-05 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912581+0000", 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00029479199999999999 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912692+0000", 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110874 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913326+0000", 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00063410300000000001 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916590+0000", 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003264543 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918212+0000", 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016214440000000001 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918252+0000", 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0562999999999998e-05 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918303+0000", 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0389000000000002e-05 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:21 2.6 2:62eebe7c:::627.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.919 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.916406573000003, 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0067672629999999999, 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912301+0000", 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1999999999999999e-07 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912305+0000", 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9770000000000002e-06 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912312+0000", 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9920000000000001e-06 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912318+0000", 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9089999999999997e-06 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084597199999999996 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913267+0000", 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103433 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913378+0000", 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011124300000000001 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916553+0000", 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031750900000000002 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918992+0000", 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00243837 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919023+0000", 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1498999999999998e-05 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919067+0000", 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4158e-05 2024-09-16T01:50:12.924 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.916380684000003, 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0068352660000000004, 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8299999999999997e-07 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912336+0000", 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0079e-05 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912345+0000", 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0639999999999994e-06 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912351+0000", 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.686e-06 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913855+0000", 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0015034829999999999 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913971+0000", 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011639 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914107+0000", 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013549999999999999 2024-09-16T01:50:12.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917094+0000", 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002987132 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919106+0000", 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020125389999999998 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919139+0000", 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2670999999999999e-05 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919161+0000", 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2238999999999998e-05 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:9 2.9 2:954a9a2b:::62b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:12.930 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.891136989000003, 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074502429999999996, 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:12.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937571+0000", 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1480000000000001e-06 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937578+0000", 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1890000000000001e-06 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937593+0000", 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4613999999999999e-05 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937602+0000", 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937662+0000", 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9778000000000003e-05 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937799+0000", 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013690299999999999 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937967+0000", 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016826100000000001 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940308+0000", 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023405750000000001 2024-09-16T01:50:12.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944953+0000", 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046456860000000004 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944987+0000", 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3997e-05 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945020+0000", 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2653000000000001e-05 2024-09-16T01:50:12.935 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.891082044000001, 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080442459999999997, 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8999999999999996e-07 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937632+0000", 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7399999999999998e-06 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937640+0000", 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.354e-06 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937647+0000", 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5050000000000004e-06 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939098+0000", 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001451473 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939454+0000", 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035574400000000001 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939603+0000", 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000149228 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940646+0000", 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001042821 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945559+0000", 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049129070000000002 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945623+0000", 2024-09-16T01:50:12.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4072999999999995e-05 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945669+0000", 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5711000000000003e-05 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:12.941 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:12.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.834+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c0ce490 con 0x7fda6c063370 2024-09-16T01:50:12.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.836+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c0ce490 con 0x7fda6c063370 2024-09-16T01:50:12.943 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:12.944 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.836+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fda6c02d5a0 0x7fda6c206b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.836+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fda6c02dae0 con 0x7fda6c02d5a0 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.836+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fda6c02d5a0 0x7fda6c206b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.837+0000 7fda6be00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fda6c02d5a0 0x7fda6c206b50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.838+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c02dae0 con 0x7fda6c02d5a0 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.864+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c000950 con 0x7fda6c02d5a0 2024-09-16T01:50:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.866+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c000950 con 0x7fda6c02d5a0 2024-09-16T01:50:12.946 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:12.946 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:12.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.866+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fda6c2075a0 0x7fda6c2079c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.866+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fda6c20cae0 con 0x7fda6c2075a0 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.866+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fda6c2075a0 0x7fda6c2079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.867+0000 7fda71600640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fda6c2075a0 0x7fda6c2079c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.868+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c20cae0 con 0x7fda6c2075a0 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.891+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c208010 con 0x7fda6c2075a0 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.892+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c208010 con 0x7fda6c2075a0 2024-09-16T01:50:12.947 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:12.948 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:12.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.893+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fda6c208730 0x7fda6c208b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:12.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.893+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fda6c209210 con 0x7fda6c208730 2024-09-16T01:50:12.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.893+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fda6c208730 0x7fda6c208b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:12.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.894+0000 7fda70c00640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fda6c208730 0x7fda6c208b90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:12.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.894+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda6c209210 con 0x7fda6c208730 2024-09-16T01:50:12.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.909+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fda6c2091e0 con 0x7fda6c208730 2024-09-16T01:50:12.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda68c00640 1 -- 172.21.15.146:0/3998514486 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda6c2091e0 con 0x7fda6c208730 2024-09-16T01:50:12.949 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fda6c150930 msgr2=0x7fda6c0c65f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fda6c150930 0x7fda6c0c65f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fda6c1f3a50 msgr2=0x7fda6c1f3e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fda6c1f3a50 0x7fda6c1f3e30 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fda6c14d520 msgr2=0x7fda6c14d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fda6c14d520 0x7fda6c14d920 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fda6c062630 msgr2=0x7fda6c062a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fda6c062630 0x7fda6c062a70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fda6c063370 msgr2=0x7fda6c02cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fda6c063370 0x7fda6c02cb50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fda6c02d5a0 msgr2=0x7fda6c206b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fda6c02d5a0 0x7fda6c206b50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fda6c2075a0 msgr2=0x7fda6c2079c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fda6c2075a0 0x7fda6c2079c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fda6c208730 msgr2=0x7fda6c208b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fda6c208730 0x7fda6c208b90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda71600640 1 -- 172.21.15.146:0/3998514486 reap_dead start 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda71600640 1 -- 172.21.15.146:0/3998514486 reap_dead start 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda71600640 1 -- 172.21.15.146:0/3998514486 reap_dead start 2024-09-16T01:50:12.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda71600640 1 -- 172.21.15.146:0/3998514486 reap_dead start 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda71600640 1 -- 172.21.15.146:0/3998514486 reap_dead start 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fda4c06c2b0 msgr2=0x7fda4c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fda4c06c2b0 0x7fda4c06e770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda58034000 tx=0x7fda5802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 msgr2=0x7fda6c0b5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.911+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 0x7fda6c0b5fa0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fda6000ec10 tx=0x7fda6000c520 comp rx=0 tx=0).stop 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.912+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 shutdown_connections 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.912+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fda4c06c2b0 0x7fda4c06e770 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.912+0000 7fda72000640 1 --2- 172.21.15.146:0/3998514486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda6c0d0490 0x7fda6c0b5fa0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.912+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 >> 172.21.15.146:0/3998514486 conn(0x7fda6c06d6f0 msgr2=0x7fda6c06e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.912+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 shutdown_connections 2024-09-16T01:50:12.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:12.912+0000 7fda72000640 1 -- 172.21.15.146:0/3998514486 wait complete. 2024-09-16T01:50:12.954 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:12.942+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:13.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.008+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fa5200c68d0 con 0x7fa5200c5eb0 2024-09-16T01:50:13.043 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:13.043+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:13.055 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.055 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:13.144 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:13.143+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:13.169 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.169 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.177 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:13.263 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:13.262+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:13.323 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.323 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.332 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:13.346 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:13.345+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:13.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.415+0000 7fa51a000640 1 -- 172.21.15.146:0/2928428522 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v40) ==== 101+0+0 (secure 0 0 0) 0x7fa51405d4e0 con 0x7fa5200c5eb0 2024-09-16T01:50:13.417 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa50406c2e0 msgr2=0x7fa50406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa50406c2e0 0x7fa50406e7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa51003d000 tx=0x7fa510008030 comp rx=0 tx=0).stop 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 msgr2=0x7fa520159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 0x7fa520159ac0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa51400cad0 tx=0x7fa51400cf90 comp rx=0 tx=0).stop 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 shutdown_connections 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa50406c2e0 0x7fa50406e7a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:13.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5201544c0 0x7fa52014a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:13.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5200c6290 0x7fa520149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:13.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 --2- 172.21.15.146:0/2928428522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5200c5eb0 0x7fa520159ac0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:13.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 >> 172.21.15.146:0/2928428522 conn(0x7fa520067b30 msgr2=0x7fa520153620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:13.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 shutdown_connections 2024-09-16T01:50:13.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:13.419+0000 7fa527200640 1 -- 172.21.15.146:0/2928428522 wait complete. 2024-09-16T01:50:13.479 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.480 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.480 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:13.479+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:13.488 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:13.548 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:13.548+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:13.592 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.592 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.601 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:13.650 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:13.649+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:13.706 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.706 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.714 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:13.751 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:13.750+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:13.852 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:13.851+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:13.858 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:13.858 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:13.867 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:13.953 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:13.952+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:14.010 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:14.011 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:14.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:14.054 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:14.053+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:14.123 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:14.124 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:14.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:14.181 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:14.180+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:14.256 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:14.255+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:14.320 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:14.320 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:14.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:14.375 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:14.374+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:14.457 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:14.456+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:14.544 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:14.544 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:14.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:14.558 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:14.557+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:14.659 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:14.658+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:14.760 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:14.758+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:14.769 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:14.770 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:14.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:14.860 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:14.859+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:14.953 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:14.953 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:14.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:14.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:14.966 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:14.964+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:15.062 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:15.061+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:15.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.135+0000 7fd442400640 1 Processor -- start 2024-09-16T01:50:15.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.136+0000 7fd442400640 1 -- start start 2024-09-16T01:50:15.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd442400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd43c0d0490 0x7fd43c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd442400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c0d0870 0x7fd43c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd442400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd43c154690 0x7fd43c154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd442400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd43c0c6160 con 0x7fd43c0d0490 2024-09-16T01:50:15.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd442400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd43c15e220 con 0x7fd43c154690 2024-09-16T01:50:15.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd442400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd43c15e500 con 0x7fd43c0d0870 2024-09-16T01:50:15.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd441000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd43c0d0490 0x7fd43c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd441000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd43c0d0490 0x7fd43c0c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50188/0 (socket says 172.21.15.146:50188) 2024-09-16T01:50:15.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd441000640 1 -- 172.21.15.146:0/354309882 learned_addr learned my addr 172.21.15.146:0/354309882 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:15.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd441a00640 1 --2- 172.21.15.146:0/354309882 >> v2:172.21.15.161:3300/0 conn(0x7fd43c154690 0x7fd43c154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.138+0000 7fd43be00640 1 --2- 172.21.15.146:0/354309882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c0d0870 0x7fd43c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.139+0000 7fd441000640 1 -- 172.21.15.146:0/354309882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c0d0870 msgr2=0x7fd43c0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.139+0000 7fd441000640 1 --2- 172.21.15.146:0/354309882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c0d0870 0x7fd43c0c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.139+0000 7fd441000640 1 -- 172.21.15.146:0/354309882 >> v2:172.21.15.161:3300/0 conn(0x7fd43c154690 msgr2=0x7fd43c154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.139+0000 7fd441000640 1 --2- 172.21.15.146:0/354309882 >> v2:172.21.15.161:3300/0 conn(0x7fd43c154690 0x7fd43c154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.139+0000 7fd441000640 1 -- 172.21.15.146:0/354309882 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43c15e7e0 con 0x7fd43c0d0490 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.140+0000 7fd441000640 1 --2- 172.21.15.146:0/354309882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c0c65f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd43c06df20 tx=0x7fd43002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c6f4260275d754 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.140+0000 7fd43b400640 1 -- 172.21.15.146:0/354309882 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430037070 con 0x7fd43c0d0490 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.140+0000 7fd43b400640 1 -- 172.21.15.146:0/354309882 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd430032440 con 0x7fd43c0d0490 2024-09-16T01:50:15.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.140+0000 7fd43b400640 1 -- 172.21.15.146:0/354309882 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430032810 con 0x7fd43c0d0490 2024-09-16T01:50:15.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 -- 172.21.15.146:0/354309882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 msgr2=0x7fd43c0c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 --2- 172.21.15.146:0/354309882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c0c65f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd43c06df20 tx=0x7fd43002a270 comp rx=0 tx=0).stop 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 -- 172.21.15.146:0/354309882 shutdown_connections 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 --2- 172.21.15.146:0/354309882 >> v2:172.21.15.161:3300/0 conn(0x7fd43c154690 0x7fd43c154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 --2- 172.21.15.146:0/354309882 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c0d0870 0x7fd43c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 --2- 172.21.15.146:0/354309882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c0c65f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 -- 172.21.15.146:0/354309882 >> 172.21.15.146:0/354309882 conn(0x7fd43c06d6f0 msgr2=0x7fd43c06e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.141+0000 7fd442400640 1 -- 172.21.15.146:0/354309882 shutdown_connections 2024-09-16T01:50:15.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.142+0000 7fd442400640 1 -- 172.21.15.146:0/354309882 wait complete. 2024-09-16T01:50:15.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.143+0000 7fd442400640 1 Processor -- start 2024-09-16T01:50:15.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.143+0000 7fd442400640 1 -- start start 2024-09-16T01:50:15.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd442400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd442400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd43c0d0870 0x7fd43c1e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd442400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 0x7fd43c14db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd442400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c155000 con 0x7fd43c0d0490 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd442400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c15eda0 con 0x7fd43c0d0870 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd442400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd43c15dec0 con 0x7fd43c154690 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd441000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd441a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 0x7fd43c14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd441a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 0x7fd43c14db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50474/0 (socket says 172.21.15.146:50474) 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 learned_addr learned my addr 172.21.15.146:0/2124808342 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:15.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.145+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd43c0d0870 0x7fd43c1e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd43c0d0870 msgr2=0x7fd43c1e3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd43c0d0870 0x7fd43c1e3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 msgr2=0x7fd43c1f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c1f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43c1f4570 con 0x7fd43c154690 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd43c0d0490 0x7fd43c1f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:15.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 0x7fd43c14db00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd434009dd0 tx=0x7fd43400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.146+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434018020 con 0x7fd43c154690 2024-09-16T01:50:15.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.147+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43c1f42d0 con 0x7fd43c154690 2024-09-16T01:50:15.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.147+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd434014480 con 0x7fd43c154690 2024-09-16T01:50:15.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.147+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd43c15c680 con 0x7fd43c154690 2024-09-16T01:50:15.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.147+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434014870 con 0x7fd43c154690 2024-09-16T01:50:15.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.148+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd418005270 con 0x7fd43c154690 2024-09-16T01:50:15.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.150+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd434005d10 con 0x7fd43c154690 2024-09-16T01:50:15.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.151+0000 7fd438c00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd41c06c2b0 0x7fd41c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.151+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 5770+0+0 (secure 0 0 0) 0x7fd43408bf50 con 0x7fd43c154690 2024-09-16T01:50:15.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.151+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v40) ==== 54+0+16 (secure 0 0 0) 0x7fd434022070 con 0x7fd43c154690 2024-09-16T01:50:15.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.151+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd418000fa0 0x7fd4180045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.151+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd418005550 con 0x7fd418000fa0 2024-09-16T01:50:15.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.152+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd41c06c2b0 0x7fd41c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.152+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd418000fa0 0x7fd4180045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.152+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd418000fa0 0x7fd4180045e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.153+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd41c06c2b0 0x7fd41c06e770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd430034000 tx=0x7fd43002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.153+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd418005550 con 0x7fd418000fa0 2024-09-16T01:50:15.163 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:15.162+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:15.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.168+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd418004fe0 con 0x7fd418000fa0 2024-09-16T01:50:15.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.170+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd418004fe0 con 0x7fd418000fa0 2024-09-16T01:50:15.171 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:15.171 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:15.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.170+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd418001d00 0x7fd418002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.170+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd41800b390 con 0x7fd418001d00 2024-09-16T01:50:15.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.170+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd418001d00 0x7fd418002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.171+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd418001d00 0x7fd418002990 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.172+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd41800b390 con 0x7fd418001d00 2024-09-16T01:50:15.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.187+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd4180031a0 con 0x7fd418001d00 2024-09-16T01:50:15.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.189+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd4180031a0 con 0x7fd418001d00 2024-09-16T01:50:15.190 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:15.190 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:15.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.189+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd418003c20 0x7fd418009c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.189+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd41800ed80 con 0x7fd418003c20 2024-09-16T01:50:15.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.189+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd418003c20 0x7fd418009c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.190+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd418003c20 0x7fd418009c80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.191+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd41800ed80 con 0x7fd418003c20 2024-09-16T01:50:15.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.215+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd41800a4e0 con 0x7fd418003c20 2024-09-16T01:50:15.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.217+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd41800a4e0 con 0x7fd418003c20 2024-09-16T01:50:15.218 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.217+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd41800af60 0x7fd41800d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.217+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd41800dd40 con 0x7fd41800af60 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.218+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd41800af60 0x7fd41800d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:15.219 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:15.220 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:15.220 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:15.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.218+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd41800af60 0x7fd41800d550 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.219+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd41800dd40 con 0x7fd41800af60 2024-09-16T01:50:15.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:15.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.246+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd41800e250 con 0x7fd41800af60 2024-09-16T01:50:15.251 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:15.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.249+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd41800e250 con 0x7fd41800af60 2024-09-16T01:50:15.251 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:15.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.249+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd41800e6a0 0x7fd418012f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.249+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd418015ed0 con 0x7fd41800e6a0 2024-09-16T01:50:15.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.249+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd41800e6a0 0x7fd418012f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.250+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd41800e6a0 0x7fd418012f50 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.251+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd418015ed0 con 0x7fd41800e6a0 2024-09-16T01:50:15.264 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:15.263+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:15.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.268+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd418015ea0 con 0x7fd41800e6a0 2024-09-16T01:50:15.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.270+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd418015ea0 con 0x7fd41800e6a0 2024-09-16T01:50:15.271 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:15.271 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:15.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.270+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd4180139f0 0x7fd418013e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.270+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd418019320 con 0x7fd4180139f0 2024-09-16T01:50:15.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.270+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd4180139f0 0x7fd418013e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.271+0000 7fd43be00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd4180139f0 0x7fd418013e70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.272+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd418019320 con 0x7fd4180139f0 2024-09-16T01:50:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.298+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd4180144c0 con 0x7fd4180139f0 2024-09-16T01:50:15.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.300+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd4180144c0 con 0x7fd4180139f0 2024-09-16T01:50:15.301 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.301+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd418014be0 0x7fd418015040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.301+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd4180156c0 con 0x7fd418014be0 2024-09-16T01:50:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.301+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd418014be0 0x7fd418015040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.302+0000 7fd441a00640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd418014be0 0x7fd418015040 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.303+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd4180156c0 con 0x7fd418014be0 2024-09-16T01:50:15.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.322+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd41800de00 con 0x7fd418014be0 2024-09-16T01:50:15.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.325+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd41800de00 con 0x7fd418014be0 2024-09-16T01:50:15.326 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:15.326 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:15.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.325+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd41801a6c0 0x7fd41801cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.325+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd41801f880 con 0x7fd41801a6c0 2024-09-16T01:50:15.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.325+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd41801a6c0 0x7fd41801cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.326+0000 7fd441000640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd41801a6c0 0x7fd41801cab0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.327+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd41801f880 con 0x7fd41801a6c0 2024-09-16T01:50:15.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.346+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd41801cff0 con 0x7fd41801a6c0 2024-09-16T01:50:15.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd438c00640 1 -- 172.21.15.146:0/2124808342 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd41801cff0 con 0x7fd41801a6c0 2024-09-16T01:50:15.349 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:15.349 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:15.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd418000fa0 msgr2=0x7fd4180045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd418000fa0 0x7fd4180045e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd418001d00 msgr2=0x7fd418002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fd418001d00 0x7fd418002990 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd418003c20 msgr2=0x7fd418009c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fd418003c20 0x7fd418009c80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd41800af60 msgr2=0x7fd41800d550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd41800af60 0x7fd41800d550 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd41800e6a0 msgr2=0x7fd418012f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fd41800e6a0 0x7fd418012f50 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 reap_dead start 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd4180139f0 msgr2=0x7fd418013e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fd4180139f0 0x7fd418013e70 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 reap_dead start 2024-09-16T01:50:15.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd418014be0 msgr2=0x7fd418015040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fd418014be0 0x7fd418015040 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd41801a6c0 msgr2=0x7fd41801cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fd41801a6c0 0x7fd41801cab0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.348+0000 7fd441a00640 1 -- 172.21.15.146:0/2124808342 reap_dead start 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd41c06c2b0 msgr2=0x7fd41c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd41c06c2b0 0x7fd41c06e770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd430034000 tx=0x7fd43002e000 comp rx=0 tx=0).stop 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 msgr2=0x7fd43c14db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 0x7fd43c14db00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd434009dd0 tx=0x7fd43400c490 comp rx=0 tx=0).stop 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 shutdown_connections 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd41c06c2b0 0x7fd41c06e770 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 --2- 172.21.15.146:0/2124808342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd43c154690 0x7fd43c14db00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 >> 172.21.15.146:0/2124808342 conn(0x7fd43c06d6f0 msgr2=0x7fd43c0c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 shutdown_connections 2024-09-16T01:50:15.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.349+0000 7fd442400640 1 -- 172.21.15.146:0/2124808342 wait complete. 2024-09-16T01:50:15.366 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:15.365+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:15.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:15.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:15.425 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T01:50:15.425 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2024-09-16T01:50:15.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 0 2024-09-16T01:50:15.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:50:15.472 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:15.471+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:15.477 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.477 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.703890845000004, 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:15.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.272337113000006, 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:15.484 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:15.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:15.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:15.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.252412891000006, 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.490 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:15.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:15.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.252392972999999, 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.252332389000003, 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:15.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:15.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:15.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.006046869000002, 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:15.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:15.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.005896355000004, 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.005876787000005, 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435720572999998, 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435552755000003, 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435533773000003, 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435514142000002, 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435380236, 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435368698000005, 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435350287000006, 2024-09-16T01:50:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.435235617000004, 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.424453889000006, 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.424337230999996, 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.608803992000006, 2024-09-16T01:50:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.608742653999997, 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:15.595 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:15.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:15.597 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:15.595+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:15.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.600+0000 7f3822000640 1 Processor -- start 2024-09-16T01:50:15.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.600+0000 7f3822000640 1 -- start start 2024-09-16T01:50:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f3822000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f381c15b190 0x7f381c0d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f3822000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c154ea0 con 0x7f381c154ac0 2024-09-16T01:50:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f3822000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c0c5340 con 0x7f381c0d0d00 2024-09-16T01:50:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f3822000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f381c0c5620 con 0x7f381c15b190 2024-09-16T01:50:15.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f3820c00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f381c154ac0 0x7f381c06d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:58530/0 (socket says 172.21.15.146:58530) 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f3820c00640 1 -- 172.21.15.146:0/847148040 learned_addr learned my addr 172.21.15.146:0/847148040 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.602+0000 7f381be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f381c15b190 0x7f381c0d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266388709 0 0) 0x7f381c0c5340 con 0x7f381c0d0d00 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3804003520 con 0x7f381c0d0d00 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573583723 0 0) 0x7f381c154ea0 con 0x7f381c154ac0 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f381c0c5340 con 0x7f381c154ac0 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381be00640 1 -- 172.21.15.146:0/847148040 >> v1:172.21.15.161:6789/0 conn(0x7f381c0d0d00 legacy=0x7f381c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381be00640 1 -- 172.21.15.146:0/847148040 >> v1:172.21.15.146:6789/0 conn(0x7f381c154ac0 legacy=0x7f381c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.603+0000 7f381be00640 1 -- 172.21.15.146:0/847148040 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c0c5900 con 0x7f381c15b190 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.605+0000 7f381be00640 1 --2- 172.21.15.146:0/847148040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c15b190 0x7f381c0d07c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f38080097a0 tx=0x7f380802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48f98302c655446 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.605+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808037070 con 0x7f381c15b190 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.605+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3808032640 con 0x7f381c15b190 2024-09-16T01:50:15.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.605+0000 7f381b400640 1 -- 172.21.15.146:0/847148040 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808032a10 con 0x7f381c15b190 2024-09-16T01:50:15.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 -- 172.21.15.146:0/847148040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c15b190 msgr2=0x7f381c0d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 --2- 172.21.15.146:0/847148040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c15b190 0x7f381c0d07c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f38080097a0 tx=0x7f380802a2c0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 -- 172.21.15.146:0/847148040 shutdown_connections 2024-09-16T01:50:15.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 --2- 172.21.15.146:0/847148040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c15b190 0x7f381c0d07c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 -- 172.21.15.146:0/847148040 >> 172.21.15.146:0/847148040 conn(0x7f381c06e400 msgr2=0x7f381c06e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 -- 172.21.15.146:0/847148040 shutdown_connections 2024-09-16T01:50:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.606+0000 7f3822000640 1 -- 172.21.15.146:0/847148040 wait complete. 2024-09-16T01:50:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.608+0000 7f3822000640 1 Processor -- start 2024-09-16T01:50:15.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.608+0000 7f3822000640 1 -- start start 2024-09-16T01:50:15.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.609+0000 7f3822000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 0x7f381c1f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.609+0000 7f3822000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c154ac0 0x7f381c14c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.609+0000 7f3822000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f381c15b190 0x7f381c14d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.609+0000 7f3822000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f381c0c61a0 con 0x7f381c154ac0 2024-09-16T01:50:15.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.609+0000 7f3822000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f381c0c5ec0 con 0x7f381c0d0d00 2024-09-16T01:50:15.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.609+0000 7f3822000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f381c0cad60 con 0x7f381c15b190 2024-09-16T01:50:15.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 0x7f381c1f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 0x7f381c1f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43112/0 (socket says 172.21.15.146:43112) 2024-09-16T01:50:15.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 -- 172.21.15.146:0/3483779957 learned_addr learned my addr 172.21.15.146:0/3483779957 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3821600640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f381c15b190 0x7f381c14d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f381be00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c154ac0 0x7f381c14c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 -- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f381c15b190 msgr2=0x7f381c14d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f381c15b190 0x7f381c14d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 -- 172.21.15.146:0/3483779957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c154ac0 msgr2=0x7f381c14c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c154ac0 0x7f381c14c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f3820c00640 1 -- 172.21.15.146:0/3483779957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c1f14f0 con 0x7f381c0d0d00 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.610+0000 7f381be00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c154ac0 0x7f381c14c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:15.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.611+0000 7f3820c00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 0x7f381c1f05a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f381000c430 tx=0x7f381000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.611+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810017070 con 0x7f381c0d0d00 2024-09-16T01:50:15.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.611+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3810005ce0 con 0x7f381c0d0d00 2024-09-16T01:50:15.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.611+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38100072f0 con 0x7f381c0d0d00 2024-09-16T01:50:15.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.612+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c14bec0 con 0x7f381c0d0d00 2024-09-16T01:50:15.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.612+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c1f0f30 con 0x7f381c0d0d00 2024-09-16T01:50:15.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.613+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c15b5e0 con 0x7f381c0d0d00 2024-09-16T01:50:15.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.615+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f381001a070 con 0x7f381c0d0d00 2024-09-16T01:50:15.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.616+0000 7f3818c00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f37fc06c380 0x7f37fc06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.616+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5770+0+0 (secure 0 0 0) 0x7f381008a9c0 con 0x7f381c0d0d00 2024-09-16T01:50:15.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.616+0000 7f381be00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f37fc06c380 0x7f37fc06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.618+0000 7f381be00640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f37fc06c380 0x7f37fc06e840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3808034000 tx=0x7f380802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.620+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f38100589f0 con 0x7f381c0d0d00 2024-09-16T01:50:15.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.685+0000 7f39d5200640 1 Processor -- start 2024-09-16T01:50:15.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.685+0000 7f39d5200640 1 -- start start 2024-09-16T01:50:15.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39d5200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f39d00cc0c0 0x7f39d015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39d5200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 0x7f39d00cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39d5200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d015e9d0 0x7f39d00cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39d5200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f39d00cbd00 con 0x7f39d00cb2f0 2024-09-16T01:50:15.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39d5200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f39d015ba70 con 0x7f39d00cc0c0 2024-09-16T01:50:15.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39d5200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f39d015e540 con 0x7f39d015e9d0 2024-09-16T01:50:15.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39cd400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 0x7f39d00cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39ce800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d015e9d0 0x7f39d00cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39cd400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 0x7f39d00cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50226/0 (socket says 172.21.15.146:50226) 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39cd400640 1 -- 172.21.15.146:0/1248213974 learned_addr learned my addr 172.21.15.146:0/1248213974 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.687+0000 7f39cde00640 1 --2- 172.21.15.146:0/1248213974 >> v2:172.21.15.161:3300/0 conn(0x7f39d00cc0c0 0x7f39d015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.688+0000 7f39cde00640 1 -- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d015e9d0 msgr2=0x7f39d00cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.688+0000 7f39cde00640 1 --2- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d015e9d0 0x7f39d00cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.688+0000 7f39cde00640 1 -- 172.21.15.146:0/1248213974 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 msgr2=0x7f39d00cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.689+0000 7f39cde00640 1 --2- 172.21.15.146:0/1248213974 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 0x7f39d00cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.689+0000 7f39cde00640 1 -- 172.21.15.146:0/1248213974 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d015b1d0 con 0x7f39d00cc0c0 2024-09-16T01:50:15.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.689+0000 7f39cd400640 1 --2- 172.21.15.146:0/1248213974 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 0x7f39d00cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:15.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.690+0000 7f39cde00640 1 --2- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cc0c0 0x7f39d015aab0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ce150 tx=0x7f39c00331d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c070c879578a65 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.691+0000 7f39cca00640 1 -- 172.21.15.146:0/1248213974 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0033b10 con 0x7f39d00cc0c0 2024-09-16T01:50:15.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.691+0000 7f39cca00640 1 -- 172.21.15.146:0/1248213974 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c0033cb0 con 0x7f39d00cc0c0 2024-09-16T01:50:15.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.691+0000 7f39cca00640 1 -- 172.21.15.146:0/1248213974 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0033e60 con 0x7f39d00cc0c0 2024-09-16T01:50:15.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 -- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cc0c0 msgr2=0x7f39d015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 --2- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cc0c0 0x7f39d015aab0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ce150 tx=0x7f39c00331d0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 -- 172.21.15.146:0/1248213974 shutdown_connections 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 --2- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d015e9d0 0x7f39d00cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 --2- 172.21.15.146:0/1248213974 >> v2:172.21.15.146:3300/0 conn(0x7f39d00cb2f0 0x7f39d00cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 --2- 172.21.15.146:0/1248213974 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cc0c0 0x7f39d015aab0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.692+0000 7f39d5200640 1 -- 172.21.15.146:0/1248213974 >> 172.21.15.146:0/1248213974 conn(0x7f39d006d560 msgr2=0x7f39d006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.693+0000 7f39d5200640 1 -- 172.21.15.146:0/1248213974 shutdown_connections 2024-09-16T01:50:15.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.693+0000 7f39d5200640 1 -- 172.21.15.146:0/1248213974 wait complete. 2024-09-16T01:50:15.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.695+0000 7f39d5200640 1 Processor -- start 2024-09-16T01:50:15.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.696+0000 7f39d5200640 1 -- start start 2024-09-16T01:50:15.697 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:15.696+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:15.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.697+0000 7f39d5200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 0x7f39d01ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.697+0000 7f39d5200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d00cc0c0 0x7f39d015d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.697+0000 7f39d5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d015e9d0 0x7f39d00c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.697+0000 7f39d5200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39d015e1d0 con 0x7f39d015e9d0 2024-09-16T01:50:15.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.697+0000 7f39d5200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f39d015b790 con 0x7f39d00cb2f0 2024-09-16T01:50:15.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.697+0000 7f39d5200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f39d00c5560 con 0x7f39d00cc0c0 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.698+0000 7f39cde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 0x7f39d01ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.698+0000 7f39cde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 0x7f39d01ecef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43128/0 (socket says 172.21.15.146:43128) 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.698+0000 7f39cde00640 1 -- 172.21.15.146:0/889355277 learned_addr learned my addr 172.21.15.146:0/889355277 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.698+0000 7f39ce800640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d015e9d0 0x7f39d00c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.698+0000 7f39cd400640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d00cc0c0 0x7f39d015d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39cde00640 1 -- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d00cc0c0 msgr2=0x7f39d015d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39cde00640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d00cc0c0 0x7f39d015d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39cde00640 1 -- 172.21.15.146:0/889355277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d015e9d0 msgr2=0x7f39d00c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39cde00640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d015e9d0 0x7f39d00c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:15.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39cde00640 1 -- 172.21.15.146:0/889355277 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d015dd80 con 0x7f39d00cb2f0 2024-09-16T01:50:15.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39cde00640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 0x7f39d01ecef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f39c0014af0 tx=0x7f39c0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.699+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c004a020 con 0x7f39d00cb2f0 2024-09-16T01:50:15.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.700+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c0045660 con 0x7f39d00cb2f0 2024-09-16T01:50:15.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.700+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0033e60 con 0x7f39d00cb2f0 2024-09-16T01:50:15.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.700+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d015bc70 con 0x7f39d00cb2f0 2024-09-16T01:50:15.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.700+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d015c230 con 0x7f39d00cb2f0 2024-09-16T01:50:15.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.702+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d00c7350 con 0x7f39d00cb2f0 2024-09-16T01:50:15.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.704+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f39c004d020 con 0x7f39d00cb2f0 2024-09-16T01:50:15.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.705+0000 7f39c6000640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39b006c2b0 0x7f39b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:15.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.705+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5770+0+0 (secure 0 0 0) 0x7f39c00b5080 con 0x7f39d00cb2f0 2024-09-16T01:50:15.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.706+0000 7f39cd400640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39b006c2b0 0x7f39b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:15.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.707+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f39d00c7350 con 0x7f39d00cb2f0 2024-09-16T01:50:15.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:15.707+0000 7f39cd400640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39b006c2b0 0x7f39b006e770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f39d0073170 tx=0x7f39bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:15.798 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:15.797+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:15.799 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:15.799 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:15.799 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:15.799 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:15.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:15.899 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:15.898+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:16.000 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:15.999+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:16.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.001+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} v 0) -- 0x7f381c0cb190 con 0x7f381c0d0d00 2024-09-16T01:50:16.035 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:16.035 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:16.036 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:16.036 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:16.036 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:16.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:16.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.058+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f39d0155060 con 0x7f39d00cb2f0 2024-09-16T01:50:16.101 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:16.100+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:16.202 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:16.201+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:16.248 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.420804214, 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.249 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:16.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.420689393000004, 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:16.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:16.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.420647523, 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:16.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:16.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.267 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.420603244000006, 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.268 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:16.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.275 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.420574367, 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:16.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:16.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.416867159999995, 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.280 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:16.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:16.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.416813179000002, 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:16.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:16.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.381397954999997, 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:16.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:16.293 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.378337806000005, 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:16.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.298 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.377478358000005, 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:16.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.377424465000004, 2024-09-16T01:50:16.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:16.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:16.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:16.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.312 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.371400492000006, 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:16.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:16.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:16.317 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.371350363999994, 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:16.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.371274030999999, 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.322 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:16.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.371242675999994, 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:16.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:16.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 55.333157221, 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:16.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 55.332365690000003, 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.337 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:16.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:16.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:16.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.343 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.308283865, 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:16.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:16.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.308211952999997, 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:16.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:16.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:16.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.977569553000002, 2024-09-16T01:50:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:16.362 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:16.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:16.364 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:16.362+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:16.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.424+0000 7f3818c00640 1 -- 172.21.15.146:0/3483779957 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]=0 set osd.7 primary-affinity to 0 (802) v41) ==== 129+0+0 (secure 0 0 0) 0x7f381005ccf0 con 0x7f381c0d0d00 2024-09-16T01:50:16.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.424+0000 7f39c6000640 1 -- 172.21.15.146:0/889355277 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v41) ==== 87+0+0 (secure 0 0 0) 0x7f39c0054030 con 0x7f39d00cb2f0 2024-09-16T01:50:16.426 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.7 primary-affinity to 0 (802) 2024-09-16T01:50:16.426 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:50:16.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39b006c2b0 msgr2=0x7f39b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f37fc06c380 msgr2=0x7f37fc06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f3822000640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f37fc06c380 0x7f37fc06e840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3808034000 tx=0x7f380802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 msgr2=0x7f381c1f05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f3822000640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 0x7f381c1f05a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f381000c430 tx=0x7f381000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f39d5200640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39b006c2b0 0x7f39b006e770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f39d0073170 tx=0x7f39bc008040 comp rx=0 tx=0).stop 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 msgr2=0x7f39d01ecef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.428+0000 7f39d5200640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 0x7f39d01ecef0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f39c0014af0 tx=0x7f39c0004a40 comp rx=0 tx=0).stop 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 shutdown_connections 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f3822000640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f37fc06c380 0x7f37fc06e840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f3822000640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f381c15b190 0x7f381c14d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f3822000640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c154ac0 0x7f381c14c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f3822000640 1 --2- 172.21.15.146:0/3483779957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f381c0d0d00 0x7f381c1f05a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 >> 172.21.15.146:0/3483779957 conn(0x7f381c06e400 msgr2=0x7f381c150d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:16.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.430+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 shutdown_connections 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 shutdown_connections 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f39d5200640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f39b006c2b0 0x7f39b006e770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f39d5200640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39d015e9d0 0x7f39d00c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f39d5200640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39d00cc0c0 0x7f39d015d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f39d5200640 1 --2- 172.21.15.146:0/889355277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39d00cb2f0 0x7f39d01ecef0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.429+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 >> 172.21.15.146:0/889355277 conn(0x7f39d006d560 msgr2=0x7f39d00cf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.431+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 shutdown_connections 2024-09-16T01:50:16.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.431+0000 7f39d5200640 1 -- 172.21.15.146:0/889355277 wait complete. 2024-09-16T01:50:16.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:16.431+0000 7f3822000640 1 -- 172.21.15.146:0/3483779957 wait complete. 2024-09-16T01:50:16.500 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:16.499+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:16.560 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:16.561 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:16.561 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:16.561 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:16.564 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:16.563+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:16.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:16.665 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:16.664+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:16.766 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:16.765+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:16.825 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:16.826 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:16.826 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:16.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:16.884 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:16.883+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:16.968 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:16.967+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:17.069 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:17.068+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:17.083 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.254560143000006, 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063070110000000004, 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825667+0000", 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825672+0000", 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6920000000000001e-06 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825689+0000", 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7039999999999999e-05 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825696+0000", 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5909999999999998e-06 2024-09-16T01:50:17.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826617+0000", 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00092117800000000004 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826697+0000", 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9598999999999995e-05 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826832+0000", 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013548 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829938+0000", 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031059059999999999 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831894+0000", 2024-09-16T01:50:17.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019552240000000002 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831942+0000", 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8389e-05 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831973+0000", 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1043000000000001e-05 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:29 2.b 2:d0630e4c:::mds0_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.254518594999993, 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087271810000000005, 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825734+0000", 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5638000000000001e-05 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825743+0000", 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7350000000000002e-06 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825750+0000", 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:17.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827333+0000", 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015834689999999999 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827472+0000", 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138796 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827572+0000", 2024-09-16T01:50:17.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9910999999999999e-05 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832328+0000", 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047559079999999997 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834353+0000", 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002025654 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834392+0000", 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8099000000000002e-05 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834435+0000", 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3169e-05 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:34 2.b 2:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.254450333999998, 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088227679999999999, 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825777+0000", 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825781+0000", 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0350000000000003e-06 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825788+0000", 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825794+0000", 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4240000000000001e-06 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827917+0000", 2024-09-16T01:50:17.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021233570000000002 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828052+0000", 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013528300000000001 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834599+0000", 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065463650000000002 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213952469000006, 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012518068, 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866276+0000", 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866303+0000", 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7229e-05 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866329+0000", 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.563e-05 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866343+0000", 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4287999999999999e-05 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867782+0000", 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014384510000000001 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867911+0000", 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012957999999999999 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868028+0000", 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117025 2024-09-16T01:50:17.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877878+0000", 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098499209999999993 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878708+0000", 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00082986900000000001 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878752+0000", 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3735000000000001e-05 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878792+0000", 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0025999999999999e-05 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:17.106 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213852188999994, 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014487969, 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:17.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866375+0000", 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9899999999999999e-07 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866379+0000", 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6519999999999997e-06 2024-09-16T01:50:17.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866388+0000", 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9579999999999996e-06 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866395+0000", 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7750000000000004e-06 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868287+0000", 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001892344 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868360+0000", 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3023999999999997e-05 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868473+0000", 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112321 2024-09-16T01:50:17.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878461+0000", 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099884610000000006 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880760+0000", 2024-09-16T01:50:17.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022984479999999998 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880823+0000", 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3276999999999997e-05 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880862+0000", 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:7 2.b 2:d70a01c6:::60c.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213822151000002, 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020806099000000002, 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.82e-07 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866408+0000", 2024-09-16T01:50:17.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0110000000000002e-06 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866415+0000", 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5830000000000003e-06 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866421+0000", 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6659999999999996e-06 2024-09-16T01:50:17.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022638100000000002 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868842+0000", 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015727699999999999 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868942+0000", 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100182 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878519+0000", 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095771399999999996 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887083+0000", 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085641750000000003 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887155+0000", 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2117000000000001e-05 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887210+0000", 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4656e-05 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:8 2.b 2:d70a01c6:::60c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213783759999998, 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020917458, 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866443+0000", 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866447+0000", 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8340000000000001e-06 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866455+0000", 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866460+0000", 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5629999999999999e-06 2024-09-16T01:50:17.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869260+0000", 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027996039999999998 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869328+0000", 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8730000000000001e-05 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:17.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869461+0000", 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264099999999999 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878525+0000", 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090635630000000002 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887259+0000", 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087339310000000003 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887305+0000", 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6538000000000001e-05 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887360+0000", 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4706000000000001e-05 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:10 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.126 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213757177999994, 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01479957, 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8100000000000003e-07 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866473+0000", 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1089999999999998e-06 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866481+0000", 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866486+0000", 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.566e-06 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870818+0000", 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004332519 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870937+0000", 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011880899999999999 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871093+0000", 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015636400000000001 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877793+0000", 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066997560000000003 2024-09-16T01:50:17.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881167+0000", 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033735029999999999 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881242+0000", 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5139e-05 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881269+0000", 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6794000000000001e-05 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:27 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213719405000006, 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014854865, 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866511+0000", 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7479999999999999e-06 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866518+0000", 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866525+0000", 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2750000000000001e-06 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875004+0000", 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084799100000000002 2024-09-16T01:50:17.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875082+0000", 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7698e-05 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875190+0000", 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010785000000000001 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877830+0000", 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026396480000000001 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881273+0000", 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034436839999999998 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881341+0000", 2024-09-16T01:50:17.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7359000000000003e-05 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881362+0000", 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0928e-05 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:33 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213693966999998, 2024-09-16T01:50:17.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020942512, 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866533+0000", 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.27e-07 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866552+0000", 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8921999999999999e-05 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866559+0000", 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2570000000000001e-06 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866564+0000", 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869714+0000", 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031497920000000002 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869828+0000", 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114331 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869968+0000", 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013971799999999999 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880969+0000", 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011001031 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887401+0000", 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064320319999999999 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887441+0000", 2024-09-16T01:50:17.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9937000000000001e-05 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887475+0000", 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4010999999999997e-05 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:2 2.5 2:a6e36b3f:::401.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213654982999998, 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014499972999999999, 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866572+0000", 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8400000000000005e-07 2024-09-16T01:50:17.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866575+0000", 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0929999999999999e-06 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866592+0000", 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7765999999999999e-05 2024-09-16T01:50:17.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866598+0000", 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4219999999999999e-06 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871180+0000", 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045822939999999998 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871272+0000", 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1308000000000001e-05 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871419+0000", 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014714 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875993+0000", 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004574486 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881013+0000", 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050202240000000002 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881046+0000", 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2944000000000002e-05 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881071+0000", 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4811999999999999e-05 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.151 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:18 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213607267, 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020977030000000001, 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1799999999999995e-07 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866623+0000", 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4709999999999999e-06 2024-09-16T01:50:17.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866630+0000", 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866635+0000", 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3079999999999998e-06 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870303+0000", 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036679210000000002 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870398+0000", 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4464999999999998e-05 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870501+0000", 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010291800000000001 2024-09-16T01:50:17.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880977+0000", 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010476014000000001 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887510+0000", 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065331759999999999 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887553+0000", 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2737000000000001e-05 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887596+0000", 2024-09-16T01:50:17.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3343000000000002e-05 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:22 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:17.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213582887000001, 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01481146, 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:17.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866644+0000", 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-09-16T01:50:17.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866646+0000", 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2139999999999999e-06 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866652+0000", 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2160000000000003e-06 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866677+0000", 2024-09-16T01:50:17.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4663000000000001e-05 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876491+0000", 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098146110000000009 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876616+0000", 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012430699999999999 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881455+0000", 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048389840000000002 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:24 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.163 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213541598999996, 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016598702999999999, 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5400000000000002e-07 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866688+0000", 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866695+0000", 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.883e-06 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866700+0000", 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7280000000000003e-06 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868221+0000", 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015209620000000001 2024-09-16T01:50:17.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868334+0000", 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113584 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883283+0000", 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014949100999999999 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:25 2.0 2:0250a5a6:::201.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.213506972999994, 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011987815000000001, 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866720+0000", 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6300000000000001e-07 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866738+0000", 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8218000000000001e-05 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866745+0000", 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7279999999999995e-06 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866751+0000", 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868589+0000", 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018382730000000001 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868709+0000", 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119571 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878707+0000", 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099980020000000006 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.2.11:5 2.0 2:03625d61:::614.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.184797484000001, 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081102819999999999, 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895434+0000", 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4909999999999999e-06 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895444+0000", 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0560000000000001e-05 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895451+0000", 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5590000000000002e-06 2024-09-16T01:50:17.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896385+0000", 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093368200000000002 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896454+0000", 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9567000000000007e-05 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896559+0000", 2024-09-16T01:50:17.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104609 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.899222+0000", 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026632000000000001 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903485+0000", 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042627999999999997 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903517+0000", 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2345999999999998e-05 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903539+0000", 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1838e-05 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:14 2.1a 2:594e21d0:::623.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.168391245000002, 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068542239999999999, 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911838+0000", 2024-09-16T01:50:17.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.435e-06 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911844+0000", 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.122e-06 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911849+0000", 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6979999999999997e-06 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914481+0000", 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026327249999999998 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914555+0000", 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3886000000000004e-05 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914697+0000", 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014193800000000001 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918088+0000", 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033908559999999998 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918610+0000", 2024-09-16T01:50:17.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052154100000000004 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918656+0000", 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-05 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918689+0000", 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3052000000000001e-05 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:23 2.18 2:1c1db442:::103.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.168257699000002, 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067043830000000004, 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911969+0000", 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911980+0000", 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1090999999999999e-05 2024-09-16T01:50:17.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911989+0000", 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6270000000000006e-06 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911994+0000", 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3639999999999998e-06 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912099+0000", 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010514900000000001 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912210+0000", 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111346 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:17.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912342+0000", 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131197 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915846+0000", 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035043269999999998 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918573+0000", 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027272289999999999 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918632+0000", 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8854000000000003e-05 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918673+0000", 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0728e-05 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.143658369000001, 2024-09-16T01:50:17.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084942400000000001, 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000004e-07 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936594+0000", 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5137999999999999e-05 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936602+0000", 2024-09-16T01:50:17.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8999999999999995e-06 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936609+0000", 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9480000000000002e-06 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938373+0000", 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001763857 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938479+0000", 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105816 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938622+0000", 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014314699999999999 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941905+0000", 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032832 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944943+0000", 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030378670000000001 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945019+0000", 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5532999999999997e-05 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945062+0000", 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180999999999999e-05 2024-09-16T01:50:17.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.143592869999999, 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085927369999999996, 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:17.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936634+0000", 2024-09-16T01:50:17.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936637+0000", 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.287e-06 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936647+0000", 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7189999999999996e-06 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936653+0000", 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.319e-06 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938867+0000", 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022140079999999999 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938935+0000", 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7644999999999994e-05 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939072+0000", 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013753600000000001 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941912+0000", 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028393350000000001 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945119+0000", 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032077160000000002 2024-09-16T01:50:17.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945183+0000", 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3723e-05 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945226+0000", 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2805000000000001e-05 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:17.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:17.204 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:17.205 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:17.206 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:17.204+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:17.306 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:17.305+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:17.321 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:17.321 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:17.321 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:17.331 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:17.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:17.415 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:17.414+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:17.479 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:17.479 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:17.479 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:17.479 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:17.479 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:17.487 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:17.508 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:17.507+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:17.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.528+0000 7ff0f5600640 1 Processor -- start 2024-09-16T01:50:17.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.528+0000 7ff0f5600640 1 -- start start 2024-09-16T01:50:17.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.529+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00c6d00 0x7ff0f00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.529+0000 7ff0f5600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f015e220 con 0x7ff0f00d0490 2024-09-16T01:50:17.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0f5600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f015e500 con 0x7ff0f00d0870 2024-09-16T01:50:17.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0f5600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f00c6160 con 0x7ff0f00c6d00 2024-09-16T01:50:17.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0ef400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff0f00d0870 0x7ff0f0154690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:53098/0 (socket says 172.21.15.146:53098) 2024-09-16T01:50:17.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0ef400640 1 -- 172.21.15.146:0/627263433 learned_addr learned my addr 172.21.15.146:0/627263433 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0efe00640 1 --1- 172.21.15.146:0/627263433 >> v1:172.21.15.146:6789/0 conn(0x7ff0f00d0490 0x7ff0f00c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:58546/0 (socket says 172.21.15.146:58546) 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/627263433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00c6d00 0x7ff0f00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1226254686 0 0) 0x7ff0f015e500 con 0x7ff0f00d0870 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0d4003520 con 0x7ff0f00d0870 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109144556 0 0) 0x7ff0f015e220 con 0x7ff0f00d0490 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f015e500 con 0x7ff0f00d0490 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0f4c00640 1 -- 172.21.15.146:0/627263433 >> v1:172.21.15.161:6789/0 conn(0x7ff0f00d0870 legacy=0x7ff0f0154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0f4c00640 1 -- 172.21.15.146:0/627263433 >> v1:172.21.15.146:6789/0 conn(0x7ff0f00d0490 legacy=0x7ff0f00c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.530+0000 7ff0f4c00640 1 -- 172.21.15.146:0/627263433 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f015e7e0 con 0x7ff0f00c6d00 2024-09-16T01:50:17.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.531+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/627263433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00c6d00 0x7ff0f00c51f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009dc0 tx=0x7ff0e802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbd5cd4473dad549 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8007740 con 0x7ff0f00c6d00 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e80078e0 con 0x7ff0f00c6d00 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0eea00640 1 -- 172.21.15.146:0/627263433 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8007cb0 con 0x7ff0f00c6d00 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 -- 172.21.15.146:0/627263433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00c6d00 msgr2=0x7ff0f00c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 --2- 172.21.15.146:0/627263433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00c6d00 0x7ff0f00c51f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009dc0 tx=0x7ff0e802a760 comp rx=0 tx=0).stop 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 -- 172.21.15.146:0/627263433 shutdown_connections 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 --2- 172.21.15.146:0/627263433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00c6d00 0x7ff0f00c51f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 -- 172.21.15.146:0/627263433 >> 172.21.15.146:0/627263433 conn(0x7ff0f006d6f0 msgr2=0x7ff0f006e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 -- 172.21.15.146:0/627263433 shutdown_connections 2024-09-16T01:50:17.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.532+0000 7ff0f5600640 1 -- 172.21.15.146:0/627263433 wait complete. 2024-09-16T01:50:17.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.535+0000 7ff0f5600640 1 Processor -- start 2024-09-16T01:50:17.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.535+0000 7ff0f5600640 1 -- start start 2024-09-16T01:50:17.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0f00c6d00 0x7ff0f01576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0f00d0490 0x7ff0f01e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 0x7ff0f00b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f5600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f015dec0 con 0x7ff0f00c6d00 2024-09-16T01:50:17.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f5600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f015eda0 con 0x7ff0f00d0490 2024-09-16T01:50:17.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f5600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f0154da0 con 0x7ff0f00d0870 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 0x7ff0f00b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.536+0000 7ff0efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0f00c6d00 0x7ff0f01576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 0x7ff0f00b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50522/0 (socket says 172.21.15.146:50522) 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 learned_addr learned my addr 172.21.15.146:0/3516552402 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0f00d0490 0x7ff0f01e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0f00d0490 msgr2=0x7ff0f01e3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff0f00d0490 0x7ff0f01e3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0f00c6d00 msgr2=0x7ff0f01576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0f00c6d00 0x7ff0f01576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f01f0620 con 0x7ff0f00d0870 2024-09-16T01:50:17.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.537+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 0x7ff0f00b3300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009ee0 tx=0x7ff0e8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.538+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8040070 con 0x7ff0f00d0870 2024-09-16T01:50:17.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.538+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8031bc0 con 0x7ff0f00d0870 2024-09-16T01:50:17.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.538+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8009450 con 0x7ff0f00d0870 2024-09-16T01:50:17.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.538+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f01e5720 con 0x7ff0f00d0870 2024-09-16T01:50:17.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.538+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f01e5ce0 con 0x7ff0f00d0870 2024-09-16T01:50:17.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.539+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff0f00c65f0 con 0x7ff0f00d0870 2024-09-16T01:50:17.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.541+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff0e8039040 con 0x7ff0f00d0870 2024-09-16T01:50:17.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.542+0000 7ff0e7e00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff0d006c300 0x7ff0d006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.542+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(42..42 src has 1..42) ==== 5770+0+0 (secure 0 0 0) 0x7ff0e80aca40 con 0x7ff0f00d0870 2024-09-16T01:50:17.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.543+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff0d006c300 0x7ff0d006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.543+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v42) ==== 54+0+16 (secure 0 0 0) 0x7ff0e80b1220 con 0x7ff0f00d0870 2024-09-16T01:50:17.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.543+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff0f00cb290 0x7ff0f01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.543+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0f0154500 con 0x7ff0f00cb290 2024-09-16T01:50:17.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.543+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff0d006c300 0x7ff0d006e7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0000f60 tx=0x7ff0e0007cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.544+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff0f00cb290 0x7ff0f01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.544+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff0f00cb290 0x7ff0f01564c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.545+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f0154500 con 0x7ff0f00cb290 2024-09-16T01:50:17.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.562+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f00cc950 con 0x7ff0f00cb290 2024-09-16T01:50:17.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.563+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f00cc950 con 0x7ff0f00cb290 2024-09-16T01:50:17.564 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:17.566 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:17.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.564+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff0f00cb670 0x7ff0f01f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.564+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff0f01e64e0 con 0x7ff0f00cb670 2024-09-16T01:50:17.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.564+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff0f00cb670 0x7ff0f01f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.565+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff0f00cb670 0x7ff0f01f37e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.565+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f01e64e0 con 0x7ff0f00cb670 2024-09-16T01:50:17.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.580+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f0156f10 con 0x7ff0f00cb670 2024-09-16T01:50:17.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.582+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f0156f10 con 0x7ff0f00cb670 2024-09-16T01:50:17.585 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.582+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff0f01f4230 0x7ff0f0062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.582+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff0f00b6260 con 0x7ff0f01f4230 2024-09-16T01:50:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.582+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff0f01f4230 0x7ff0f0062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.583+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff0f01f4230 0x7ff0f0062630 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.584+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f00b6260 con 0x7ff0f01f4230 2024-09-16T01:50:17.608 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:17.607+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:17.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.610+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f0062e40 con 0x7ff0f01f4230 2024-09-16T01:50:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.612+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f0062e40 con 0x7ff0f01f4230 2024-09-16T01:50:17.613 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.612+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff0f0152630 0x7ff0f0152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.612+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff0f0062dc0 con 0x7ff0f0152630 2024-09-16T01:50:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.613+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff0f0152630 0x7ff0f0152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.613+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff0f0152630 0x7ff0f0152a50 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.614+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f0062dc0 con 0x7ff0f0152630 2024-09-16T01:50:17.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.628+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f0062dc0 con 0x7ff0f0152630 2024-09-16T01:50:17.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.630+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f0062dc0 con 0x7ff0f0152630 2024-09-16T01:50:17.631 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:17.632 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:17.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.631+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff0f002cb50 0x7ff0f002cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.631+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff0f0062dc0 con 0x7ff0f002cb50 2024-09-16T01:50:17.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.631+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff0f002cb50 0x7ff0f002cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.632+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff0f002cb50 0x7ff0f002cf70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.633+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f0062dc0 con 0x7ff0f002cb50 2024-09-16T01:50:17.634 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:17.634 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.806233594000005, 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073505619999999997, 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825023+0000", 2024-09-16T01:50:17.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1150000000000004e-06 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825036+0000", 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3022e-05 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:50:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3779e-05 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3620000000000006e-06 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825161+0000", 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010298600000000001 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825279+0000", 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118087 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825418+0000", 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013899299999999999 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829035+0000", 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036175130000000002 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832264+0000", 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003228908 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832322+0000", 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7751000000000003e-05 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832369+0000", 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7046000000000003e-05 2024-09-16T01:50:17.640 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.806146588999994, 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0091091320000000007, 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:17.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4900000000000005e-07 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825113+0000", 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.229e-06 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825125+0000", 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2493e-05 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825133+0000", 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6340000000000001e-06 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825778+0000", 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064556499999999996 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825887+0000", 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108416 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826017+0000", 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013044 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829740+0000", 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037227200000000001 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834119+0000", 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043795930000000002 2024-09-16T01:50:17.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834175+0000", 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5492999999999999e-05 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834215+0000", 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9700000000000003e-05 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.805952376999997, 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061249809999999998, 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.34e-07 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825303+0000", 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.96e-06 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825311+0000", 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6000000000000001e-06 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825317+0000", 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6139999999999997e-06 2024-09-16T01:50:17.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828779+0000", 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034614210000000001 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829036+0000", 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00025713999999999999 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:17.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829173+0000", 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000136797 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831180+0000", 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020068859999999998 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831366+0000", 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018597399999999999 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831398+0000", 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2518999999999998e-05 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831425+0000", 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6636e-05 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.805926584000005, 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006225062, 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:17.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2e-07 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825335+0000", 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4339999999999998e-06 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825342+0000", 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6900000000000003e-06 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825348+0000", 2024-09-16T01:50:17.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5470000000000004e-06 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829429+0000", 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040815720000000003 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829541+0000", 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112285 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829647+0000", 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105592 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831435+0000", 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017880330000000001 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831474+0000", 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:17.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831518+0000", 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3516999999999997e-05 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831551+0000", 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3062000000000003e-05 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.805896868999994, 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062900040000000001, 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:17.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825356+0000", 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825359+0000", 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0620000000000001e-06 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825366+0000", 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6899999999999992e-06 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825380+0000", 2024-09-16T01:50:17.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3193000000000001e-05 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829978+0000", 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045982879999999999 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830056+0000", 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8466000000000004e-05 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:17.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830226+0000", 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000169944 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831440+0000", 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012132790000000001 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831590+0000", 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015076400000000001 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831621+0000", 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0440999999999999e-05 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831645+0000", 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4383e-05 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.804943390999995, 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080128709999999995, 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.665 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826310+0000", 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3200000000000003e-07 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826315+0000", 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6980000000000003e-06 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826328+0000", 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2568999999999999e-05 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826336+0000", 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9670000000000001e-06 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826916+0000", 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058005899999999996 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827058+0000", 2024-09-16T01:50:17.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000142533 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827196+0000", 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013736299999999999 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829940+0000", 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027445600000000001 2024-09-16T01:50:17.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834254+0000", 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043136210000000001 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834292+0000", 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8266000000000003e-05 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834322+0000", 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9303e-05 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 73.804897482000001, 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080982150000000006, 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.671 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5799999999999999e-07 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826368+0000", 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3152999999999999e-05 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826378+0000", 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826385+0000", 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5440000000000004e-06 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827629+0000", 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001243886 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827855+0000", 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000226615 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834453+0000", 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006597548 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:14 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 68.770700395000006, 2024-09-16T01:50:17.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013360132, 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:17.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6600000000000002e-07 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860560+0000", 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7919999999999993e-06 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860569+0000", 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3999999999999998e-06 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860576+0000", 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8589999999999999e-06 2024-09-16T01:50:17.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860646+0000", 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9676000000000002e-05 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143559 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860917+0000", 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012731900000000001 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862638+0000", 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017212670000000001 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873802+0000", 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011164074 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873866+0000", 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4381999999999994e-05 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873912+0000", 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5337999999999999e-05 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.682 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:31 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 68.770512173, 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013335217, 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860741+0000", 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9899999999999999e-07 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860748+0000", 2024-09-16T01:50:17.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3250000000000002e-06 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860761+0000", 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2958e-05 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860769+0000", 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2230000000000007e-06 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861199+0000", 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000429178 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861286+0000", 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7436999999999996e-05 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861415+0000", 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012907000000000001 2024-09-16T01:50:17.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863136+0000", 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001720627 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873957+0000", 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010821809999999999 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874029+0000", 2024-09-16T01:50:17.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1641000000000001e-05 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874075+0000", 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6048999999999999e-05 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:19 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:17.688 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 68.770468002000001, 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013415967000000001, 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:17.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860785+0000", 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6300000000000005e-07 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5279999999999997e-06 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860797+0000", 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.418e-06 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860802+0000", 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.344e-06 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861655+0000", 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085269000000000002 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861767+0000", 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011184900000000001 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861914+0000", 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014756700000000001 2024-09-16T01:50:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863730+0000", 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001815564 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874127+0000", 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010396924 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874163+0000", 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6285999999999998e-05 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874200+0000", 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7134000000000001e-05 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:6 2.1f 2:fa28013d:::615.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:17.694 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 62.734613549999999, 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006133548, 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896640+0000", 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896645+0000", 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6540000000000004e-06 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896655+0000", 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0316000000000001e-05 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896664+0000", 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2090000000000005e-06 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897019+0000", 2024-09-16T01:50:17.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035541199999999999 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897088+0000", 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9388999999999996e-05 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897195+0000", 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010608299999999999 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900354+0000", 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031591919999999999 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902602+0000", 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002248468 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902659+0000", 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6889999999999999e-05 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902772+0000", 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112972 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:9 2.6 2:62978f0a:::617.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 62.733822320000002, 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063594389999999997, 2024-09-16T01:50:17.700 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897431+0000", 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8899999999999998e-07 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897437+0000", 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3060000000000003e-06 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897447+0000", 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9920000000000002e-06 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897455+0000", 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.627e-06 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897523+0000", 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7912999999999998e-05 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897652+0000", 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000129259 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897777+0000", 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012468599999999999 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901149+0000", 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033725389999999999 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903683+0000", 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025335539999999999 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903730+0000", 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7339999999999997e-05 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903789+0000", 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9234000000000002e-05 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.705 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:10 2.6 2:62978f0a:::617.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 62.733529289000003, 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061735380000000001, 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897724+0000", 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0139999999999999e-06 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897730+0000", 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7629999999999997e-06 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897740+0000", 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0499999999999999e-05 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897748+0000", 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1519999999999992e-06 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898076+0000", 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00032781800000000001 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898143+0000", 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6905999999999999e-05 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898267+0000", 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123979 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901274+0000", 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00300716 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903796+0000", 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025223210000000001 2024-09-16T01:50:17.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903859+0000", 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2638000000000006e-05 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903896+0000", 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7286999999999997e-05 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 62.733437574, 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060644100000000001, 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:17.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5099999999999999e-07 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897831+0000", 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5534000000000001e-05 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897850+0000", 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8825999999999999e-05 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897858+0000", 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3939999999999996e-06 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898448+0000", 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059040600000000003 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898526+0000", 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7600000000000002e-05 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898646+0000", 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011967400000000001 2024-09-16T01:50:17.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903214+0000", 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045682589999999999 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903787+0000", 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00057291199999999997 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903842+0000", 2024-09-16T01:50:17.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4747999999999998e-05 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903879+0000", 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7305999999999998e-05 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:8 2.6 2:6642f7a8:::620.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:17.717 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 56.719436905000002, 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0071307610000000002, 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911816+0000", 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2299999999999999e-07 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911822+0000", 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0460000000000002e-06 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911846+0000", 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4193999999999999e-05 2024-09-16T01:50:17.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911857+0000", 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.042e-05 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912714+0000", 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085704200000000003 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912790+0000", 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6019999999999994e-05 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912934+0000", 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014454 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916495+0000", 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035605770000000001 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918865+0000", 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002370451 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918906+0000", 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0123999999999997e-05 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918946+0000", 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.722 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:12 2.1f 2:ffe2d39d:::622.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 56.718994457999997, 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060451970000000004, 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912259+0000", 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0729999999999999e-06 2024-09-16T01:50:17.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912265+0000", 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4409999999999998e-06 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912276+0000", 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0798000000000001e-05 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912286+0000", 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0176999999999999e-05 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912581+0000", 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00029479199999999999 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912692+0000", 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110874 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913326+0000", 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00063410300000000001 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916590+0000", 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003264543 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918212+0000", 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016214440000000001 2024-09-16T01:50:17.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918252+0000", 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0562999999999998e-05 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918303+0000", 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0389000000000002e-05 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:21 2.6 2:62eebe7c:::627.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 56.718952057000003, 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0067672629999999999, 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912301+0000", 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1999999999999999e-07 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912305+0000", 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9770000000000002e-06 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912312+0000", 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9920000000000001e-06 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912318+0000", 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9089999999999997e-06 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084597199999999996 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913267+0000", 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103433 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913378+0000", 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011124300000000001 2024-09-16T01:50:17.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916553+0000", 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031750900000000002 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918992+0000", 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00243837 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919023+0000", 2024-09-16T01:50:17.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1498999999999998e-05 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919067+0000", 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4158e-05 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 56.718926168000003, 2024-09-16T01:50:17.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0068352660000000004, 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:17.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8299999999999997e-07 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912336+0000", 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0079e-05 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912345+0000", 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0639999999999994e-06 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912351+0000", 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.686e-06 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913855+0000", 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0015034829999999999 2024-09-16T01:50:17.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913971+0000", 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011639 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914107+0000", 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013549999999999999 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917094+0000", 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002987132 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919106+0000", 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020125389999999998 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919139+0000", 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2670999999999999e-05 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919161+0000", 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2238999999999998e-05 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:9 2.9 2:954a9a2b:::62b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.693682473000003, 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074502429999999996, 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.740 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937571+0000", 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1480000000000001e-06 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937578+0000", 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1890000000000001e-06 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937593+0000", 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4613999999999999e-05 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937602+0000", 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937662+0000", 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9778000000000003e-05 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937799+0000", 2024-09-16T01:50:17.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013690299999999999 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937967+0000", 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016826100000000001 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940308+0000", 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023405750000000001 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944953+0000", 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046456860000000004 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944987+0000", 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3997e-05 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945020+0000", 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2653000000000001e-05 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.745 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.693627528, 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080442459999999997, 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8999999999999996e-07 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937632+0000", 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7399999999999998e-06 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937640+0000", 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.354e-06 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937647+0000", 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5050000000000004e-06 2024-09-16T01:50:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939098+0000", 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001451473 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939454+0000", 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035574400000000001 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939603+0000", 2024-09-16T01:50:17.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000149228 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940646+0000", 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001042821 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945559+0000", 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049129070000000002 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:17.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945623+0000", 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4072999999999995e-05 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945669+0000", 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5711000000000003e-05 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:17.751 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:17.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.646+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f01570c0 con 0x7ff0f002cb50 2024-09-16T01:50:17.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.648+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f01570c0 con 0x7ff0f002cb50 2024-09-16T01:50:17.753 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:17.754 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.648+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff0f0204b60 0x7ff0f0206f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.648+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff0f0062dc0 con 0x7ff0f0204b60 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.649+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff0f0204b60 0x7ff0f0206f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.649+0000 7ff0ef400640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff0f0204b60 0x7ff0f0206f50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.650+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f0062dc0 con 0x7ff0f0204b60 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.676+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f0062dc0 con 0x7ff0f0204b60 2024-09-16T01:50:17.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.678+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f0062dc0 con 0x7ff0f0204b60 2024-09-16T01:50:17.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:17.756 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.678+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff0f02079a0 0x7ff0f0209d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.678+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff0f0063410 con 0x7ff0f02079a0 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.679+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff0f02079a0 0x7ff0f0209d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.679+0000 7ff0f4c00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff0f02079a0 0x7ff0f0209d90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.680+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f0063410 con 0x7ff0f02079a0 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.693+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f0063410 con 0x7ff0f02079a0 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.695+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f0063410 con 0x7ff0f02079a0 2024-09-16T01:50:17.757 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:17.758 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:17.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.695+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff0f020a7e0 0x7ff0f020cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:17.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.695+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff0f020fa90 con 0x7ff0f020a7e0 2024-09-16T01:50:17.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.696+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff0f020a7e0 0x7ff0f020cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:17.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.696+0000 7ff0efe00640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff0f020a7e0 0x7ff0f020cbd0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:17.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.697+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0f020fa90 con 0x7ff0f020a7e0 2024-09-16T01:50:17.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.710+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff0f020d1d0 con 0x7ff0f020a7e0 2024-09-16T01:50:17.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0e7e00640 1 -- 172.21.15.146:0/3516552402 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff0f020d1d0 con 0x7ff0f020a7e0 2024-09-16T01:50:17.759 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff0f00cb290 msgr2=0x7ff0f01564c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff0f00cb290 0x7ff0f01564c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff0f00cb670 msgr2=0x7ff0f01f37e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff0f00cb670 0x7ff0f01f37e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff0f01f4230 msgr2=0x7ff0f0062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff0f01f4230 0x7ff0f0062630 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff0f0152630 msgr2=0x7ff0f0152a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff0f0152630 0x7ff0f0152a50 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff0f002cb50 msgr2=0x7ff0f002cf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff0f002cb50 0x7ff0f002cf70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff0f0204b60 msgr2=0x7ff0f0206f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff0f0204b60 0x7ff0f0206f50 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff0f02079a0 msgr2=0x7ff0f0209d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff0f02079a0 0x7ff0f0209d90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff0f020a7e0 msgr2=0x7ff0f020cbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff0f020a7e0 0x7ff0f020cbd0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 reap_dead start 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 reap_dead start 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 reap_dead start 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.712+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 reap_dead start 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f4c00640 1 -- 172.21.15.146:0/3516552402 reap_dead start 2024-09-16T01:50:17.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff0d006c300 msgr2=0x7ff0d006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff0d006c300 0x7ff0d006e7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0000f60 tx=0x7ff0e0007cf0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 msgr2=0x7ff0f00b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 0x7ff0f00b3300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009ee0 tx=0x7ff0e8004a40 comp rx=0 tx=0).stop 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 shutdown_connections 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff0d006c300 0x7ff0d006e7c0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0000f60 tx=0x7ff0e0007cf0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 --2- 172.21.15.146:0/3516552402 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff0f00d0870 0x7ff0f00b3300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 >> 172.21.15.146:0/3516552402 conn(0x7ff0f006d6f0 msgr2=0x7ff0f00cee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 shutdown_connections 2024-09-16T01:50:17.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:17.713+0000 7ff0f5600640 1 -- 172.21.15.146:0/3516552402 wait complete. 2024-09-16T01:50:17.763 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:17.752+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:17.853 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:17.853+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:17.865 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:17.865 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:17.874 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:17.955 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:17.954+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:17.978 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:17.978 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:17.987 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:18.071 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:18.070+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:18.131 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.132 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:18.181 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:18.180+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:18.258 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:18.257+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:18.285 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.285 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.294 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:18.359 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:18.358+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:18.440 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.440 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.449 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:18.455 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:50:18.493 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:18.492+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:18.554 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.555 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.561 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:18.560+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:18.563 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:18.662 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:18.661+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:18.707 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.707 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.715 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:18.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.714+0000 7fbd37e00640 1 Processor -- start 2024-09-16T01:50:18.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.715+0000 7fbd37e00640 1 -- start start 2024-09-16T01:50:18.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd37e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd3815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd37e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3806d140 con 0x7fbd380cc040 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd37e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3806d420 con 0x7fbd380c75a0 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd37e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3815b880 con 0x7fbd380c6b70 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd3815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd3815b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50538/0 (socket says 172.21.15.146:50538) 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbd380c75a0 0x7fbd380cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:53112/0 (socket says 172.21.15.146:53112) 2024-09-16T01:50:18.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36a00640 1 -- 172.21.15.146:0/2145127121 learned_addr learned my addr 172.21.15.146:0/2145127121 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:18.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36a00640 1 -- 172.21.15.146:0/2145127121 >> v1:172.21.15.161:6789/0 conn(0x7fbd380c75a0 legacy=0x7fbd380cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:18.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36a00640 1 -- 172.21.15.146:0/2145127121 >> v1:172.21.15.146:6789/0 conn(0x7fbd380cc040 legacy=0x7fbd38165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:18.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.717+0000 7fbd36a00640 1 -- 172.21.15.146:0/2145127121 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3806d700 con 0x7fbd380c6b70 2024-09-16T01:50:18.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.719+0000 7fbd36a00640 1 --2- 172.21.15.146:0/2145127121 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd3815b340 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbd280098a0 tx=0x7fbd2802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41791fb0683b82a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:18.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.719+0000 7fbd35600640 1 -- 172.21.15.146:0/2145127121 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28037070 con 0x7fbd380c6b70 2024-09-16T01:50:18.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.719+0000 7fbd35600640 1 -- 172.21.15.146:0/2145127121 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd28032440 con 0x7fbd380c6b70 2024-09-16T01:50:18.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.719+0000 7fbd35600640 1 -- 172.21.15.146:0/2145127121 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28032810 con 0x7fbd380c6b70 2024-09-16T01:50:18.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.720+0000 7fbd37e00640 1 -- 172.21.15.146:0/2145127121 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 msgr2=0x7fbd3815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.720+0000 7fbd37e00640 1 --2- 172.21.15.146:0/2145127121 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd3815b340 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbd280098a0 tx=0x7fbd2802a250 comp rx=0 tx=0).stop 2024-09-16T01:50:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.720+0000 7fbd37e00640 1 -- 172.21.15.146:0/2145127121 shutdown_connections 2024-09-16T01:50:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.720+0000 7fbd37e00640 1 --2- 172.21.15.146:0/2145127121 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd3815b340 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.720+0000 7fbd37e00640 1 -- 172.21.15.146:0/2145127121 >> 172.21.15.146:0/2145127121 conn(0x7fbd380ccc20 msgr2=0x7fbd380cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.721+0000 7fbd37e00640 1 -- 172.21.15.146:0/2145127121 shutdown_connections 2024-09-16T01:50:18.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.721+0000 7fbd37e00640 1 -- 172.21.15.146:0/2145127121 wait complete. 2024-09-16T01:50:18.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.723+0000 7fbd37e00640 1 Processor -- start 2024-09-16T01:50:18.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.723+0000 7fbd37e00640 1 -- start start 2024-09-16T01:50:18.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd381f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 0x7fbd38156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd380cc040 0x7fbd38157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbd380cb050 con 0x7fbd380cc040 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd380cab20 con 0x7fbd380c75a0 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbd380cbc80 con 0x7fbd380c6b70 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd36000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 0x7fbd38156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd36000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 0x7fbd38156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43146/0 (socket says 172.21.15.146:43146) 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd36000640 1 -- 172.21.15.146:0/3986708487 learned_addr learned my addr 172.21.15.146:0/3986708487 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.725+0000 7fbd37400640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd380cc040 0x7fbd38157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36a00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd381f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:18.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36000640 1 -- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 msgr2=0x7fbd381f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:18.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36000640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd381f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36000640 1 -- 172.21.15.146:0/3986708487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd380cc040 msgr2=0x7fbd38157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36000640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd380cc040 0x7fbd38157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36000640 1 -- 172.21.15.146:0/3986708487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd381f4db0 con 0x7fbd380c75a0 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd37400640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd380cc040 0x7fbd38157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36a00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd381f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.726+0000 7fbd36000640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 0x7fbd38156f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400c480 tx=0x7fbd2400c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.727+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd24018070 con 0x7fbd380c75a0 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.727+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd24007480 con 0x7fbd380c75a0 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.727+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240047e0 con 0x7fbd380c75a0 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.727+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd38156420 con 0x7fbd380c75a0 2024-09-16T01:50:18.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.727+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd380c9900 con 0x7fbd380c75a0 2024-09-16T01:50:18.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.728+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd3806cd20 con 0x7fbd380c75a0 2024-09-16T01:50:18.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.730+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbd24004980 con 0x7fbd380c75a0 2024-09-16T01:50:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.731+0000 7fbd2f400640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd1806c2b0 0x7fbd1806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:18.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.731+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5770+0+0 (secure 0 0 0) 0x7fbd2408a770 con 0x7fbd380c75a0 2024-09-16T01:50:18.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.732+0000 7fbd36a00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd1806c2b0 0x7fbd1806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:18.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.732+0000 7fbd36a00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd1806c2b0 0x7fbd1806e770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd28034000 tx=0x7fbd2802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:18.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:18.735+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbd24004c70 con 0x7fbd380c75a0 2024-09-16T01:50:18.763 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:18.762+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.820 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.828 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:18.869 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:18.868+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:18.965 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:18.964+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:18.972 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:18.972 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:18.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:19.066 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:19.065+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:19.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.115+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fbd3806da90 con 0x7fbd380c75a0 2024-09-16T01:50:19.168 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:19.167+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:19.252 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:19.253 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:19.269 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:19.268+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:19.270 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:19.370 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:19.369+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:19.437 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:19.437 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:19.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.444+0000 7fbd2f400640 1 -- 172.21.15.146:0/3986708487 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v43) ==== 97+0+0 (secure 0 0 0) 0x7fbd24058e90 con 0x7fbd380c75a0 2024-09-16T01:50:19.447 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:50:19.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd1806c2b0 msgr2=0x7fbd1806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd1806c2b0 0x7fbd1806e770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd28034000 tx=0x7fbd2802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 msgr2=0x7fbd38156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 0x7fbd38156f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400c480 tx=0x7fbd2400c940 comp rx=0 tx=0).stop 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 shutdown_connections 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd1806c2b0 0x7fbd1806e770 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd380cc040 0x7fbd38157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd380c75a0 0x7fbd38156f00 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400c480 tx=0x7fbd2400c940 comp rx=0 tx=0).stop 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 --2- 172.21.15.146:0/3986708487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd380c6b70 0x7fbd381f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.448+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 >> 172.21.15.146:0/3986708487 conn(0x7fbd380ccc20 msgr2=0x7fbd380c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:19.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.450+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 shutdown_connections 2024-09-16T01:50:19.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.450+0000 7fbd37e00640 1 -- 172.21.15.146:0/3986708487 wait complete. 2024-09-16T01:50:19.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:19.476 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:19.475+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:19.572 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:19.571+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:19.640 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:19.640 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:19.653 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:19.673 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:19.672+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:19.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:19.774 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:19.773+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:19.836 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:19.836 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:19.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:19.875 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:19.874+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:19.976 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:19.975+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:19.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.995+0000 7f23c4a00640 1 Processor -- start 2024-09-16T01:50:19.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.995+0000 7f23c4a00640 1 -- start start 2024-09-16T01:50:19.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.997+0000 7f23c4a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f23c00c6d00 0x7f23c00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:19.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.997+0000 7f23c4a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c015e220 con 0x7f23c00d0870 2024-09-16T01:50:19.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.997+0000 7f23c4a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c015e500 con 0x7f23c00d0490 2024-09-16T01:50:19.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.997+0000 7f23c4a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f23c00c6160 con 0x7f23c00c6d00 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.998+0000 7f23bf400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f23c00d0490 0x7f23c00c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:53116/0 (socket says 172.21.15.146:53116) 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.998+0000 7f23bf400640 1 -- 172.21.15.146:0/1516095888 learned_addr learned my addr 172.21.15.146:0/1516095888 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.998+0000 7f23bfe00640 1 --2- 172.21.15.146:0/1516095888 >> v2:172.21.15.146:3300/0 conn(0x7f23c00c6d00 0x7f23c00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.998+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545766286 0 0) 0x7f23c015e500 con 0x7f23c00d0490 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.998+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23a4003520 con 0x7f23c00d0490 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 598476883 0 0) 0x7f23a4003520 con 0x7f23c00d0490 2024-09-16T01:50:20.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 >> v2:172.21.15.146:3300/0 conn(0x7f23c00c6d00 msgr2=0x7f23c00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:20.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 --2- 172.21.15.146:0/1516095888 >> v2:172.21.15.146:3300/0 conn(0x7f23c00c6d00 0x7f23c00c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 >> v1:172.21.15.146:6789/0 conn(0x7f23c00d0870 legacy=0x7f23c0154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c015e7e0 con 0x7f23c00d0490 2024-09-16T01:50:20.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f23b0001670 con 0x7f23c00d0490 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 >> v1:172.21.15.161:6789/0 conn(0x7f23c00d0490 legacy=0x7f23c00c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23a40039b0 0x7f23a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23a40043b0 0x7f23a4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23a4005080 0x7f23a40141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23c00c6160 con 0x7f23a4005080 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23a4014740 con 0x7f23a40043b0 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:19.999+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f23a4014940 con 0x7f23a40039b0 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bea00640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23a40043b0 0x7f23a4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bfe00640 1 --2- 172.21.15.146:0/1516095888 >> v2:172.21.15.146:3300/0 conn(0x7f23c00c6d00 0x7f23c00c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bea00640 1 -- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23a40039b0 msgr2=0x7f23a4003d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bea00640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23a40039b0 0x7f23a4003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bea00640 1 -- 172.21.15.146:0/1516095888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23a4005080 msgr2=0x7f23a40141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bea00640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23a4005080 0x7f23a40141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bea00640 1 -- 172.21.15.146:0/1516095888 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c015e7e0 con 0x7f23a40043b0 2024-09-16T01:50:20.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.000+0000 7f23bfe00640 1 -- 172.21.15.146:0/1516095888 reap_dead start 2024-09-16T01:50:20.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.002+0000 7f23bea00640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23a40043b0 0x7f23a4004880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001830 tx=0x7f23a4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f093c8912a7f8dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.002+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac037070 con 0x7f23a40043b0 2024-09-16T01:50:20.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.002+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23ac02abe0 con 0x7f23a40043b0 2024-09-16T01:50:20.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.002+0000 7f23be000640 1 -- 172.21.15.146:0/1516095888 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac001430 con 0x7f23a40043b0 2024-09-16T01:50:20.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 -- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23a40043b0 msgr2=0x7f23a4004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23a40043b0 0x7f23a4004880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001830 tx=0x7f23a4005830 comp rx=0 tx=0).stop 2024-09-16T01:50:20.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 -- 172.21.15.146:0/1516095888 shutdown_connections 2024-09-16T01:50:20.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 --2- 172.21.15.146:0/1516095888 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23a40043b0 0x7f23a4004880 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 -- 172.21.15.146:0/1516095888 >> 172.21.15.146:0/1516095888 conn(0x7f23c006d6f0 msgr2=0x7f23c006e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:20.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 -- 172.21.15.146:0/1516095888 shutdown_connections 2024-09-16T01:50:20.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.003+0000 7f23c4a00640 1 -- 172.21.15.146:0/1516095888 wait complete. 2024-09-16T01:50:20.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.006+0000 7f23c4a00640 1 Processor -- start 2024-09-16T01:50:20.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.006+0000 7f23c4a00640 1 -- start start 2024-09-16T01:50:20.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23c4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23c00b5270 0x7f23c014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23c4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23c01557b0 0x7f23c014db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23c4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c00b5650 0x7f23c014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23c4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23c015dec0 con 0x7f23c00b5270 2024-09-16T01:50:20.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23c4a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23c015eda0 con 0x7f23c00b5650 2024-09-16T01:50:20.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23c4a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f23c00c65f0 con 0x7f23c01557b0 2024-09-16T01:50:20.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23bea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23c01557b0 0x7f23c014db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23bea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23c01557b0 0x7f23c014db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50572/0 (socket says 172.21.15.146:50572) 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.008+0000 7f23bea00640 1 -- 172.21.15.146:0/2236110108 learned_addr learned my addr 172.21.15.146:0/2236110108 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23c00b5270 0x7f23c014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c00b5650 0x7f23c014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bea00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c00b5650 msgr2=0x7f23c014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23c00b5650 0x7f23c014e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bea00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23c00b5270 msgr2=0x7f23c014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23c00b5270 0x7f23c014fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bea00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c0150130 con 0x7f23c01557b0 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23c00b5270 0x7f23c014fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23c01557b0 0x7f23c014db00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001950 tx=0x7f23ac001230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac03f020 con 0x7f23c01557b0 2024-09-16T01:50:20.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23ac005270 con 0x7f23c01557b0 2024-09-16T01:50:20.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.009+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac037040 con 0x7f23c01557b0 2024-09-16T01:50:20.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.010+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c01552f0 con 0x7f23c01557b0 2024-09-16T01:50:20.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.010+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c01fd8b0 con 0x7f23c01557b0 2024-09-16T01:50:20.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.011+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f23c00d0ff0 con 0x7f23c01557b0 2024-09-16T01:50:20.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.013+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f23ac042020 con 0x7f23c01557b0 2024-09-16T01:50:20.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.014+0000 7f23b7400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f23a006c2a0 0x7f23a006e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.014+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 5770+0+0 (secure 0 0 0) 0x7f23ac0aaa90 con 0x7f23c01557b0 2024-09-16T01:50:20.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.015+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v43) ==== 54+0+16 (secure 0 0 0) 0x7f23c00d0ff0 con 0x7f23c01557b0 2024-09-16T01:50:20.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.015+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f23a006c2a0 0x7f23a006e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.015+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23c01fdc20 0x7f23c02009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.015+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23c01555d0 con 0x7f23c01fdc20 2024-09-16T01:50:20.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.016+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23c01fdc20 0x7f23c02009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.016+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f23a006c2a0 0x7f23a006e760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007a20 tx=0x7f23b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.017+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23c01fdc20 0x7f23c02009a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.018+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c01555d0 con 0x7f23c01fdc20 2024-09-16T01:50:20.033 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:20.033 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:20.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.048+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c00c6980 con 0x7f23c01fdc20 2024-09-16T01:50:20.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.051+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c00c6980 con 0x7f23c01fdc20 2024-09-16T01:50:20.052 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.051+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23c02013f0 0x7f23c00b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.051+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f23c02017d0 con 0x7f23c02013f0 2024-09-16T01:50:20.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.051+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23c02013f0 0x7f23c00b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.052+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23c02013f0 0x7f23c00b3d10 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.053+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c02017d0 con 0x7f23c02013f0 2024-09-16T01:50:20.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.082+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c00b44a0 con 0x7f23c02013f0 2024-09-16T01:50:20.083 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:20.083+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:20.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.085+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c00b44a0 con 0x7f23c02013f0 2024-09-16T01:50:20.086 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:20.086 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:20.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.085+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23c0062b40 0x7f23c0062f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.085+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f23c00cc3f0 con 0x7f23c0062b40 2024-09-16T01:50:20.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.085+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23c0062b40 0x7f23c0062f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.086+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23c0062b40 0x7f23c0062f60 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.087+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23ac097e50 con 0x7f23c0062b40 2024-09-16T01:50:20.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.115+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c00cbfa0 con 0x7f23c0062b40 2024-09-16T01:50:20.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.118+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c00cbfa0 con 0x7f23c0062b40 2024-09-16T01:50:20.119 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:20.120 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:20.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.118+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f23c00c8e00 0x7f23c00c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.118+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f23c00cfe60 con 0x7f23c00c8e00 2024-09-16T01:50:20.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.119+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f23c00c8e00 0x7f23c00c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.120+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f23c00c8e00 0x7f23c00c9260 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.121+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c00cfe60 con 0x7f23c00c8e00 2024-09-16T01:50:20.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.148+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c00cfb80 con 0x7f23c00c8e00 2024-09-16T01:50:20.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.150+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c00cfb80 con 0x7f23c00c8e00 2024-09-16T01:50:20.151 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:20.152 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:20.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.151+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f23c002cb50 0x7f23c002cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.151+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f23c006e3e0 con 0x7f23c002cb50 2024-09-16T01:50:20.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.151+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f23c002cb50 0x7f23c002cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.152+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f23c002cb50 0x7f23c002cfb0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.153+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c006e3e0 con 0x7f23c002cb50 2024-09-16T01:50:20.178 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:20.177+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:20.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.181+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c006e3e0 con 0x7f23c002cb50 2024-09-16T01:50:20.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.183+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c006e3e0 con 0x7f23c002cb50 2024-09-16T01:50:20.184 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:20.185 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:20.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.184+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f23c0211d20 0x7f23c0214110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.184+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f23c002db70 con 0x7f23c0211d20 2024-09-16T01:50:20.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.184+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f23c0211d20 0x7f23c0214110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.185+0000 7f23bea00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f23c0211d20 0x7f23c0214110 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.186+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c002db70 con 0x7f23c0211d20 2024-09-16T01:50:20.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.213+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c0214650 con 0x7f23c0211d20 2024-09-16T01:50:20.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.215+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c0214650 con 0x7f23c0211d20 2024-09-16T01:50:20.216 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:20.217 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:20.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.216+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f23c0214d70 0x7f23c02151b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.216+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f23c021a0d0 con 0x7f23c0214d70 2024-09-16T01:50:20.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.216+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f23c0214d70 0x7f23c02151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.217+0000 7f23bfe00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f23c0214d70 0x7f23c02151b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.218+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c021a0d0 con 0x7f23c0214d70 2024-09-16T01:50:20.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.244+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c0215770 con 0x7f23c0214d70 2024-09-16T01:50:20.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.247+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c0215770 con 0x7f23c0214d70 2024-09-16T01:50:20.248 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.247+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f23c0215e90 0x7f23c02162f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.247+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f23c0216b50 con 0x7f23c0215e90 2024-09-16T01:50:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.247+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f23c0215e90 0x7f23c02162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.248+0000 7f23bf400640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f23c0215e90 0x7f23c02162f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:20.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.249+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23c0216b50 con 0x7f23c0215e90 2024-09-16T01:50:20.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.262+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f23c0216b50 con 0x7f23c0215e90 2024-09-16T01:50:20.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23b7400640 1 -- 172.21.15.146:0/2236110108 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23c0216b50 con 0x7f23c0215e90 2024-09-16T01:50:20.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23c01fdc20 msgr2=0x7f23c02009a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23c01fdc20 0x7f23c02009a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23c02013f0 msgr2=0x7f23c00b3d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23c02013f0 0x7f23c00b3d10 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23c0062b40 msgr2=0x7f23c0062f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23c0062b40 0x7f23c0062f60 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f23c00c8e00 msgr2=0x7f23c00c9260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f23c00c8e00 0x7f23c00c9260 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f23c002cb50 msgr2=0x7f23c002cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f23c002cb50 0x7f23c002cfb0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f23c0211d20 msgr2=0x7f23c0214110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f23c0211d20 0x7f23c0214110 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f23c0214d70 msgr2=0x7f23c02151b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f23c0214d70 0x7f23c02151b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f23c0215e90 msgr2=0x7f23c02162f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f23c0215e90 0x7f23c02162f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f23a006c2a0 msgr2=0x7f23a006e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f23a006c2a0 0x7f23a006e760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f23b0007a20 tx=0x7f23b0009040 comp rx=0 tx=0).stop 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23c01557b0 msgr2=0x7f23c014db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.265+0000 7f23c4a00640 1 --2- 172.21.15.146:0/2236110108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23c01557b0 0x7f23c014db00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001950 tx=0x7f23ac001230 comp rx=0 tx=0).stop 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23bfe00640 1 -- 172.21.15.146:0/2236110108 reap_dead start 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23bfe00640 1 -- 172.21.15.146:0/2236110108 reap_dead start 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23bfe00640 1 -- 172.21.15.146:0/2236110108 reap_dead start 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23bfe00640 1 -- 172.21.15.146:0/2236110108 reap_dead start 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23bfe00640 1 -- 172.21.15.146:0/2236110108 reap_dead start 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 shutdown_connections 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 >> 172.21.15.146:0/2236110108 conn(0x7f23c006d6f0 msgr2=0x7f23c00ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 shutdown_connections 2024-09-16T01:50:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:20.266+0000 7f23c4a00640 1 -- 172.21.15.146:0/2236110108 wait complete. 2024-09-16T01:50:20.279 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:20.278+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.519122690000003, 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.087568958000006, 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.298 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:20.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.067644736000005, 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.304 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:20.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:20.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:20.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.067624817999999, 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.067564234000002, 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.821278714000002, 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.821128200000004, 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.821108632000005, 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250952417999997, 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250784600000003, 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250765618000003, 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250745987000002, 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250612081, 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250600543000004, 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250582132000005, 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.250467462000003, 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.239685734000005, 2024-09-16T01:50:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.239569075999995, 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 71.424035837000005, 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:20.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 71.423974498999996, 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:20.415 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:20.420 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:20.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:20.423 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:20.421+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:20.522 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:20.522+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:20.623 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:20.622+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:20.630 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:20.630 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:20.630 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:20.630 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:20.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:20.723 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:20.722+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:20.824 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:20.823+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:20.853 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:20.853 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:20.853 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:20.853 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:20.853 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:20.867 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:20.925 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:20.924+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:21.026 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:21.025+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:21.085 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.257732928999999, 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:21.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:21.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:21.092 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.257618108000003, 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:21.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:21.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.257576237999999, 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:21.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.257531959000005, 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:21.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:21.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.257503081999999, 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:21.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:21.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.253795874999994, 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.114 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:21.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:21.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.253741894000001, 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:21.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:21.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.218326669999996, 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:21.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.215266521000004, 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:21.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:21.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.214407073000004, 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:21.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:21.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.214353180000003, 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.138 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:21.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:21.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.208329207000006, 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.144 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:21.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.208279078999993, 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:21.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.208202745999998, 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:21.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.208171390999993, 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:21.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:21.162 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.170085936, 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:21.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.169294405000002, 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.167 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:21.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 55.145212579999999, 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:21.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:21.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 55.145140668000003, 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:21.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:21.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:21.183 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.814498268000001, 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:21.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:21.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:21.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:21.190 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:21.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:21.192 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:21.190+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:21.292 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:21.291+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:21.393 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:21.392+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:21.458 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:21.458 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:21.458 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T01:50:21.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T01:50:21.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:50:21.486 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:21.487 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:21.496 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:21.494+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:21.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:21.594 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:21.593+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:21.697 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:21.694+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.694+0000 7f5a4be00640 1 Processor -- start 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.694+0000 7f5a4be00640 1 -- start start 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.696+0000 7f5a4be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5a4c0c75a0 0x7f5a4c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.696+0000 7f5a4be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a4c0cc040 0x7f5a4c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.696+0000 7f5a4be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4c06d140 con 0x7f5a4c0c6b70 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.696+0000 7f5a4be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5a4c15b880 con 0x7f5a4c0c75a0 2024-09-16T01:50:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.696+0000 7f5a4be00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cab20 con 0x7f5a4c0cc040 2024-09-16T01:50:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.697+0000 7f5a4aa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5a4c0c6b70 0x7f5a4c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:58560/0 (socket says 172.21.15.146:58560) 2024-09-16T01:50:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.697+0000 7f5a4aa00640 1 -- 172.21.15.146:0/591520823 learned_addr learned my addr 172.21.15.146:0/591520823 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.697+0000 7f5a4a000640 1 --2- 172.21.15.146:0/591520823 >> v2:172.21.15.146:3300/0 conn(0x7f5a4c0c75a0 0x7f5a4c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.697+0000 7f5a4b400640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a4c0cc040 0x7f5a4c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.697+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056860320 0 0) 0x7f5a4c06d140 con 0x7f5a4c0c6b70 2024-09-16T01:50:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.697+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a30003520 con 0x7f5a4c0c6b70 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1980925373 0 0) 0x7f5a30003520 con 0x7f5a4c0c6b70 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a4c0cc040 msgr2=0x7f5a4c0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a4c0cc040 0x7f5a4c0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 >> v2:172.21.15.146:3300/0 conn(0x7f5a4c0c75a0 msgr2=0x7f5a4c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 --2- 172.21.15.146:0/591520823 >> v2:172.21.15.146:3300/0 conn(0x7f5a4c0c75a0 0x7f5a4c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c06d420 con 0x7f5a4c0c6b70 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f5a3c001670 con 0x7f5a4c0c6b70 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 >> v1:172.21.15.146:6789/0 conn(0x7f5a4c0c6b70 legacy=0x7f5a4c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a300039b0 0x7f5a30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a30004800 0x7f5a30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c15b880 con 0x7f5a30004800 2024-09-16T01:50:21.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cab20 con 0x7f5a300039b0 2024-09-16T01:50:21.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.698+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3000bf30 con 0x7f5a300051c0 2024-09-16T01:50:21.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4aa00640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a300039b0 0x7f5a30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4b400640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4a000640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a30004800 0x7f5a30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4b400640 1 -- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a300039b0 msgr2=0x7f5a30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4b400640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a300039b0 0x7f5a30003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4b400640 1 -- 172.21.15.146:0/591520823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a30004800 msgr2=0x7f5a30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4b400640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a30004800 0x7f5a30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4b400640 1 -- 172.21.15.146:0/591520823 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c06d420 con 0x7f5a300051c0 2024-09-16T01:50:21.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.699+0000 7f5a4a000640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a30004800 0x7f5a30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:21.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.702+0000 7f5a4b400640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a3000b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c069570 tx=0x7f5a30005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da00340e9eb62971 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:21.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.702+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44036070 con 0x7f5a300051c0 2024-09-16T01:50:21.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.702+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a44031600 con 0x7f5a300051c0 2024-09-16T01:50:21.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.702+0000 7f5a49600640 1 -- 172.21.15.146:0/591520823 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44009bd0 con 0x7f5a300051c0 2024-09-16T01:50:21.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.703+0000 7f5a4be00640 1 -- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 msgr2=0x7f5a3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.703+0000 7f5a4be00640 1 --2- 172.21.15.146:0/591520823 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a3000b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c069570 tx=0x7f5a30005830 comp rx=0 tx=0).stop 2024-09-16T01:50:21.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.703+0000 7f5a4b400640 1 -- 172.21.15.146:0/591520823 reap_dead start 2024-09-16T01:50:21.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.703+0000 7f5a4be00640 1 -- 172.21.15.146:0/591520823 shutdown_connections 2024-09-16T01:50:21.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.703+0000 7f5a4be00640 1 -- 172.21.15.146:0/591520823 >> 172.21.15.146:0/591520823 conn(0x7f5a4c0ccc20 msgr2=0x7f5a4c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:21.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.703+0000 7f5a4be00640 1 -- 172.21.15.146:0/591520823 shutdown_connections 2024-09-16T01:50:21.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.704+0000 7f5a4be00640 1 -- 172.21.15.146:0/591520823 wait complete. 2024-09-16T01:50:21.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.706+0000 7f5a4be00640 1 Processor -- start 2024-09-16T01:50:21.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.706+0000 7f5a4be00640 1 -- start start 2024-09-16T01:50:21.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.708+0000 7f5a4be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a4c15e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.709+0000 7f5a4be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a4c0cfa80 0x7f5a4c15a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.709+0000 7f5a4be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 0x7f5a4c1571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.709+0000 7f5a4a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a4c0cfa80 0x7f5a4c15a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.709+0000 7f5a4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 0x7f5a4c1571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 0x7f5a4c1571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50326/0 (socket says 172.21.15.146:50326) 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a4c0cfa80 0x7f5a4c15a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43180/0 (socket says 172.21.15.146:43180) 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4b400640 1 -- 172.21.15.146:0/1008268729 learned_addr learned my addr 172.21.15.146:0/1008268729 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4aa00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a4c15e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cb050 con 0x7f5a4c15ac90 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cad70 con 0x7f5a4c0cfa80 2024-09-16T01:50:21.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.710+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cbc80 con 0x7f5a300051c0 2024-09-16T01:50:21.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4b400640 1 -- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 msgr2=0x7f5a4c15e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4b400640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a4c15e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4b400640 1 -- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a4c0cfa80 msgr2=0x7f5a4c15a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4b400640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a4c0cfa80 0x7f5a4c15a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4b400640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c1f0d20 con 0x7f5a4c15ac90 2024-09-16T01:50:21.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4aa00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a4c15e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:21.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.711+0000 7f5a4b400640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 0x7f5a4c1571e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5a4402a470 tx=0x7f5a44009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:21.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.712+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44036070 con 0x7f5a4c15ac90 2024-09-16T01:50:21.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.712+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a44043070 con 0x7f5a4c15ac90 2024-09-16T01:50:21.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.712+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44009bd0 con 0x7f5a4c15ac90 2024-09-16T01:50:21.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.712+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4c1f0fc0 con 0x7f5a4c15ac90 2024-09-16T01:50:21.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.712+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4c156700 con 0x7f5a4c15ac90 2024-09-16T01:50:21.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.716+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a4c06cd20 con 0x7f5a4c15ac90 2024-09-16T01:50:21.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.716+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5a440309e0 con 0x7f5a4c15ac90 2024-09-16T01:50:21.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.718+0000 7f5a42a00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a2c06c330 0x7f5a2c06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.718+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5770+0+0 (secure 0 0 0) 0x7f5a440abfb0 con 0x7f5a4c15ac90 2024-09-16T01:50:21.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.720+0000 7f5a4aa00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a2c06c330 0x7f5a2c06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.724+0000 7f5a4aa00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a2c06c330 0x7f5a2c06e7f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c000fb0 tx=0x7f5a3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:21.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.728+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5a4407a100 con 0x7f5a4c15ac90 2024-09-16T01:50:21.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.750+0000 7fde17200640 1 Processor -- start 2024-09-16T01:50:21.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.751+0000 7fde17200640 1 -- start start 2024-09-16T01:50:21.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde17200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde1815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde17200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde180c75a0 0x7fde180cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde17200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde180cbab0 0x7fde180cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde17200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fde1815b880 con 0x7fde180cbab0 2024-09-16T01:50:21.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde17200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fde180cab20 con 0x7fde180c75a0 2024-09-16T01:50:21.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde17200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fde1806d140 con 0x7fde180c6b70 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde16800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde180cbab0 0x7fde180cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde16800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde180cbab0 0x7fde180cc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50342/0 (socket says 172.21.15.146:50342) 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.753+0000 7fde16800640 1 -- 172.21.15.146:0/3110251059 learned_addr learned my addr 172.21.15.146:0/3110251059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.754+0000 7fde15400640 1 --2- 172.21.15.146:0/3110251059 >> v2:172.21.15.161:3300/0 conn(0x7fde180c75a0 0x7fde180cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.754+0000 7fde16800640 1 -- 172.21.15.146:0/3110251059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 msgr2=0x7fde1815b340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.754+0000 7fde16800640 1 --2- 172.21.15.146:0/3110251059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde1815b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.754+0000 7fde16800640 1 -- 172.21.15.146:0/3110251059 >> v2:172.21.15.161:3300/0 conn(0x7fde180c75a0 msgr2=0x7fde180cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.754+0000 7fde16800640 1 --2- 172.21.15.146:0/3110251059 >> v2:172.21.15.161:3300/0 conn(0x7fde180c75a0 0x7fde180cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.754+0000 7fde16800640 1 -- 172.21.15.146:0/3110251059 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1806d420 con 0x7fde180cbab0 2024-09-16T01:50:21.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.756+0000 7fde16800640 1 --2- 172.21.15.146:0/3110251059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde180cc040 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde1806dda0 tx=0x7fde1002a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af780b60a6886c92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:21.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.756+0000 7fde14a00640 1 -- 172.21.15.146:0/3110251059 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10036070 con 0x7fde180cbab0 2024-09-16T01:50:21.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.756+0000 7fde14a00640 1 -- 172.21.15.146:0/3110251059 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde10031600 con 0x7fde180cbab0 2024-09-16T01:50:21.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.757+0000 7fde14a00640 1 -- 172.21.15.146:0/3110251059 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde100319d0 con 0x7fde180cbab0 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.757+0000 7fde17200640 1 -- 172.21.15.146:0/3110251059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 msgr2=0x7fde180cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.757+0000 7fde17200640 1 --2- 172.21.15.146:0/3110251059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde180cc040 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde1806dda0 tx=0x7fde1002a590 comp rx=0 tx=0).stop 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 -- 172.21.15.146:0/3110251059 shutdown_connections 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 --2- 172.21.15.146:0/3110251059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde180cc040 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 --2- 172.21.15.146:0/3110251059 >> v2:172.21.15.161:3300/0 conn(0x7fde180c75a0 0x7fde180cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 --2- 172.21.15.146:0/3110251059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde1815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 -- 172.21.15.146:0/3110251059 >> 172.21.15.146:0/3110251059 conn(0x7fde180ccc20 msgr2=0x7fde180cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:21.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 -- 172.21.15.146:0/3110251059 shutdown_connections 2024-09-16T01:50:21.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.758+0000 7fde17200640 1 -- 172.21.15.146:0/3110251059 wait complete. 2024-09-16T01:50:21.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.761+0000 7fde17200640 1 Processor -- start 2024-09-16T01:50:21.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.761+0000 7fde17200640 1 -- start start 2024-09-16T01:50:21.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.762+0000 7fde17200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde181f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde17200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde180c75a0 0x7fde181f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde17200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde181f7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde17200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fde180cb050 con 0x7fde180cbab0 2024-09-16T01:50:21.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde17200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fde180cad70 con 0x7fde180c75a0 2024-09-16T01:50:21.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde17200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fde1806cd20 con 0x7fde180c6b70 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde15e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde181f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde15400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde180c75a0 0x7fde181f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde15400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde180c75a0 0x7fde181f7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43204/0 (socket says 172.21.15.146:43204) 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.763+0000 7fde15400640 1 -- 172.21.15.146:0/636018659 learned_addr learned my addr 172.21.15.146:0/636018659 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.764+0000 7fde16800640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde181f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.765+0000 7fde16800640 1 -- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 msgr2=0x7fde181f88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.765+0000 7fde16800640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde181f88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.765+0000 7fde16800640 1 -- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde180c75a0 msgr2=0x7fde181f7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:21.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.765+0000 7fde16800640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde180c75a0 0x7fde181f7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:21.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.765+0000 7fde16800640 1 -- 172.21.15.146:0/636018659 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde181f9260 con 0x7fde180cbab0 2024-09-16T01:50:21.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.765+0000 7fde16800640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde181f7860 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fde10030000 tx=0x7fde10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:21.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.766+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10036050 con 0x7fde180cbab0 2024-09-16T01:50:21.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.766+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde10042070 con 0x7fde180cbab0 2024-09-16T01:50:21.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.766+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10007bd0 con 0x7fde180cbab0 2024-09-16T01:50:21.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.766+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde181f9500 con 0x7fde180cbab0 2024-09-16T01:50:21.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.766+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1815a4d0 con 0x7fde180cbab0 2024-09-16T01:50:21.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.768+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde180cb700 con 0x7fde180cbab0 2024-09-16T01:50:21.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.770+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde10047460 con 0x7fde180cbab0 2024-09-16T01:50:21.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.771+0000 7fde0e000640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddf806c300 0x7fddf806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:21.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.772+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5770+0+0 (secure 0 0 0) 0x7fde100ac190 con 0x7fde180cbab0 2024-09-16T01:50:21.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.772+0000 7fde15e00640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddf806c300 0x7fddf806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:21.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.773+0000 7fde15e00640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddf806c300 0x7fddf806e7c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fde08006fd0 tx=0x7fde08009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:21.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:21.780+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde1007a2c0 con 0x7fde180cbab0 2024-09-16T01:50:21.796 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:21.795+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:21.817 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:21.817 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:21.817 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:21.817 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:21.817 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:21.835 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:21.898 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:21.897+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:21.999 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:21.998+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:22.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.059+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fde180ca280 con 0x7fde180cbab0 2024-09-16T01:50:22.066 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:22.066 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:22.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:22.066 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.238331216999995, 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063070110000000004, 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825667+0000", 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825672+0000", 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6920000000000001e-06 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825689+0000", 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7039999999999999e-05 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825696+0000", 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5909999999999998e-06 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826617+0000", 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00092117800000000004 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826697+0000", 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9598999999999995e-05 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826832+0000", 2024-09-16T01:50:22.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013548 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829938+0000", 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031059059999999999 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831894+0000", 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019552240000000002 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831942+0000", 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8389e-05 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831973+0000", 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1043000000000001e-05 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:29 2.b 2:d0630e4c:::mds0_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:22.072 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.238289668999997, 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087271810000000005, 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825734+0000", 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5638000000000001e-05 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825743+0000", 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7350000000000002e-06 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825750+0000", 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827333+0000", 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015834689999999999 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827472+0000", 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138796 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827572+0000", 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9910999999999999e-05 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832328+0000", 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047559079999999997 2024-09-16T01:50:22.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834353+0000", 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002025654 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834392+0000", 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8099000000000002e-05 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834435+0000", 2024-09-16T01:50:22.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3169e-05 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:34 2.b 2:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.238221408000001, 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088227679999999999, 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825777+0000", 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:22.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825781+0000", 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0350000000000003e-06 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825788+0000", 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825794+0000", 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4240000000000001e-06 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827917+0000", 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021233570000000002 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828052+0000", 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013528300000000001 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834599+0000", 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065463650000000002 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197723542999995, 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012518068, 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.082 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866276+0000", 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866303+0000", 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7229e-05 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866329+0000", 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.563e-05 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866343+0000", 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4287999999999999e-05 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867782+0000", 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014384510000000001 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867911+0000", 2024-09-16T01:50:22.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012957999999999999 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868028+0000", 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117025 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877878+0000", 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098499209999999993 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878708+0000", 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00082986900000000001 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878752+0000", 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3735000000000001e-05 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878792+0000", 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0025999999999999e-05 2024-09-16T01:50:22.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197623262999997, 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014487969, 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866375+0000", 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9899999999999999e-07 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866379+0000", 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6519999999999997e-06 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866388+0000", 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9579999999999996e-06 2024-09-16T01:50:22.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866395+0000", 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7750000000000004e-06 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868287+0000", 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001892344 2024-09-16T01:50:22.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868360+0000", 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3023999999999997e-05 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868473+0000", 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112321 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878461+0000", 2024-09-16T01:50:22.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099884610000000006 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880760+0000", 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022984479999999998 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880823+0000", 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3276999999999997e-05 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880862+0000", 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:7 2.b 2:d70a01c6:::60c.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197593225000006, 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020806099000000002, 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.094 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.82e-07 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866408+0000", 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0110000000000002e-06 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866415+0000", 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5830000000000003e-06 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866421+0000", 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6659999999999996e-06 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022638100000000002 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868842+0000", 2024-09-16T01:50:22.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015727699999999999 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868942+0000", 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100182 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878519+0000", 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095771399999999996 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887083+0000", 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085641750000000003 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887155+0000", 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2117000000000001e-05 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887210+0000", 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4656e-05 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.102 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:8 2.b 2:d70a01c6:::60c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197554834000002, 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020917458, 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866443+0000", 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866447+0000", 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8340000000000001e-06 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866455+0000", 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866460+0000", 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5629999999999999e-06 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869260+0000", 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027996039999999998 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869328+0000", 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8730000000000001e-05 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869461+0000", 2024-09-16T01:50:22.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264099999999999 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878525+0000", 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090635630000000002 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887259+0000", 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087339310000000003 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887305+0000", 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6538000000000001e-05 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887360+0000", 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4706000000000001e-05 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:10 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.108 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197528251999998, 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01479957, 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8100000000000003e-07 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866473+0000", 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1089999999999998e-06 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866481+0000", 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866486+0000", 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.566e-06 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870818+0000", 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004332519 2024-09-16T01:50:22.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870937+0000", 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011880899999999999 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871093+0000", 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015636400000000001 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877793+0000", 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066997560000000003 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881167+0000", 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033735029999999999 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881242+0000", 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5139e-05 2024-09-16T01:50:22.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881269+0000", 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6794000000000001e-05 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:27 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197490478999995, 2024-09-16T01:50:22.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014854865, 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866511+0000", 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7479999999999999e-06 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866518+0000", 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866525+0000", 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2750000000000001e-06 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875004+0000", 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084799100000000002 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875082+0000", 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7698e-05 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875190+0000", 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010785000000000001 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877830+0000", 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026396480000000001 2024-09-16T01:50:22.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881273+0000", 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034436839999999998 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881341+0000", 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7359000000000003e-05 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881362+0000", 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0928e-05 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:33 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197465041000001, 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020942512, 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.120 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866533+0000", 2024-09-16T01:50:22.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.27e-07 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866552+0000", 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8921999999999999e-05 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866559+0000", 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2570000000000001e-06 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866564+0000", 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869714+0000", 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031497920000000002 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869828+0000", 2024-09-16T01:50:22.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114331 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869968+0000", 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013971799999999999 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880969+0000", 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011001031 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887401+0000", 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064320319999999999 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887441+0000", 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9937000000000001e-05 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887475+0000", 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4010999999999997e-05 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.125 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:2 2.5 2:a6e36b3f:::401.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197426057000001, 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014499972999999999, 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866572+0000", 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8400000000000005e-07 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866575+0000", 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0929999999999999e-06 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866592+0000", 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7765999999999999e-05 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866598+0000", 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4219999999999999e-06 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871180+0000", 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045822939999999998 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871272+0000", 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1308000000000001e-05 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871419+0000", 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014714 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875993+0000", 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004574486 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881013+0000", 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050202240000000002 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881046+0000", 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2944000000000002e-05 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881071+0000", 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4811999999999999e-05 2024-09-16T01:50:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:18 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197378341000004, 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020977030000000001, 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1799999999999995e-07 2024-09-16T01:50:22.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866623+0000", 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4709999999999999e-06 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866630+0000", 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866635+0000", 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3079999999999998e-06 2024-09-16T01:50:22.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870303+0000", 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036679210000000002 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870398+0000", 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4464999999999998e-05 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:22.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870501+0000", 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010291800000000001 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880977+0000", 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010476014000000001 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887510+0000", 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065331759999999999 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887553+0000", 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2737000000000001e-05 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887596+0000", 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3343000000000002e-05 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:22 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.137 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197353961000005, 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01481146, 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866644+0000", 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866646+0000", 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2139999999999999e-06 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866652+0000", 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2160000000000003e-06 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866677+0000", 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4663000000000001e-05 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876491+0000", 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098146110000000009 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876616+0000", 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012430699999999999 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881455+0000", 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048389840000000002 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:24 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197312672999999, 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016598702999999999, 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5400000000000002e-07 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866688+0000", 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-09-16T01:50:22.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866695+0000", 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.883e-06 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866700+0000", 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7280000000000003e-06 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868221+0000", 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015209620000000001 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868334+0000", 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113584 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883283+0000", 2024-09-16T01:50:22.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014949100999999999 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:25 2.0 2:0250a5a6:::201.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.197278046999998, 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011987815000000001, 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866720+0000", 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6300000000000001e-07 2024-09-16T01:50:22.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866738+0000", 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8218000000000001e-05 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866745+0000", 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7279999999999995e-06 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866751+0000", 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868589+0000", 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018382730000000001 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868709+0000", 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119571 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878707+0000", 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099980020000000006 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.2.11:5 2.0 2:03625d61:::614.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.168568558000004, 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081102819999999999, 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.150 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:22.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895434+0000", 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4909999999999999e-06 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895444+0000", 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0560000000000001e-05 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895451+0000", 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5590000000000002e-06 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896385+0000", 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093368200000000002 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896454+0000", 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9567000000000007e-05 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896559+0000", 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104609 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.899222+0000", 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026632000000000001 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903485+0000", 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042627999999999997 2024-09-16T01:50:22.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903517+0000", 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2345999999999998e-05 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903539+0000", 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1838e-05 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:14 2.1a 2:594e21d0:::623.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.152162318999999, 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068542239999999999, 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911838+0000", 2024-09-16T01:50:22.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.435e-06 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911844+0000", 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.122e-06 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911849+0000", 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6979999999999997e-06 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914481+0000", 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026327249999999998 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914555+0000", 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3886000000000004e-05 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914697+0000", 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014193800000000001 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918088+0000", 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033908559999999998 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918610+0000", 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052154100000000004 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918656+0000", 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-05 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918689+0000", 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3052000000000001e-05 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:23 2.18 2:1c1db442:::103.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.152028772999998, 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067043830000000004, 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.161 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911969+0000", 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911980+0000", 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1090999999999999e-05 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911989+0000", 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6270000000000006e-06 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911994+0000", 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3639999999999998e-06 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912099+0000", 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010514900000000001 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912210+0000", 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111346 2024-09-16T01:50:22.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912342+0000", 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131197 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915846+0000", 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035043269999999998 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918573+0000", 2024-09-16T01:50:22.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027272289999999999 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918632+0000", 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8854000000000003e-05 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918673+0000", 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0728e-05 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.166 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.127429442999997, 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084942400000000001, 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000004e-07 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936594+0000", 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5137999999999999e-05 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936602+0000", 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8999999999999995e-06 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936609+0000", 2024-09-16T01:50:22.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9480000000000002e-06 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938373+0000", 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001763857 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938479+0000", 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105816 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938622+0000", 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014314699999999999 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941905+0000", 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032832 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944943+0000", 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030378670000000001 2024-09-16T01:50:22.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945019+0000", 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5532999999999997e-05 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945062+0000", 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180999999999999e-05 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.127363944000003, 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085927369999999996, 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936634+0000", 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936637+0000", 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.287e-06 2024-09-16T01:50:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936647+0000", 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7189999999999996e-06 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936653+0000", 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.319e-06 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938867+0000", 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022140079999999999 2024-09-16T01:50:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938935+0000", 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7644999999999994e-05 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939072+0000", 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013753600000000001 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941912+0000", 2024-09-16T01:50:22.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028393350000000001 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945119+0000", 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032077160000000002 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945183+0000", 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3723e-05 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945226+0000", 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2805000000000001e-05 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.178 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:22.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.071+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f5a4c0ca280 con 0x7f5a4c15ac90 2024-09-16T01:50:22.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.072+0000 7f5a42a00640 1 -- 172.21.15.146:0/1008268729 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v43) ==== 81+0+34703 (secure 0 0 0) 0x7f5a4407e400 con 0x7f5a4c15ac90 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 43, 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: "fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: "created": "2024-09-16T01:48:34.399638+0000", 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: "modified": "2024-09-16T01:50:19.435177+0000", 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: "last_up_change": "2024-09-16T01:48:45.748664+0000", 2024-09-16T01:50:22.180 INFO:teuthology.orchestra.run.smithi146.stdout: "last_in_change": "2024-09-16T01:50:04.358386+0000", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_num": 5806080, 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_set": [ 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "nodeep-scrub", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "noscrub", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "pglog_hardlimit", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "purged_snapdirs", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_deletes", 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "sortbitwise" 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_version": 6, 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T01:50:22.181 INFO:teuthology.orchestra.run.smithi146.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_snapshot": "", 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_max": 3, 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd": 8, 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "require_min_compat_client": "luminous", 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "min_compat_client": "luminous", 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "require_osd_release": "squid", 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_crimson": false, 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "pools": [ 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 1, 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": ".mgr", 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:46.570355+0000", 2024-09-16T01:50:22.182 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:50:22.183 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 1, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 1, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 1, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 1, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 1, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "18", 2024-09-16T01:50:22.184 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:50:22.185 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.186 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_max": 32, 2024-09-16T01:50:22.187 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 1 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": {} 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 6.9000000953674316, 2024-09-16T01:50:22.188 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 6.9000000953674316, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.28999999165534973, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 2, 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_metadata", 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:56.008811+0000", 2024-09-16T01:50:22.189 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 32, 2024-09-16T01:50:22.190 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 32, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 32, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 32, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 32, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:50:22.191 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "21", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:50:22.192 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:50:22.193 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:50:22.194 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_bias": 4, 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 16, 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_priority": 5 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": "cephfs" 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:50:22.195 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.9700000286102295, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.9700000286102295, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2.25, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2.25, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 3, 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_data", 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:57.207121+0000", 2024-09-16T01:50:22.196 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:50:22.197 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 64, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 64, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 64, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 64, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 64, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "22", 2024-09-16T01:50:22.198 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:50:22.199 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.200 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 64, 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: "target_size_ratio": 0.89999997615814209 2024-09-16T01:50:22.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "data": "cephfs" 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.3200000524520874, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.3200000524520874, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 1.5, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 1.5, 2024-09-16T01:50:22.202 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "osds": [ 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.203 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6824", 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6825", 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6826", 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6827", 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6830", 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.206 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6831", 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6828", 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6829", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6825/61440064", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6827/61440064", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6831/61440064", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6829/61440064", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.209 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6808", 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6809", 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6810", 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6811", 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.211 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6814", 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6815", 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.212 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6812", 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6813", 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.213 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6809/2375201615", 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6811/2375201615", 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6815/2375201615", 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6813/2375201615", 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.214 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6800", 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6801", 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6802", 2024-09-16T01:50:22.216 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6803", 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6806", 2024-09-16T01:50:22.217 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6807", 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6804", 2024-09-16T01:50:22.218 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6805", 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6801/1818492127", 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6803/1818492127", 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6807/1818492127", 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6805/1818492127", 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.219 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:22.220 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 35, 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6816", 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6817", 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6818", 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6819", 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.222 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6822", 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6823", 2024-09-16T01:50:22.223 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6820", 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6821", 2024-09-16T01:50:22.224 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6817/1633804394", 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6819/1633804394", 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1633804394", 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6821/1633804394", 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:50:22.225 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0.8634185791015625, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 13, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.226 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6800", 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6801", 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.227 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6802", 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6803", 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.228 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6806", 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6807", 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.229 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6804", 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6805", 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.230 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6801/154390265", 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6803/154390265", 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6807/154390265", 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6805/154390265", 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.231 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6824", 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6825", 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6826", 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6827", 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6830", 2024-09-16T01:50:22.234 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6831", 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6828", 2024-09-16T01:50:22.235 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6829", 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6825/356357279", 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6827/356357279", 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6831/356357279", 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6829/356357279", 2024-09-16T01:50:22.236 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.237 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6808", 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6809", 2024-09-16T01:50:22.238 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6810", 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6811", 2024-09-16T01:50:22.239 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6814", 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.240 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6815", 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6812", 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.241 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6813", 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6809/3661001615", 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6811/3661001615", 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6815/3661001615", 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6813/3661001615", 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 0, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 35, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6816", 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6817", 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6818", 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6819", 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:22.245 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6822", 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6823", 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6820", 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6821", 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6817/1279166573", 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6819/1279166573", 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6823/1279166573", 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6821/1279166573", 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_xinfo": [ 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:50:22.248 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.135308+0000", 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.249 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.290750+0000", 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.712744+0000", 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.988678+0000", 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.251 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.198114+0000", 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.252 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.085332+0000", 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.394321+0000", 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.236983+0000", 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap": [], 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_items": [ 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.20", 2024-09-16T01:50:22.254 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 6, 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 5 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_primaries": [], 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_temp": [], 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_temp": [], 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklist": {}, 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "range_blocklist": {}, 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profiles": { 2024-09-16T01:50:22.255 INFO:teuthology.orchestra.run.smithi146.stdout: "default": { 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-failure-domain": "osd", 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "k": "2", 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "m": "1", 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin": "jerasure", 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "technique": "reed_sol_van" 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps_queue": [], 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "new_removed_snaps": [], 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "new_purged_snaps": [], 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_node_flags": {}, 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "device_class_flags": {}, 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode": { 2024-09-16T01:50:22.256 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_enabled": false, 2024-09-16T01:50:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_bucket_count": 0, 2024-09-16T01:50:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "degraded_stretch_mode": 0, 2024-09-16T01:50:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "recovering_stretch_mode": 0, 2024-09-16T01:50:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_bucket": 0 2024-09-16T01:50:22.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:22.257 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:22.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a2c06c330 msgr2=0x7f5a2c06e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a2c06c330 0x7f5a2c06e7f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c000fb0 tx=0x7f5a3c009040 comp rx=0 tx=0).stop 2024-09-16T01:50:22.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 msgr2=0x7f5a4c1571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 0x7f5a4c1571e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5a4402a470 tx=0x7f5a44009750 comp rx=0 tx=0).stop 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 shutdown_connections 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a2c06c330 0x7f5a2c06e7f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a4c15ac90 0x7f5a4c1571e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a4c0cfa80 0x7f5a4c15a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 --2- 172.21.15.146:0/1008268729 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a300051c0 0x7f5a4c15e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.076+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 >> 172.21.15.146:0/1008268729 conn(0x7f5a4c0ccc20 msgr2=0x7f5a4c0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.077+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 shutdown_connections 2024-09-16T01:50:22.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.077+0000 7f5a4be00640 1 -- 172.21.15.146:0/1008268729 wait complete. 2024-09-16T01:50:22.260 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:22.261 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:22.178+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:22.262 INFO:tasks.thrashosds.thrasher:j is {'epoch': 43, 'fsid': '4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d', 'created': '2024-09-16T01:48:34.399638+0000', 'modified': '2024-09-16T01:50:19.435177+0000', 'last_up_change': '2024-09-16T01:48:45.748664+0000', 'last_in_change': '2024-09-16T01:50:04.358386+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:48:46.570355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:48:56.008811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:48:57.207121+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e7758ddf-75ff-43c7-abae-3647b170f7c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 61440064}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 61440064}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 61440064}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 61440064}]}, 'public_addr': '172.21.15.146:6825/61440064', 'cluster_addr': '172.21.15.146:6827/61440064', 'heartbeat_back_addr': '172.21.15.146:6831/61440064', 'heartbeat_front_addr': '172.21.15.146:6829/61440064', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a065e77e-26f0-4245-abe8-e36c2fb4436b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 2375201615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 2375201615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 2375201615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 2375201615}]}, 'public_addr': '172.21.15.146:6809/2375201615', 'cluster_addr': '172.21.15.146:6811/2375201615', 'heartbeat_back_addr': '172.21.15.146:6815/2375201615', 'heartbeat_front_addr': '172.21.15.146:6813/2375201615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fefb6d8f-6799-4344-a977-85ab983ec0cb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 1818492127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 1818492127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 1818492127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 1818492127}]}, 'public_addr': '172.21.15.146:6801/1818492127', 'cluster_addr': '172.21.15.146:6803/1818492127', 'heartbeat_back_addr': '172.21.15.146:6807/1818492127', 'heartbeat_front_addr': '172.21.15.146:6805/1818492127', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9d1a2f96-efcb-4a73-b5db-ec5e2a63122a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 1633804394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 1633804394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1633804394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 1633804394}]}, 'public_addr': '172.21.15.146:6817/1633804394', 'cluster_addr': '172.21.15.146:6819/1633804394', 'heartbeat_back_addr': '172.21.15.146:6823/1633804394', 'heartbeat_front_addr': '172.21.15.146:6821/1633804394', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c', 'up': 1, 'in': 1, 'weight': 0.8634185791015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6800', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6801', 'nonce': 154390265}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6802', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6803', 'nonce': 154390265}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6806', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6807', 'nonce': 154390265}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6804', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6805', 'nonce': 154390265}]}, 'public_addr': '172.21.15.161:6801/154390265', 'cluster_addr': '172.21.15.161:6803/154390265', 'heartbeat_back_addr': '172.21.15.161:6807/154390265', 'heartbeat_front_addr': '172.21.15.161:6805/154390265', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '11562dd5-d537-4d88-a282-e8687caebad6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6824', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6825', 'nonce': 356357279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6826', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6827', 'nonce': 356357279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6830', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6831', 'nonce': 356357279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6828', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6829', 'nonce': 356357279}]}, 'public_addr': '172.21.15.161:6825/356357279', 'cluster_addr': '172.21.15.161:6827/356357279', 'heartbeat_back_addr': '172.21.15.161:6831/356357279', 'heartbeat_front_addr': '172.21.15.161:6829/356357279', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b31adf01-6768-49cf-a232-127b9d0d2dc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6808', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6809', 'nonce': 3661001615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6810', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6811', 'nonce': 3661001615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6814', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6815', 'nonce': 3661001615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6812', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6813', 'nonce': 3661001615}]}, 'public_addr': '172.21.15.161:6809/3661001615', 'cluster_addr': '172.21.15.161:6811/3661001615', 'heartbeat_back_addr': '172.21.15.161:6815/3661001615', 'heartbeat_front_addr': '172.21.15.161:6813/3661001615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '67c95c28-b600-430c-9474-a902cf46a73b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6816', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6817', 'nonce': 1279166573}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6818', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6819', 'nonce': 1279166573}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6822', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6823', 'nonce': 1279166573}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6820', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6821', 'nonce': 1279166573}]}, 'public_addr': '172.21.15.161:6817/1279166573', 'cluster_addr': '172.21.15.161:6819/1279166573', 'heartbeat_back_addr': '172.21.15.161:6823/1279166573', 'heartbeat_front_addr': '172.21.15.161:6821/1279166573', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.135308+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.290750+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.712744+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.988678+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.198114+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.085332+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.394321+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.236983+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.20', 'mappings': [{'from': 6, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T01:50:22.262 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T01:50:22.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:22.302 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:22.301+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:22.376 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:22.376 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:22.376 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:22.376 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:22.381 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:22.380+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:22.385 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:22.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.448+0000 7fde0e000640 1 -- 172.21.15.146:0/636018659 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v44) ==== 91+0+0 (secure 0 0 0) 0x7fde1007e5c0 con 0x7fde180cbab0 2024-09-16T01:50:22.450 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:50:22.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddf806c300 msgr2=0x7fddf806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddf806c300 0x7fddf806e7c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fde08006fd0 tx=0x7fde08009060 comp rx=0 tx=0).stop 2024-09-16T01:50:22.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 msgr2=0x7fde181f7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde181f7860 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fde10030000 tx=0x7fde10004a40 comp rx=0 tx=0).stop 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 shutdown_connections 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddf806c300 0x7fddf806e7c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde180cbab0 0x7fde181f7860 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde180c75a0 0x7fde181f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 --2- 172.21.15.146:0/636018659 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fde180c6b70 0x7fde181f88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 >> 172.21.15.146:0/636018659 conn(0x7fde180ccc20 msgr2=0x7fde180cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 shutdown_connections 2024-09-16T01:50:22.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.453+0000 7fde17200640 1 -- 172.21.15.146:0/636018659 wait complete. 2024-09-16T01:50:22.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.480+0000 7f84cbe00640 1 Processor -- start 2024-09-16T01:50:22.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.480+0000 7f84cbe00640 1 -- start start 2024-09-16T01:50:22.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.481+0000 7f84cbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f84cc0d0870 0x7f84cc154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.481+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0c6d00 0x7f84cc154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.481+0000 7f84cbe00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84cc15e220 con 0x7f84cc0d0490 2024-09-16T01:50:22.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.481+0000 7f84cbe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f84cc0c6160 con 0x7f84cc0d0870 2024-09-16T01:50:22.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.481+0000 7f84cbe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f84cc15e500 con 0x7f84cc0c6d00 2024-09-16T01:50:22.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84caa00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f84cc0d0490 0x7f84cc0c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:53118/0 (socket says 172.21.15.146:53118) 2024-09-16T01:50:22.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84caa00640 1 -- 172.21.15.146:0/2160618168 learned_addr learned my addr 172.21.15.146:0/2160618168 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:22.483 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:22.482+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:22.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84cb400640 1 --2- 172.21.15.146:0/2160618168 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0c6d00 0x7f84cc154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84ca000640 1 --2- 172.21.15.146:0/2160618168 >> v2:172.21.15.146:3300/0 conn(0x7f84cc0d0870 0x7f84cc154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84cb400640 1 -- 172.21.15.146:0/2160618168 >> v2:172.21.15.146:3300/0 conn(0x7f84cc0d0870 msgr2=0x7f84cc154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:22.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84cb400640 1 --2- 172.21.15.146:0/2160618168 >> v2:172.21.15.146:3300/0 conn(0x7f84cc0d0870 0x7f84cc154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84c9600640 1 -- 172.21.15.146:0/2160618168 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804745421 0 0) 0x7f84cc15e220 con 0x7f84cc0d0490 2024-09-16T01:50:22.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84cb400640 1 -- 172.21.15.146:0/2160618168 >> v1:172.21.15.161:6789/0 conn(0x7f84cc0d0490 legacy=0x7f84cc0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.482+0000 7f84cb400640 1 -- 172.21.15.146:0/2160618168 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84cc15e7e0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.483+0000 7f84cb400640 1 --2- 172.21.15.146:0/2160618168 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0c6d00 0x7f84cc154bd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009d90 tx=0x7f84c402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6efd4daa33f82b3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.483+0000 7f84c9600640 1 -- 172.21.15.146:0/2160618168 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007740 con 0x7f84cc0c6d00 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.483+0000 7f84c9600640 1 -- 172.21.15.146:0/2160618168 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40078e0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84c9600640 1 -- 172.21.15.146:0/2160618168 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4007cb0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84cbe00640 1 -- 172.21.15.146:0/2160618168 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0c6d00 msgr2=0x7f84cc154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84cbe00640 1 --2- 172.21.15.146:0/2160618168 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0c6d00 0x7f84cc154bd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009d90 tx=0x7f84c402a6f0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84cbe00640 1 -- 172.21.15.146:0/2160618168 shutdown_connections 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84cbe00640 1 --2- 172.21.15.146:0/2160618168 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0c6d00 0x7f84cc154bd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84cbe00640 1 --2- 172.21.15.146:0/2160618168 >> v2:172.21.15.146:3300/0 conn(0x7f84cc0d0870 0x7f84cc154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.484+0000 7f84cbe00640 1 -- 172.21.15.146:0/2160618168 >> 172.21.15.146:0/2160618168 conn(0x7f84cc06d6f0 msgr2=0x7f84cc06e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.485+0000 7f84cbe00640 1 -- 172.21.15.146:0/2160618168 shutdown_connections 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.485+0000 7f84cbe00640 1 -- 172.21.15.146:0/2160618168 wait complete. 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.486+0000 7f84cbe00640 1 Processor -- start 2024-09-16T01:50:22.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 -- start start 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 0x7f84cc1e4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84cc0d0490 0x7f84cc1f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0d0870 0x7f84cc1f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc15dec0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc15eda0 con 0x7f84cc0d0490 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.487+0000 7f84cbe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f84cc0c51f0 con 0x7f84cc0d0870 2024-09-16T01:50:22.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 0x7f84cc1e4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 0x7f84cc1e4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50360/0 (socket says 172.21.15.146:50360) 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84cb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0d0870 0x7f84cc1f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 -- 172.21.15.146:0/3662031860 learned_addr learned my addr 172.21.15.146:0/3662031860 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84cc0d0490 0x7f84cc1f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0d0870 msgr2=0x7f84cc1f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84cc0d0870 0x7f84cc1f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84cc0d0490 msgr2=0x7f84cc1f09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84cc0d0490 0x7f84cc1f09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84cc0b4780 con 0x7f84cc0c6d00 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.488+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 0x7f84cc1e4ef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00c500 tx=0x7f84bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.489+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc018070 con 0x7f84cc0c6d00 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.489+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84bc007560 con 0x7f84cc0c6d00 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.489+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc004a20 con 0x7f84cc0c6d00 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.489+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84cc1f4670 con 0x7f84cc0c6d00 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.489+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84cc151c20 con 0x7f84cc0c6d00 2024-09-16T01:50:22.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.490+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f84cc0d0ff0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.491+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f84bc01b070 con 0x7f84cc0c6d00 2024-09-16T01:50:22.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.492+0000 7f84c2a00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f84ac06c2b0 0x7f84ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.492+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5770+0+0 (secure 0 0 0) 0x7f84bc08a7e0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.493+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f84ac06c2b0 0x7f84ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.493+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v44) ==== 54+0+16 (secure 0 0 0) 0x7f84cc0d0ff0 con 0x7f84cc0c6d00 2024-09-16T01:50:22.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.493+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f84cc1574a0 0x7f84cc157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.493+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84cc15e220 con 0x7f84cc1574a0 2024-09-16T01:50:22.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.493+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f84ac06c2b0 0x7f84ac06e770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84cc155a40 tx=0x7f84b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.493+0000 7f84cb400640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f84cc1574a0 0x7f84cc157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.494+0000 7f84cb400640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f84cc1574a0 0x7f84cc157880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.495+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc15e220 con 0x7f84cc1574a0 2024-09-16T01:50:22.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.509+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc0cc3f0 con 0x7f84cc1574a0 2024-09-16T01:50:22.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.511+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc0cc3f0 con 0x7f84cc1574a0 2024-09-16T01:50:22.513 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.511+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f84cc157dc0 0x7f84cc14dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.511+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f84cc1e6990 con 0x7f84cc157dc0 2024-09-16T01:50:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.512+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f84cc157dc0 0x7f84cc14dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.512+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f84cc157dc0 0x7f84cc14dd70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.513+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc1e6990 con 0x7f84cc157dc0 2024-09-16T01:50:22.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.528+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc1e5430 con 0x7f84cc157dc0 2024-09-16T01:50:22.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.529+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc1e5430 con 0x7f84cc157dc0 2024-09-16T01:50:22.531 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:22.531 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:22.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.530+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f84cc14e420 0x7f84cc0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.530+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f84cc0b3b70 con 0x7f84cc14e420 2024-09-16T01:50:22.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.530+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f84cc14e420 0x7f84cc0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.531+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f84cc14e420 0x7f84cc0b3630 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.531+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc0b3b70 con 0x7f84cc14e420 2024-09-16T01:50:22.532 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:22.533 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:22.533 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:22.533 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:22.533 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:22.542 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:22.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.546+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc0b3f70 con 0x7f84cc14e420 2024-09-16T01:50:22.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.547+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc0b3f70 con 0x7f84cc14e420 2024-09-16T01:50:22.549 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:22.549 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:22.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.547+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f84cc062f00 0x7f84cc1e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.547+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f84cc063380 con 0x7f84cc062f00 2024-09-16T01:50:22.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.548+0000 7f84cb400640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f84cc062f00 0x7f84cc1e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.548+0000 7f84cb400640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f84cc062f00 0x7f84cc1e6c70 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.549+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc063380 con 0x7f84cc062f00 2024-09-16T01:50:22.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.563+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc1e7400 con 0x7f84cc062f00 2024-09-16T01:50:22.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.565+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc1e7400 con 0x7f84cc062f00 2024-09-16T01:50:22.566 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:22.566 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:22.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.565+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f84cc1e79d0 0x7f84cc02cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.565+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f84cc1e7400 con 0x7f84cc1e79d0 2024-09-16T01:50:22.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.565+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f84cc1e79d0 0x7f84cc02cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.566+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f84cc1e79d0 0x7f84cc02cb50 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.566+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc1e7400 con 0x7f84cc1e79d0 2024-09-16T01:50:22.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.580+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc1e7400 con 0x7f84cc1e79d0 2024-09-16T01:50:22.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.582+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc1e7400 con 0x7f84cc1e79d0 2024-09-16T01:50:22.583 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:22.583 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:22.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.582+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f84cc02d5a0 0x7f84cc207060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.582+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f84cc0635e0 con 0x7f84cc02d5a0 2024-09-16T01:50:22.584 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:22.582+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:22.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.582+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f84cc02d5a0 0x7f84cc207060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.583+0000 7f84ca000640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f84cc02d5a0 0x7f84cc207060 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.583+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc0635e0 con 0x7f84cc02d5a0 2024-09-16T01:50:22.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.597+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc0635e0 con 0x7f84cc02d5a0 2024-09-16T01:50:22.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.599+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc0635e0 con 0x7f84cc02d5a0 2024-09-16T01:50:22.600 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:22.601 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:22.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.599+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f84cc207ab0 0x7f84cc209ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.599+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f84cc02da20 con 0x7f84cc207ab0 2024-09-16T01:50:22.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.599+0000 7f84cb400640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f84cc207ab0 0x7f84cc209ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.600+0000 7f84cb400640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f84cc207ab0 0x7f84cc209ea0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.601+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc02da20 con 0x7f84cc207ab0 2024-09-16T01:50:22.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.615+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc02da20 con 0x7f84cc207ab0 2024-09-16T01:50:22.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.617+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc02da20 con 0x7f84cc207ab0 2024-09-16T01:50:22.619 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:22.619 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:22.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.618+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f84cc20a8f0 0x7f84cc20cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:22.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.618+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f84cc20fbb0 con 0x7f84cc20a8f0 2024-09-16T01:50:22.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.618+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f84cc20a8f0 0x7f84cc20cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:22.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.619+0000 7f84caa00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f84cc20a8f0 0x7f84cc20cce0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:22.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.619+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84cc20fbb0 con 0x7f84cc20a8f0 2024-09-16T01:50:22.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.633+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f84cc20fb80 con 0x7f84cc20a8f0 2024-09-16T01:50:22.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84c2a00640 1 -- 172.21.15.146:0/3662031860 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f84cc20fb80 con 0x7f84cc20a8f0 2024-09-16T01:50:22.636 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f84cc1574a0 msgr2=0x7f84cc157880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f84cc1574a0 0x7f84cc157880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f84cc157dc0 msgr2=0x7f84cc14dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f84cc157dc0 0x7f84cc14dd70 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f84cc14e420 msgr2=0x7f84cc0b3630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f84cc14e420 0x7f84cc0b3630 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f84cc062f00 msgr2=0x7f84cc1e6c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f84cc062f00 0x7f84cc1e6c70 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f84cc1e79d0 msgr2=0x7f84cc02cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f84cc1e79d0 0x7f84cc02cb50 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f84cc02d5a0 msgr2=0x7f84cc207060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f84cc02d5a0 0x7f84cc207060 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f84cc207ab0 msgr2=0x7f84cc209ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f84cc207ab0 0x7f84cc209ea0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f84cc20a8f0 msgr2=0x7f84cc20cce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f84cc20a8f0 0x7f84cc20cce0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cb400640 1 -- 172.21.15.146:0/3662031860 reap_dead start 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cb400640 1 -- 172.21.15.146:0/3662031860 reap_dead start 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cb400640 1 -- 172.21.15.146:0/3662031860 reap_dead start 2024-09-16T01:50:22.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cb400640 1 -- 172.21.15.146:0/3662031860 reap_dead start 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cb400640 1 -- 172.21.15.146:0/3662031860 reap_dead start 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f84ac06c2b0 msgr2=0x7f84ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f84ac06c2b0 0x7f84ac06e770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84cc155a40 tx=0x7f84b8008040 comp rx=0 tx=0).stop 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 msgr2=0x7f84cc1e4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.635+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 0x7f84cc1e4ef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00c500 tx=0x7f84bc00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.636+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 shutdown_connections 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.636+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f84ac06c2b0 0x7f84ac06e770 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.636+0000 7f84cbe00640 1 --2- 172.21.15.146:0/3662031860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84cc0c6d00 0x7f84cc1e4ef0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.636+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 >> 172.21.15.146:0/3662031860 conn(0x7f84cc06d6f0 msgr2=0x7f84cc0ccf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.636+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 shutdown_connections 2024-09-16T01:50:22.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:22.636+0000 7f84cbe00640 1 -- 172.21.15.146:0/3662031860 wait complete. 2024-09-16T01:50:22.648 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:22.649 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:22.657 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:22.693 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:22.692+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:22.761 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:22.762 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:22.770 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:22.814 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:22.813+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:22.873 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:22.874 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:22.882 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:22.886 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:22.885+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:22.986 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:22.985+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:23.028 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:23.028 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:23.037 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:23.122 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:23.121+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:23.181 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:23.181 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:23.188 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:23.187+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:23.189 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:23.289 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:23.288+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:23.332 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:23.579 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:23.581 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:23.582 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:23.580+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:23.682 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:23.680+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:23.686 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:23.687 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:23.695 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:23.782 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:23.780+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:23.839 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:23.839 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:23.848 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:23.892 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:23.891+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:23.951 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:23.951 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:23.960 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:24.001 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:24.000+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:24.084 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:24.083+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:24.105 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:24.105 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:24.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:24.185 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:24.183+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:24.286 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:24.285+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:24.311 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:24.311 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:24.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:24.416 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:24.414+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:24.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:50:24.528 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:24.526+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:24.588 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:24.587+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:24.640 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:24.641 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:24.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:24.661 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.697+0000 7f75b7600640 1 Processor -- start 2024-09-16T01:50:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.698+0000 7f75b7600640 1 -- start start 2024-09-16T01:50:24.700 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:24.698+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.699+0000 7f75b7600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b015aab0 0x7f75b00cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.699+0000 7f75b7600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b015af00 con 0x7f75b00cb2f0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.699+0000 7f75b7600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b015b1d0 con 0x7f75b00cc0c0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.699+0000 7f75b7600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75b00cbd00 con 0x7f75b015aab0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.700+0000 7f75b4c00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f75b00cc0c0 0x7f75b015e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56834/0 (socket says 172.21.15.146:56834) 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.700+0000 7f75b4c00640 1 -- 172.21.15.146:0/4177800333 learned_addr learned my addr 172.21.15.146:0/4177800333 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.700+0000 7f75afe00640 1 --1- 172.21.15.146:0/4177800333 >> v1:172.21.15.146:6789/0 conn(0x7f75b00cb2f0 0x7f75b0163420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:37108/0 (socket says 172.21.15.146:37108) 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.700+0000 7f75b5600640 1 --2- 172.21.15.146:0/4177800333 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b015aab0 0x7f75b00cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.700+0000 7f75af400640 1 -- 172.21.15.146:0/4177800333 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4044093425 0 0) 0x7f75b015b1d0 con 0x7f75b00cc0c0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.700+0000 7f75af400640 1 -- 172.21.15.146:0/4177800333 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7598003520 con 0x7f75b00cc0c0 2024-09-16T01:50:24.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.701+0000 7f75b5600640 1 -- 172.21.15.146:0/4177800333 >> v1:172.21.15.161:6789/0 conn(0x7f75b00cc0c0 legacy=0x7f75b015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.701+0000 7f75b5600640 1 -- 172.21.15.146:0/4177800333 >> v1:172.21.15.146:6789/0 conn(0x7f75b00cb2f0 legacy=0x7f75b0163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.701+0000 7f75b5600640 1 -- 172.21.15.146:0/4177800333 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b015b4b0 con 0x7f75b015aab0 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.702+0000 7f75b5600640 1 --2- 172.21.15.146:0/4177800333 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b015aab0 0x7f75b00cc4a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009dc0 tx=0x7f75a802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b7e8ff8d666e16 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.702+0000 7f75af400640 1 -- 172.21.15.146:0/4177800333 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8007740 con 0x7f75b015aab0 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.702+0000 7f75af400640 1 -- 172.21.15.146:0/4177800333 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a80078e0 con 0x7f75b015aab0 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.702+0000 7f75af400640 1 -- 172.21.15.146:0/4177800333 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8007cf0 con 0x7f75b015aab0 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.703+0000 7f75b7600640 1 -- 172.21.15.146:0/4177800333 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b015aab0 msgr2=0x7f75b00cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.703+0000 7f75b7600640 1 --2- 172.21.15.146:0/4177800333 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b015aab0 0x7f75b00cc4a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009dc0 tx=0x7f75a802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.704+0000 7f75b7600640 1 -- 172.21.15.146:0/4177800333 shutdown_connections 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.704+0000 7f75b7600640 1 --2- 172.21.15.146:0/4177800333 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b015aab0 0x7f75b00cc4a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.704+0000 7f75b7600640 1 -- 172.21.15.146:0/4177800333 >> 172.21.15.146:0/4177800333 conn(0x7f75b006d560 msgr2=0x7f75b006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:24.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.704+0000 7f75b7600640 1 -- 172.21.15.146:0/4177800333 shutdown_connections 2024-09-16T01:50:24.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.704+0000 7f75b7600640 1 -- 172.21.15.146:0/4177800333 wait complete. 2024-09-16T01:50:24.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.706+0000 7f75b7600640 1 Processor -- start 2024-09-16T01:50:24.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.707+0000 7f75b7600640 1 -- start start 2024-09-16T01:50:24.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.708+0000 7f75b7600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 0x7f75b01e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.708+0000 7f75b7600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cc0c0 0x7f75b0155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.708+0000 7f75b7600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b015aab0 0x7f75b01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.708+0000 7f75b7600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f75b015e540 con 0x7f75b00cb2f0 2024-09-16T01:50:24.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.708+0000 7f75b7600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75b015ba70 con 0x7f75b015aab0 2024-09-16T01:50:24.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.708+0000 7f75b7600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75b015e1d0 con 0x7f75b00cc0c0 2024-09-16T01:50:24.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.709+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 0x7f75b01e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.709+0000 7f75b4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 0x7f75b01e4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47614/0 (socket says 172.21.15.146:47614) 2024-09-16T01:50:24.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.709+0000 7f75b4c00640 1 -- 172.21.15.146:0/845560637 learned_addr learned my addr 172.21.15.146:0/845560637 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:24.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.709+0000 7f75b5600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b015aab0 0x7f75b01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.709+0000 7f75afe00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cc0c0 0x7f75b0155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b4c00640 1 -- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cc0c0 msgr2=0x7f75b0155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b4c00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cc0c0 0x7f75b0155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b4c00640 1 -- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b015aab0 msgr2=0x7f75b01573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b4c00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b015aab0 0x7f75b01573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b4c00640 1 -- 172.21.15.146:0/845560637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b0157da0 con 0x7f75b00cb2f0 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b5600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b015aab0 0x7f75b01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:24.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75b4c00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 0x7f75b01e4f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f75a400c530 tx=0x7f75a400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.710+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4017070 con 0x7f75b00cb2f0 2024-09-16T01:50:24.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.711+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a4005ce0 con 0x7f75b00cb2f0 2024-09-16T01:50:24.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.711+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4017070 con 0x7f75b00cb2f0 2024-09-16T01:50:24.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.712+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b0158080 con 0x7f75b00cb2f0 2024-09-16T01:50:24.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.712+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b01e62f0 con 0x7f75b00cb2f0 2024-09-16T01:50:24.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.714+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b00c5560 con 0x7f75b00cb2f0 2024-09-16T01:50:24.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.716+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f75a401a070 con 0x7f75b00cb2f0 2024-09-16T01:50:24.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.717+0000 7f75acc00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759006c4e0 0x7f759006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.717+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5770+0+0 (secure 0 0 0) 0x7f75a408a310 con 0x7f75b00cb2f0 2024-09-16T01:50:24.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.718+0000 7f75afe00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759006c4e0 0x7f759006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.718+0000 7f75afe00640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759006c4e0 0x7f759006e9a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f75b01e5c30 tx=0x7f759c000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.724+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f75a40583f0 con 0x7f75b00cb2f0 2024-09-16T01:50:24.791 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:24.789+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:24.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.853+0000 7f80f9800640 1 Processor -- start 2024-09-16T01:50:24.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.853+0000 7f80f9800640 1 -- start start 2024-09-16T01:50:24.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f9800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40d0870 0x7f80f4154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f9800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f80f40c6d00 0x7f80f4154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f9800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80f415e220 con 0x7f80f40d0490 2024-09-16T01:50:24.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f9800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f80f40c6160 con 0x7f80f40c6d00 2024-09-16T01:50:24.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f9800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f80f415e500 con 0x7f80f40d0870 2024-09-16T01:50:24.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f3e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f80f40d0490 0x7f80f40c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:37110/0 (socket says 172.21.15.146:37110) 2024-09-16T01:50:24.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f3e00640 1 -- 172.21.15.146:0/2632201993 learned_addr learned my addr 172.21.15.146:0/2632201993 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:24.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.855+0000 7f80f8e00640 1 --2- 172.21.15.146:0/2632201993 >> v2:172.21.15.161:3300/0 conn(0x7f80f40c6d00 0x7f80f4154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.856+0000 7f80f3400640 1 --2- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40d0870 0x7f80f4154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.856+0000 7f80f8e00640 1 -- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40d0870 msgr2=0x7f80f4154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:24.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.856+0000 7f80f8e00640 1 --2- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40d0870 0x7f80f4154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.856+0000 7f80f8e00640 1 -- 172.21.15.146:0/2632201993 >> v1:172.21.15.146:6789/0 conn(0x7f80f40d0490 legacy=0x7f80f40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:24.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.857+0000 7f80f8e00640 1 -- 172.21.15.146:0/2632201993 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f415e7e0 con 0x7f80f40c6d00 2024-09-16T01:50:24.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.858+0000 7f80f8e00640 1 --2- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40c6d00 0x7f80f4154bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009d90 tx=0x7f80ec02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb6e2eb56fc42498 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.858+0000 7f80f2a00640 1 -- 172.21.15.146:0/2632201993 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec007740 con 0x7f80f40c6d00 2024-09-16T01:50:24.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.858+0000 7f80f2a00640 1 -- 172.21.15.146:0/2632201993 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ec0078e0 con 0x7f80f40c6d00 2024-09-16T01:50:24.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.858+0000 7f80f2a00640 1 -- 172.21.15.146:0/2632201993 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec007cb0 con 0x7f80f40c6d00 2024-09-16T01:50:24.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 -- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40c6d00 msgr2=0x7f80f4154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 --2- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40c6d00 0x7f80f4154bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009d90 tx=0x7f80ec02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 -- 172.21.15.146:0/2632201993 shutdown_connections 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 --2- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40d0870 0x7f80f4154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 --2- 172.21.15.146:0/2632201993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40c6d00 0x7f80f4154bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 -- 172.21.15.146:0/2632201993 >> 172.21.15.146:0/2632201993 conn(0x7f80f406d6f0 msgr2=0x7f80f406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 -- 172.21.15.146:0/2632201993 shutdown_connections 2024-09-16T01:50:24.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.859+0000 7f80f9800640 1 -- 172.21.15.146:0/2632201993 wait complete. 2024-09-16T01:50:24.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.861+0000 7f80f9800640 1 Processor -- start 2024-09-16T01:50:24.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.861+0000 7f80f9800640 1 -- start start 2024-09-16T01:50:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.862+0000 7f80f9800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40c6d00 0x7f80f4148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.862+0000 7f80f9800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40d0490 0x7f80f41f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f9800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 0x7f80f4157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40c6d00 0x7f80f4148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40c6d00 0x7f80f4148fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47264/0 (socket says 172.21.15.146:47264) 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 0x7f80f4157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 0x7f80f4157450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47626/0 (socket says 172.21.15.146:47626) 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40d0490 0x7f80f41f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40d0490 0x7f80f41f0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54318/0 (socket says 172.21.15.146:54318) 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.863+0000 7f80f3e00640 1 -- 172.21.15.146:0/3155816696 learned_addr learned my addr 172.21.15.146:0/3155816696 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.864+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f80f415dec0 con 0x7f80f40d0870 2024-09-16T01:50:24.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.864+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f80f415eda0 con 0x7f80f40d0490 2024-09-16T01:50:24.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.864+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f80f40c51f0 con 0x7f80f40c6d00 2024-09-16T01:50:24.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.864+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40c6d00 msgr2=0x7f80f4148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.864+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f80f40c6d00 0x7f80f4148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.865+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40d0490 msgr2=0x7f80f41f0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:24.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.865+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f80f40d0490 0x7f80f41f0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:24.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.865+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f4151b90 con 0x7f80f40d0870 2024-09-16T01:50:24.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.865+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 0x7f80f4157450 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80ec031000 tx=0x7f80ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.865+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec040070 con 0x7f80f40d0870 2024-09-16T01:50:24.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.866+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ec031c00 con 0x7f80f40d0870 2024-09-16T01:50:24.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.866+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f40c7ff0 con 0x7f80f40d0870 2024-09-16T01:50:24.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.866+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec009450 con 0x7f80f40d0870 2024-09-16T01:50:24.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.866+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f414ec40 con 0x7f80f40d0870 2024-09-16T01:50:24.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.867+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f80f40c65f0 con 0x7f80f40d0870 2024-09-16T01:50:24.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.868+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f80ec039040 con 0x7f80f40d0870 2024-09-16T01:50:24.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.869+0000 7f80ebe00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f80d406c2b0 0x7f80d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.869+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5770+0+0 (secure 0 0 0) 0x7f80ec0acb10 con 0x7f80f40d0870 2024-09-16T01:50:24.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.870+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f80d406c2b0 0x7f80d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.870+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f80d406c2b0 0x7f80d406e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80e4001720 tx=0x7f80e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.872+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v44) ==== 54+0+16 (secure 0 0 0) 0x7f80ec07b2d0 con 0x7f80f40d0870 2024-09-16T01:50:24.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.872+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f80f414ef20 0x7f80f4152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.872+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80f414f300 con 0x7f80f414ef20 2024-09-16T01:50:24.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.873+0000 7f80f3400640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f80f414ef20 0x7f80f4152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.873+0000 7f80f3400640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f80f414ef20 0x7f80f4152ca0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.874+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f414f300 con 0x7f80f414ef20 2024-09-16T01:50:24.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.890+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f40cc3f0 con 0x7f80f414ef20 2024-09-16T01:50:24.892 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:24.890+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:24.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.893+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f40cc3f0 con 0x7f80f414ef20 2024-09-16T01:50:24.895 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:24.895 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:24.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.893+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f80f41f41b0 0x7f80f41f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.893+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f80f41f4ad0 con 0x7f80f41f41b0 2024-09-16T01:50:24.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.894+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f80f41f41b0 0x7f80f41f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.895+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f80f41f41b0 0x7f80f41f4590 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.896+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f41f4ad0 con 0x7f80f41f41b0 2024-09-16T01:50:24.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.923+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f41524a0 con 0x7f80f41f41b0 2024-09-16T01:50:24.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.928+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f41524a0 con 0x7f80f41f41b0 2024-09-16T01:50:24.930 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.928+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f80f4062b40 0x7f80f4062f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.928+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f80f41e6560 con 0x7f80f4062b40 2024-09-16T01:50:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.929+0000 7f80f3400640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f80f4062b40 0x7f80f4062f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.930+0000 7f80f3400640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f80f4062b40 0x7f80f4062f20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:24.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.931+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f41e6560 con 0x7f80f4062b40 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.135444453999995, 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:50:24.966 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:24.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:24.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:50:24.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:24.973 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.135338404999999, 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:24.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:50:24.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:24.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.135283779000005, 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:24.979 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:50:24.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:24.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:50:24.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.135215446000004, 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:24.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:50:24.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:50:24.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:24.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:24.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.100960243000003, 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:24.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:50:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:24.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.996 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.100777045000001, 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:50:24.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:50:25.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:25.002 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.100723036999995, 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:25.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:50:25.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.100690596999996, 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:50:25.008 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:50:25.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:50:25.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.097322595999998, 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:50:25.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.097251920000005, 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:25.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:50:25.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.097215544999997, 2024-09-16T01:50:25.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.027 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.096587802000002, 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:50:25.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:25.032 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.096533195000006, 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:25.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.036 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024394786999999, 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:50:25.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024336474999998, 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:50:25.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024262399999998, 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:50:25.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024203778999997, 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:50:25.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:25.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024181845000001, 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:50:25.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:25.066 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024151484999997, 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:25.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:50:25.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:50:25.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.024124905999997, 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:50:25.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:50:25.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:25.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.077 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:25.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.963+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f41f0ad0 con 0x7f80f4062b40 2024-09-16T01:50:25.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.965+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f41f0ad0 con 0x7f80f4062b40 2024-09-16T01:50:25.078 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:25.079 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:25.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.966+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f80f4155790 0x7f80f4155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:25.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.966+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f80f41f08a0 con 0x7f80f4155790 2024-09-16T01:50:25.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.967+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f80f4155790 0x7f80f4155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:25.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.967+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f80f4155790 0x7f80f4155bd0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:25.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.968+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f41f08a0 con 0x7f80f4155790 2024-09-16T01:50:25.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.982+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f41e5390 con 0x7f80f4155790 2024-09-16T01:50:25.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:25.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.984+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f41e5390 con 0x7f80f4155790 2024-09-16T01:50:25.081 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.984+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f80f402cb50 0x7f80f402cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.984+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f80f41e5390 con 0x7f80f402cb50 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.986+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f80f402cb50 0x7f80f402cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.986+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f80f402cb50 0x7f80f402cfb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:24.987+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f41e5390 con 0x7f80f402cb50 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.001+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f41e5390 con 0x7f80f402cb50 2024-09-16T01:50:25.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.003+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f41e5390 con 0x7f80f402cb50 2024-09-16T01:50:25.084 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:25.084 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:25.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.003+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f80f4205040 0x7f80f4207430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:25.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.003+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f80f41e5390 con 0x7f80f4205040 2024-09-16T01:50:25.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.004+0000 7f80f3400640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f80f4205040 0x7f80f4207430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:25.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.004+0000 7f80f3400640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f80f4205040 0x7f80f4207430 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:25.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.005+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f41e5390 con 0x7f80f4205040 2024-09-16T01:50:25.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.019+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f41e5390 con 0x7f80f4205040 2024-09-16T01:50:25.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.021+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f41e5390 con 0x7f80f4205040 2024-09-16T01:50:25.089 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:25.090 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:25.082+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:25.090 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:25.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.021+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f80f4207e80 0x7f80f420a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.021+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f80f41f1520 con 0x7f80f4207e80 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.022+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f80f4207e80 0x7f80f420a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.022+0000 7f80f8e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f80f4207e80 0x7f80f420a270 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.023+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f41f1520 con 0x7f80f4207e80 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.037+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f40756f0 con 0x7f80f4207e80 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.039+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f40756f0 con 0x7f80f4207e80 2024-09-16T01:50:25.091 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:25.092 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:25.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.039+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f80f420acc0 0x7f80f420d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:25.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.039+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f80f420ff30 con 0x7f80f420acc0 2024-09-16T01:50:25.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.040+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f80f420acc0 0x7f80f420d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:25.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.040+0000 7f80f3e00640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f80f420acc0 0x7f80f420d0b0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:25.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.041+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80f420ff30 con 0x7f80f420acc0 2024-09-16T01:50:25.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.055+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f80f420d5f0 con 0x7f80f420acc0 2024-09-16T01:50:25.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80ebe00640 1 -- 172.21.15.146:0/3155816696 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f80f420d5f0 con 0x7f80f420acc0 2024-09-16T01:50:25.093 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:25.094 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:25.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f80f414ef20 msgr2=0x7f80f4152ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f80f414ef20 0x7f80f4152ca0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f80f41f41b0 msgr2=0x7f80f41f4590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f80f41f41b0 0x7f80f41f4590 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f80f4062b40 msgr2=0x7f80f4062f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f80f4062b40 0x7f80f4062f20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f80f4155790 msgr2=0x7f80f4155bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f80f4155790 0x7f80f4155bd0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f80f402cb50 msgr2=0x7f80f402cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f80f402cb50 0x7f80f402cfb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f80f4205040 msgr2=0x7f80f4207430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f80f4205040 0x7f80f4207430 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f80f4207e80 msgr2=0x7f80f420a270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f80f4207e80 0x7f80f420a270 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f80f420acc0 msgr2=0x7f80f420d0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f80f420acc0 0x7f80f420d0b0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.057+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 reap_dead start 2024-09-16T01:50:25.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 reap_dead start 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 reap_dead start 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 reap_dead start 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f8e00640 1 -- 172.21.15.146:0/3155816696 reap_dead start 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f80d406c2b0 msgr2=0x7f80d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f80d406c2b0 0x7f80d406e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80e4001720 tx=0x7f80e4009040 comp rx=0 tx=0).stop 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 msgr2=0x7f80f4157450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 0x7f80f4157450 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80ec031000 tx=0x7f80ec004a40 comp rx=0 tx=0).stop 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 shutdown_connections 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f80d406c2b0 0x7f80d406e770 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80e4001720 tx=0x7f80e4009040 comp rx=0 tx=0).stop 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 --2- 172.21.15.146:0/3155816696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80f40d0870 0x7f80f4157450 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 >> 172.21.15.146:0/3155816696 conn(0x7f80f406d6f0 msgr2=0x7f80f4154460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 shutdown_connections 2024-09-16T01:50:25.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.058+0000 7f80f9800640 1 -- 172.21.15.146:0/3155816696 wait complete. 2024-09-16T01:50:25.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.073+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f75b01553a0 con 0x7f75b00cb2f0 2024-09-16T01:50:25.184 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:25.182+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:25.285 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:25.283+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:25.290 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:25.290 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:25.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:25.385 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:25.384+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:25.486 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:25.484+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:25.541 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:25.541 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:25.541 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:25.541 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:25.541 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:25.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:25.587 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:25.586+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.595+0000 7f75acc00640 1 -- 172.21.15.146:0/845560637 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v45) ==== 101+0+0 (secure 0 0 0) 0x7f75a405c6f0 con 0x7f75b00cb2f0 2024-09-16T01:50:25.597 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.598+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759006c4e0 msgr2=0x7f759006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.598+0000 7f75b7600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759006c4e0 0x7f759006e9a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f75b01e5c30 tx=0x7f759c000ed0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 msgr2=0x7f75b01e4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 0x7f75b01e4f30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f75a400c530 tx=0x7f75a400c9f0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 shutdown_connections 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759006c4e0 0x7f759006e9a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b015aab0 0x7f75b01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75b00cc0c0 0x7f75b0155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 --2- 172.21.15.146:0/845560637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b00cb2f0 0x7f75b01e4f30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:25.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.599+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 >> 172.21.15.146:0/845560637 conn(0x7f75b006d560 msgr2=0x7f75b00cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:25.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.600+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 shutdown_connections 2024-09-16T01:50:25.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:25.600+0000 7f75b7600640 1 -- 172.21.15.146:0/845560637 wait complete. 2024-09-16T01:50:25.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:50:25.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:50:25.719 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:25.718+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:25.789 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:25.787+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:25.812 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.041477968999999, 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:25.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:25.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.609924237000001, 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:25.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:25.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.826 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.590000015000001, 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:25.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.589980096999994, 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:25.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:25.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:25.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.589919512999998, 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:25.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:25.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:25.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.343633992999997, 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:25.845 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:25.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:25.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:25.851 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.343483479, 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:25.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.343463911000001, 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:25.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:25.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:25.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.773307697000007, 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:25.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:25.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.773139878999999, 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:25.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.773120896999998, 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.876 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:25.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.773101265999998, 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:25.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:25.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.772967359999996, 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.772955822, 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:25.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:25.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.772937411000001, 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:25.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.772822740999999, 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:25.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:25.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:25.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.762041013000001, 2024-09-16T01:50:25.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:25.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:25.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.761924355000005, 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:25.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:25.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.946391116000001, 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:25.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.946329778000006, 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:25.930 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:25.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:25.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:25.935 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:25.935 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:25.937 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:25.935+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:25.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:26.038 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:26.036+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:26.139 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:26.137+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:26.162 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:26.162 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:26.162 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:26.162 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:26.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:26.239 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:26.237+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:26.339 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:26.338+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:26.397 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:26.397 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:26.397 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:26.397 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:26.397 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:26.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:26.458 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:26.456+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:26.542 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:26.541+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:26.582 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.755110693000006, 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:26.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:26.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.754995871999995, 2024-09-16T01:50:26.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:26.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:26.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:26.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:26.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.754954002000005, 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:26.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:26.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.754909722999997, 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:26.604 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:26.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:26.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:26.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.754880846000006, 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:26.610 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:26.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:26.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.751173639000001, 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:26.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:26.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:26.619 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.751119657999993, 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:26.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.715704434000003, 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.623 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:26.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:26.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.712644284999996, 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:26.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:26.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:26.632 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.711784836999996, 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.711730943999996, 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.638 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:26.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:26.643 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.705706970999998, 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:26.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:26.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.705656843, 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:26.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:26.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.705580510000004, 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:26.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.705549155, 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:26.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:26.662 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 65.667463699999999, 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:26.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:26.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 65.666672168999995, 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:26.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.642590343999998, 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:26.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:26.677 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.642518432000003, 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:26.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.311876032000001, 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:26.683 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:26.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:26.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:26.689 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:26.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:26.692 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:26.690+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:26.792 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:26.791+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:26.893 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:26.891+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:26.936 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:26.937 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:26.937 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:26.937 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:26.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:26.994 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:26.993+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:27.095 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:27.093+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:27.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:27.196 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:27.195+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:27.211 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:27.211 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:27.211 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:27.211 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:27.211 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:27.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:27.263 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:27.264 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:27.264 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T01:50:27.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 0.05 2024-09-16T01:50:27.297 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:27.296+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:27.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.351+0000 7ff439600640 1 Processor -- start 2024-09-16T01:50:27.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.351+0000 7ff439600640 1 -- start start 2024-09-16T01:50:27.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.353+0000 7ff439600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff42c007340 0x7ff42c04ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.353+0000 7ff439600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42c050520 con 0x7ff42c04f870 2024-09-16T01:50:27.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.353+0000 7ff439600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42c050800 con 0x7ff42c0d8bb0 2024-09-16T01:50:27.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.353+0000 7ff439600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff42c04f070 con 0x7ff42c007340 2024-09-16T01:50:27.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.354+0000 7ff433e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff42c0d8bb0 0x7ff42c00e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56838/0 (socket says 172.21.15.146:56838) 2024-09-16T01:50:27.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.354+0000 7ff433e00640 1 -- 172.21.15.146:0/3149166936 learned_addr learned my addr 172.21.15.146:0/3149166936 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:27.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.354+0000 7ff433400640 1 --2- 172.21.15.146:0/3149166936 >> v2:172.21.15.161:3300/0 conn(0x7ff42c007340 0x7ff42c04ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205581860 0 0) 0x7ff42c050800 con 0x7ff42c0d8bb0 2024-09-16T01:50:27.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff41c003520 con 0x7ff42c0d8bb0 2024-09-16T01:50:27.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090238982 0 0) 0x7ff42c050520 con 0x7ff42c04f870 2024-09-16T01:50:27.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff42c050800 con 0x7ff42c04f870 2024-09-16T01:50:27.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff433400640 1 -- 172.21.15.146:0/3149166936 >> v1:172.21.15.161:6789/0 conn(0x7ff42c0d8bb0 legacy=0x7ff42c00e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff433400640 1 -- 172.21.15.146:0/3149166936 >> v1:172.21.15.146:6789/0 conn(0x7ff42c04f870 legacy=0x7ff42c0e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.355+0000 7ff433400640 1 -- 172.21.15.146:0/3149166936 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c04f350 con 0x7ff42c007340 2024-09-16T01:50:27.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.357+0000 7ff433400640 1 --2- 172.21.15.146:0/3149166936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c007340 0x7ff42c04ea00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4200097a0 tx=0x7ff42002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b9f1bb7472dfc10 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.358+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420037070 con 0x7ff42c007340 2024-09-16T01:50:27.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.358+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff420032440 con 0x7ff42c007340 2024-09-16T01:50:27.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.358+0000 7ff432a00640 1 -- 172.21.15.146:0/3149166936 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4200327f0 con 0x7ff42c007340 2024-09-16T01:50:27.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.359+0000 7ff439600640 1 -- 172.21.15.146:0/3149166936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c007340 msgr2=0x7ff42c04ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.359+0000 7ff439600640 1 --2- 172.21.15.146:0/3149166936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c007340 0x7ff42c04ea00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff4200097a0 tx=0x7ff42002a230 comp rx=0 tx=0).stop 2024-09-16T01:50:27.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.360+0000 7ff439600640 1 -- 172.21.15.146:0/3149166936 shutdown_connections 2024-09-16T01:50:27.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.360+0000 7ff439600640 1 --2- 172.21.15.146:0/3149166936 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c007340 0x7ff42c04ea00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.360+0000 7ff439600640 1 -- 172.21.15.146:0/3149166936 >> 172.21.15.146:0/3149166936 conn(0x7ff42c054470 msgr2=0x7ff42c054880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:27.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.360+0000 7ff439600640 1 -- 172.21.15.146:0/3149166936 shutdown_connections 2024-09-16T01:50:27.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.360+0000 7ff439600640 1 -- 172.21.15.146:0/3149166936 wait complete. 2024-09-16T01:50:27.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.363+0000 7ff439600640 1 Processor -- start 2024-09-16T01:50:27.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.364+0000 7ff439600640 1 -- start start 2024-09-16T01:50:27.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.365+0000 7ff439600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff42c007340 0x7ff42c0d4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.365+0000 7ff439600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff42c04f870 0x7ff42c0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.365+0000 7ff439600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c0d8bb0 0x7ff42c175080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.365+0000 7ff439600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c049340 con 0x7ff42c007340 2024-09-16T01:50:27.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.365+0000 7ff439600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c0491c0 con 0x7ff42c0d8bb0 2024-09-16T01:50:27.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.365+0000 7ff439600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff42c00eae0 con 0x7ff42c04f870 2024-09-16T01:50:27.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff42c007340 0x7ff42c0d4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff42c007340 0x7ff42c0d4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47640/0 (socket says 172.21.15.146:47640) 2024-09-16T01:50:27.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433e00640 1 -- 172.21.15.146:0/2266474893 learned_addr learned my addr 172.21.15.146:0/2266474893 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:27.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff42c04f870 0x7ff42c0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433400640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c0d8bb0 msgr2=0x7ff42c175080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:50:27.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff42c0d8bb0 0x7ff42c175080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.366+0000 7ff433400640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff42c007340 msgr2=0x7ff42c0d4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.367+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff42c007340 0x7ff42c0d4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.367+0000 7ff433400640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c0d52c0 con 0x7ff42c04f870 2024-09-16T01:50:27.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.367+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff42c007340 0x7ff42c0d4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:27.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.368+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff42c04f870 0x7ff42c0d09b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0498e0 tx=0x7ff4200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.368+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420037050 con 0x7ff42c04f870 2024-09-16T01:50:27.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.368+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff420043070 con 0x7ff42c04f870 2024-09-16T01:50:27.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.368+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420006e10 con 0x7ff42c04f870 2024-09-16T01:50:27.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.369+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42c0d3f20 con 0x7ff42c04f870 2024-09-16T01:50:27.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.369+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42c0d41c0 con 0x7ff42c04f870 2024-09-16T01:50:27.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.370+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff42c1755c0 con 0x7ff42c04f870 2024-09-16T01:50:27.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.373+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff420032a30 con 0x7ff42c04f870 2024-09-16T01:50:27.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.374+0000 7ff427e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff41406c380 0x7ff41406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.375+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 5770+0+0 (secure 0 0 0) 0x7ff4200ac060 con 0x7ff42c04f870 2024-09-16T01:50:27.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.375+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v45) ==== 54+0+16 (secure 0 0 0) 0x7ff42c1755c0 con 0x7ff42c04f870 2024-09-16T01:50:27.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.375+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff41406c380 0x7ff41406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.376+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff42c0d4400 0x7ff42c0d77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.376+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff42c0d5e60 con 0x7ff42c0d4400 2024-09-16T01:50:27.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.376+0000 7ff438c00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff42c0d4400 0x7ff42c0d77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.377+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff41406c380 0x7ff41406e840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff428000f60 tx=0x7ff428009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.377+0000 7ff438c00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff42c0d4400 0x7ff42c0d77c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.379+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c0d5e60 con 0x7ff42c0d4400 2024-09-16T01:50:27.397 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:27.396+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:27.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.409+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c0dda30 con 0x7ff42c0d4400 2024-09-16T01:50:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.412+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c0dda30 con 0x7ff42c0d4400 2024-09-16T01:50:27.417 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.412+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff42c0ceed0 0x7ff42c177bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.412+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff42c0539f0 con 0x7ff42c0ceed0 2024-09-16T01:50:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.413+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff42c0ceed0 0x7ff42c177bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.414+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff42c0ceed0 0x7ff42c177bf0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.415+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c0539f0 con 0x7ff42c0ceed0 2024-09-16T01:50:27.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.445+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c054980 con 0x7ff42c0ceed0 2024-09-16T01:50:27.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.448+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c054980 con 0x7ff42c0ceed0 2024-09-16T01:50:27.450 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:27.450 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:27.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.448+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff42c178640 0x7ff42c0d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.448+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff42c004fc0 con 0x7ff42c178640 2024-09-16T01:50:27.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.449+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff42c178640 0x7ff42c0d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.449+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff42c178640 0x7ff42c0d63d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.451+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c004fc0 con 0x7ff42c178640 2024-09-16T01:50:27.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.478+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c0d4810 con 0x7ff42c178640 2024-09-16T01:50:27.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.481+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c0d4810 con 0x7ff42c178640 2024-09-16T01:50:27.483 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:27.483 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:27.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.481+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff42c001180 0x7ff42c184fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.481+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff42c0d71e0 con 0x7ff42c001180 2024-09-16T01:50:27.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.483+0000 7ff438c00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff42c001180 0x7ff42c184fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.484+0000 7ff438c00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff42c001180 0x7ff42c184fb0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.485+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c0d71e0 con 0x7ff42c001180 2024-09-16T01:50:27.497 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:27.497 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:27.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.669547594999997, 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063070110000000004, 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.498 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825667+0000", 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825672+0000", 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6920000000000001e-06 2024-09-16T01:50:27.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825689+0000", 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7039999999999999e-05 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825696+0000", 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5909999999999998e-06 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826617+0000", 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00092117800000000004 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826697+0000", 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9598999999999995e-05 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826832+0000", 2024-09-16T01:50:27.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013548 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829938+0000", 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031059059999999999 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831894+0000", 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019552240000000002 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831942+0000", 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8389e-05 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831973+0000", 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1043000000000001e-05 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.504 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:29 2.b 2:d0630e4c:::mds0_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.669506046999999, 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087271810000000005, 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825734+0000", 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5638000000000001e-05 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825743+0000", 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7350000000000002e-06 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825750+0000", 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827333+0000", 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015834689999999999 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827472+0000", 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138796 2024-09-16T01:50:27.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827572+0000", 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9910999999999999e-05 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832328+0000", 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047559079999999997 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834353+0000", 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002025654 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834392+0000", 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8099000000000002e-05 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834435+0000", 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3169e-05 2024-09-16T01:50:27.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:34 2.b 2:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.669437786000003, 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088227679999999999, 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825777+0000", 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:27.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825781+0000", 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0350000000000003e-06 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825788+0000", 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825794+0000", 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4240000000000001e-06 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827917+0000", 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021233570000000002 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828052+0000", 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013528300000000001 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834599+0000", 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065463650000000002 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628939920999997, 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012518068, 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.515 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866276+0000", 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866303+0000", 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7229e-05 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866329+0000", 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.563e-05 2024-09-16T01:50:27.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866343+0000", 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4287999999999999e-05 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867782+0000", 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014384510000000001 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867911+0000", 2024-09-16T01:50:27.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012957999999999999 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868028+0000", 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117025 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877878+0000", 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098499209999999993 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878708+0000", 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00082986900000000001 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878752+0000", 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3735000000000001e-05 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878792+0000", 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0025999999999999e-05 2024-09-16T01:50:27.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628839640999999, 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014487969, 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866375+0000", 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9899999999999999e-07 2024-09-16T01:50:27.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866379+0000", 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6519999999999997e-06 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866388+0000", 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9579999999999996e-06 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866395+0000", 2024-09-16T01:50:27.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7750000000000004e-06 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868287+0000", 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001892344 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868360+0000", 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3023999999999997e-05 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868473+0000", 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112321 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878461+0000", 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099884610000000006 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880760+0000", 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022984479999999998 2024-09-16T01:50:27.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880823+0000", 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3276999999999997e-05 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880862+0000", 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:7 2.b 2:d70a01c6:::60c.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628809602999993, 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020806099000000002, 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.82e-07 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866408+0000", 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0110000000000002e-06 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866415+0000", 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5830000000000003e-06 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866421+0000", 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6659999999999996e-06 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022638100000000002 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868842+0000", 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015727699999999999 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868942+0000", 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100182 2024-09-16T01:50:27.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878519+0000", 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095771399999999996 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887083+0000", 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085641750000000003 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887155+0000", 2024-09-16T01:50:27.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2117000000000001e-05 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887210+0000", 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4656e-05 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:8 2.b 2:d70a01c6:::60c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:27.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628771212000004, 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020917458, 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866443+0000", 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866447+0000", 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8340000000000001e-06 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866455+0000", 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866460+0000", 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5629999999999999e-06 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869260+0000", 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027996039999999998 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869328+0000", 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8730000000000001e-05 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869461+0000", 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264099999999999 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878525+0000", 2024-09-16T01:50:27.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090635630000000002 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887259+0000", 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087339310000000003 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887305+0000", 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6538000000000001e-05 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887360+0000", 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4706000000000001e-05 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:10 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.62874463, 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01479957, 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8100000000000003e-07 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866473+0000", 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1089999999999998e-06 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866481+0000", 2024-09-16T01:50:27.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866486+0000", 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.566e-06 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870818+0000", 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004332519 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870937+0000", 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011880899999999999 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871093+0000", 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015636400000000001 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877793+0000", 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066997560000000003 2024-09-16T01:50:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881167+0000", 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033735029999999999 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881242+0000", 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5139e-05 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881269+0000", 2024-09-16T01:50:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6794000000000001e-05 2024-09-16T01:50:27.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.544 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:27 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628706856999997, 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014854865, 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866511+0000", 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7479999999999999e-06 2024-09-16T01:50:27.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866518+0000", 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866525+0000", 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2750000000000001e-06 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875004+0000", 2024-09-16T01:50:27.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084799100000000002 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875082+0000", 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7698e-05 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875190+0000", 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010785000000000001 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877830+0000", 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026396480000000001 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881273+0000", 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034436839999999998 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881341+0000", 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7359000000000003e-05 2024-09-16T01:50:27.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881362+0000", 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0928e-05 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:33 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628681419000003, 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020942512, 2024-09-16T01:50:27.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866533+0000", 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.27e-07 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866552+0000", 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8921999999999999e-05 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866559+0000", 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2570000000000001e-06 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866564+0000", 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869714+0000", 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031497920000000002 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869828+0000", 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114331 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869968+0000", 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013971799999999999 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880969+0000", 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011001031 2024-09-16T01:50:27.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887401+0000", 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064320319999999999 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887441+0000", 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9937000000000001e-05 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887475+0000", 2024-09-16T01:50:27.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4010999999999997e-05 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:2 2.5 2:a6e36b3f:::401.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628642435000003, 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014499972999999999, 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.556 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866572+0000", 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8400000000000005e-07 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866575+0000", 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0929999999999999e-06 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866592+0000", 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7765999999999999e-05 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866598+0000", 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4219999999999999e-06 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871180+0000", 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045822939999999998 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871272+0000", 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1308000000000001e-05 2024-09-16T01:50:27.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871419+0000", 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014714 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875993+0000", 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004574486 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881013+0000", 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050202240000000002 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881046+0000", 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2944000000000002e-05 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881071+0000", 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4811999999999999e-05 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:18 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628594719000006, 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020977030000000001, 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:27.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1799999999999995e-07 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866623+0000", 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4709999999999999e-06 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866630+0000", 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866635+0000", 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3079999999999998e-06 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870303+0000", 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036679210000000002 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870398+0000", 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4464999999999998e-05 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870501+0000", 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010291800000000001 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880977+0000", 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010476014000000001 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887510+0000", 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065331759999999999 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887553+0000", 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2737000000000001e-05 2024-09-16T01:50:27.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887596+0000", 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3343000000000002e-05 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:22 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628570339000007, 2024-09-16T01:50:27.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01481146, 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:27.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866644+0000", 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866646+0000", 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2139999999999999e-06 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866652+0000", 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2160000000000003e-06 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866677+0000", 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4663000000000001e-05 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876491+0000", 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098146110000000009 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876616+0000", 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012430699999999999 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881455+0000", 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048389840000000002 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.571 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:24 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628529051000001, 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016598702999999999, 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5400000000000002e-07 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866688+0000", 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-09-16T01:50:27.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866695+0000", 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.883e-06 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866700+0000", 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7280000000000003e-06 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868221+0000", 2024-09-16T01:50:27.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015209620000000001 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868334+0000", 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113584 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883283+0000", 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014949100999999999 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:25 2.0 2:0250a5a6:::201.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.628494425, 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011987815000000001, 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866720+0000", 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6300000000000001e-07 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866738+0000", 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8218000000000001e-05 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866745+0000", 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7279999999999995e-06 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866751+0000", 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-09-16T01:50:27.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868589+0000", 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018382730000000001 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868709+0000", 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119571 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878707+0000", 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099980020000000006 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.2.11:5 2.0 2:03625d61:::614.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.599784936000006, 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081102819999999999, 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895434+0000", 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4909999999999999e-06 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895444+0000", 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0560000000000001e-05 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895451+0000", 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5590000000000002e-06 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896385+0000", 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093368200000000002 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896454+0000", 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9567000000000007e-05 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896559+0000", 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104609 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.899222+0000", 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026632000000000001 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903485+0000", 2024-09-16T01:50:27.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042627999999999997 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903517+0000", 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2345999999999998e-05 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903539+0000", 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1838e-05 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:14 2.1a 2:594e21d0:::623.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.583378697000001, 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068542239999999999, 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911838+0000", 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.435e-06 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911844+0000", 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.122e-06 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911849+0000", 2024-09-16T01:50:27.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6979999999999997e-06 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914481+0000", 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026327249999999998 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914555+0000", 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3886000000000004e-05 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:27.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914697+0000", 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014193800000000001 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918088+0000", 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033908559999999998 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918610+0000", 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052154100000000004 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918656+0000", 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-05 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918689+0000", 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3052000000000001e-05 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:23 2.18 2:1c1db442:::103.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.583245151, 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067043830000000004, 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911969+0000", 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911980+0000", 2024-09-16T01:50:27.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1090999999999999e-05 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911989+0000", 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6270000000000006e-06 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911994+0000", 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3639999999999998e-06 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912099+0000", 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010514900000000001 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912210+0000", 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111346 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912342+0000", 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131197 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915846+0000", 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035043269999999998 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918573+0000", 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027272289999999999 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918632+0000", 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8854000000000003e-05 2024-09-16T01:50:27.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918673+0000", 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0728e-05 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.558645820999999, 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084942400000000001, 2024-09-16T01:50:27.597 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000004e-07 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936594+0000", 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5137999999999999e-05 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936602+0000", 2024-09-16T01:50:27.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8999999999999995e-06 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936609+0000", 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9480000000000002e-06 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938373+0000", 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001763857 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938479+0000", 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105816 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938622+0000", 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014314699999999999 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941905+0000", 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032832 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944943+0000", 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030378670000000001 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945019+0000", 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5532999999999997e-05 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945062+0000", 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180999999999999e-05 2024-09-16T01:50:27.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.558580321999997, 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085927369999999996, 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936634+0000", 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936637+0000", 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.287e-06 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936647+0000", 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7189999999999996e-06 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936653+0000", 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.319e-06 2024-09-16T01:50:27.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938867+0000", 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022140079999999999 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938935+0000", 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7644999999999994e-05 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939072+0000", 2024-09-16T01:50:27.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013753600000000001 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941912+0000", 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028393350000000001 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945119+0000", 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032077160000000002 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:27.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945183+0000", 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3723e-05 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945226+0000", 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2805000000000001e-05 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:27.608 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:27.610 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:27.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.512+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c003860 con 0x7ff42c001180 2024-09-16T01:50:27.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.514+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c003860 con 0x7ff42c001180 2024-09-16T01:50:27.611 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.514+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff42c003590 0x7ff42c187a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.514+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff42c18a7e0 con 0x7ff42c003590 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.515+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff42c003590 0x7ff42c187a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.516+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff42c003590 0x7ff42c187a10 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.516+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c18a7e0 con 0x7ff42c003590 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.531+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c187f50 con 0x7ff42c003590 2024-09-16T01:50:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.533+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c187f50 con 0x7ff42c003590 2024-09-16T01:50:27.613 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:27.613 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:27.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.533+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff42c188670 0x7ff42c18aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.533+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff42c187f50 con 0x7ff42c188670 2024-09-16T01:50:27.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.534+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff42c188670 0x7ff42c18aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.534+0000 7ff433400640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff42c188670 0x7ff42c18aa60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.535+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c187f50 con 0x7ff42c188670 2024-09-16T01:50:27.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.539+0000 7f171d800640 1 Processor -- start 2024-09-16T01:50:27.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.539+0000 7f171d800640 1 -- start start 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f171d800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f171815bc50 0x7f171815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f171d800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1718154ea0 con 0x7f171815b190 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f171d800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171815c5c0 con 0x7f1718154ac0 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f171d800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f17180c5340 con 0x7f171815bc50 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f1717e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1718154ac0 0x7f171806d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56854/0 (socket says 172.21.15.146:56854) 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f1717e00640 1 -- 172.21.15.146:0/41484085 learned_addr learned my addr 172.21.15.146:0/41484085 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.541+0000 7f171ce00640 1 --2- 172.21.15.146:0/41484085 >> v2:172.21.15.161:3300/0 conn(0x7f171815bc50 0x7f171815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.542+0000 7f1716a00640 1 -- 172.21.15.146:0/41484085 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194993421 0 0) 0x7f1718154ea0 con 0x7f171815b190 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.542+0000 7f171ce00640 1 -- 172.21.15.146:0/41484085 >> v1:172.21.15.161:6789/0 conn(0x7f1718154ac0 legacy=0x7f171806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.542+0000 7f171ce00640 1 -- 172.21.15.146:0/41484085 >> v1:172.21.15.146:6789/0 conn(0x7f171815b190 legacy=0x7f17180d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.542+0000 7f171ce00640 1 -- 172.21.15.146:0/41484085 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17180c5620 con 0x7f171815bc50 2024-09-16T01:50:27.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.543+0000 7f171ce00640 1 --2- 172.21.15.146:0/41484085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815bc50 0x7f171815c080 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1710009d90 tx=0x7f171002a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f72b1ebe83362e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f1716a00640 1 -- 172.21.15.146:0/41484085 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710037070 con 0x7f171815bc50 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f1716a00640 1 -- 172.21.15.146:0/41484085 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1710007740 con 0x7f171815bc50 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f1716a00640 1 -- 172.21.15.146:0/41484085 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710007b10 con 0x7f171815bc50 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f171d800640 1 -- 172.21.15.146:0/41484085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815bc50 msgr2=0x7f171815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f171d800640 1 --2- 172.21.15.146:0/41484085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815bc50 0x7f171815c080 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1710009d90 tx=0x7f171002a780 comp rx=0 tx=0).stop 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f171d800640 1 -- 172.21.15.146:0/41484085 shutdown_connections 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f171d800640 1 --2- 172.21.15.146:0/41484085 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815bc50 0x7f171815c080 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.544+0000 7f171d800640 1 -- 172.21.15.146:0/41484085 >> 172.21.15.146:0/41484085 conn(0x7f171806e400 msgr2=0x7f171806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.545+0000 7f171d800640 1 -- 172.21.15.146:0/41484085 shutdown_connections 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.545+0000 7f171d800640 1 -- 172.21.15.146:0/41484085 wait complete. 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.546+0000 7f171d800640 1 Processor -- start 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.547+0000 7f171d800640 1 -- start start 2024-09-16T01:50:27.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.547+0000 7f171d800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 0x7f1718151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.547+0000 7f171d800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815b190 0x7f17180c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f171d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f171815bc50 0x7f17180c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f171d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f17180c5ec0 con 0x7f171815bc50 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f171d800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f17180c5be0 con 0x7f171815b190 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f171d800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f17180c61a0 con 0x7f1718154ac0 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 0x7f1718151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 0x7f1718151660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47278/0 (socket says 172.21.15.146:47278) 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 -- 172.21.15.146:0/4034604687 learned_addr learned my addr 172.21.15.146:0/4034604687 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717400640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815b190 0x7f17180c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f171ce00640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f171815bc50 0x7f17180c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 -- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815b190 msgr2=0x7f17180c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815b190 0x7f17180c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 -- 172.21.15.146:0/4034604687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f171815bc50 msgr2=0x7f17180c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f171815bc50 0x7f17180c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.548+0000 7f1717e00640 1 -- 172.21.15.146:0/4034604687 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171815a470 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f171ce00640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f171815bc50 0x7f17180c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f1717e00640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 0x7f1718151660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f170800c4d0 tx=0x7f170800c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708017070 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1708005ce0 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17180ca370 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f171815da60 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.549+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708007270 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.551+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f0005290 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.551+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f170801a070 con 0x7f1718154ac0 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.552+0000 7f170fe00640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f16f806c2b0 0x7f16f806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.552+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 5770+0+0 (secure 0 0 0) 0x7f170808a710 con 0x7f1718154ac0 2024-09-16T01:50:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.553+0000 7f1717400640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f16f806c2b0 0x7f16f806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.553+0000 7f1717400640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f16f806c2b0 0x7f16f806e770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1718151ee0 tx=0x7f1704009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.557+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1708058750 con 0x7f1718154ac0 2024-09-16T01:50:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.560+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c18afa0 con 0x7ff42c188670 2024-09-16T01:50:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.563+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c18afa0 con 0x7ff42c188670 2024-09-16T01:50:27.622 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:27.623 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:27.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.563+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff42c18b6c0 0x7ff42c18bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.563+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff42c190a50 con 0x7ff42c18b6c0 2024-09-16T01:50:27.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.563+0000 7ff438c00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff42c18b6c0 0x7ff42c18bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.564+0000 7ff438c00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff42c18b6c0 0x7ff42c18bb20 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.564+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c190a50 con 0x7ff42c18b6c0 2024-09-16T01:50:27.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.582+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c18f8d0 con 0x7ff42c18b6c0 2024-09-16T01:50:27.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.584+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c18f8d0 con 0x7ff42c18b6c0 2024-09-16T01:50:27.624 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.584+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff42c18fff0 0x7ff42c190450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.584+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff42c193dc0 con 0x7ff42c18fff0 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.585+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff42c18fff0 0x7ff42c190450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.585+0000 7ff433e00640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff42c18fff0 0x7ff42c190450 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.586+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c193dc0 con 0x7ff42c18fff0 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.600+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff42c18c060 con 0x7ff42c18fff0 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff427e00640 1 -- 172.21.15.146:0/2266474893 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff42c18c060 con 0x7ff42c18fff0 2024-09-16T01:50:27.626 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:27.627 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:27.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff42c0d4400 msgr2=0x7ff42c0d77c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff42c0d4400 0x7ff42c0d77c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff42c0ceed0 msgr2=0x7ff42c177bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff42c0ceed0 0x7ff42c177bf0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff42c178640 msgr2=0x7ff42c0d63d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff42c178640 0x7ff42c0d63d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff42c001180 msgr2=0x7ff42c184fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff42c001180 0x7ff42c184fb0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff42c003590 msgr2=0x7ff42c187a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff42c003590 0x7ff42c187a10 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff42c188670 msgr2=0x7ff42c18aa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.602+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff42c188670 0x7ff42c18aa60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff42c18b6c0 msgr2=0x7ff42c18bb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff42c18b6c0 0x7ff42c18bb20 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff42c18fff0 msgr2=0x7ff42c190450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff42c18fff0 0x7ff42c190450 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff438c00640 1 -- 172.21.15.146:0/2266474893 reap_dead start 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff438c00640 1 -- 172.21.15.146:0/2266474893 reap_dead start 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff41406c380 msgr2=0x7ff41406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff41406c380 0x7ff41406e840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff428000f60 tx=0x7ff428009040 comp rx=0 tx=0).stop 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff42c04f870 msgr2=0x7ff42c0d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff439600640 1 --2- 172.21.15.146:0/2266474893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff42c04f870 0x7ff42c0d09b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0498e0 tx=0x7ff4200046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff438c00640 1 -- 172.21.15.146:0/2266474893 reap_dead start 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff438c00640 1 -- 172.21.15.146:0/2266474893 reap_dead start 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.603+0000 7ff438c00640 1 -- 172.21.15.146:0/2266474893 reap_dead start 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.604+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 shutdown_connections 2024-09-16T01:50:27.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.604+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 >> 172.21.15.146:0/2266474893 conn(0x7ff42c054470 msgr2=0x7ff42c051f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:27.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.604+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 shutdown_connections 2024-09-16T01:50:27.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.604+0000 7ff439600640 1 -- 172.21.15.146:0/2266474893 wait complete. 2024-09-16T01:50:27.630 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:27.608+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:27.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:50:27.720 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:27.718+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:27.722 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:27.722 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:27.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:27.811 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:27.809+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:27.836 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:27.836 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:27.844 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:27.912 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:27.910+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:27.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.913+0000 7fd4f7200640 1 Processor -- start 2024-09-16T01:50:27.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.913+0000 7fd4f7200640 1 -- start start 2024-09-16T01:50:27.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.914+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7f16f0000bc0 con 0x7f16f806c2b0 2024-09-16T01:50:27.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4f015b340 0x7fd4f0166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f006d140 con 0x7fd4f00c75a0 2024-09-16T01:50:27.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f7200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f006d420 con 0x7fd4f00c6b70 2024-09-16T01:50:27.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd4f00cab20 con 0x7fd4f015b340 2024-09-16T01:50:27.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f5400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd4f00c75a0 0x7fd4f00cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:37140/0 (socket says 172.21.15.146:37140) 2024-09-16T01:50:27.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f5400640 1 -- 172.21.15.146:0/4017612468 learned_addr learned my addr 172.21.15.146:0/4017612468 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:moved 0 / 194 (0%) 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:avg 24.25 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:stddev 6.21993 -> 6.21993 (expected baseline 4.60638) 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:min osd.7 with 16 -> 16 pgs (0.659794 -> 0.659794 * mean) 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:max osd.0 with 35 -> 35 pgs (1.4433 -> 1.4433 * mean) 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:27.919 INFO:teuthology.orchestra.run.smithi146.stdout:oload 110 2024-09-16T01:50:27.920 INFO:teuthology.orchestra.run.smithi146.stdout:max_change 0.05 2024-09-16T01:50:27.920 INFO:teuthology.orchestra.run.smithi146.stdout:max_change_osds 4 2024-09-16T01:50:27.920 INFO:teuthology.orchestra.run.smithi146.stdout:average_utilization 0.0003 2024-09-16T01:50:27.920 INFO:teuthology.orchestra.run.smithi146.stdout:overload_utilization 0.0003 2024-09-16T01:50:27.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7f170fe00640 1 -- 172.21.15.146:0/4034604687 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+295 (secure 0 0 0) 0x7f16f0000bc0 con 0x7f16f806c2b0 2024-09-16T01:50:27.921 INFO:teuthology.orchestra.run.smithi146.stderr:no change 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.915+0000 7fd4f6800640 1 --2- 172.21.15.146:0/4017612468 >> v2:172.21.15.146:3300/0 conn(0x7fd4f015b340 0x7fd4f0166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.916+0000 7fd4f6800640 1 -- 172.21.15.146:0/4017612468 >> v1:172.21.15.161:6789/0 conn(0x7fd4f00c6b70 legacy=0x7fd4f00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.916+0000 7fd4f6800640 1 -- 172.21.15.146:0/4017612468 >> v1:172.21.15.146:6789/0 conn(0x7fd4f00c75a0 legacy=0x7fd4f00cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.916+0000 7fd4f6800640 1 -- 172.21.15.146:0/4017612468 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f006d700 con 0x7fd4f015b340 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.916+0000 7fd4f4a00640 1 -- 172.21.15.146:0/4017612468 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352378334 0 0) 0x7fd4f006d140 con 0x7fd4f00c75a0 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.917+0000 7fd4f6800640 1 --2- 172.21.15.146:0/4017612468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f015b340 0x7fd4f0166170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009d90 tx=0x7fd4ec02a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212b92c662b4648e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.918+0000 7fd4f4a00640 1 -- 172.21.15.146:0/4017612468 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec007740 con 0x7fd4f015b340 2024-09-16T01:50:27.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.918+0000 7fd4f4a00640 1 -- 172.21.15.146:0/4017612468 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ec0078e0 con 0x7fd4f015b340 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.918+0000 7fd4f4a00640 1 -- 172.21.15.146:0/4017612468 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec007cb0 con 0x7fd4f015b340 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 -- 172.21.15.146:0/4017612468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f015b340 msgr2=0x7fd4f0166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 --2- 172.21.15.146:0/4017612468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f015b340 0x7fd4f0166170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009d90 tx=0x7fd4ec02a580 comp rx=0 tx=0).stop 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f16f806c2b0 msgr2=0x7f16f806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f16f806c2b0 0x7f16f806e770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1718151ee0 tx=0x7f1704009040 comp rx=0 tx=0).stop 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 msgr2=0x7f1718151660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 0x7f1718151660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f170800c4d0 tx=0x7f170800c990 comp rx=0 tx=0).stop 2024-09-16T01:50:27.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 -- 172.21.15.146:0/4017612468 shutdown_connections 2024-09-16T01:50:27.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 --2- 172.21.15.146:0/4017612468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f015b340 0x7fd4f0166170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 -- 172.21.15.146:0/4017612468 >> 172.21.15.146:0/4017612468 conn(0x7fd4f00ccc20 msgr2=0x7fd4f00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:27.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 shutdown_connections 2024-09-16T01:50:27.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 -- 172.21.15.146:0/4017612468 shutdown_connections 2024-09-16T01:50:27.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7fd4f7200640 1 -- 172.21.15.146:0/4017612468 wait complete. 2024-09-16T01:50:27.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f16f806c2b0 0x7f16f806e770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f171815bc50 0x7f17180c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f171815b190 0x7f17180c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 --2- 172.21.15.146:0/4034604687 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1718154ac0 0x7f1718151660 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 >> 172.21.15.146:0/4034604687 conn(0x7f171806e400 msgr2=0x7f171814ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:27.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 shutdown_connections 2024-09-16T01:50:27.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.919+0000 7f171d800640 1 -- 172.21.15.146:0/4034604687 wait complete. 2024-09-16T01:50:27.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.921+0000 7fd4f7200640 1 Processor -- start 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.922+0000 7fd4f7200640 1 -- start start 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f00c6b70 0x7fd4f0072af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f7200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4f00c75a0 0x7fd4f0157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f7200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 0x7fd4f015a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f015b7c0 con 0x7fd4f00c6b70 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f7200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f00cb050 con 0x7fd4f015b340 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f7200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f00cbc80 con 0x7fd4f00c75a0 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f6800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 0x7fd4f015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4f00c75a0 0x7fd4f0157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f6800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 0x7fd4f015a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54366/0 (socket says 172.21.15.146:54366) 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f6800640 1 -- 172.21.15.146:0/2728718899 learned_addr learned my addr 172.21.15.146:0/2728718899 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.923+0000 7fd4f5e00640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f00c6b70 0x7fd4f0072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f6800640 1 -- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4f00c75a0 msgr2=0x7fd4f0157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f6800640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4f00c75a0 0x7fd4f0157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f6800640 1 -- 172.21.15.146:0/2728718899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f00c6b70 msgr2=0x7fd4f0072af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f6800640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f00c6b70 0x7fd4f0072af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f6800640 1 -- 172.21.15.146:0/2728718899 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f01f4b60 con 0x7fd4f015b340 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f5400640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4f00c75a0 0x7fd4f0157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4f6800640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 0x7fd4f015a150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009eb0 tx=0x7fd4ec004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec040070 con 0x7fd4f015b340 2024-09-16T01:50:27.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ec031c00 con 0x7fd4f015b340 2024-09-16T01:50:27.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.924+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec003990 con 0x7fd4f015b340 2024-09-16T01:50:27.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.925+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f01563b0 con 0x7fd4f015b340 2024-09-16T01:50:27.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.925+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f0156930 con 0x7fd4f015b340 2024-09-16T01:50:27.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.926+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f006cd20 con 0x7fd4f015b340 2024-09-16T01:50:27.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.928+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd4ec039040 con 0x7fd4f015b340 2024-09-16T01:50:27.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.928+0000 7fd4ea000640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4d406c2e0 0x7fd4d406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:27.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.929+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5770+0+0 (secure 0 0 0) 0x7fd4ec0ac9c0 con 0x7fd4f015b340 2024-09-16T01:50:27.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.929+0000 7fd4f5e00640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4d406c2e0 0x7fd4d406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:27.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.930+0000 7fd4f5e00640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4d406c2e0 0x7fd4d406e7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4007fd0 tx=0x7fd4e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:27.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:27.933+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd4ec07b160 con 0x7fd4f015b340 2024-09-16T01:50:27.947 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T01:50:27.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-16T01:50:27.950 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:27.950 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:27.959 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:28.037 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:28.032+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:28.062 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.063 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.072 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:28.114 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:28.112+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:28.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.141+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd4f00ca280 con 0x7fd4f015b340 2024-09-16T01:50:28.175 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.176 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.185 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.222+0000 7f13c5e00640 1 Processor -- start 2024-09-16T01:50:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.223+0000 7f13c5e00640 1 -- start start 2024-09-16T01:50:28.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13c0154ac0 0x7f13c015b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c00d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c0154ea0 con 0x7f13c00d0d00 2024-09-16T01:50:28.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f13c015b6d0 con 0x7f13c0154ac0 2024-09-16T01:50:28.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f13c00c5340 con 0x7f13c006d820 2024-09-16T01:50:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c4a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13c0154ac0 0x7f13c015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13bfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c00d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f13c00d0d00 0x7f13c015bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56872/0 (socket says 172.21.15.146:56872) 2024-09-16T01:50:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.224+0000 7f13c5400640 1 -- 172.21.15.146:0/251627035 learned_addr learned my addr 172.21.15.146:0/251627035 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13bf400640 1 -- 172.21.15.146:0/251627035 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105957186 0 0) 0x7f13c0154ea0 con 0x7f13c00d0d00 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13bf400640 1 -- 172.21.15.146:0/251627035 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13a8003520 con 0x7f13c00d0d00 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13bfe00640 1 -- 172.21.15.146:0/251627035 >> v2:172.21.15.146:3300/0 conn(0x7f13c0154ac0 msgr2=0x7f13c015b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13bfe00640 1 --2- 172.21.15.146:0/251627035 >> v2:172.21.15.146:3300/0 conn(0x7f13c0154ac0 0x7f13c015b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13bfe00640 1 -- 172.21.15.146:0/251627035 >> v1:172.21.15.161:6789/0 conn(0x7f13c00d0d00 legacy=0x7f13c015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13bfe00640 1 -- 172.21.15.146:0/251627035 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c00c5620 con 0x7f13c006d820 2024-09-16T01:50:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.225+0000 7f13c4a00640 1 --2- 172.21.15.146:0/251627035 >> v2:172.21.15.146:3300/0 conn(0x7f13c0154ac0 0x7f13c015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:28.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.226+0000 7f13bfe00640 1 --2- 172.21.15.146:0/251627035 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c00d07c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0097a0 tx=0x7f13ac02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a99cece454b8b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:28.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.227+0000 7f13bf400640 1 -- 172.21.15.146:0/251627035 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac037070 con 0x7f13c006d820 2024-09-16T01:50:28.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.227+0000 7f13bf400640 1 -- 172.21.15.146:0/251627035 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ac032440 con 0x7f13c006d820 2024-09-16T01:50:28.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.227+0000 7f13bf400640 1 -- 172.21.15.146:0/251627035 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac032850 con 0x7f13c006d820 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 -- 172.21.15.146:0/251627035 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 msgr2=0x7f13c00d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 --2- 172.21.15.146:0/251627035 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c00d07c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0097a0 tx=0x7f13ac02a250 comp rx=0 tx=0).stop 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 -- 172.21.15.146:0/251627035 shutdown_connections 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 --2- 172.21.15.146:0/251627035 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c00d07c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 --2- 172.21.15.146:0/251627035 >> v2:172.21.15.146:3300/0 conn(0x7f13c0154ac0 0x7f13c015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 -- 172.21.15.146:0/251627035 >> 172.21.15.146:0/251627035 conn(0x7f13c006e400 msgr2=0x7f13c006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 -- 172.21.15.146:0/251627035 shutdown_connections 2024-09-16T01:50:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.228+0000 7f13c5e00640 1 -- 172.21.15.146:0/251627035 wait complete. 2024-09-16T01:50:28.231 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:28.229+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.231+0000 7f13c5e00640 1 Processor -- start 2024-09-16T01:50:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.231+0000 7f13c5e00640 1 -- start start 2024-09-16T01:50:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.232+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c014b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.232+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c00d0d00 0x7f13c014bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.232+0000 7f13c5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 0x7f13c015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.232+0000 7f13c5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00c5ec0 con 0x7f13c00d0d00 2024-09-16T01:50:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.232+0000 7f13c5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f13c00c5be0 con 0x7f13c0154ac0 2024-09-16T01:50:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.232+0000 7f13c5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f13c00c61a0 con 0x7f13c006d820 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 0x7f13c015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c014b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c014b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47286/0 (socket says 172.21.15.146:47286) 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c4a00640 1 -- 172.21.15.146:0/736469336 learned_addr learned my addr 172.21.15.146:0/736469336 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 0x7f13c015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54372/0 (socket says 172.21.15.146:54372) 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13bfe00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c00d0d00 0x7f13c014bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 -- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 msgr2=0x7f13c014b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c014b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 -- 172.21.15.146:0/736469336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c00d0d00 msgr2=0x7f13c014bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c00d0d00 0x7f13c014bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.233+0000 7f13c5400640 1 -- 172.21.15.146:0/736469336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c015a5b0 con 0x7f13c0154ac0 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13c5400640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 0x7f13c015dca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f13b800caa0 tx=0x7f13b800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13c4a00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c014b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8017070 con 0x7f13c0154ac0 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b800aba0 con 0x7f13c0154ac0 2024-09-16T01:50:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8012400 con 0x7f13c0154ac0 2024-09-16T01:50:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c015a850 con 0x7f13c0154ac0 2024-09-16T01:50:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.234+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c014e130 con 0x7f13c0154ac0 2024-09-16T01:50:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.235+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c00cad60 con 0x7f13c0154ac0 2024-09-16T01:50:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.236+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f13b801a070 con 0x7f13c0154ac0 2024-09-16T01:50:28.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.238+0000 7f13bcc00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a006c300 0x7f13a006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.238+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5770+0+0 (secure 0 0 0) 0x7f13b808b1e0 con 0x7f13c0154ac0 2024-09-16T01:50:28.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.238+0000 7f13c4a00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a006c300 0x7f13a006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.239+0000 7f13c4a00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a006c300 0x7f13a006e7c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009560 tx=0x7f13b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:28.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.241+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f13b8059210 con 0x7f13c0154ac0 2024-09-16T01:50:28.290 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.290 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.299 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:28.315 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:28.314+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:28.416 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:28.414+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:28.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.427+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7f13c01535a0 con 0x7f13a006c300 2024-09-16T01:50:28.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.428+0000 7f13bcc00640 1 -- 172.21.15.146:0/736469336 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+295 (secure 0 0 0) 0x7f13c01535a0 con 0x7f13a006c300 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:moved 0 / 194 (0%) 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:avg 24.25 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:stddev 6.21993 -> 6.21993 (expected baseline 4.60638) 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:min osd.7 with 16 -> 16 pgs (0.659794 -> 0.659794 * mean) 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:max osd.0 with 35 -> 35 pgs (1.4433 -> 1.4433 * mean) 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:oload 110 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:max_change 0.05 2024-09-16T01:50:28.430 INFO:teuthology.orchestra.run.smithi146.stdout:max_change_osds 4 2024-09-16T01:50:28.431 INFO:teuthology.orchestra.run.smithi146.stdout:average_utilization 0.0003 2024-09-16T01:50:28.431 INFO:teuthology.orchestra.run.smithi146.stdout:overload_utilization 0.0003 2024-09-16T01:50:28.431 INFO:teuthology.orchestra.run.smithi146.stderr:no change 2024-09-16T01:50:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a006c300 msgr2=0x7f13a006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a006c300 0x7f13a006e7c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009560 tx=0x7f13b4009190 comp rx=0 tx=0).stop 2024-09-16T01:50:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 msgr2=0x7f13c015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 0x7f13c015dca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f13b800caa0 tx=0x7f13b800cf60 comp rx=0 tx=0).stop 2024-09-16T01:50:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 shutdown_connections 2024-09-16T01:50:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f13a006c300 0x7f13a006e7c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13c0154ac0 0x7f13c015dca0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c00d0d00 0x7f13c014bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 --2- 172.21.15.146:0/736469336 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f13c006d820 0x7f13c014b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 >> 172.21.15.146:0/736469336 conn(0x7f13c006e400 msgr2=0x7f13c01528f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 shutdown_connections 2024-09-16T01:50:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.430+0000 7f13c5e00640 1 -- 172.21.15.146:0/736469336 wait complete. 2024-09-16T01:50:28.445 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.445 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.454 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:28.457 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T01:50:28.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-09-16T01:50:28.544 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:28.542+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:28.559 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.559 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.568 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:28.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.614+0000 7fd4ea000640 1 -- 172.21.15.146:0/2728718899 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v46) ==== 87+0+0 (secure 0 0 0) 0x7fd4ec07aa60 con 0x7fd4f015b340 2024-09-16T01:50:28.618 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:28.617+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:28.618 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:50:28.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4d406c2e0 msgr2=0x7fd4d406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4d406c2e0 0x7fd4d406e7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4007fd0 tx=0x7fd4e4009040 comp rx=0 tx=0).stop 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 msgr2=0x7fd4f015a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 0x7fd4f015a150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009eb0 tx=0x7fd4ec004a40 comp rx=0 tx=0).stop 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 shutdown_connections 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd4d406c2e0 0x7fd4d406e7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4f015b340 0x7fd4f015a150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4f00c75a0 0x7fd4f0157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.619+0000 7fd4f7200640 1 --2- 172.21.15.146:0/2728718899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4f00c6b70 0x7fd4f0072af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.620+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 >> 172.21.15.146:0/2728718899 conn(0x7fd4f00ccc20 msgr2=0x7fd4f00cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:28.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.620+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 shutdown_connections 2024-09-16T01:50:28.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.620+0000 7fd4f7200640 1 -- 172.21.15.146:0/2728718899 wait complete. 2024-09-16T01:50:28.673 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.673 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.673+0000 7ff82a400640 1 Processor -- start 2024-09-16T01:50:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.674+0000 7ff82a400640 1 -- start start 2024-09-16T01:50:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff82a400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff82415b190 0x7ff8240d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff82a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8240d0d00 0x7ff82415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff82a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff824154ea0 con 0x7ff824154ac0 2024-09-16T01:50:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff82a400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff8240c5340 con 0x7ff82415b190 2024-09-16T01:50:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff82a400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff8240c5620 con 0x7ff8240d0d00 2024-09-16T01:50:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff829000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff824154ac0 0x7ff82406d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:37156/0 (socket says 172.21.15.146:37156) 2024-09-16T01:50:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff823e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff82415b190 0x7ff8240d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff829000640 1 -- 172.21.15.146:0/2038664727 learned_addr learned my addr 172.21.15.146:0/2038664727 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.675+0000 7ff829a00640 1 --2- 172.21.15.146:0/2038664727 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8240d0d00 0x7ff82415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.676+0000 7ff829a00640 1 -- 172.21.15.146:0/2038664727 >> v2:172.21.15.161:3300/0 conn(0x7ff82415b190 msgr2=0x7ff8240d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.676+0000 7ff829a00640 1 --2- 172.21.15.146:0/2038664727 >> v2:172.21.15.161:3300/0 conn(0x7ff82415b190 0x7ff8240d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.676+0000 7ff829a00640 1 -- 172.21.15.146:0/2038664727 >> v1:172.21.15.146:6789/0 conn(0x7ff824154ac0 legacy=0x7ff82406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.676+0000 7ff829a00640 1 -- 172.21.15.146:0/2038664727 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8240c5900 con 0x7ff8240d0d00 2024-09-16T01:50:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.676+0000 7ff823e00640 1 --2- 172.21.15.146:0/2038664727 >> v2:172.21.15.161:3300/0 conn(0x7ff82415b190 0x7ff8240d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.677+0000 7ff829a00640 1 --2- 172.21.15.146:0/2038664727 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8240d0d00 0x7ff82415bc50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009d90 tx=0x7ff81c02a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ce293f5ab37c185 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.678+0000 7ff823400640 1 -- 172.21.15.146:0/2038664727 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c007740 con 0x7ff8240d0d00 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.678+0000 7ff823400640 1 -- 172.21.15.146:0/2038664727 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c0078e0 con 0x7ff8240d0d00 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.678+0000 7ff823400640 1 -- 172.21.15.146:0/2038664727 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c007cb0 con 0x7ff8240d0d00 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.678+0000 7ff82a400640 1 -- 172.21.15.146:0/2038664727 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8240d0d00 msgr2=0x7ff82415bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.678+0000 7ff82a400640 1 --2- 172.21.15.146:0/2038664727 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8240d0d00 0x7ff82415bc50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009d90 tx=0x7ff81c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.679+0000 7ff82a400640 1 -- 172.21.15.146:0/2038664727 shutdown_connections 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.679+0000 7ff82a400640 1 --2- 172.21.15.146:0/2038664727 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff8240d0d00 0x7ff82415bc50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.679+0000 7ff82a400640 1 --2- 172.21.15.146:0/2038664727 >> v2:172.21.15.161:3300/0 conn(0x7ff82415b190 0x7ff8240d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.679+0000 7ff82a400640 1 -- 172.21.15.146:0/2038664727 >> 172.21.15.146:0/2038664727 conn(0x7ff82406e400 msgr2=0x7ff82406e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.679+0000 7ff82a400640 1 -- 172.21.15.146:0/2038664727 shutdown_connections 2024-09-16T01:50:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.679+0000 7ff82a400640 1 -- 172.21.15.146:0/2038664727 wait complete. 2024-09-16T01:50:28.682 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:28.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.681+0000 7ff82a400640 1 Processor -- start 2024-09-16T01:50:28.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.682+0000 7ff82a400640 1 -- start start 2024-09-16T01:50:28.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff82a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 0x7ff8240cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff82a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff824154ac0 0x7ff8240c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff82a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff82415b190 0x7ff8240c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff82a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff8240c61a0 con 0x7ff824154ac0 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff82a400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff8240c5ec0 con 0x7ff8240d0d00 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff82a400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff8240cad60 con 0x7ff82415b190 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff829000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 0x7ff8240cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff829000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 0x7ff8240cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54388/0 (socket says 172.21.15.146:54388) 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff829000640 1 -- 172.21.15.146:0/3491096505 learned_addr learned my addr 172.21.15.146:0/3491096505 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.683+0000 7ff823e00640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff824154ac0 0x7ff8240c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829a00640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff82415b190 0x7ff8240c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829000640 1 -- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff82415b190 msgr2=0x7ff8240c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829000640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff82415b190 0x7ff8240c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829000640 1 -- 172.21.15.146:0/3491096505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff824154ac0 msgr2=0x7ff8240c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829000640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff824154ac0 0x7ff8240c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829000640 1 -- 172.21.15.146:0/3491096505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8240ca0c0 con 0x7ff8240d0d00 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff829000640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 0x7ff8240cbd50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff81800c400 tx=0x7ff81800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818017070 con 0x7ff8240d0d00 2024-09-16T01:50:28.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff818005ce0 con 0x7ff8240d0d00 2024-09-16T01:50:28.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8240ca360 con 0x7ff8240d0d00 2024-09-16T01:50:28.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.684+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180072f0 con 0x7ff8240d0d00 2024-09-16T01:50:28.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.685+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8241f0800 con 0x7ff8240d0d00 2024-09-16T01:50:28.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.686+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8241f0310 con 0x7ff8240d0d00 2024-09-16T01:50:28.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.687+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff81801a070 con 0x7ff8240d0d00 2024-09-16T01:50:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.688+0000 7ff820c00640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff80406c2b0 0x7ff80406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.689+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5770+0+0 (secure 0 0 0) 0x7ff81808a8b0 con 0x7ff8240d0d00 2024-09-16T01:50:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.689+0000 7ff823e00640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff80406c2b0 0x7ff80406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:28.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.689+0000 7ff823e00640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff80406c2b0 0x7ff80406e770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff8240cb4d0 tx=0x7ff810008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:28.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.692+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff818058fd0 con 0x7ff8240d0d00 2024-09-16T01:50:28.723 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:28.722+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:28.820 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:28.819+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:28.826 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.826 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.835 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:28.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.882+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}) -- 0x7ff82415b5e0 con 0x7ff80406c2b0 2024-09-16T01:50:28.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.882+0000 7ff820c00640 1 -- 172.21.15.146:0/3491096505 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+292 (secure 0 0 0) 0x7ff82415b5e0 con 0x7ff80406c2b0 2024-09-16T01:50:28.884 INFO:teuthology.orchestra.run.smithi146.stdout:moved 0 / 194 (0%) 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:avg 24.25 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:stddev 6.21993 -> 6.21993 (expected baseline 4.60638) 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:min osd.7 with 16 -> 16 pgs (0.659794 -> 0.659794 * mean) 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:max osd.0 with 35 -> 35 pgs (1.4433 -> 1.4433 * mean) 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:oload 110 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:max_change 1 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:max_change_osds 4 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:average_utilization 0.0003 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stdout:overload_utilization 0.0003 2024-09-16T01:50:28.885 INFO:teuthology.orchestra.run.smithi146.stderr:no change 2024-09-16T01:50:28.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff80406c2b0 msgr2=0x7ff80406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff80406c2b0 0x7ff80406e770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff8240cb4d0 tx=0x7ff810008040 comp rx=0 tx=0).stop 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 msgr2=0x7ff8240cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 0x7ff8240cbd50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff81800c400 tx=0x7ff81800c8c0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 shutdown_connections 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff80406c2b0 0x7ff80406e770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff82415b190 0x7ff8240c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff824154ac0 0x7ff8240c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 --2- 172.21.15.146:0/3491096505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8240d0d00 0x7ff8240cbd50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:28.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.885+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 >> 172.21.15.146:0/3491096505 conn(0x7ff82406e400 msgr2=0x7ff8241e3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:28.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.886+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 shutdown_connections 2024-09-16T01:50:28.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:28.886+0000 7ff82a400640 1 -- 172.21.15.146:0/3491096505 wait complete. 2024-09-16T01:50:28.905 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T01:50:28.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-16T01:50:28.921 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:28.920+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:28.979 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:28.979 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:28.988 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:29.022 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:29.021+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:29.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.106+0000 7fcb55600640 1 Processor -- start 2024-09-16T01:50:29.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.106+0000 7fcb55600640 1 -- start start 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.107+0000 7fcb55600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb5015bc50 0x7fcb5015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.107+0000 7fcb55600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb50154ea0 con 0x7fcb5015b190 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.107+0000 7fcb55600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb500c5340 con 0x7fcb50154ac0 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.107+0000 7fcb55600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5015c600 con 0x7fcb5015bc50 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb4f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcb5015b190 0x7fcb500d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:37166/0 (socket says 172.21.15.146:37166) 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb4f400640 1 -- 172.21.15.146:0/1221967444 learned_addr learned my addr 172.21.15.146:0/1221967444 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:29.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb54c00640 1 --2- 172.21.15.146:0/1221967444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb5015bc50 0x7fcb5015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb4ea00640 1 -- 172.21.15.146:0/1221967444 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174879453 0 0) 0x7fcb500c5340 con 0x7fcb50154ac0 2024-09-16T01:50:29.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb54c00640 1 -- 172.21.15.146:0/1221967444 >> v1:172.21.15.161:6789/0 conn(0x7fcb50154ac0 legacy=0x7fcb5006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb54c00640 1 -- 172.21.15.146:0/1221967444 >> v1:172.21.15.146:6789/0 conn(0x7fcb5015b190 legacy=0x7fcb500d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.108+0000 7fcb54c00640 1 -- 172.21.15.146:0/1221967444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb500c5620 con 0x7fcb5015bc50 2024-09-16T01:50:29.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.110+0000 7fcb54c00640 1 --2- 172.21.15.146:0/1221967444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb5015bc50 0x7fcb5015c0c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009dc0 tx=0x7fcb4802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88764bf47b1269b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.110+0000 7fcb4ea00640 1 -- 172.21.15.146:0/1221967444 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48007740 con 0x7fcb5015bc50 2024-09-16T01:50:29.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.110+0000 7fcb4ea00640 1 -- 172.21.15.146:0/1221967444 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb480078e0 con 0x7fcb5015bc50 2024-09-16T01:50:29.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.110+0000 7fcb4ea00640 1 -- 172.21.15.146:0/1221967444 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48007cb0 con 0x7fcb5015bc50 2024-09-16T01:50:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 -- 172.21.15.146:0/1221967444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb5015bc50 msgr2=0x7fcb5015c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 --2- 172.21.15.146:0/1221967444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb5015bc50 0x7fcb5015c0c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009dc0 tx=0x7fcb4802a660 comp rx=0 tx=0).stop 2024-09-16T01:50:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 -- 172.21.15.146:0/1221967444 shutdown_connections 2024-09-16T01:50:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 --2- 172.21.15.146:0/1221967444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb5015bc50 0x7fcb5015c0c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 -- 172.21.15.146:0/1221967444 >> 172.21.15.146:0/1221967444 conn(0x7fcb5006e400 msgr2=0x7fcb5006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 -- 172.21.15.146:0/1221967444 shutdown_connections 2024-09-16T01:50:29.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.111+0000 7fcb55600640 1 -- 172.21.15.146:0/1221967444 wait complete. 2024-09-16T01:50:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.114+0000 7fcb55600640 1 Processor -- start 2024-09-16T01:50:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.114+0000 7fcb55600640 1 -- start start 2024-09-16T01:50:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.115+0000 7fcb55600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 0x7fcb501e5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.115+0000 7fcb55600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5015b190 0x7fcb500b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.115+0000 7fcb55600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb5015bc50 0x7fcb501e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.115+0000 7fcb55600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb500c5ec0 con 0x7fcb5015b190 2024-09-16T01:50:29.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.115+0000 7fcb55600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcb500c5be0 con 0x7fcb5015bc50 2024-09-16T01:50:29.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.115+0000 7fcb55600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcb500c61a0 con 0x7fcb50154ac0 2024-09-16T01:50:29.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 0x7fcb501e5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 0x7fcb501e5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47312/0 (socket says 172.21.15.146:47312) 2024-09-16T01:50:29.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 -- 172.21.15.146:0/3654650580 learned_addr learned my addr 172.21.15.146:0/3654650580 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:29.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4f400640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5015b190 0x7fcb500b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb54c00640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb5015bc50 0x7fcb501e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 -- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb5015bc50 msgr2=0x7fcb501e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb5015bc50 0x7fcb501e5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 -- 172.21.15.146:0/3654650580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5015b190 msgr2=0x7fcb500b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5015b190 0x7fcb500b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 -- 172.21.15.146:0/3654650580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb500cb5e0 con 0x7fcb50154ac0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4f400640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5015b190 0x7fcb500b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.116+0000 7fcb4fe00640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 0x7fcb501e5fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000c440 tx=0x7fcb4000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.117+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40017070 con 0x7fcb50154ac0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.117+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb40005480 con 0x7fcb50154ac0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.117+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40003830 con 0x7fcb50154ac0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.117+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb500cb8a0 con 0x7fcb50154ac0 2024-09-16T01:50:29.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.117+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb500cbe60 con 0x7fcb50154ac0 2024-09-16T01:50:29.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.118+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb500cff00 con 0x7fcb50154ac0 2024-09-16T01:50:29.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.120+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcb4001a070 con 0x7fcb50154ac0 2024-09-16T01:50:29.122 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:29.122+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:29.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.121+0000 7fcb47e00640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb3006c2b0 0x7fcb3006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.121+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 5770+0+0 (secure 0 0 0) 0x7fcb4008acc0 con 0x7fcb50154ac0 2024-09-16T01:50:29.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.121+0000 7fcb4f400640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb3006c2b0 0x7fcb3006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.122+0000 7fcb4f400640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb3006c2b0 0x7fcb3006e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c007fd0 tx=0x7fcb3c007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.128+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcb500cff00 con 0x7fcb50154ac0 2024-09-16T01:50:29.137 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:29.137 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:29.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:29.224 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:29.222+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:29.325 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:29.324+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:29.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.326+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) -- 0x7fcb5015b5e0 con 0x7fcb3006c2b0 2024-09-16T01:50:29.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.327+0000 7fcb47e00640 1 -- 172.21.15.146:0/3654650580 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+292 (secure 0 0 0) 0x7fcb5015b5e0 con 0x7fcb3006c2b0 2024-09-16T01:50:29.329 INFO:teuthology.orchestra.run.smithi146.stdout:moved 0 / 194 (0%) 2024-09-16T01:50:29.329 INFO:teuthology.orchestra.run.smithi146.stdout:avg 24.25 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:stddev 6.21993 -> 6.21993 (expected baseline 4.60638) 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:min osd.7 with 16 -> 16 pgs (0.659794 -> 0.659794 * mean) 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:max osd.0 with 35 -> 35 pgs (1.4433 -> 1.4433 * mean) 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:oload 110 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:max_change 3 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:max_change_osds 4 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:average_utilization 0.0003 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stdout:overload_utilization 0.0003 2024-09-16T01:50:29.330 INFO:teuthology.orchestra.run.smithi146.stderr:no change 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb3006c2b0 msgr2=0x7fcb3006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb3006c2b0 0x7fcb3006e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c007fd0 tx=0x7fcb3c007d60 comp rx=0 tx=0).stop 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 msgr2=0x7fcb501e5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 0x7fcb501e5fb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000c440 tx=0x7fcb4000c900 comp rx=0 tx=0).stop 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 shutdown_connections 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb3006c2b0 0x7fcb3006e770 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb5015bc50 0x7fcb501e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5015b190 0x7fcb500b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 --2- 172.21.15.146:0/3654650580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcb50154ac0 0x7fcb501e5fb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 >> 172.21.15.146:0/3654650580 conn(0x7fcb5006e400 msgr2=0x7fcb50153ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:29.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 shutdown_connections 2024-09-16T01:50:29.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.330+0000 7fcb55600640 1 -- 172.21.15.146:0/3654650580 wait complete. 2024-09-16T01:50:29.350 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-16T01:50:29.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-09-16T01:50:29.389 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:29.389 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:29.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:29.450 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:29.448+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:29.528 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:29.526+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:29.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.614+0000 7f0a77e00640 1 Processor -- start 2024-09-16T01:50:29.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.614+0000 7f0a77e00640 1 -- start start 2024-09-16T01:50:29.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.616+0000 7f0a77e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a780d0340 0x7f0a7815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.617+0000 7f0a77e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a780b6150 con 0x7f0a780d0720 2024-09-16T01:50:29.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.617+0000 7f0a77e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a7815a940 con 0x7f0a780cc260 2024-09-16T01:50:29.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.617+0000 7f0a77e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0a780cb020 con 0x7f0a780d0340 2024-09-16T01:50:29.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.617+0000 7f0a76a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a780d0340 0x7f0a7815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.617+0000 7f0a76a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a780d0340 0x7f0a7815b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54394/0 (socket says 172.21.15.146:54394) 2024-09-16T01:50:29.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.617+0000 7f0a76a00640 1 -- 172.21.15.146:0/2129742661 learned_addr learned my addr 172.21.15.146:0/2129742661 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:29.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.618+0000 7f0a76a00640 1 -- 172.21.15.146:0/2129742661 >> v1:172.21.15.161:6789/0 conn(0x7f0a780cc260 legacy=0x7f0a781650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.618+0000 7f0a76a00640 1 -- 172.21.15.146:0/2129742661 >> v1:172.21.15.146:6789/0 conn(0x7f0a780d0720 legacy=0x7f0a780c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.618+0000 7f0a76a00640 1 -- 172.21.15.146:0/2129742661 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a780cb270 con 0x7f0a780d0340 2024-09-16T01:50:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.618+0000 7f0a75600640 1 -- 172.21.15.146:0/2129742661 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2064454238 0 0) 0x7f0a780b6150 con 0x7f0a780d0720 2024-09-16T01:50:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.619+0000 7f0a76a00640 1 --2- 172.21.15.146:0/2129742661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a7815b3e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009870 tx=0x7f0a6802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d43a2f791f4708c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.619+0000 7f0a75600640 1 -- 172.21.15.146:0/2129742661 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68037070 con 0x7f0a780d0340 2024-09-16T01:50:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.619+0000 7f0a75600640 1 -- 172.21.15.146:0/2129742661 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6802ae50 con 0x7f0a780d0340 2024-09-16T01:50:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.619+0000 7f0a75600640 1 -- 172.21.15.146:0/2129742661 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68032860 con 0x7f0a780d0340 2024-09-16T01:50:29.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.620+0000 7f0a77e00640 1 -- 172.21.15.146:0/2129742661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 msgr2=0x7f0a7815b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.620+0000 7f0a77e00640 1 --2- 172.21.15.146:0/2129742661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a7815b3e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009870 tx=0x7f0a6802a300 comp rx=0 tx=0).stop 2024-09-16T01:50:29.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.620+0000 7f0a77e00640 1 -- 172.21.15.146:0/2129742661 shutdown_connections 2024-09-16T01:50:29.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.620+0000 7f0a77e00640 1 --2- 172.21.15.146:0/2129742661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a7815b3e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.620+0000 7f0a77e00640 1 -- 172.21.15.146:0/2129742661 >> 172.21.15.146:0/2129742661 conn(0x7f0a7806e0d0 msgr2=0x7f0a7806e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:29.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.621+0000 7f0a77e00640 1 -- 172.21.15.146:0/2129742661 shutdown_connections 2024-09-16T01:50:29.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.621+0000 7f0a77e00640 1 -- 172.21.15.146:0/2129742661 wait complete. 2024-09-16T01:50:29.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.623+0000 7f0a77e00640 1 Processor -- start 2024-09-16T01:50:29.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.624+0000 7f0a77e00640 1 -- start start 2024-09-16T01:50:29.627 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:29.627 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:29.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.625+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a780cc260 0x7f0a78152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a781f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a780d0720 0x7f0a7814ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.628 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:29.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f0a76a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a780cc260 0x7f0a78152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f0a76a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a780cc260 0x7f0a78152630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47316/0 (socket says 172.21.15.146:47316) 2024-09-16T01:50:29.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f0a77400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a780d0720 0x7f0a7814ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.626+0000 7f0a77400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a780d0720 0x7f0a7814ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47710/0 (socket says 172.21.15.146:47710) 2024-09-16T01:50:29.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.627+0000 7f0a76000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a781f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.627+0000 7f0a76a00640 1 -- 172.21.15.146:0/575901807 learned_addr learned my addr 172.21.15.146:0/575901807 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:29.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.627+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0a780cb550 con 0x7f0a780d0720 2024-09-16T01:50:29.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.627+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7815aec0 con 0x7f0a780d0340 2024-09-16T01:50:29.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.627+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0a780c6a10 con 0x7f0a780cc260 2024-09-16T01:50:29.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a76000640 1 -- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a780cc260 msgr2=0x7f0a78152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a76000640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a780cc260 0x7f0a78152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a76000640 1 -- 172.21.15.146:0/575901807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a780d0720 msgr2=0x7f0a7814ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a76000640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a780d0720 0x7f0a7814ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a76000640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a781f3a20 con 0x7f0a780d0340 2024-09-16T01:50:29.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a76000640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a781f04b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0a780d0b30 tx=0x7f0a6400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64017070 con 0x7f0a780d0340 2024-09-16T01:50:29.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a640035f0 con 0x7f0a780d0340 2024-09-16T01:50:29.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a640039e0 con 0x7f0a780d0340 2024-09-16T01:50:29.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.628+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781552f0 con 0x7f0a780d0340 2024-09-16T01:50:29.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.629+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7815a640 con 0x7f0a780d0340 2024-09-16T01:50:29.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:29.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.630+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a780c7730 con 0x7f0a780d0340 2024-09-16T01:50:29.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.631+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0a6401a070 con 0x7f0a780d0340 2024-09-16T01:50:29.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.633+0000 7f0a6ea00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a5806c2b0 0x7f0a5806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.633+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5770+0+0 (secure 0 0 0) 0x7f0a6408a870 con 0x7f0a780d0340 2024-09-16T01:50:29.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.634+0000 7f0a76a00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a5806c2b0 0x7f0a5806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.634+0000 7f0a76a00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a5806c2b0 0x7f0a5806e770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a680090b0 tx=0x7f0a6802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.637+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0a64058820 con 0x7f0a780d0340 2024-09-16T01:50:29.645 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:29.734 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:29.732+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:29.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.818+0000 7fed73e00640 1 Processor -- start 2024-09-16T01:50:29.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.818+0000 7fed73e00640 1 -- start start 2024-09-16T01:50:29.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.819+0000 7fed73e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740c6d00 0x7fed740c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed73e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed7415e220 con 0x7fed740d0870 2024-09-16T01:50:29.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed73e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed7415e500 con 0x7fed740d0490 2024-09-16T01:50:29.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed73e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fed740c6160 con 0x7fed740c6d00 2024-09-16T01:50:29.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed72a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fed740d0490 0x7fed740c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56898/0 (socket says 172.21.15.146:56898) 2024-09-16T01:50:29.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed72a00640 1 -- 172.21.15.146:0/1077424319 learned_addr learned my addr 172.21.15.146:0/1077424319 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:29.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed73400640 1 --2- 172.21.15.146:0/1077424319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740c6d00 0x7fed740c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed71600640 1 -- 172.21.15.146:0/1077424319 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1825115888 0 0) 0x7fed7415e500 con 0x7fed740d0490 2024-09-16T01:50:29.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.820+0000 7fed71600640 1 -- 172.21.15.146:0/1077424319 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed58003520 con 0x7fed740d0490 2024-09-16T01:50:29.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.821+0000 7fed73400640 1 -- 172.21.15.146:0/1077424319 >> v1:172.21.15.161:6789/0 conn(0x7fed740d0490 legacy=0x7fed740c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.821+0000 7fed73400640 1 -- 172.21.15.146:0/1077424319 >> v1:172.21.15.146:6789/0 conn(0x7fed740d0870 legacy=0x7fed74154690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:50:29.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.821+0000 7fed73400640 1 -- 172.21.15.146:0/1077424319 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7415e7e0 con 0x7fed740c6d00 2024-09-16T01:50:29.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.822+0000 7fed73400640 1 --2- 172.21.15.146:0/1077424319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740c6d00 0x7fed740c51f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009dc0 tx=0x7fed6c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=435f369eb27a3105 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.822+0000 7fed71600640 1 -- 172.21.15.146:0/1077424319 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c007740 con 0x7fed740c6d00 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.822+0000 7fed71600640 1 -- 172.21.15.146:0/1077424319 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6c0078e0 con 0x7fed740c6d00 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.822+0000 7fed71600640 1 -- 172.21.15.146:0/1077424319 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c007cb0 con 0x7fed740c6d00 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.823+0000 7fed73e00640 1 -- 172.21.15.146:0/1077424319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740c6d00 msgr2=0x7fed740c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.823+0000 7fed73e00640 1 --2- 172.21.15.146:0/1077424319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740c6d00 0x7fed740c51f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009dc0 tx=0x7fed6c02a710 comp rx=0 tx=0).stop 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.823+0000 7fed73e00640 1 -- 172.21.15.146:0/1077424319 shutdown_connections 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.823+0000 7fed73e00640 1 --2- 172.21.15.146:0/1077424319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740c6d00 0x7fed740c51f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.823+0000 7fed73e00640 1 -- 172.21.15.146:0/1077424319 >> 172.21.15.146:0/1077424319 conn(0x7fed7406d6f0 msgr2=0x7fed7406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.824+0000 7fed73e00640 1 -- 172.21.15.146:0/1077424319 shutdown_connections 2024-09-16T01:50:29.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.824+0000 7fed73e00640 1 -- 172.21.15.146:0/1077424319 wait complete. 2024-09-16T01:50:29.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.825+0000 7fed73e00640 1 Processor -- start 2024-09-16T01:50:29.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.826+0000 7fed73e00640 1 -- start start 2024-09-16T01:50:29.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.826+0000 7fed73e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed740c6d00 0x7fed741f8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.826+0000 7fed73e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed740d0490 0x7fed741533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed73e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740d0870 0x7fed740b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed73e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fed7415dec0 con 0x7fed740d0490 2024-09-16T01:50:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed73e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed7415eda0 con 0x7fed740c6d00 2024-09-16T01:50:29.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed73e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fed74154da0 con 0x7fed740d0870 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed72000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed740d0490 0x7fed741533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed72a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed740c6d00 0x7fed741f8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed72000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed740d0490 0x7fed741533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47714/0 (socket says 172.21.15.146:47714) 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed72000640 1 -- 172.21.15.146:0/1880959620 learned_addr learned my addr 172.21.15.146:0/1880959620 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.827+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740d0870 0x7fed740b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed72a00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740d0870 msgr2=0x7fed740b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed740d0870 0x7fed740b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed72a00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed740d0490 msgr2=0x7fed741533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed740d0490 0x7fed741533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:29.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed72a00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed741500b0 con 0x7fed740c6d00 2024-09-16T01:50:29.831 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed740c6d00 0x7fed741f8c70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fed6400c430 tx=0x7fed6400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.828+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed64017070 con 0x7fed740c6d00 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.829+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed64005ce0 con 0x7fed740c6d00 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.829+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed74150740 con 0x7fed740c6d00 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.829+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed741e6be0 con 0x7fed740c6d00 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.830+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed640072f0 con 0x7fed740c6d00 2024-09-16T01:50:29.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.830+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fed74150c60 con 0x7fed740c6d00 2024-09-16T01:50:29.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.834+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fed6401a070 con 0x7fed740c6d00 2024-09-16T01:50:29.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.835+0000 7fed6aa00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fed5406c2b0 0x7fed5406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.835+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5770+0+0 (secure 0 0 0) 0x7fed6408a960 con 0x7fed740c6d00 2024-09-16T01:50:29.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.836+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v46) ==== 54+0+16 (secure 0 0 0) 0x7fed74150c60 con 0x7fed740c6d00 2024-09-16T01:50:29.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.836+0000 7fed72000640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fed5406c2b0 0x7fed5406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.836+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fed740b6070 0x7fed7414de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.836+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed7414fe60 con 0x7fed740b6070 2024-09-16T01:50:29.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.837+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fed740b6070 0x7fed7414de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.837+0000 7fed72000640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fed5406c2b0 0x7fed5406e770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed7414fdf0 tx=0x7fed600090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.838+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fed740b6070 0x7fed7414de40 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.839+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed7414fe60 con 0x7fed740b6070 2024-09-16T01:50:29.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.869+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed740c65f0 con 0x7fed740b6070 2024-09-16T01:50:29.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.872+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed740c65f0 con 0x7fed740b6070 2024-09-16T01:50:29.874 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:29.874 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:29.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.872+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fed7414eba0 0x7fed7414efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.872+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fed740c7a30 con 0x7fed7414eba0 2024-09-16T01:50:29.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.873+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fed7414eba0 0x7fed7414efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.873+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fed7414eba0 0x7fed7414efc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.875+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed740c7a30 con 0x7fed7414eba0 2024-09-16T01:50:29.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.904+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed740c5d40 con 0x7fed7414eba0 2024-09-16T01:50:29.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.907+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed740c5d40 con 0x7fed7414eba0 2024-09-16T01:50:29.909 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:29.909 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:29.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.907+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fed74062630 0x7fed74062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.907+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fed7414fa10 con 0x7fed74062630 2024-09-16T01:50:29.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.908+0000 7fed72000640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fed74062630 0x7fed74062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.909+0000 7fed72000640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fed74062630 0x7fed74062a70 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.910+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed7414fa10 con 0x7fed74062630 2024-09-16T01:50:29.933 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:29.931+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.938+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed740cbe20 con 0x7fed74062630 2024-09-16T01:50:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.941+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed740cbe20 con 0x7fed74062630 2024-09-16T01:50:29.944 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.941+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fed7402c640 0x7fed7402cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.942+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fed741e67d0 con 0x7fed7402c640 2024-09-16T01:50:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.942+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fed7402c640 0x7fed7402cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.943+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fed7402c640 0x7fed7402cac0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.944+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed741e67d0 con 0x7fed7402c640 2024-09-16T01:50:29.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.957+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}) -- 0x7f0a780c9b00 con 0x7f0a5806c2b0 2024-09-16T01:50:29.960 INFO:teuthology.orchestra.run.smithi146.stdout:moved 0 / 194 (0%) 2024-09-16T01:50:29.960 INFO:teuthology.orchestra.run.smithi146.stdout:avg 24.25 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:stddev 6.21993 -> 6.21993 (expected baseline 4.60638) 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:min osd.7 with 16 -> 16 pgs (0.659794 -> 0.659794 * mean) 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:max osd.0 with 35 -> 35 pgs (1.4433 -> 1.4433 * mean) 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:oload 1000 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:max_change 0.05 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:max_change_osds 4 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:average_utilization 0.0003 2024-09-16T01:50:29.961 INFO:teuthology.orchestra.run.smithi146.stdout:overload_utilization 0.0029 2024-09-16T01:50:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.958+0000 7f0a6ea00640 1 -- 172.21.15.146:0/575901807 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 no change) ==== 17+0+296 (secure 0 0 0) 0x7f0a780c9b00 con 0x7f0a5806c2b0 2024-09-16T01:50:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:no change 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:29.964 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.135166316999999, 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:50:29.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:50:29.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:29.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.135060268000004, 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:29.971 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:29.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:50:29.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:50:29.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.135005641999996, 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:29.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:50:29.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:50:29.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:29.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:50:29.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:50:29.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.983 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:29.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.134937308999994, 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:29.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:29.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:29.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:29.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:29.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:50:29.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:29.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:29.991 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.100682105999994, 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:29.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:50:29.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:29.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:50:29.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.997 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.100498908000006, 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:50:29.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:50:30.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:30.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:50:30.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.004 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.100444899999999, 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:50:30.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:50:30.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:50:30.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:50:30.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:50:30.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:50:30.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:50:30.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.100412460000001, 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:30.020 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.097044459000003, 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:50:30.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.096973782999996, 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:50:30.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:50:30.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.096937408000002, 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.096309665000007, 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:50:30.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.096255057999997, 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:50:30.039 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:50:30.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.024116649999996, 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:50:30.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:50:30.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.024058338000003, 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:30.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:50:30.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.023984263000003, 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:50:30.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:50:30.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.961+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a5806c2b0 msgr2=0x7f0a5806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.961+0000 7f0a77e00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a5806c2b0 0x7f0a5806e770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a680090b0 tx=0x7f0a6802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:30.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.961+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 msgr2=0x7f0a781f04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.961+0000 7f0a77e00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a781f04b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0a780d0b30 tx=0x7f0a6400c890 comp rx=0 tx=0).stop 2024-09-16T01:50:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:50:30.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.023925641999995, 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:50:30.062 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:50:30.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.023903708000006, 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:50:30.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.023873347999995, 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:50:30.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:50:30.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:50:30.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.023846769000002, 2024-09-16T01:50:30.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:30.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:50:30.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:50:30.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:50:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.085 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 shutdown_connections 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a5806c2b0 0x7f0a5806e770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a780d0720 0x7f0a7814ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a780d0340 0x7f0a781f04b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 --2- 172.21.15.146:0/575901807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a780cc260 0x7f0a78152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 >> 172.21.15.146:0/575901807 conn(0x7f0a7806e0d0 msgr2=0x7f0a780cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 shutdown_connections 2024-09-16T01:50:30.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.962+0000 7f0a77e00640 1 -- 172.21.15.146:0/575901807 wait complete. 2024-09-16T01:50:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.972+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed7402d2d0 con 0x7fed7402c640 2024-09-16T01:50:30.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.975+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed7402d2d0 con 0x7fed7402c640 2024-09-16T01:50:30.088 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:30.088 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.975+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fed7402d760 0x7fed741e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.975+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fed7402dba0 con 0x7fed7402d760 2024-09-16T01:50:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.977+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fed7402d760 0x7fed741e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.978+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fed7402d760 0x7fed741e50c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.978+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed7402dba0 con 0x7fed7402d760 2024-09-16T01:50:30.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.993+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed7402dba0 con 0x7fed7402d760 2024-09-16T01:50:30.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.995+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed7402dba0 con 0x7fed7402d760 2024-09-16T01:50:30.091 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.995+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fed741e5b10 0x7fed741e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.995+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fed741e64b0 con 0x7fed741e5b10 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.996+0000 7fed72000640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fed741e5b10 0x7fed741e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.996+0000 7fed72000640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fed741e5b10 0x7fed741e5f70 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:29.997+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed741e64b0 con 0x7fed741e5b10 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.021+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed741e64b0 con 0x7fed741e5b10 2024-09-16T01:50:30.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.023+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed741e64b0 con 0x7fed741e5b10 2024-09-16T01:50:30.093 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.024+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fed7420bb50 0x7fed7420df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.024+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fed74210de0 con 0x7fed7420bb50 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.024+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fed7420bb50 0x7fed7420df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.025+0000 7fed73400640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fed7420bb50 0x7fed7420df40 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.025+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed74210de0 con 0x7fed7420bb50 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.052+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed7420e480 con 0x7fed7420bb50 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.054+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed7420e480 con 0x7fed7420bb50 2024-09-16T01:50:30.094 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.054+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fed7420eba0 0x7fed7420f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.054+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fed742141b0 con 0x7fed7420eba0 2024-09-16T01:50:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.055+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fed7420eba0 0x7fed7420f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.055+0000 7fed72a00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fed7420eba0 0x7fed7420f000 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.056+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed742141b0 con 0x7fed7420eba0 2024-09-16T01:50:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.083+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fed74213030 con 0x7fed7420eba0 2024-09-16T01:50:30.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed6aa00640 1 -- 172.21.15.146:0/1880959620 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed74213030 con 0x7fed7420eba0 2024-09-16T01:50:30.096 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:30.097 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:30.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fed740b6070 msgr2=0x7fed7414de40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fed740b6070 0x7fed7414de40 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fed7414eba0 msgr2=0x7fed7414efc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fed7414eba0 0x7fed7414efc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fed74062630 msgr2=0x7fed74062a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fed74062630 0x7fed74062a70 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fed7402c640 msgr2=0x7fed7402cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fed7402c640 0x7fed7402cac0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fed7402d760 msgr2=0x7fed741e50c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fed7402d760 0x7fed741e50c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fed741e5b10 msgr2=0x7fed741e5f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fed741e5b10 0x7fed741e5f70 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fed7420bb50 msgr2=0x7fed7420df40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fed7420bb50 0x7fed7420df40 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fed7420eba0 msgr2=0x7fed7420f000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.085+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fed7420eba0 0x7fed7420f000 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fed5406c2b0 msgr2=0x7fed5406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fed5406c2b0 0x7fed5406e770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed7414fdf0 tx=0x7fed600090e0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed740c6d00 msgr2=0x7fed741f8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 --2- 172.21.15.146:0/1880959620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed740c6d00 0x7fed741f8c70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fed6400c430 tx=0x7fed6400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73400640 1 -- 172.21.15.146:0/1880959620 reap_dead start 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73400640 1 -- 172.21.15.146:0/1880959620 reap_dead start 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73400640 1 -- 172.21.15.146:0/1880959620 reap_dead start 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73400640 1 -- 172.21.15.146:0/1880959620 reap_dead start 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73400640 1 -- 172.21.15.146:0/1880959620 reap_dead start 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 shutdown_connections 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 >> 172.21.15.146:0/1880959620 conn(0x7fed7406d6f0 msgr2=0x7fed740cb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:30.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 shutdown_connections 2024-09-16T01:50:30.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.086+0000 7fed73e00640 1 -- 172.21.15.146:0/1880959620 wait complete. 2024-09-16T01:50:30.100 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:30.088+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:30.191 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:30.189+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:30.271 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:30.272 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:30.272 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:30.272 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:30.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:30.292 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:30.291+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:30.393 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:30.391+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:30.494 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:30.492+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:30.594 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:30.594 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:30.594 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:30.594 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:30.594 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:30.595 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:30.593+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:30.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:30.649 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:50:30.704 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:30.702+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:30.795 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:30.794+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:30.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.832+0000 7fe0c8200640 1 Processor -- start 2024-09-16T01:50:30.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.832+0000 7fe0c8200640 1 -- start start 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.834+0000 7fe0c8200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c00c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.834+0000 7fe0c8200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c006c950 con 0x7fe0c015e040 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.834+0000 7fe0c8200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c00707e0 con 0x7fe0c0154d00 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.834+0000 7fe0c8200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c0070a60 con 0x7fe0c0071010 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0c5800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe0c0154d00 0x7fe0c00c8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56902/0 (socket says 172.21.15.146:56902) 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0c5800640 1 -- 172.21.15.146:0/3018903793 learned_addr learned my addr 172.21.15.146:0/3018903793 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:30.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0c6200640 1 --2- 172.21.15.146:0/3018903793 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c00c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0bfe00640 1 -- 172.21.15.146:0/3018903793 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090986982 0 0) 0x7fe0b4001110 con 0x7fe0c0154d00 2024-09-16T01:50:30.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0bfe00640 1 -- 172.21.15.146:0/3018903793 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a8003520 con 0x7fe0c0154d00 2024-09-16T01:50:30.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0c6200640 1 -- 172.21.15.146:0/3018903793 >> v1:172.21.15.161:6789/0 conn(0x7fe0c0154d00 legacy=0x7fe0c00c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0c6200640 1 -- 172.21.15.146:0/3018903793 >> v1:172.21.15.146:6789/0 conn(0x7fe0c015e040 legacy=0x7fe0c00cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.835+0000 7fe0c6200640 1 -- 172.21.15.146:0/3018903793 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c006c390 con 0x7fe0c0071010 2024-09-16T01:50:30.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.837+0000 7fe0c6200640 1 --2- 172.21.15.146:0/3018903793 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c00c7bc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8009dc0 tx=0x7fe0b802a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8a704be37ba66f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.837+0000 7fe0bfe00640 1 -- 172.21.15.146:0/3018903793 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b802ae60 con 0x7fe0c0071010 2024-09-16T01:50:30.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.838+0000 7fe0bfe00640 1 -- 172.21.15.146:0/3018903793 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b8007740 con 0x7fe0c0071010 2024-09-16T01:50:30.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.838+0000 7fe0bfe00640 1 -- 172.21.15.146:0/3018903793 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8007af0 con 0x7fe0c0071010 2024-09-16T01:50:30.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.838+0000 7fe0c8200640 1 -- 172.21.15.146:0/3018903793 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 msgr2=0x7fe0c00c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.838+0000 7fe0c8200640 1 --2- 172.21.15.146:0/3018903793 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c00c7bc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8009dc0 tx=0x7fe0b802a680 comp rx=0 tx=0).stop 2024-09-16T01:50:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.839+0000 7fe0c8200640 1 -- 172.21.15.146:0/3018903793 shutdown_connections 2024-09-16T01:50:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.839+0000 7fe0c8200640 1 --2- 172.21.15.146:0/3018903793 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c00c7bc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.839+0000 7fe0c8200640 1 -- 172.21.15.146:0/3018903793 >> 172.21.15.146:0/3018903793 conn(0x7fe0c0067950 msgr2=0x7fe0c0067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.839+0000 7fe0c8200640 1 -- 172.21.15.146:0/3018903793 shutdown_connections 2024-09-16T01:50:30.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.839+0000 7fe0c8200640 1 -- 172.21.15.146:0/3018903793 wait complete. 2024-09-16T01:50:30.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.841+0000 7fe0c8200640 1 Processor -- start 2024-09-16T01:50:30.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.841+0000 7fe0c8200640 1 -- start start 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:30.844 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.073072838, 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:30.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:30.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.851 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.641519106, 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:30.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:30.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.621594884, 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:30.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:30.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.621574966, 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:30.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:30.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:30.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.621514382, 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:30.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:30.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:30.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.375228862, 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:30.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:30.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.375078348, 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:30.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:30.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.37505878, 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:30.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:30.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:30.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.804902566, 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:30.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.804734748, 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:30.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:30.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:30.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.804715766, 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.908 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:30.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:30.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:30.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:30.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:30.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:30.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:30.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:30.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:30.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.804696135, 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:30.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:30.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:30.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:30.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:30.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:30.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:30.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:30.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:30.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:30.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:30.924 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.804562229, 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.925 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:30.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:30.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:30.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.804550691, 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.931 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:30.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:30.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:30.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.80453228, 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.937 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:30.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:30.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.80441761, 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:30.943 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:30.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.949 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.793635882, 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:30.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:30.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.955 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.79351922399999, 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:30.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:30.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:30.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.977985985000004, 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:30.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:30.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:30.966 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.977924646999995, 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:30.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:30.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:30.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:30.971 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:30.971 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.842+0000 7fe0c8200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c01f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.842+0000 7fe0c8200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0c0154d00 0x7fe0c0157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.842+0000 7fe0c8200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0c015e040 0x7fe0c00cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.842+0000 7fe0c8200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c00c75a0 con 0x7fe0c015e040 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.842+0000 7fe0c8200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0070d40 con 0x7fe0c0154d00 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.842+0000 7fe0c8200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c01550e0 con 0x7fe0c0071010 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c01f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c4e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0c0154d00 0x7fe0c0157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c01f42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47350/0 (socket says 172.21.15.146:47350) 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 -- 172.21.15.146:0/1876716780 learned_addr learned my addr 172.21.15.146:0/1876716780 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c6200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0c015e040 0x7fe0c00cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 -- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0c0154d00 msgr2=0x7fe0c0157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0c0154d00 0x7fe0c0157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 -- 172.21.15.146:0/1876716780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0c015e040 msgr2=0x7fe0c00cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0c015e040 0x7fe0c00cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 -- 172.21.15.146:0/1876716780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c0150950 con 0x7fe0c0071010 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c6200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0c015e040 0x7fe0c00cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.843+0000 7fe0c5800640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c01f42e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe0b40012b0 tx=0x7fe0b400cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.844+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4018070 con 0x7fe0c0071010 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.844+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b40072d0 con 0x7fe0c0071010 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.844+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b40047e0 con 0x7fe0c0071010 2024-09-16T01:50:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.844+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c01f4820 con 0x7fe0c0071010 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.844+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c0158450 con 0x7fe0c0071010 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.846+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c015c830 con 0x7fe0c0071010 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.847+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe0b4004980 con 0x7fe0c0071010 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.848+0000 7fe0bd600640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0a006c2e0 0x7fe0a006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.848+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 5770+0+0 (secure 0 0 0) 0x7fe0b408a8d0 con 0x7fe0c0071010 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.849+0000 7fe0c4e00640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0a006c2e0 0x7fe0a006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.849+0000 7fe0c4e00640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0a006c2e0 0x7fe0a006e7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe0c01545d0 tx=0x7fe0ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:30.852+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe0c015c830 con 0x7fe0c0071010 2024-09-16T01:50:30.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:30.977 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:30.974+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:31.077 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:31.076+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:31.178 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:31.176+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:31.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.187+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe0c00c7bc0 con 0x7fe0c0071010 2024-09-16T01:50:31.279 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:31.278+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:31.286 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:31.287 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:31.287 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:31.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:31.380 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:31.379+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:31.481 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:31.480+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:31.545 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:31.546 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:31.546 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:31.546 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:31.546 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:31.564 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:31.611 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:31.609+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:31.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.631+0000 7fe0bd600640 1 -- 172.21.15.146:0/1876716780 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v47) ==== 97+0+0 (secure 0 0 0) 0x7fe0c00707e0 con 0x7fe0c0071010 2024-09-16T01:50:31.634 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:50:31.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0a006c2e0 msgr2=0x7fe0a006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:31.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0a006c2e0 0x7fe0a006e7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe0c01545d0 tx=0x7fe0ac009040 comp rx=0 tx=0).stop 2024-09-16T01:50:31.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 msgr2=0x7fe0c01f42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c01f42e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe0b40012b0 tx=0x7fe0b400cbe0 comp rx=0 tx=0).stop 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 shutdown_connections 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0a006c2e0 0x7fe0a006e7a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0c015e040 0x7fe0c00cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0c0154d00 0x7fe0c0157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 --2- 172.21.15.146:0/1876716780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0c0071010 0x7fe0c01f42e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.634+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 >> 172.21.15.146:0/1876716780 conn(0x7fe0c0067950 msgr2=0x7fe0c015bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.635+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 shutdown_connections 2024-09-16T01:50:31.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:31.635+0000 7fe0c8200640 1 -- 172.21.15.146:0/1876716780 wait complete. 2024-09-16T01:50:31.682 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:31.681+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:31.783 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:31.781+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:31.811 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:31.811 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.983451251000005, 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:31.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:31.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.983336429999994, 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:31.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:31.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.823 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.983294560000004, 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:31.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.829 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.983250280999997, 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:31.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:31.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:31.835 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.983221404000005, 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:31.839 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.979514197, 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:31.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.979460216000007, 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.844 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:31.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.944044992000002, 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:31.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.852 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.940984842999995, 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:31.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.940125394999995, 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.857 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:31.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:31.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.940071501999995, 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.863 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:31.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.934047528999997, 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:31.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.933997400999999, 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:31.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.933921068000004, 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:31.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.933889712999999, 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.882 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.886 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.895804257999998, 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:31.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.895012726999994, 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.891 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:31.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:31.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 65.870930901999998, 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:31.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:31.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:31.901 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 65.870858990000002, 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:31.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.54021659, 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:31.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:31.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:31.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:31.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:31.913 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:31.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:31.915 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:31.913+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:32.016 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:32.015+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:32.111 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:32.117 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:32.116+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:32.137 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:32.137 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:32.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:32.238 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:32.236+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:32.318 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:32.318+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:32.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.328+0000 7f7dabe00640 1 Processor -- start 2024-09-16T01:50:32.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.328+0000 7f7dabe00640 1 -- start start 2024-09-16T01:50:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.330+0000 7f7dabe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.330+0000 7f7dabe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7dac0c6d00 0x7f7dac154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.330+0000 7f7dabe00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dac15e220 con 0x7f7dac0d0490 2024-09-16T01:50:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.330+0000 7f7dabe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7dac0c6160 con 0x7f7dac0c6d00 2024-09-16T01:50:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.330+0000 7f7dabe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7dac15e500 con 0x7f7dac0d0870 2024-09-16T01:50:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daa000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daaa00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7dac0d0490 0x7f7dac0c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56912/0 (socket says 172.21.15.146:56912) 2024-09-16T01:50:32.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daaa00640 1 -- 172.21.15.146:0/47220399 learned_addr learned my addr 172.21.15.146:0/47220399 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daa000640 1 --2- 172.21.15.146:0/47220399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:47354/0 (socket says 172.21.15.146:47354) 2024-09-16T01:50:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7dab400640 1 --2- 172.21.15.146:0/47220399 >> v2:172.21.15.161:3300/0 conn(0x7f7dac0c6d00 0x7f7dac154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daa000640 1 -- 172.21.15.146:0/47220399 >> v2:172.21.15.161:3300/0 conn(0x7f7dac0c6d00 msgr2=0x7f7dac154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daa000640 1 --2- 172.21.15.146:0/47220399 >> v2:172.21.15.161:3300/0 conn(0x7f7dac0c6d00 0x7f7dac154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daa000640 1 -- 172.21.15.146:0/47220399 >> v1:172.21.15.161:6789/0 conn(0x7f7dac0d0490 legacy=0x7f7dac0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7daa000640 1 -- 172.21.15.146:0/47220399 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac15e7e0 con 0x7f7dac0d0870 2024-09-16T01:50:32.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.331+0000 7f7dab400640 1 --2- 172.21.15.146:0/47220399 >> v2:172.21.15.161:3300/0 conn(0x7f7dac0c6d00 0x7f7dac154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:32.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.333+0000 7f7daa000640 1 --2- 172.21.15.146:0/47220399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac154690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9bd90efb24c98 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.333+0000 7f7da9600640 1 -- 172.21.15.146:0/47220399 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98037070 con 0x7f7dac0d0870 2024-09-16T01:50:32.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.333+0000 7f7da9600640 1 -- 172.21.15.146:0/47220399 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d98032440 con 0x7f7dac0d0870 2024-09-16T01:50:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.333+0000 7f7da9600640 1 -- 172.21.15.146:0/47220399 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98032810 con 0x7f7dac0d0870 2024-09-16T01:50:32.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.334+0000 7f7dabe00640 1 -- 172.21.15.146:0/47220399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 msgr2=0x7f7dac154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.334+0000 7f7dabe00640 1 --2- 172.21.15.146:0/47220399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac154690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802a210 comp rx=0 tx=0).stop 2024-09-16T01:50:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.334+0000 7f7dabe00640 1 -- 172.21.15.146:0/47220399 shutdown_connections 2024-09-16T01:50:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.335+0000 7f7dabe00640 1 --2- 172.21.15.146:0/47220399 >> v2:172.21.15.161:3300/0 conn(0x7f7dac0c6d00 0x7f7dac154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.335+0000 7f7dabe00640 1 --2- 172.21.15.146:0/47220399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac154690 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.335+0000 7f7dabe00640 1 -- 172.21.15.146:0/47220399 >> 172.21.15.146:0/47220399 conn(0x7f7dac06d6f0 msgr2=0x7f7dac06e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:32.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.335+0000 7f7dabe00640 1 -- 172.21.15.146:0/47220399 shutdown_connections 2024-09-16T01:50:32.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.335+0000 7f7dabe00640 1 -- 172.21.15.146:0/47220399 wait complete. 2024-09-16T01:50:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.338+0000 7f7dabe00640 1 Processor -- start 2024-09-16T01:50:32.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.338+0000 7f7dabe00640 1 -- start start 2024-09-16T01:50:32.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.339+0000 7f7dabe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 0x7f7dac150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.339+0000 7f7dabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dac0d0490 0x7f7dac1e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7daaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 0x7f7dac150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7daaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 0x7f7dac150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54428/0 (socket says 172.21.15.146:54428) 2024-09-16T01:50:32.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7daa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dac0d0490 0x7f7dac1e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7daa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dac0d0490 0x7f7dac1e3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47738/0 (socket says 172.21.15.146:47738) 2024-09-16T01:50:32.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7dabe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7dabe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac15dec0 con 0x7f7dac0d0490 2024-09-16T01:50:32.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7dabe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac15eda0 con 0x7f7dac0c6d00 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7dabe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7dac0c51f0 con 0x7f7dac0d0870 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.340+0000 7f7daaa00640 1 -- 172.21.15.146:0/2195901717 learned_addr learned my addr 172.21.15.146:0/2195901717 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7daaa00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 msgr2=0x7f7dac14f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7dac0d0870 0x7f7dac14f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7daaa00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dac0d0490 msgr2=0x7f7dac1e3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dac0d0490 0x7f7dac1e3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7daaa00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac1501d0 con 0x7f7dac0c6d00 2024-09-16T01:50:32.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.341+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dac0d0490 0x7f7dac1e3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:32.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.342+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 0x7f7dac150f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00c400 tx=0x7f7d9c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.342+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c017070 con 0x7f7dac0c6d00 2024-09-16T01:50:32.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.342+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9c005ce0 con 0x7f7dac0c6d00 2024-09-16T01:50:32.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.342+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c0072f0 con 0x7f7dac0c6d00 2024-09-16T01:50:32.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.343+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dac1f11b0 con 0x7f7dac0c6d00 2024-09-16T01:50:32.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.343+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dac150750 con 0x7f7dac0c6d00 2024-09-16T01:50:32.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.344+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7dac0c65f0 con 0x7f7dac0c6d00 2024-09-16T01:50:32.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.346+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7d9c01a070 con 0x7f7dac0c6d00 2024-09-16T01:50:32.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.348+0000 7f7da3400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d8c06c2b0 0x7f7d8c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.348+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5770+0+0 (secure 0 0 0) 0x7f7d9c08a920 con 0x7f7dac0c6d00 2024-09-16T01:50:32.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.348+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v47) ==== 54+0+16 (secure 0 0 0) 0x7f7d9c08f220 con 0x7f7dac0c6d00 2024-09-16T01:50:32.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.349+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d8c06c2b0 0x7f7d8c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.349+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7dac1e5d00 0x7f7dac14f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.349+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7dac1e4170 con 0x7f7dac1e5d00 2024-09-16T01:50:32.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.349+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7dac1e5d00 0x7f7dac14f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.350+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d8c06c2b0 0x7f7d8c06e770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d98034000 tx=0x7f7d9802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.350+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7dac1e5d00 0x7f7dac14f5f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.351+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac1e4170 con 0x7f7dac1e5d00 2024-09-16T01:50:32.368 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:32.369 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:32.369 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:32.369 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:32.369 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:32.381 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:32.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.380+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac0cc3f0 con 0x7f7dac1e5d00 2024-09-16T01:50:32.386 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:32.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.383+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac0cc3f0 con 0x7f7dac1e5d00 2024-09-16T01:50:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.384+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7dac1f3b20 0x7f7dac1f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.384+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7dac152560 con 0x7f7dac1f3b20 2024-09-16T01:50:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.384+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7dac1f3b20 0x7f7dac1f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.385+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7dac1f3b20 0x7f7dac1f3f00 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.385+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac152560 con 0x7f7dac1f3b20 2024-09-16T01:50:32.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.400+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac0ce350 con 0x7f7dac1f3b20 2024-09-16T01:50:32.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.403+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac0ce350 con 0x7f7dac1f3b20 2024-09-16T01:50:32.405 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.403+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7dac14e030 0x7f7dac14e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.403+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7dac14e970 con 0x7f7dac14e030 2024-09-16T01:50:32.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.404+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7dac14e030 0x7f7dac14e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.405+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7dac14e030 0x7f7dac14e430 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.406+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac14e970 con 0x7f7dac14e030 2024-09-16T01:50:32.419 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:32.418+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:32.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.433+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac062630 con 0x7f7dac14e030 2024-09-16T01:50:32.437 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:32.438 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:32.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.435+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac062630 con 0x7f7dac14e030 2024-09-16T01:50:32.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.436+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7dac063110 0x7f7dac15b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.436+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7dac063570 con 0x7f7dac063110 2024-09-16T01:50:32.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.436+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7dac063110 0x7f7dac15b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.437+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7dac063110 0x7f7dac15b9b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.437+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac063570 con 0x7f7dac063110 2024-09-16T01:50:32.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.454+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac15c1c0 con 0x7f7dac063110 2024-09-16T01:50:32.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.456+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac15c1c0 con 0x7f7dac063110 2024-09-16T01:50:32.458 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:32.459 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:32.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.457+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7dac02c640 0x7f7dac02ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.457+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7dac063570 con 0x7f7dac02c640 2024-09-16T01:50:32.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.458+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7dac02c640 0x7f7dac02ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.458+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7dac02c640 0x7f7dac02ca60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.459+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac063570 con 0x7f7dac02c640 2024-09-16T01:50:32.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.486+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac063570 con 0x7f7dac02c640 2024-09-16T01:50:32.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.488+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac063570 con 0x7f7dac02c640 2024-09-16T01:50:32.490 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:32.492 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:32.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.489+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7dac02d4b0 0x7f7dac207130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.489+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7dac02d910 con 0x7f7dac02d4b0 2024-09-16T01:50:32.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.489+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7dac02d4b0 0x7f7dac207130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.490+0000 7f7daa000640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7dac02d4b0 0x7f7dac207130 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.490+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac02d910 con 0x7f7dac02d4b0 2024-09-16T01:50:32.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.517+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac02d910 con 0x7f7dac02d4b0 2024-09-16T01:50:32.520 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:32.518+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:32.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.519+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac02d910 con 0x7f7dac02d4b0 2024-09-16T01:50:32.521 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:32.522 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:32.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.519+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7dac207b80 0x7f7dac209f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.519+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7dac20cd40 con 0x7f7dac207b80 2024-09-16T01:50:32.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.520+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7dac207b80 0x7f7dac209f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.520+0000 7f7dab400640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7dac207b80 0x7f7dac209f70 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.521+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac20cd40 con 0x7f7dac207b80 2024-09-16T01:50:32.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.548+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac20a4b0 con 0x7f7dac207b80 2024-09-16T01:50:32.550 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:32.550 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:32.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.550+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac20a4b0 con 0x7f7dac207b80 2024-09-16T01:50:32.552 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:32.552 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:32.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.550+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7dac20c330 0x7f7dac20c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:32.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.550+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7dac2100e0 con 0x7f7dac20c330 2024-09-16T01:50:32.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.551+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7dac20c330 0x7f7dac20c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:32.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.551+0000 7f7daaa00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7dac20c330 0x7f7dac20c770 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:32.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.552+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7dac2100e0 con 0x7f7dac20c330 2024-09-16T01:50:32.564 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:32.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.580+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7dac20ef60 con 0x7f7dac20c330 2024-09-16T01:50:32.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7da3400640 1 -- 172.21.15.146:0/2195901717 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7dac20ef60 con 0x7f7dac20c330 2024-09-16T01:50:32.584 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:32.586 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:32.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7dac1e5d00 msgr2=0x7f7dac14f5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7dac1e5d00 0x7f7dac14f5f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7dac1f3b20 msgr2=0x7f7dac1f3f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7dac1f3b20 0x7f7dac1f3f00 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7dac14e030 msgr2=0x7f7dac14e430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7dac14e030 0x7f7dac14e430 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7dac063110 msgr2=0x7f7dac15b9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7dac063110 0x7f7dac15b9b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7dac02c640 msgr2=0x7f7dac02ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7dac02c640 0x7f7dac02ca60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7dac02d4b0 msgr2=0x7f7dac207130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7dac02d4b0 0x7f7dac207130 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7dac207b80 msgr2=0x7f7dac209f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7dac207b80 0x7f7dac209f70 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7dac20c330 msgr2=0x7f7dac20c770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7dac20c330 0x7f7dac20c770 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dab400640 1 -- 172.21.15.146:0/2195901717 reap_dead start 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dab400640 1 -- 172.21.15.146:0/2195901717 reap_dead start 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dab400640 1 -- 172.21.15.146:0/2195901717 reap_dead start 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dab400640 1 -- 172.21.15.146:0/2195901717 reap_dead start 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dab400640 1 -- 172.21.15.146:0/2195901717 reap_dead start 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d8c06c2b0 msgr2=0x7f7d8c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.582+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d8c06c2b0 0x7f7d8c06e770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d98034000 tx=0x7f7d9802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:32.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 msgr2=0x7f7dac150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 0x7f7dac150f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00c400 tx=0x7f7d9c00c8c0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 shutdown_connections 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d8c06c2b0 0x7f7d8c06e770 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 --2- 172.21.15.146:0/2195901717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7dac0c6d00 0x7f7dac150f50 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00c400 tx=0x7f7d9c00c8c0 comp rx=0 tx=0).stop 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 >> 172.21.15.146:0/2195901717 conn(0x7f7dac06d6f0 msgr2=0x7f7dac0ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 shutdown_connections 2024-09-16T01:50:32.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:32.583+0000 7f7dabe00640 1 -- 172.21.15.146:0/2195901717 wait complete. 2024-09-16T01:50:32.651 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:32.649+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:32.685 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:32.685 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:32.694 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:32.722 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:32.720+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:32.798 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:32.798 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:32.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:32.852 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:32.851+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:32.911 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:32.912 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:32.920 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:32.924 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:32.922+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:33.025 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:33.024+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:33.063 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.063 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.072 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:33.156 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:33.155+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:33.216 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.216 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.224 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:33.265 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:33.264+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:33.328 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:33.327+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:33.368 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.368 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.377 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:33.429 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:33.427+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:33.522 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.522 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.529 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:33.528+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:33.531 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:33.630 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:33.628+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:33.654 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:50:33.676 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.676 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.685 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:33.731 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:33.730+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:33.790 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.790 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.799 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:33.832 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:33.830+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.903+0000 7f002fe00640 1 Processor -- start 2024-09-16T01:50:33.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.903+0000 7f002fe00640 1 -- start start 2024-09-16T01:50:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.904+0000 7f002fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f00300c6b70 0x7f003015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.904+0000 7f002fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 0x7f00300cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.904+0000 7f002fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f003006d140 con 0x7f00300cbab0 2024-09-16T01:50:33.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.904+0000 7f002fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f003015b880 con 0x7f00300c6b70 2024-09-16T01:50:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.904+0000 7f002fe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f00300cab20 con 0x7f00300c75a0 2024-09-16T01:50:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.905+0000 7f002e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 0x7f00300cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.905+0000 7f002e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 0x7f00300cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50608/0 (socket says 172.21.15.146:50608) 2024-09-16T01:50:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.905+0000 7f002e000640 1 -- 172.21.15.146:0/4066630539 learned_addr learned my addr 172.21.15.146:0/4066630539 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.905+0000 7f002ea00640 1 --2- 172.21.15.146:0/4066630539 >> v2:172.21.15.146:3300/0 conn(0x7f00300c6b70 0x7f003015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:33.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.905+0000 7f002e000640 1 -- 172.21.15.146:0/4066630539 >> v2:172.21.15.146:3300/0 conn(0x7f00300c6b70 msgr2=0x7f003015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:33.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.906+0000 7f002e000640 1 --2- 172.21.15.146:0/4066630539 >> v2:172.21.15.146:3300/0 conn(0x7f00300c6b70 0x7f003015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:33.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.906+0000 7f002e000640 1 -- 172.21.15.146:0/4066630539 >> v1:172.21.15.146:6789/0 conn(0x7f00300cbab0 legacy=0x7f00300cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:33.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.906+0000 7f002d600640 1 -- 172.21.15.146:0/4066630539 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357713834 0 0) 0x7f003006d140 con 0x7f00300cbab0 2024-09-16T01:50:33.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.906+0000 7f002e000640 1 -- 172.21.15.146:0/4066630539 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003006d420 con 0x7f00300c75a0 2024-09-16T01:50:33.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.906+0000 7f002ea00640 1 --2- 172.21.15.146:0/4066630539 >> v2:172.21.15.146:3300/0 conn(0x7f00300c6b70 0x7f003015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.907+0000 7f002e000640 1 --2- 172.21.15.146:0/4066630539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 0x7f00300cb570 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f001c009870 tx=0x7f001c02a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37b34589566832ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.908+0000 7f002d600640 1 -- 172.21.15.146:0/4066630539 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c037070 con 0x7f00300c75a0 2024-09-16T01:50:33.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.908+0000 7f002d600640 1 -- 172.21.15.146:0/4066630539 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001c032440 con 0x7f00300c75a0 2024-09-16T01:50:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.908+0000 7f002d600640 1 -- 172.21.15.146:0/4066630539 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c032810 con 0x7f00300c75a0 2024-09-16T01:50:33.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.908+0000 7f002fe00640 1 -- 172.21.15.146:0/4066630539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 msgr2=0x7f00300cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.908+0000 7f002fe00640 1 --2- 172.21.15.146:0/4066630539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 0x7f00300cb570 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f001c009870 tx=0x7f001c02a150 comp rx=0 tx=0).stop 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.909+0000 7f002fe00640 1 -- 172.21.15.146:0/4066630539 shutdown_connections 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.909+0000 7f002fe00640 1 --2- 172.21.15.146:0/4066630539 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300c75a0 0x7f00300cb570 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.909+0000 7f002fe00640 1 --2- 172.21.15.146:0/4066630539 >> v2:172.21.15.146:3300/0 conn(0x7f00300c6b70 0x7f003015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.909+0000 7f002fe00640 1 -- 172.21.15.146:0/4066630539 >> 172.21.15.146:0/4066630539 conn(0x7f00300ccc20 msgr2=0x7f00300cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.909+0000 7f002fe00640 1 -- 172.21.15.146:0/4066630539 shutdown_connections 2024-09-16T01:50:33.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.909+0000 7f002fe00640 1 -- 172.21.15.146:0/4066630539 wait complete. 2024-09-16T01:50:33.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.911+0000 7f002fe00640 1 Processor -- start 2024-09-16T01:50:33.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.911+0000 7f002fe00640 1 -- start start 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 0x7f00300ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00300c75a0 0x7f003015d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300cbab0 0x7f003015da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00300cb050 con 0x7f00300c75a0 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002fe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00300cad70 con 0x7f00300c6b70 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002fe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f003006cd20 con 0x7f00300cbab0 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 0x7f00300ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:33.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 0x7f00300ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34484/0 (socket says 172.21.15.146:34484) 2024-09-16T01:50:33.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002ea00640 1 -- 172.21.15.146:0/1389068331 learned_addr learned my addr 172.21.15.146:0/1389068331 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:33.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300cbab0 0x7f003015da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:33.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002e000640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00300c75a0 0x7f003015d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.913+0000 7f002ea00640 1 -- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300cbab0 msgr2=0x7f003015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002ea00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300cbab0 0x7f003015da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002ea00640 1 -- 172.21.15.146:0/1389068331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00300c75a0 msgr2=0x7f003015d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002ea00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00300c75a0 0x7f003015d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002ea00640 1 -- 172.21.15.146:0/1389068331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00301f5210 con 0x7f00300c6b70 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002f400640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300cbab0 0x7f003015da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002ea00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 0x7f00300ce670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f002000e960 tx=0x7f002000ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:33.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020017070 con 0x7f00300c6b70 2024-09-16T01:50:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002000ca00 con 0x7f00300c6b70 2024-09-16T01:50:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000cdd0 con 0x7f00300c6b70 2024-09-16T01:50:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.914+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00301f54f0 con 0x7f00300c6b70 2024-09-16T01:50:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.915+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00301f5ab0 con 0x7f00300c6b70 2024-09-16T01:50:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.916+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f003006da90 con 0x7f00300c6b70 2024-09-16T01:50:33.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.917+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f002001a070 con 0x7f00300c6b70 2024-09-16T01:50:33.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.918+0000 7f0026a00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f001006c2b0 0x7f001006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:33.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.918+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5770+0+0 (secure 0 0 0) 0x7f002008b2b0 con 0x7f00300c6b70 2024-09-16T01:50:33.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.919+0000 7f002e000640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f001006c2b0 0x7f001006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:33.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.919+0000 7f002e000640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f001006c2b0 0x7f001006e770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f001c034000 tx=0x7f001c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:33.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:33.922+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0020059380 con 0x7f00300c6b70 2024-09-16T01:50:33.933 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:33.931+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:33.944 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:33.944 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:33.953 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:34.034 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:34.033+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:34.058 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:34.058 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:34.067 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:34.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.114+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f00300cda30 con 0x7f00300c6b70 2024-09-16T01:50:34.151 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:34.150+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:34.212 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:34.212 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:34.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:34.262 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:34.261+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:34.336 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:34.335+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:34.436 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:34.435+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:34.489 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:34.489 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:34.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:34.549 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:34.547+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:34.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:34.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.645+0000 7f0026a00640 1 -- 172.21.15.146:0/1389068331 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v48) ==== 91+0+0 (secure 0 0 0) 0x7f002005d680 con 0x7f00300c6b70 2024-09-16T01:50:34.651 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:50:34.651 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:34.650+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:34.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f001006c2b0 msgr2=0x7f001006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:34.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f001006c2b0 0x7f001006e770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f001c034000 tx=0x7f001c02e000 comp rx=0 tx=0).stop 2024-09-16T01:50:34.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 msgr2=0x7f00300ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:34.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 0x7f00300ce670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f002000e960 tx=0x7f002000ee20 comp rx=0 tx=0).stop 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 shutdown_connections 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f001006c2b0 0x7f001006e770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00300cbab0 0x7f003015da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00300c75a0 0x7f003015d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 --2- 172.21.15.146:0/1389068331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00300c6b70 0x7f00300ce670 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.650+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 >> 172.21.15.146:0/1389068331 conn(0x7f00300ccc20 msgr2=0x7f00300cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.651+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 shutdown_connections 2024-09-16T01:50:34.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.651+0000 7f002fe00640 1 -- 172.21.15.146:0/1389068331 wait complete. 2024-09-16T01:50:34.681 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:34.681 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:34.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:34.740 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:34.739+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:34.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.797+0000 7f9bbb400640 1 Processor -- start 2024-09-16T01:50:34.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.797+0000 7f9bbb400640 1 -- start start 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.798+0000 7f9bbb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 0x7f9bb415b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.798+0000 7f9bbb400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9bb40cca70 0x7f9bb4073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.798+0000 7f9bbb400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9bb40cafa0 0x7f9bb4073880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.798+0000 7f9bbb400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9bb415de60 con 0x7f9bb40cca70 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bbb400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9bb415e0b0 con 0x7f9bb40cafa0 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bbb400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb415e390 con 0x7f9bb4154c10 2024-09-16T01:50:34.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bb8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 0x7f9bb415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bb8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 0x7f9bb415b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50620/0 (socket says 172.21.15.146:50620) 2024-09-16T01:50:34.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bb8a00640 1 -- 172.21.15.146:0/533232847 learned_addr learned my addr 172.21.15.146:0/533232847 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bb9400640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.161:3300/0 conn(0x7f9bb40cafa0 0x7f9bb4073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bb8a00640 1 -- 172.21.15.146:0/533232847 >> v2:172.21.15.161:3300/0 conn(0x7f9bb40cafa0 msgr2=0x7f9bb4073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.799+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.161:3300/0 conn(0x7f9bb40cafa0 0x7f9bb4073880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.800+0000 7f9bb8a00640 1 -- 172.21.15.146:0/533232847 >> v2:172.21.15.146:3300/0 conn(0x7f9bb40cca70 msgr2=0x7f9bb4073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.800+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.146:3300/0 conn(0x7f9bb40cca70 0x7f9bb4073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.800+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.146:3300/0 conn(0x7f9bb40cca70 0x7f9bb4073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.800+0000 7f9bb8a00640 1 -- 172.21.15.146:0/533232847 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb415e670 con 0x7f9bb4154c10 2024-09-16T01:50:34.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.800+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.146:3300/0 conn(0x7f9bb40cca70 0x7f9bb4073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.801+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/533232847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 0x7f9bb415b370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40d1310 tx=0x7f9ba802a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89f5805b81a84601 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.801+0000 7f9bb3400640 1 -- 172.21.15.146:0/533232847 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8037070 con 0x7f9bb4154c10 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.801+0000 7f9bb3400640 1 -- 172.21.15.146:0/533232847 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba8032440 con 0x7f9bb4154c10 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bb3400640 1 -- 172.21.15.146:0/533232847 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8032810 con 0x7f9bb4154c10 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 -- 172.21.15.146:0/533232847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 msgr2=0x7f9bb415b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 --2- 172.21.15.146:0/533232847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 0x7f9bb415b370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40d1310 tx=0x7f9ba802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 -- 172.21.15.146:0/533232847 shutdown_connections 2024-09-16T01:50:34.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.161:3300/0 conn(0x7f9bb40cafa0 0x7f9bb4073880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 --2- 172.21.15.146:0/533232847 >> v2:172.21.15.146:3300/0 conn(0x7f9bb40cca70 0x7f9bb4073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 --2- 172.21.15.146:0/533232847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb4154c10 0x7f9bb415b370 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.802+0000 7f9bbb400640 1 -- 172.21.15.146:0/533232847 >> 172.21.15.146:0/533232847 conn(0x7f9bb406ce80 msgr2=0x7f9bb406d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:34.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.803+0000 7f9bbb400640 1 -- 172.21.15.146:0/533232847 shutdown_connections 2024-09-16T01:50:34.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.803+0000 7f9bbb400640 1 -- 172.21.15.146:0/533232847 wait complete. 2024-09-16T01:50:34.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.804+0000 7f9bbb400640 1 Processor -- start 2024-09-16T01:50:34.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 -- start start 2024-09-16T01:50:34.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 0x7f9bb41e3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb40cca70 0x7f9bb415c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb4154c10 0x7f9bb40b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4154ff0 con 0x7f9bb4154c10 2024-09-16T01:50:34.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb415ec30 con 0x7f9bb40cca70 2024-09-16T01:50:34.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.805+0000 7f9bbb400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb40cb6f0 con 0x7f9bb40cafa0 2024-09-16T01:50:34.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 0x7f9bb41e3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb40cca70 0x7f9bb415c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 0x7f9bb41e3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50630/0 (socket says 172.21.15.146:50630) 2024-09-16T01:50:34.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 -- 172.21.15.146:0/3974458194 learned_addr learned my addr 172.21.15.146:0/3974458194 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:34.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb4154c10 0x7f9bb40b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb40cca70 msgr2=0x7f9bb415c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:34.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9bb40cca70 0x7f9bb415c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb4154c10 msgr2=0x7f9bb40b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb4154c10 0x7f9bb40b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb8a00640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb41e4e10 con 0x7f9bb40cafa0 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.806+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb4154c10 0x7f9bb40b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.807+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 0x7f9bb41e3c20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ba802a220 tx=0x7f9ba80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.807+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8041020 con 0x7f9bb40cafa0 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.807+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba80306c0 con 0x7f9bb40cafa0 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.807+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb4150400 con 0x7f9bb40cafa0 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.807+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8037040 con 0x7f9bb40cafa0 2024-09-16T01:50:34.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.807+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb4150f10 con 0x7f9bb40cafa0 2024-09-16T01:50:34.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.809+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9b90005270 con 0x7f9bb40cafa0 2024-09-16T01:50:34.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.809+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9ba8044020 con 0x7f9bb40cafa0 2024-09-16T01:50:34.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.810+0000 7f9bb0c00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b9406c2b0 0x7f9b9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.811+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f9ba80abd30 con 0x7f9bb40cafa0 2024-09-16T01:50:34.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.811+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v48) ==== 54+0+16 (secure 0 0 0) 0x7f9ba804b030 con 0x7f9bb40cafa0 2024-09-16T01:50:34.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.811+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b9406c2b0 0x7f9b9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.811+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9b90000fa0 0x7f9b900045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.811+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b90004b20 con 0x7f9b90000fa0 2024-09-16T01:50:34.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.812+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b9406c2b0 0x7f9b9406e770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4076570 tx=0x7f9ba0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.812+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9b90000fa0 0x7f9b900045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.812+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9b90000fa0 0x7f9b900045e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.814+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b90004b20 con 0x7f9b90000fa0 2024-09-16T01:50:34.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.830+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b90005090 con 0x7f9b90000fa0 2024-09-16T01:50:34.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.834+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b90005090 con 0x7f9b90000fa0 2024-09-16T01:50:34.836 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:34.837 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:34.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.835+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9b90001c70 0x7f9b90002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.835+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9b900022b0 con 0x7f9b90001c70 2024-09-16T01:50:34.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.836+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9b90001c70 0x7f9b90002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.838+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9b90001c70 0x7f9b90002990 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.840 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:34.839+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:34.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.839+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b900022b0 con 0x7f9b90001c70 2024-09-16T01:50:34.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.869+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b900031a0 con 0x7f9b90001c70 2024-09-16T01:50:34.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.872+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b900031a0 con 0x7f9b90001c70 2024-09-16T01:50:34.875 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:34.875 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:34.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.873+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9b90003a80 0x7f9b9000bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.873+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9b9000eb30 con 0x7f9b90003a80 2024-09-16T01:50:34.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.874+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9b90003a80 0x7f9b9000bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.875+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9b90003a80 0x7f9b9000bce0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.876+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b9000eb30 con 0x7f9b90003a80 2024-09-16T01:50:34.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.906+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b9000c5b0 con 0x7f9b90003a80 2024-09-16T01:50:34.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.908+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b9000c5b0 con 0x7f9b90003a80 2024-09-16T01:50:34.911 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:34.911 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:34.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.909+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9b9000e870 0x7f9b9000c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.909+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9b90012510 con 0x7f9b9000e870 2024-09-16T01:50:34.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.910+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9b9000e870 0x7f9b9000c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.912+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9b9000e870 0x7f9b9000c570 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.913+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b90012510 con 0x7f9b9000e870 2024-09-16T01:50:34.941 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:34.940+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:34.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.942+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b9000cef0 con 0x7f9b9000e870 2024-09-16T01:50:34.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.945+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b9000cef0 con 0x7f9b9000e870 2024-09-16T01:50:34.947 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:34.947 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:34.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.945+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9b9000d610 0x7f9b90010d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:34.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.945+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9b9000da50 con 0x7f9b9000d610 2024-09-16T01:50:34.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.946+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9b9000d610 0x7f9b90010d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:34.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.946+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9b9000d610 0x7f9b90010d80 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:34.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.947+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b9000da50 con 0x7f9b9000d610 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.123651937999995, 2024-09-16T01:50:34.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:34.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:50:34.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.123545888999999, 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:50:34.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:50:34.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:34.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.123491263000005, 2024-09-16T01:50:34.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.123422930000004, 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:50:34.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.974 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.089167727000003, 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:50:34.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:50:34.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.088984529000001, 2024-09-16T01:50:34.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:34.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:50:34.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:50:34.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.088930520999995, 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:50:34.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:50:34.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:34.991 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.088898080999996, 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:50:34.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:50:34.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.085530079999998, 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:34.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:50:35.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.085459404000005, 2024-09-16T01:50:35.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:50:35.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.005 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.085423028999998, 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:50:35.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.084795286000002, 2024-09-16T01:50:35.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:50:35.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:50:35.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.084740679000006, 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:50:35.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:50:35.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:50:35.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012602271000006, 2024-09-16T01:50:35.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:50:35.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:50:35.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012543958999998, 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:50:35.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012469883999998, 2024-09-16T01:50:35.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:50:35.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:50:35.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012411263000004, 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:50:35.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.041 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012389329000001, 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:50:35.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:50:35.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:50:35.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012358969000005, 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:50:35.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:50:35.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:35.052 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.012332389999997, 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:35.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:50:35.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.057 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.058 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.058 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:35.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:35.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.971+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b9000da50 con 0x7f9b9000d610 2024-09-16T01:50:35.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.973+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b9000da50 con 0x7f9b9000d610 2024-09-16T01:50:35.060 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.973+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9b900117d0 0x7f9b90011c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.973+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9b90012260 con 0x7f9b900117d0 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.974+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9b900117d0 0x7f9b90011c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.974+0000 7f9bb3e00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9b900117d0 0x7f9b90011c50 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.975+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b90012260 con 0x7f9b900117d0 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.989+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b90012260 con 0x7f9b900117d0 2024-09-16T01:50:35.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.991+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b90012260 con 0x7f9b900117d0 2024-09-16T01:50:35.062 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.991+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9b90016c50 0x7f9b90019040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.991+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9b9001bff0 con 0x7f9b90016c50 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.991+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9b90016c50 0x7f9b90019040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.992+0000 7f9bb9400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9b90016c50 0x7f9b90019040 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:34.993+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b9001bff0 con 0x7f9b90016c50 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.006+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b9001bfc0 con 0x7f9b90016c50 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.008+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b9001bfc0 con 0x7f9b90016c50 2024-09-16T01:50:35.063 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:35.064 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:35.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.008+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9b9001b220 0x7f9b9001b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.008+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9b9001f3c0 con 0x7f9b9001b220 2024-09-16T01:50:35.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.009+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9b9001b220 0x7f9b9001b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.009+0000 7f9bb8a00640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9b9001b220 0x7f9b9001b680 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.010+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b9001f3c0 con 0x7f9b9001b220 2024-09-16T01:50:35.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.023+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9b900195d0 con 0x7f9b9001b220 2024-09-16T01:50:35.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.025+0000 7f9bb0c00640 1 -- 172.21.15.146:0/3974458194 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9b900195d0 con 0x7f9b9001b220 2024-09-16T01:50:35.065 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:35.066 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:35.058+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9b90000fa0 msgr2=0x7f9b900045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9b90000fa0 0x7f9b900045e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9b90001c70 msgr2=0x7f9b90002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9b90001c70 0x7f9b90002990 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9b90003a80 msgr2=0x7f9b9000bce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9b90003a80 0x7f9b9000bce0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9b9000e870 msgr2=0x7f9b9000c570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9b9000e870 0x7f9b9000c570 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9b9000d610 msgr2=0x7f9b90010d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9b9000d610 0x7f9b90010d80 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9b900117d0 msgr2=0x7f9b90011c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9b900117d0 0x7f9b90011c50 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9b90016c50 msgr2=0x7f9b90019040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9b90016c50 0x7f9b90019040 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9b9001b220 msgr2=0x7f9b9001b680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9b9001b220 0x7f9b9001b680 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bb9400640 1 -- 172.21.15.146:0/3974458194 reap_dead start 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bb9400640 1 -- 172.21.15.146:0/3974458194 reap_dead start 2024-09-16T01:50:35.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bb9400640 1 -- 172.21.15.146:0/3974458194 reap_dead start 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bb9400640 1 -- 172.21.15.146:0/3974458194 reap_dead start 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bb9400640 1 -- 172.21.15.146:0/3974458194 reap_dead start 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b9406c2b0 msgr2=0x7f9b9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b9406c2b0 0x7f9b9406e770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4076570 tx=0x7f9ba0008040 comp rx=0 tx=0).stop 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 msgr2=0x7f9bb41e3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.026+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 0x7f9bb41e3c20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ba802a220 tx=0x7f9ba80046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.027+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 shutdown_connections 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.027+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b9406c2b0 0x7f9b9406e770 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.027+0000 7f9bbb400640 1 --2- 172.21.15.146:0/3974458194 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9bb40cafa0 0x7f9bb41e3c20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.027+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 >> 172.21.15.146:0/3974458194 conn(0x7f9bb406ce80 msgr2=0x7f9bb4153730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.027+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 shutdown_connections 2024-09-16T01:50:35.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.027+0000 7f9bbb400640 1 -- 172.21.15.146:0/3974458194 wait complete. 2024-09-16T01:50:35.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:35.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:35.091 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T01:50:35.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-16T01:50:35.160 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:35.160+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:35.261 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:35.260+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.286+0000 7f036f200640 1 Processor -- start 2024-09-16T01:50:35.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.286+0000 7f036f200640 1 -- start start 2024-09-16T01:50:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.287+0000 7f036f200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f03700c75a0 0x7f037015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.287+0000 7f036f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037006d140 con 0x7f03700c6b70 2024-09-16T01:50:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.287+0000 7f036f200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037006d420 con 0x7f03700cc040 2024-09-16T01:50:35.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.287+0000 7f036f200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f037015b880 con 0x7f03700c75a0 2024-09-16T01:50:35.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036de00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f03700c6b70 0x7f03700cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43790/0 (socket says 172.21.15.146:43790) 2024-09-16T01:50:35.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036de00640 1 -- 172.21.15.146:0/3248513765 learned_addr learned my addr 172.21.15.146:0/3248513765 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:35.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036d400640 1 --2- 172.21.15.146:0/3248513765 >> v2:172.21.15.161:3300/0 conn(0x7f03700c75a0 0x7f037015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225527317 0 0) 0x7f037006d140 con 0x7f03700c6b70 2024-09-16T01:50:35.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0354003520 con 0x7f03700c6b70 2024-09-16T01:50:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699369965 0 0) 0x7f037006d420 con 0x7f03700cc040 2024-09-16T01:50:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.288+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f037006d140 con 0x7f03700cc040 2024-09-16T01:50:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.289+0000 7f036d400640 1 -- 172.21.15.146:0/3248513765 >> v1:172.21.15.161:6789/0 conn(0x7f03700cc040 legacy=0x7f0370165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.289+0000 7f036d400640 1 -- 172.21.15.146:0/3248513765 >> v1:172.21.15.146:6789/0 conn(0x7f03700c6b70 legacy=0x7f03700cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.289+0000 7f036d400640 1 -- 172.21.15.146:0/3248513765 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037006d700 con 0x7f03700c75a0 2024-09-16T01:50:35.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.290+0000 7f036d400640 1 --2- 172.21.15.146:0/3248513765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700c75a0 0x7f037015b340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f035c0097a0 tx=0x7f035c02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75fd38c3af4075ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.292 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:35.292 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:35.292 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:35.292 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.290+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c037070 con 0x7f03700c75a0 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.290+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f035c032440 con 0x7f03700c75a0 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.290+0000 7f036ca00640 1 -- 172.21.15.146:0/3248513765 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c032810 con 0x7f03700c75a0 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 -- 172.21.15.146:0/3248513765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700c75a0 msgr2=0x7f037015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 --2- 172.21.15.146:0/3248513765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700c75a0 0x7f037015b340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f035c0097a0 tx=0x7f035c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 -- 172.21.15.146:0/3248513765 shutdown_connections 2024-09-16T01:50:35.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 --2- 172.21.15.146:0/3248513765 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700c75a0 0x7f037015b340 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 -- 172.21.15.146:0/3248513765 >> 172.21.15.146:0/3248513765 conn(0x7f03700ccc20 msgr2=0x7f03700cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 -- 172.21.15.146:0/3248513765 shutdown_connections 2024-09-16T01:50:35.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.291+0000 7f036f200640 1 -- 172.21.15.146:0/3248513765 wait complete. 2024-09-16T01:50:35.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.293+0000 7f036f200640 1 Processor -- start 2024-09-16T01:50:35.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.293+0000 7f036f200640 1 -- start start 2024-09-16T01:50:35.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.294+0000 7f036f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03700c6b70 0x7f03700cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.294+0000 7f036f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f03700c75a0 0x7f03701f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.294+0000 7f036f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700cc040 0x7f03701f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.294+0000 7f036f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03700cb050 con 0x7f03700c6b70 2024-09-16T01:50:35.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.294+0000 7f036f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f03700cab20 con 0x7f03700cc040 2024-09-16T01:50:35.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.294+0000 7f036f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f03700cbc80 con 0x7f03700c75a0 2024-09-16T01:50:35.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03700c6b70 0x7f03700cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03700c6b70 0x7f03700cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47478/0 (socket says 172.21.15.146:47478) 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 -- 172.21.15.146:0/3701715547 learned_addr learned my addr 172.21.15.146:0/3701715547 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036e800640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700cc040 0x7f03701f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036d400640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f03700c75a0 0x7f03701f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 -- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f03700c75a0 msgr2=0x7f03701f88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f03700c75a0 0x7f03701f88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 -- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700cc040 msgr2=0x7f03701f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03700cc040 0x7f03701f8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.295+0000 7f036de00640 1 -- 172.21.15.146:0/3701715547 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037006f0b0 con 0x7f03700c6b70 2024-09-16T01:50:35.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.296+0000 7f036de00640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03700c6b70 0x7f03700cf710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f036000c410 tx=0x7f036000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.297+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360017070 con 0x7f03700c6b70 2024-09-16T01:50:35.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.297+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0360005ce0 con 0x7f03700c6b70 2024-09-16T01:50:35.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.297+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03600072d0 con 0x7f03700c6b70 2024-09-16T01:50:35.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.297+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037006f350 con 0x7f03700c6b70 2024-09-16T01:50:35.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.297+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0370072e00 con 0x7f03700c6b70 2024-09-16T01:50:35.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.298+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f03700c6fe0 con 0x7f03700c6b70 2024-09-16T01:50:35.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.300+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f036001a070 con 0x7f03700c6b70 2024-09-16T01:50:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.301+0000 7f0366000640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f035006c330 0x7f035006e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.301+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f036008a700 con 0x7f03700c6b70 2024-09-16T01:50:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.301+0000 7f0366000640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0350074610 0x7f0350076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.301+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f036001ae50 con 0x7f0350074610 2024-09-16T01:50:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.301+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0360091050 con 0x7f03700c6b70 2024-09-16T01:50:35.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.302+0000 7f036e800640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0350074610 0x7f0350076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.302+0000 7f036d400640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f035006c330 0x7f035006e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.303+0000 7f036e800640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0350074610 0x7f0350076a50 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.303+0000 7f036d400640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f035006c330 0x7f035006e7f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f035c034000 tx=0x7f035c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.303+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f036001ae50 con 0x7f0350074610 2024-09-16T01:50:35.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:35.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.325+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f037006cd20 con 0x7f0350074610 2024-09-16T01:50:35.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.325+0000 7f0366000640 1 -- 172.21.15.146:0/3701715547 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f037006cd20 con 0x7f0350074610 2024-09-16T01:50:35.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0350074610 msgr2=0x7f0350076a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036f200640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0350074610 0x7f0350076a50 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f035006c330 msgr2=0x7f035006e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036f200640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f035006c330 0x7f035006e7f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f035c034000 tx=0x7f035c02e000 comp rx=0 tx=0).stop 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03700c6b70 msgr2=0x7f03700cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036f200640 1 --2- 172.21.15.146:0/3701715547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03700c6b70 0x7f03700cf710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f036000c410 tx=0x7f036000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.326+0000 7f036e800640 1 -- 172.21.15.146:0/3701715547 reap_dead start 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.327+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 shutdown_connections 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.327+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 >> 172.21.15.146:0/3701715547 conn(0x7f03700ccc20 msgr2=0x7f03700c53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.327+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 shutdown_connections 2024-09-16T01:50:35.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.327+0000 7f036f200640 1 -- 172.21.15.146:0/3701715547 wait complete. 2024-09-16T01:50:35.351 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-16T01:50:35.362 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:35.361+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:35.464 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:35.462+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:35.553 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:35.553 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:35.553 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:35.553 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:35.553 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:35.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.561+0000 7f88d5400640 1 Processor -- start 2024-09-16T01:50:35.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.562+0000 7f88d5400640 1 -- start start 2024-09-16T01:50:35.565 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:35.564+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:35.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.563+0000 7f88d5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d015b340 0x7f88d0166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.563+0000 7f88d5400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d006d140 con 0x7f88d00c75a0 2024-09-16T01:50:35.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.563+0000 7f88d5400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88d006d420 con 0x7f88d00c6b70 2024-09-16T01:50:35.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.564+0000 7f88d5400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f88d00cab20 con 0x7f88d015b340 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.564+0000 7f88cfe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f88d00c6b70 0x7f88d00cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:59986/0 (socket says 172.21.15.146:59986) 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.564+0000 7f88cfe00640 1 -- 172.21.15.146:0/1834127031 learned_addr learned my addr 172.21.15.146:0/1834127031 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.564+0000 7f88d4a00640 1 --2- 172.21.15.146:0/1834127031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d015b340 0x7f88d0166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.564+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785419840 0 0) 0x7f88d006d140 con 0x7f88d00c75a0 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.564+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88b4003520 con 0x7f88d00c75a0 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.565+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931697605 0 0) 0x7f88d006d420 con 0x7f88d00c6b70 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.565+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d006d140 con 0x7f88d00c6b70 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.565+0000 7f88d4a00640 1 -- 172.21.15.146:0/1834127031 >> v1:172.21.15.161:6789/0 conn(0x7f88d00c6b70 legacy=0x7f88d00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.565+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2036837389 0 0) 0x7f88b4003520 con 0x7f88d00c75a0 2024-09-16T01:50:35.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.565+0000 7f88d4a00640 1 -- 172.21.15.146:0/1834127031 >> v1:172.21.15.146:6789/0 conn(0x7f88d00c75a0 legacy=0x7f88d00cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.565+0000 7f88d4a00640 1 -- 172.21.15.146:0/1834127031 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d006d700 con 0x7f88d015b340 2024-09-16T01:50:35.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.566+0000 7f88d4a00640 1 --2- 172.21.15.146:0/1834127031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d015b340 0x7f88d0166170 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009dc0 tx=0x7f88c802a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f22410e0732a1e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.566+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c802ae60 con 0x7f88d015b340 2024-09-16T01:50:35.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.567+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c8007740 con 0x7f88d015b340 2024-09-16T01:50:35.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.567+0000 7f88cea00640 1 -- 172.21.15.146:0/1834127031 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8007b10 con 0x7f88d015b340 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.567+0000 7f88d5400640 1 -- 172.21.15.146:0/1834127031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d015b340 msgr2=0x7f88d0166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.567+0000 7f88d5400640 1 --2- 172.21.15.146:0/1834127031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d015b340 0x7f88d0166170 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009dc0 tx=0x7f88c802a640 comp rx=0 tx=0).stop 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.568+0000 7f88d5400640 1 -- 172.21.15.146:0/1834127031 shutdown_connections 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.568+0000 7f88d5400640 1 --2- 172.21.15.146:0/1834127031 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d015b340 0x7f88d0166170 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.568+0000 7f88d5400640 1 -- 172.21.15.146:0/1834127031 >> 172.21.15.146:0/1834127031 conn(0x7f88d00ccc20 msgr2=0x7f88d00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.568+0000 7f88d5400640 1 -- 172.21.15.146:0/1834127031 shutdown_connections 2024-09-16T01:50:35.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.568+0000 7f88d5400640 1 -- 172.21.15.146:0/1834127031 wait complete. 2024-09-16T01:50:35.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:35.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.570+0000 7f88d5400640 1 Processor -- start 2024-09-16T01:50:35.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.571+0000 7f88d5400640 1 -- start start 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.572+0000 7f88d5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d00c6b70 0x7f88d0150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.572+0000 7f88d5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d00c75a0 0x7f88d015e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.572+0000 7f88d5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88d015b340 0x7f88d01f0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.572+0000 7f88d5400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f88d015b7c0 con 0x7f88d015b340 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.572+0000 7f88d5400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f88d00cb050 con 0x7f88d00c6b70 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.572+0000 7f88d5400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f88d00cbc80 con 0x7f88d00c75a0 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d00c6b70 0x7f88d0150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d00c6b70 0x7f88d0150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34520/0 (socket says 172.21.15.146:34520) 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cfe00640 1 -- 172.21.15.146:0/4163023174 learned_addr learned my addr 172.21.15.146:0/4163023174 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cf400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d00c75a0 0x7f88d015e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88d4a00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88d015b340 0x7f88d01f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cfe00640 1 -- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d00c75a0 msgr2=0x7f88d015e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cfe00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d00c75a0 0x7f88d015e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.573+0000 7f88cfe00640 1 -- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88d015b340 msgr2=0x7f88d01f0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88cfe00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88d015b340 0x7f88d01f0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88cfe00640 1 -- 172.21.15.146:0/4163023174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d00ced50 con 0x7f88d00c6b70 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88cf400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f88d00c75a0 0x7f88d015e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88d4a00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88d015b340 0x7f88d01f0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88cfe00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d00c6b70 0x7f88d0150a90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88c000c3c0 tx=0x7f88c000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0017070 con 0x7f88d00c6b70 2024-09-16T01:50:35.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88c0005ce0 con 0x7f88d00c6b70 2024-09-16T01:50:35.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c0007310 con 0x7f88d00c6b70 2024-09-16T01:50:35.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.574+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d01f3ea0 con 0x7f88d00c6b70 2024-09-16T01:50:35.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.575+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d01f4460 con 0x7f88d00c6b70 2024-09-16T01:50:35.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.577+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88d01510d0 con 0x7f88d00c6b70 2024-09-16T01:50:35.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.577+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f88c001a070 con 0x7f88d00c6b70 2024-09-16T01:50:35.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.578+0000 7f88c7e00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f88b006c2b0 0x7f88b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.578+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f88c008a720 con 0x7f88d00c6b70 2024-09-16T01:50:35.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.579+0000 7f88c7e00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f88b0074570 0x7f88b00769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.579+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88b0077080 con 0x7f88b0074570 2024-09-16T01:50:35.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.579+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f88c0091050 con 0x7f88d00c6b70 2024-09-16T01:50:35.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.579+0000 7f88cf400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f88b006c2b0 0x7f88b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.579+0000 7f88d4a00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f88b0074570 0x7f88b00769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.580+0000 7f88cf400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f88b006c2b0 0x7f88b006e770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f88d00cc750 tx=0x7f88bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.582+0000 7f88d4a00640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f88b0074570 0x7f88b00769b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.583+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88b0077080 con 0x7f88b0074570 2024-09-16T01:50:35.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.616+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f88d006cd20 con 0x7f88b0074570 2024-09-16T01:50:35.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.617+0000 7f88c7e00640 1 -- 172.21.15.146:0/4163023174 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f88d006cd20 con 0x7f88b0074570 2024-09-16T01:50:35.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.617+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f88b0074570 msgr2=0x7f88b00769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.617+0000 7f88d5400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f88b0074570 0x7f88b00769b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.617+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f88b006c2b0 msgr2=0x7f88b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.617+0000 7f88d5400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f88b006c2b0 0x7f88b006e770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f88d00cc750 tx=0x7f88bc009040 comp rx=0 tx=0).stop 2024-09-16T01:50:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d00c6b70 msgr2=0x7f88d0150a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d5400640 1 --2- 172.21.15.146:0/4163023174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88d00c6b70 0x7f88d0150a90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f88c000c3c0 tx=0x7f88c000c880 comp rx=0 tx=0).stop 2024-09-16T01:50:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d4a00640 1 -- 172.21.15.146:0/4163023174 reap_dead start 2024-09-16T01:50:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 shutdown_connections 2024-09-16T01:50:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 >> 172.21.15.146:0/4163023174 conn(0x7f88d00ccc20 msgr2=0x7f88d00cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 shutdown_connections 2024-09-16T01:50:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.618+0000 7f88d5400640 1 -- 172.21.15.146:0/4163023174 wait complete. 2024-09-16T01:50:35.645 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-16T01:50:35.665 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:35.665+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:35.757 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:35.757 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:35.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:35.757 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:35.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.986211744, 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.758 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:35.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:35.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:35.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.554658012, 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:35.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:35.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.53473379, 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:35.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.778 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.534713872, 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:35.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.534653288, 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.785 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:35.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:35.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.791 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.288367768, 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:35.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:35.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:35.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.288217254, 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:35.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:35.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.288197686, 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:35.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:35.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.810 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.718041472, 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:35.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:35.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:35.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:35.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717873654, 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.817 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:35.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.822 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717854672, 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.823 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:35.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:35.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:35.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717835041, 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:35.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717701135, 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:35.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.841 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717689597, 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.842 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:35.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:35.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717671186, 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:35.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.717556516, 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:35.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:35.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.706774788, 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:35.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.70665812999999, 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:35.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:35.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.891124891000004, 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.872 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:35.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:35.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 86.891063552999995, 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:35.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:35.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:35.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:35.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:35.882 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:35.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:35.884 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:35.882+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:35.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.882+0000 7f9611c00640 1 Processor -- start 2024-09-16T01:50:35.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.883+0000 7f9611c00640 1 -- start start 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f9611c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f960c0c6b70 0x7f960c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f9611c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 0x7f960c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f9611c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c0cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f9611c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f960c15b880 con 0x7f960c0c75a0 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f9611c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f960c0cab20 con 0x7f960c0c6b70 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f9611c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f960c06d140 con 0x7f960c0cbab0 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f960b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 0x7f960c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f960be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f960c0c6b70 0x7f960c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f960b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 0x7f960c0cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47488/0 (socket says 172.21.15.146:47488) 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f960be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f960c0c6b70 0x7f960c15b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34524/0 (socket says 172.21.15.146:34524) 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.884+0000 7f960b400640 1 -- 172.21.15.146:0/3871988305 learned_addr learned my addr 172.21.15.146:0/3871988305 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f9611200640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c0cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f960be00640 1 -- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 msgr2=0x7f960c0cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f960be00640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c0cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f960be00640 1 -- 172.21.15.146:0/3871988305 >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 msgr2=0x7f960c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f960be00640 1 --2- 172.21.15.146:0/3871988305 >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 0x7f960c0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f960be00640 1 -- 172.21.15.146:0/3871988305 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c06d420 con 0x7f960c0c6b70 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f9611200640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c0cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.885+0000 7f960b400640 1 --2- 172.21.15.146:0/3871988305 >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 0x7f960c0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.886+0000 7f960be00640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c6b70 0x7f960c15b340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f960c075ec0 tx=0x7f95fc029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4c4588f1a168321 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.886+0000 7f960aa00640 1 -- 172.21.15.146:0/3871988305 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc037070 con 0x7f960c0c6b70 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.886+0000 7f960aa00640 1 -- 172.21.15.146:0/3871988305 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95fc02ad30 con 0x7f960c0c6b70 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.886+0000 7f960aa00640 1 -- 172.21.15.146:0/3871988305 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc032660 con 0x7f960c0c6b70 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 -- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c6b70 msgr2=0x7f960c15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c6b70 0x7f960c15b340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f960c075ec0 tx=0x7f95fc029fe0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 -- 172.21.15.146:0/3871988305 shutdown_connections 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c0cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 --2- 172.21.15.146:0/3871988305 >> v2:172.21.15.146:3300/0 conn(0x7f960c0c75a0 0x7f960c0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 --2- 172.21.15.146:0/3871988305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c6b70 0x7f960c15b340 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 -- 172.21.15.146:0/3871988305 >> 172.21.15.146:0/3871988305 conn(0x7f960c0ccc20 msgr2=0x7f960c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 -- 172.21.15.146:0/3871988305 shutdown_connections 2024-09-16T01:50:35.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.887+0000 7f9611c00640 1 -- 172.21.15.146:0/3871988305 wait complete. 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.889+0000 7f9611c00640 1 Processor -- start 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.889+0000 7f9611c00640 1 -- start start 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f9611c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f960c0c6b70 0x7f960c1f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f9611c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c75a0 0x7f960c15da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f9611c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f9611c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f960c0cb050 con 0x7f960c0c6b70 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f9611c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f960c0cad70 con 0x7f960c0c75a0 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f9611c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f960c06cd20 con 0x7f960c0cbab0 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f960be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f960c0c6b70 0x7f960c1f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f960be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f960c0c6b70 0x7f960c1f1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47490/0 (socket says 172.21.15.146:47490) 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f960be00640 1 -- 172.21.15.146:0/893371961 learned_addr learned my addr 172.21.15.146:0/893371961 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.890+0000 7f960b400640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c75a0 0x7f960c15da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f9611200640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960be00640 1 -- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 msgr2=0x7f960c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960be00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f960c0cbab0 0x7f960c15bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960be00640 1 -- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c75a0 msgr2=0x7f960c15da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960be00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c75a0 0x7f960c15da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960be00640 1 -- 172.21.15.146:0/893371961 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c0ac5c0 con 0x7f960c0c6b70 2024-09-16T01:50:35.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960b400640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f960c0c75a0 0x7f960c15da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:35.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.891+0000 7f960be00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f960c0c6b70 0x7f960c1f1590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009890 tx=0x7f95fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.892+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc041020 con 0x7f960c0c6b70 2024-09-16T01:50:35.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.892+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95fc03d620 con 0x7f960c0c6b70 2024-09-16T01:50:35.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.892+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc037040 con 0x7f960c0c6b70 2024-09-16T01:50:35.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.893+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960c0c2b50 con 0x7f960c0c6b70 2024-09-16T01:50:35.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.893+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960c1f0ab0 con 0x7f960c0c6b70 2024-09-16T01:50:35.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.894+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f960c15eb30 con 0x7f960c0c6b70 2024-09-16T01:50:35.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.895+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f95fc044020 con 0x7f960c0c6b70 2024-09-16T01:50:35.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.897+0000 7f9603e00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95ec06c2e0 0x7f95ec06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.897+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f95fc0ab830 con 0x7f960c0c6b70 2024-09-16T01:50:35.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.897+0000 7f9603e00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f95ec074540 0x7f95ec076980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:35.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.897+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95fc008eb0 con 0x7f95ec074540 2024-09-16T01:50:35.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.897+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f95fc0b2050 con 0x7f960c0c6b70 2024-09-16T01:50:35.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.898+0000 7f9611200640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f95ec074540 0x7f95ec076980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.898+0000 7f960b400640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95ec06c2e0 0x7f95ec06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:35.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.899+0000 7f960b400640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95ec06c2e0 0x7f95ec06e7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95f80096f0 tx=0x7f95f8009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.899+0000 7f9611200640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f95ec074540 0x7f95ec076980 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:35.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.900+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f95fc008eb0 con 0x7f95ec074540 2024-09-16T01:50:35.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.931+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f960c0ca280 con 0x7f95ec074540 2024-09-16T01:50:35.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.931+0000 7f9603e00640 1 -- 172.21.15.146:0/893371961 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f960c0ca280 con 0x7f95ec074540 2024-09-16T01:50:35.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f95ec074540 msgr2=0x7f95ec076980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f95ec074540 0x7f95ec076980 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95ec06c2e0 msgr2=0x7f95ec06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f95ec06c2e0 0x7f95ec06e7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95f80096f0 tx=0x7f95f8009240 comp rx=0 tx=0).stop 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f960c0c6b70 msgr2=0x7f960c1f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 --2- 172.21.15.146:0/893371961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f960c0c6b70 0x7f960c1f1590 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009890 tx=0x7f95fc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611200640 1 -- 172.21.15.146:0/893371961 reap_dead start 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 shutdown_connections 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.932+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 >> 172.21.15.146:0/893371961 conn(0x7f960c0ccc20 msgr2=0x7f960c0ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.933+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 shutdown_connections 2024-09-16T01:50:35.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:35.933+0000 7f9611c00640 1 -- 172.21.15.146:0/893371961 wait complete. 2024-09-16T01:50:35.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-16T01:50:35.984 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:35.983+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:36.085 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:36.083+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:36.086 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:36.086 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:36.086 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:36.086 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:36.105 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:36.186 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:36.185+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:36.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.189+0000 7fd42a400640 1 Processor -- start 2024-09-16T01:50:36.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.189+0000 7fd42a400640 1 -- start start 2024-09-16T01:50:36.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.190+0000 7fd42a400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd424070950 0x7fd4240c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.190+0000 7fd42a400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4240c8c00 0x7fd4240c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.190+0000 7fd42a400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4240c33c0 con 0x7fd4241528e0 2024-09-16T01:50:36.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.190+0000 7fd42a400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd4241596d0 con 0x7fd424070950 2024-09-16T01:50:36.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.190+0000 7fd42a400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd4240c36a0 con 0x7fd4240c8c00 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.191+0000 7fd422a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd424070950 0x7fd4240c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.191+0000 7fd423400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd4241528e0 0x7fd424158fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:59992/0 (socket says 172.21.15.146:59992) 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.191+0000 7fd423400640 1 -- 172.21.15.146:0/1305895265 learned_addr learned my addr 172.21.15.146:0/1305895265 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.191+0000 7fd423e00640 1 --2- 172.21.15.146:0/1305895265 >> v2:172.21.15.161:3300/0 conn(0x7fd4240c8c00 0x7fd4240c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.191+0000 7fd422000640 1 -- 172.21.15.146:0/1305895265 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3645205872 0 0) 0x7fd4240c33c0 con 0x7fd4241528e0 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.191+0000 7fd422000640 1 -- 172.21.15.146:0/1305895265 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd408003520 con 0x7fd4241528e0 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.192+0000 7fd423e00640 1 -- 172.21.15.146:0/1305895265 >> v2:172.21.15.146:3300/0 conn(0x7fd424070950 msgr2=0x7fd4240c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.192+0000 7fd423e00640 1 --2- 172.21.15.146:0/1305895265 >> v2:172.21.15.146:3300/0 conn(0x7fd424070950 0x7fd4240c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.192+0000 7fd423e00640 1 -- 172.21.15.146:0/1305895265 >> v1:172.21.15.161:6789/0 conn(0x7fd4241528e0 legacy=0x7fd424158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.192+0000 7fd423e00640 1 -- 172.21.15.146:0/1305895265 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4241587c0 con 0x7fd4240c8c00 2024-09-16T01:50:36.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.192+0000 7fd422a00640 1 --2- 172.21.15.146:0/1305895265 >> v2:172.21.15.146:3300/0 conn(0x7fd424070950 0x7fd4240c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:36.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.193+0000 7fd423e00640 1 --2- 172.21.15.146:0/1305895265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4240c8c00 0x7fd4240c93c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4240cb830 tx=0x7fd41c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=744d8b2d4ca4434 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.193+0000 7fd422000640 1 -- 172.21.15.146:0/1305895265 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c007740 con 0x7fd4240c8c00 2024-09-16T01:50:36.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.193+0000 7fd422000640 1 -- 172.21.15.146:0/1305895265 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41c0078e0 con 0x7fd4240c8c00 2024-09-16T01:50:36.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.193+0000 7fd422000640 1 -- 172.21.15.146:0/1305895265 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c007cb0 con 0x7fd4240c8c00 2024-09-16T01:50:36.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 -- 172.21.15.146:0/1305895265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4240c8c00 msgr2=0x7fd4240c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 --2- 172.21.15.146:0/1305895265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4240c8c00 0x7fd4240c93c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4240cb830 tx=0x7fd41c02a710 comp rx=0 tx=0).stop 2024-09-16T01:50:36.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 -- 172.21.15.146:0/1305895265 shutdown_connections 2024-09-16T01:50:36.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 --2- 172.21.15.146:0/1305895265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4240c8c00 0x7fd4240c93c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 --2- 172.21.15.146:0/1305895265 >> v2:172.21.15.146:3300/0 conn(0x7fd424070950 0x7fd4240c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 -- 172.21.15.146:0/1305895265 >> 172.21.15.146:0/1305895265 conn(0x7fd424067950 msgr2=0x7fd424067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 -- 172.21.15.146:0/1305895265 shutdown_connections 2024-09-16T01:50:36.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.194+0000 7fd42a400640 1 -- 172.21.15.146:0/1305895265 wait complete. 2024-09-16T01:50:36.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.196+0000 7fd42a400640 1 Processor -- start 2024-09-16T01:50:36.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.196+0000 7fd42a400640 1 -- start start 2024-09-16T01:50:36.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.197+0000 7fd42a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd424070950 0x7fd4241f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.197+0000 7fd42a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4240c8c00 0x7fd4241f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.197+0000 7fd42a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4241528e0 0x7fd424146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.197+0000 7fd42a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd42415bca0 con 0x7fd4241528e0 2024-09-16T01:50:36.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.197+0000 7fd42a400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4240c3980 con 0x7fd424070950 2024-09-16T01:50:36.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.197+0000 7fd42a400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd42414ab70 con 0x7fd4240c8c00 2024-09-16T01:50:36.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4241528e0 0x7fd424146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd422a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4240c8c00 0x7fd4241f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4241528e0 0x7fd424146b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47508/0 (socket says 172.21.15.146:47508) 2024-09-16T01:50:36.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423e00640 1 -- 172.21.15.146:0/2473116811 learned_addr learned my addr 172.21.15.146:0/2473116811 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:36.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd424070950 0x7fd4241f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423400640 1 -- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4240c8c00 msgr2=0x7fd4241f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd4240c8c00 0x7fd4241f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423400640 1 -- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4241528e0 msgr2=0x7fd424146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4241528e0 0x7fd424146b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.198+0000 7fd423400640 1 -- 172.21.15.146:0/2473116811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42414b5f0 con 0x7fd424070950 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.199+0000 7fd423400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd424070950 0x7fd4241f2150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd41400c390 tx=0x7fd414001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.199+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414017070 con 0x7fd424070950 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.199+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd414005ce0 con 0x7fd424070950 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.199+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4140072f0 con 0x7fd424070950 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.199+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42414ada0 con 0x7fd424070950 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.199+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42415b730 con 0x7fd424070950 2024-09-16T01:50:36.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.200+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd42414bc00 con 0x7fd424070950 2024-09-16T01:50:36.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.202+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd41401a070 con 0x7fd424070950 2024-09-16T01:50:36.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.202+0000 7fd41b400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd40406c2b0 0x7fd40406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.203+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7fd41408a8c0 con 0x7fd424070950 2024-09-16T01:50:36.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.203+0000 7fd41b400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd404074570 0x7fd4040769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.203+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd414015e70 con 0x7fd404074570 2024-09-16T01:50:36.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.203+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fd41408ac70 con 0x7fd424070950 2024-09-16T01:50:36.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.203+0000 7fd422a00640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd40406c2b0 0x7fd40406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.203+0000 7fd423e00640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd404074570 0x7fd4040769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.204+0000 7fd422a00640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd40406c2b0 0x7fd40406e770 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd424157270 tx=0x7fd410008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.205+0000 7fd423e00640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd404074570 0x7fd4040769b0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.207+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd414015e70 con 0x7fd404074570 2024-09-16T01:50:36.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.226+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fd4240c4fa0 con 0x7fd404074570 2024-09-16T01:50:36.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.231+0000 7fd41b400640 1 -- 172.21.15.146:0/2473116811 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fd4240c4fa0 con 0x7fd404074570 2024-09-16T01:50:36.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.231+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd404074570 msgr2=0x7fd4040769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.231+0000 7fd42a400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fd404074570 0x7fd4040769b0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd40406c2b0 msgr2=0x7fd40406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd40406c2b0 0x7fd40406e770 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd424157270 tx=0x7fd410008040 comp rx=0 tx=0).stop 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd424070950 msgr2=0x7fd4241f2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 --2- 172.21.15.146:0/2473116811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd424070950 0x7fd4241f2150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd41400c390 tx=0x7fd414001e70 comp rx=0 tx=0).stop 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd423e00640 1 -- 172.21.15.146:0/2473116811 reap_dead start 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 shutdown_connections 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 >> 172.21.15.146:0/2473116811 conn(0x7fd424067950 msgr2=0x7fd4240b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 shutdown_connections 2024-09-16T01:50:36.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.232+0000 7fd42a400640 1 -- 172.21.15.146:0/2473116811 wait complete. 2024-09-16T01:50:36.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-16T01:50:36.300 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:36.298+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:36.343 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:36.344 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:36.344 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:36.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:36.387 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:36.386+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:36.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.467+0000 7f8e93200640 1 Processor -- start 2024-09-16T01:50:36.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.467+0000 7f8e93200640 1 -- start start 2024-09-16T01:50:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e93200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8e940c5eb0 0x7f8e940cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e93200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e9415aa70 0x7f8e94154910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e93200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e9415af00 con 0x7f8e940c6290 2024-09-16T01:50:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e93200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8e9406dbd0 con 0x7f8e940c5eb0 2024-09-16T01:50:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e93200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8e9415b1e0 con 0x7f8e9415aa70 2024-09-16T01:50:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e92800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8e940c6290 0x7f8e9406c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:59994/0 (socket says 172.21.15.146:59994) 2024-09-16T01:50:36.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e92800640 1 -- 172.21.15.146:0/2717473297 learned_addr learned my addr 172.21.15.146:0/2717473297 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.470+0000 7f8e91e00640 1 --2- 172.21.15.146:0/2717473297 >> v2:172.21.15.146:3300/0 conn(0x7f8e940c5eb0 0x7f8e940cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e91400640 1 --2- 172.21.15.146:0/2717473297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e9415aa70 0x7f8e94154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e90a00640 1 -- 172.21.15.146:0/2717473297 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1985421249 0 0) 0x7f8e9415af00 con 0x7f8e940c6290 2024-09-16T01:50:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e90a00640 1 -- 172.21.15.146:0/2717473297 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e78003520 con 0x7f8e940c6290 2024-09-16T01:50:36.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e91400640 1 -- 172.21.15.146:0/2717473297 >> v2:172.21.15.146:3300/0 conn(0x7f8e940c5eb0 msgr2=0x7f8e940cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e91400640 1 --2- 172.21.15.146:0/2717473297 >> v2:172.21.15.146:3300/0 conn(0x7f8e940c5eb0 0x7f8e940cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e91400640 1 -- 172.21.15.146:0/2717473297 >> v1:172.21.15.161:6789/0 conn(0x7f8e940c6290 legacy=0x7f8e9406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.471+0000 7f8e91400640 1 -- 172.21.15.146:0/2717473297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9415b4c0 con 0x7f8e9415aa70 2024-09-16T01:50:36.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.472+0000 7f8e91e00640 1 --2- 172.21.15.146:0/2717473297 >> v2:172.21.15.146:3300/0 conn(0x7f8e940c5eb0 0x7f8e940cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:36.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.473+0000 7f8e91400640 1 --2- 172.21.15.146:0/2717473297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e9415aa70 0x7f8e94154910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8e800097a0 tx=0x7f8e8002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f98778d874038a2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.473+0000 7f8e90a00640 1 -- 172.21.15.146:0/2717473297 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80037070 con 0x7f8e9415aa70 2024-09-16T01:50:36.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.473+0000 7f8e90a00640 1 -- 172.21.15.146:0/2717473297 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e80032440 con 0x7f8e9415aa70 2024-09-16T01:50:36.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.473+0000 7f8e90a00640 1 -- 172.21.15.146:0/2717473297 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80032810 con 0x7f8e9415aa70 2024-09-16T01:50:36.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.474+0000 7f8e93200640 1 -- 172.21.15.146:0/2717473297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e9415aa70 msgr2=0x7f8e94154910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.474+0000 7f8e93200640 1 --2- 172.21.15.146:0/2717473297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e9415aa70 0x7f8e94154910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8e800097a0 tx=0x7f8e8002a230 comp rx=0 tx=0).stop 2024-09-16T01:50:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.475+0000 7f8e93200640 1 -- 172.21.15.146:0/2717473297 shutdown_connections 2024-09-16T01:50:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.475+0000 7f8e93200640 1 --2- 172.21.15.146:0/2717473297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e9415aa70 0x7f8e94154910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.475+0000 7f8e93200640 1 --2- 172.21.15.146:0/2717473297 >> v2:172.21.15.146:3300/0 conn(0x7f8e940c5eb0 0x7f8e940cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.475+0000 7f8e93200640 1 -- 172.21.15.146:0/2717473297 >> 172.21.15.146:0/2717473297 conn(0x7f8e94067b30 msgr2=0x7f8e94067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.475+0000 7f8e93200640 1 -- 172.21.15.146:0/2717473297 shutdown_connections 2024-09-16T01:50:36.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.475+0000 7f8e93200640 1 -- 172.21.15.146:0/2717473297 wait complete. 2024-09-16T01:50:36.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.478+0000 7f8e93200640 1 Processor -- start 2024-09-16T01:50:36.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.478+0000 7f8e93200640 1 -- start start 2024-09-16T01:50:36.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.479+0000 7f8e93200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e940c5eb0 0x7f8e94153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.479+0000 7f8e93200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e940c6290 0x7f8e941e2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.479+0000 7f8e93200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e941e34d0 0x7f8e941f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.479+0000 7f8e93200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9406cfa0 con 0x7f8e940c6290 2024-09-16T01:50:36.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.479+0000 7f8e93200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9406c370 con 0x7f8e940c5eb0 2024-09-16T01:50:36.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.479+0000 7f8e93200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8e9406d120 con 0x7f8e941e34d0 2024-09-16T01:50:36.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e940c5eb0 0x7f8e94153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e940c5eb0 0x7f8e94153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34568/0 (socket says 172.21.15.146:34568) 2024-09-16T01:50:36.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 -- 172.21.15.146:0/1818590012 learned_addr learned my addr 172.21.15.146:0/1818590012 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e92800640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e941e34d0 0x7f8e941f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91400640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e940c6290 0x7f8e941e2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 -- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e941e34d0 msgr2=0x7f8e941f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8e941e34d0 0x7f8e941f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 -- 172.21.15.146:0/1818590012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e940c6290 msgr2=0x7f8e941e2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e940c6290 0x7f8e941e2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.480+0000 7f8e91e00640 1 -- 172.21.15.146:0/1818590012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e941f8150 con 0x7f8e940c5eb0 2024-09-16T01:50:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.481+0000 7f8e91e00640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e940c5eb0 0x7f8e94153a90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400e960 tx=0x7f8e8400ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.481+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e84017070 con 0x7f8e940c5eb0 2024-09-16T01:50:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.481+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e8400ca00 con 0x7f8e940c5eb0 2024-09-16T01:50:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.481+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e941f8430 con 0x7f8e940c5eb0 2024-09-16T01:50:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.481+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e941f89f0 con 0x7f8e940c5eb0 2024-09-16T01:50:36.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.481+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400cdb0 con 0x7f8e940c5eb0 2024-09-16T01:50:36.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.483+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e6c001030 con 0x7f8e940c5eb0 2024-09-16T01:50:36.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.485+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8e8401a070 con 0x7f8e940c5eb0 2024-09-16T01:50:36.488 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:36.487+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:36.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.486+0000 7f8e8a000640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8e7406c2b0 0x7f8e7406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.486+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f8e8408b780 con 0x7f8e940c5eb0 2024-09-16T01:50:36.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.487+0000 7f8e8a000640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8e74074570 0x7f8e740769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.487+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e74077080 con 0x7f8e74074570 2024-09-16T01:50:36.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.487+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f8e8408bb30 con 0x7f8e940c5eb0 2024-09-16T01:50:36.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.487+0000 7f8e91400640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8e7406c2b0 0x7f8e7406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.487+0000 7f8e92800640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8e74074570 0x7f8e740769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.488+0000 7f8e92800640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8e74074570 0x7f8e740769b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.488+0000 7f8e91400640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8e7406c2b0 0x7f8e7406e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e80034000 tx=0x7f8e8002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.489+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e74077080 con 0x7f8e74074570 2024-09-16T01:50:36.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.523+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f8e6c002dd0 con 0x7f8e74074570 2024-09-16T01:50:36.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.523+0000 7f8e8a000640 1 -- 172.21.15.146:0/1818590012 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f8e6c002dd0 con 0x7f8e74074570 2024-09-16T01:50:36.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8e74074570 msgr2=0x7f8e740769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8e74074570 0x7f8e740769b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8e7406c2b0 msgr2=0x7f8e7406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8e7406c2b0 0x7f8e7406e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e80034000 tx=0x7f8e8002e000 comp rx=0 tx=0).stop 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e940c5eb0 msgr2=0x7f8e94153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 --2- 172.21.15.146:0/1818590012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e940c5eb0 0x7f8e94153a90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400e960 tx=0x7f8e8400ee20 comp rx=0 tx=0).stop 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e92800640 1 -- 172.21.15.146:0/1818590012 reap_dead start 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 shutdown_connections 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.524+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 >> 172.21.15.146:0/1818590012 conn(0x7f8e94067b30 msgr2=0x7f8e9406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.525+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 shutdown_connections 2024-09-16T01:50:36.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.525+0000 7f8e93200640 1 -- 172.21.15.146:0/1818590012 wait complete. 2024-09-16T01:50:36.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-16T01:50:36.589 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:36.588+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.774752617000004, 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:36.602 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:36.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.774637795999993, 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.608 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:36.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.774595926000003, 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:36.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:36.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:36.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.774551646999996, 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.620 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:36.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.625 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.774522770000004, 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:36.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.770815562999999, 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:36.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:36.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.770761582000006, 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:36.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:36.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:36.640 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.735346358000001, 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:36.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:36.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.732286208999994, 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:36.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:36.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.731426760999994, 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:36.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:36.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:36.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:36.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.731372867999994, 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:36.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:36.661 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.725348894999996, 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:36.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.725298766999998, 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:36.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.725222434000003, 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.671 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.675 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.725191078999998, 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:36.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 75.687105623999997, 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.680 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:36.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 75.686314092999993, 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:36.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:36.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.662232267999997, 2024-09-16T01:50:36.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:36.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:36.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 70.662160356000001, 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.695 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:36.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:36.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.331517955999999, 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:36.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:36.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:36.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:36.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:36.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:36.707 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:36.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:50:36.709 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:36.710 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:36.707+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:36.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.715+0000 7fc19b200640 1 Processor -- start 2024-09-16T01:50:36.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.715+0000 7fc19b200640 1 -- start start 2024-09-16T01:50:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc19b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 0x7fc1940c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc19b200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc1940d0fd0 0x7fc1940ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc19b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc194155430 con 0x7fc19415edc0 2024-09-16T01:50:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc19b200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc19415aea0 con 0x7fc1940d0fd0 2024-09-16T01:50:36.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc19b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc19415b180 con 0x7fc1940d0bf0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc193e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 0x7fc1940c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc193e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 0x7fc1940c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50708/0 (socket says 172.21.15.146:50708) 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc193e00640 1 -- 172.21.15.146:0/2780722116 learned_addr learned my addr 172.21.15.146:0/2780722116 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.717+0000 7fc18be00640 1 --2- 172.21.15.146:0/2780722116 >> v2:172.21.15.161:3300/0 conn(0x7fc1940d0fd0 0x7fc1940ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.718+0000 7fc193e00640 1 -- 172.21.15.146:0/2780722116 >> v2:172.21.15.161:3300/0 conn(0x7fc1940d0fd0 msgr2=0x7fc1940ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.718+0000 7fc193e00640 1 --2- 172.21.15.146:0/2780722116 >> v2:172.21.15.161:3300/0 conn(0x7fc1940d0fd0 0x7fc1940ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.718+0000 7fc193e00640 1 -- 172.21.15.146:0/2780722116 >> v1:172.21.15.146:6789/0 conn(0x7fc19415edc0 legacy=0x7fc1940b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.718+0000 7fc193e00640 1 -- 172.21.15.146:0/2780722116 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19415b460 con 0x7fc1940d0bf0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.719+0000 7fc193e00640 1 --2- 172.21.15.146:0/2780722116 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 0x7fc1940c9fe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc184009870 tx=0x7fc18402a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f0fed0e210aa99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.719+0000 7fc193400640 1 -- 172.21.15.146:0/2780722116 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184037070 con 0x7fc1940d0bf0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.719+0000 7fc193400640 1 -- 172.21.15.146:0/2780722116 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18402ae50 con 0x7fc1940d0bf0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.719+0000 7fc193400640 1 -- 172.21.15.146:0/2780722116 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184032860 con 0x7fc1940d0bf0 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 -- 172.21.15.146:0/2780722116 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 msgr2=0x7fc1940c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 --2- 172.21.15.146:0/2780722116 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 0x7fc1940c9fe0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc184009870 tx=0x7fc18402a300 comp rx=0 tx=0).stop 2024-09-16T01:50:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 -- 172.21.15.146:0/2780722116 shutdown_connections 2024-09-16T01:50:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 --2- 172.21.15.146:0/2780722116 >> v2:172.21.15.161:3300/0 conn(0x7fc1940d0fd0 0x7fc1940ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 --2- 172.21.15.146:0/2780722116 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0bf0 0x7fc1940c9fe0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 -- 172.21.15.146:0/2780722116 >> 172.21.15.146:0/2780722116 conn(0x7fc19406c900 msgr2=0x7fc19406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.720+0000 7fc19b200640 1 -- 172.21.15.146:0/2780722116 shutdown_connections 2024-09-16T01:50:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.721+0000 7fc19b200640 1 -- 172.21.15.146:0/2780722116 wait complete. 2024-09-16T01:50:36.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.723+0000 7fc19b200640 1 Processor -- start 2024-09-16T01:50:36.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.724+0000 7fc19b200640 1 -- start start 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.725+0000 7fc19b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1940d0bf0 0x7fc1941f7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.725+0000 7fc19b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0fd0 0x7fc1941577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.725+0000 7fc19b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc19415edc0 0x7fc1941f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.725+0000 7fc19b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940b3790 con 0x7fc19415edc0 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.725+0000 7fc19b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415ba20 con 0x7fc1940d0bf0 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.725+0000 7fc19b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc1940b3910 con 0x7fc1940d0fd0 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1940d0bf0 0x7fc1941f7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1940d0bf0 0x7fc1941f7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34582/0 (socket says 172.21.15.146:34582) 2024-09-16T01:50:36.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 -- 172.21.15.146:0/4062613750 learned_addr learned my addr 172.21.15.146:0/4062613750 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:36.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 -- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0fd0 msgr2=0x7fc1941577c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:50:36.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc1940d0fd0 0x7fc1941577c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 -- 172.21.15.146:0/4062613750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc19415edc0 msgr2=0x7fc1941f5c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc19415edc0 0x7fc1941f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.726+0000 7fc193e00640 1 -- 172.21.15.146:0/4062613750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc194158620 con 0x7fc1940d0bf0 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.727+0000 7fc193e00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1940d0bf0 0x7fc1941f7f70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc184000c00 tx=0x7fc1840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.727+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184041070 con 0x7fc1940d0bf0 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.727+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18403c660 con 0x7fc1940d0bf0 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.727+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184037040 con 0x7fc1940d0bf0 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.727+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc194158340 con 0x7fc1940d0bf0 2024-09-16T01:50:36.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.728+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1941f8b10 con 0x7fc1940d0bf0 2024-09-16T01:50:36.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.730+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc1940b5d90 con 0x7fc1940d0bf0 2024-09-16T01:50:36.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.731+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc184044070 con 0x7fc1940d0bf0 2024-09-16T01:50:36.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.732+0000 7fc190c00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc17006c2b0 0x7fc17006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.732+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7fc184077b50 con 0x7fc1940d0bf0 2024-09-16T01:50:36.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.732+0000 7fc190c00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc170075150 0x7fc1700775f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.732+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc170077cc0 con 0x7fc170075150 2024-09-16T01:50:36.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.732+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc1840764e0 con 0x7fc1940d0bf0 2024-09-16T01:50:36.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.732+0000 7fc18be00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc17006c2b0 0x7fc17006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.733+0000 7fc199200640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc170075150 0x7fc1700775f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.734+0000 7fc18be00640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc17006c2b0 0x7fc17006e770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc19406d160 tx=0x7fc17c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.734+0000 7fc199200640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc170075150 0x7fc1700775f0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.736+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc170077cc0 con 0x7fc170075150 2024-09-16T01:50:36.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.760+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fc1941f84b0 con 0x7fc170075150 2024-09-16T01:50:36.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.760+0000 7fc190c00640 1 -- 172.21.15.146:0/4062613750 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fc1941f84b0 con 0x7fc170075150 2024-09-16T01:50:36.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.761+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc170075150 msgr2=0x7fc1700775f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.761+0000 7fc19b200640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc170075150 0x7fc1700775f0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc17006c2b0 msgr2=0x7fc17006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc17006c2b0 0x7fc17006e770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc19406d160 tx=0x7fc17c009190 comp rx=0 tx=0).stop 2024-09-16T01:50:36.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1940d0bf0 msgr2=0x7fc1941f7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 --2- 172.21.15.146:0/4062613750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1940d0bf0 0x7fc1941f7f70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc184000c00 tx=0x7fc1840046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc199200640 1 -- 172.21.15.146:0/4062613750 reap_dead start 2024-09-16T01:50:36.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 shutdown_connections 2024-09-16T01:50:36.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 >> 172.21.15.146:0/4062613750 conn(0x7fc19406c900 msgr2=0x7fc1940ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 shutdown_connections 2024-09-16T01:50:36.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.762+0000 7fc19b200640 1 -- 172.21.15.146:0/4062613750 wait complete. 2024-09-16T01:50:36.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-16T01:50:36.810 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:36.809+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:36.911 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:36.910+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:36.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.925+0000 7f750d600640 1 Processor -- start 2024-09-16T01:50:36.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.926+0000 7f750d600640 1 -- start start 2024-09-16T01:50:36.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f750d600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75080c75a0 0x7f750815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f750d600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75080cc040 0x7f75080cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f750d600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f750806d140 con 0x7f75080c6b70 2024-09-16T01:50:36.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f750d600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f750815b880 con 0x7f75080c75a0 2024-09-16T01:50:36.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f750d600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f75080cab20 con 0x7f75080cc040 2024-09-16T01:50:36.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f7507e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f75080c6b70 0x7f75080cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:59998/0 (socket says 172.21.15.146:59998) 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.927+0000 7f7507e00640 1 -- 172.21.15.146:0/3672726823 learned_addr learned my addr 172.21.15.146:0/3672726823 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.928+0000 7f750cc00640 1 --2- 172.21.15.146:0/3672726823 >> v2:172.21.15.161:3300/0 conn(0x7f75080cc040 0x7f75080cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.928+0000 7f7507400640 1 --2- 172.21.15.146:0/3672726823 >> v2:172.21.15.146:3300/0 conn(0x7f75080c75a0 0x7f750815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.928+0000 7f7506a00640 1 -- 172.21.15.146:0/3672726823 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169822650 0 0) 0x7f750806d140 con 0x7f75080c6b70 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.928+0000 7f7506a00640 1 -- 172.21.15.146:0/3672726823 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74ec003520 con 0x7f75080c6b70 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.929+0000 7f7507400640 1 -- 172.21.15.146:0/3672726823 >> v2:172.21.15.161:3300/0 conn(0x7f75080cc040 msgr2=0x7f75080cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.929+0000 7f7507400640 1 --2- 172.21.15.146:0/3672726823 >> v2:172.21.15.161:3300/0 conn(0x7f75080cc040 0x7f75080cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.929+0000 7f7507400640 1 -- 172.21.15.146:0/3672726823 >> v1:172.21.15.161:6789/0 conn(0x7f75080c6b70 legacy=0x7f75080cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.929+0000 7f7507400640 1 -- 172.21.15.146:0/3672726823 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750806d420 con 0x7f75080c75a0 2024-09-16T01:50:36.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.930+0000 7f7507400640 1 --2- 172.21.15.146:0/3672726823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080c75a0 0x7f750815b340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f750806ee20 tx=0x7f74f402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad18b1502be1efde server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.930+0000 7f7506a00640 1 -- 172.21.15.146:0/3672726823 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4037070 con 0x7f75080c75a0 2024-09-16T01:50:36.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.930+0000 7f7506a00640 1 -- 172.21.15.146:0/3672726823 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f4032440 con 0x7f75080c75a0 2024-09-16T01:50:36.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.930+0000 7f7506a00640 1 -- 172.21.15.146:0/3672726823 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4032850 con 0x7f75080c75a0 2024-09-16T01:50:36.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.931+0000 7f750d600640 1 -- 172.21.15.146:0/3672726823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080c75a0 msgr2=0x7f750815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.931+0000 7f750d600640 1 --2- 172.21.15.146:0/3672726823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080c75a0 0x7f750815b340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f750806ee20 tx=0x7f74f402a130 comp rx=0 tx=0).stop 2024-09-16T01:50:36.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.931+0000 7f750d600640 1 -- 172.21.15.146:0/3672726823 shutdown_connections 2024-09-16T01:50:36.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.931+0000 7f750d600640 1 --2- 172.21.15.146:0/3672726823 >> v2:172.21.15.161:3300/0 conn(0x7f75080cc040 0x7f75080cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.931+0000 7f750d600640 1 --2- 172.21.15.146:0/3672726823 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080c75a0 0x7f750815b340 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.931+0000 7f750d600640 1 -- 172.21.15.146:0/3672726823 >> 172.21.15.146:0/3672726823 conn(0x7f75080ccc20 msgr2=0x7f75080cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:36.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.932+0000 7f750d600640 1 -- 172.21.15.146:0/3672726823 shutdown_connections 2024-09-16T01:50:36.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.932+0000 7f750d600640 1 -- 172.21.15.146:0/3672726823 wait complete. 2024-09-16T01:50:36.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.933+0000 7f750d600640 1 Processor -- start 2024-09-16T01:50:36.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.934+0000 7f750d600640 1 -- start start 2024-09-16T01:50:36.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.934+0000 7f750d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c6b70 0x7f75080c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.934+0000 7f750d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75080c75a0 0x7f75081f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f750d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080cc040 0x7f75081f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f750d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f75080cb050 con 0x7f75080cc040 2024-09-16T01:50:36.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f750d600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75080cad70 con 0x7f75080c6b70 2024-09-16T01:50:36.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f750d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75080cbc80 con 0x7f75080c75a0 2024-09-16T01:50:36.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f7507e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c6b70 0x7f75080c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f7507e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c6b70 0x7f75080c53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34594/0 (socket says 172.21.15.146:34594) 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f7507e00640 1 -- 172.21.15.146:0/821226084 learned_addr learned my addr 172.21.15.146:0/821226084 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.935+0000 7f7507400640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75080c75a0 0x7f75081f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f750cc00640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080cc040 0x7f75081f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f7507400640 1 -- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c6b70 msgr2=0x7f75080c53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f7507400640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c6b70 0x7f75080c53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f7507400640 1 -- 172.21.15.146:0/821226084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080cc040 msgr2=0x7f75081f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:36.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f7507400640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080cc040 0x7f75081f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:36.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f7507400640 1 -- 172.21.15.146:0/821226084 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75080c2970 con 0x7f75080c75a0 2024-09-16T01:50:36.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.936+0000 7f7507400640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75080c75a0 0x7f75081f41f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f75080ca7a0 tx=0x7f74f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.937+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4041030 con 0x7f75080c75a0 2024-09-16T01:50:36.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.937+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f403d4e0 con 0x7f75080c75a0 2024-09-16T01:50:36.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.937+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4037040 con 0x7f75080c75a0 2024-09-16T01:50:36.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.937+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7508158240 con 0x7f75080c75a0 2024-09-16T01:50:36.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.937+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75081579a0 con 0x7f75080c75a0 2024-09-16T01:50:36.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.939+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f750806cd20 con 0x7f75080c75a0 2024-09-16T01:50:36.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.940+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f74f403d890 con 0x7f75080c75a0 2024-09-16T01:50:36.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.941+0000 7f74ffe00640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f74e806c2b0 0x7f74e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:36.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.941+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f74f40abc30 con 0x7f75080c75a0 2024-09-16T01:50:36.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.942+0000 7f7507e00640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f74e806c2b0 0x7f74e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:36.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.942+0000 7f7507e00640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f74e806c2b0 0x7f74e806e770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f74f8005880 tx=0x7f74f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:36.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:36.944+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f74f407a2b0 con 0x7f75080c75a0 2024-09-16T01:50:36.946 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:36.946 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:36.947 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:36.947 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:36.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:37.014 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:37.012+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.045+0000 7f29a0c00640 1 Processor -- start 2024-09-16T01:50:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.046+0000 7f29a0c00640 1 -- start start 2024-09-16T01:50:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.047+0000 7f29a0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f299c0c75a0 0x7f299c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.047+0000 7f29a0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299c06d140 con 0x7f299c0cc040 2024-09-16T01:50:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.047+0000 7f29a0c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f299c06d420 con 0x7f299c0c6b70 2024-09-16T01:50:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.047+0000 7f29a0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f299c15b880 con 0x7f299c0c75a0 2024-09-16T01:50:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.048+0000 7f299b400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f299c0c6b70 0x7f299c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:60002/0 (socket says 172.21.15.146:60002) 2024-09-16T01:50:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.048+0000 7f299b400640 1 -- 172.21.15.146:0/3460640369 learned_addr learned my addr 172.21.15.146:0/3460640369 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.048+0000 7f299aa00640 1 --2- 172.21.15.146:0/3460640369 >> v2:172.21.15.146:3300/0 conn(0x7f299c0c75a0 0x7f299c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.048+0000 7f299a000640 1 -- 172.21.15.146:0/3460640369 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638669824 0 0) 0x7f299c06d140 con 0x7f299c0cc040 2024-09-16T01:50:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.048+0000 7f299a000640 1 -- 172.21.15.146:0/3460640369 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2980003520 con 0x7f299c0cc040 2024-09-16T01:50:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.049+0000 7f299a000640 1 -- 172.21.15.146:0/3460640369 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2183734953 0 0) 0x7f299c06d420 con 0x7f299c0c6b70 2024-09-16T01:50:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.049+0000 7f299aa00640 1 -- 172.21.15.146:0/3460640369 >> v1:172.21.15.161:6789/0 conn(0x7f299c0c6b70 legacy=0x7f299c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.049+0000 7f299aa00640 1 -- 172.21.15.146:0/3460640369 >> v1:172.21.15.146:6789/0 conn(0x7f299c0cc040 legacy=0x7f299c165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.049+0000 7f299aa00640 1 -- 172.21.15.146:0/3460640369 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c06d700 con 0x7f299c0c75a0 2024-09-16T01:50:37.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.050+0000 7f299aa00640 1 --2- 172.21.15.146:0/3460640369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c15b340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29840097a0 tx=0x7f298402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4c9d1d388cf7b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.051+0000 7f299a000640 1 -- 172.21.15.146:0/3460640369 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984037070 con 0x7f299c0c75a0 2024-09-16T01:50:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.051+0000 7f299a000640 1 -- 172.21.15.146:0/3460640369 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2984032440 con 0x7f299c0c75a0 2024-09-16T01:50:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.051+0000 7f299a000640 1 -- 172.21.15.146:0/3460640369 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984032810 con 0x7f299c0c75a0 2024-09-16T01:50:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.052+0000 7f29a0c00640 1 -- 172.21.15.146:0/3460640369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 msgr2=0x7f299c15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.052+0000 7f29a0c00640 1 --2- 172.21.15.146:0/3460640369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c15b340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29840097a0 tx=0x7f298402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.053+0000 7f29a0c00640 1 -- 172.21.15.146:0/3460640369 shutdown_connections 2024-09-16T01:50:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.053+0000 7f29a0c00640 1 --2- 172.21.15.146:0/3460640369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c15b340 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.053+0000 7f29a0c00640 1 -- 172.21.15.146:0/3460640369 >> 172.21.15.146:0/3460640369 conn(0x7f299c0ccc20 msgr2=0x7f299c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.053+0000 7f29a0c00640 1 -- 172.21.15.146:0/3460640369 shutdown_connections 2024-09-16T01:50:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.053+0000 7f29a0c00640 1 -- 172.21.15.146:0/3460640369 wait complete. 2024-09-16T01:50:37.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.055+0000 7f29a0c00640 1 Processor -- start 2024-09-16T01:50:37.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.055+0000 7f29a0c00640 1 -- start start 2024-09-16T01:50:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f29a0c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f299c0c6b70 0x7f299c1e4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f29a0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f29a0c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f299c0cc040 0x7f299c1f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f29a0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f299c0cb050 con 0x7f299c0c75a0 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f29a0c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f299c0cab20 con 0x7f299c0c6b70 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f29a0c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f299c0cbc80 con 0x7f299c0cc040 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f299b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f299c0c6b70 0x7f299c1e4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f299aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.057+0000 7f299b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f299c0c6b70 0x7f299c1e4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34602/0 (socket says 172.21.15.146:34602) 2024-09-16T01:50:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.058+0000 7f299b400640 1 -- 172.21.15.146:0/3619162448 learned_addr learned my addr 172.21.15.146:0/3619162448 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.058+0000 7f299be00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f299c0cc040 0x7f299c1f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.058+0000 7f299be00640 1 -- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f299c0c6b70 msgr2=0x7f299c1e4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.058+0000 7f299be00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f299c0c6b70 0x7f299c1e4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.058+0000 7f299be00640 1 -- 172.21.15.146:0/3619162448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 msgr2=0x7f299c1554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f299be00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c1554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f299be00640 1 -- 172.21.15.146:0/3619162448 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c1e3e60 con 0x7f299c0cc040 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f299aa00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f299c0c75a0 0x7f299c1554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f299be00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f299c0cc040 0x7f299c1f8270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f298c00c760 tx=0x7f298c00cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c003b70 con 0x7f299c0cc040 2024-09-16T01:50:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f298c003d30 con 0x7f299c0cc040 2024-09-16T01:50:37.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.059+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c00aa90 con 0x7f299c0cc040 2024-09-16T01:50:37.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.060+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299c1562b0 con 0x7f299c0cc040 2024-09-16T01:50:37.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.060+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f299c156850 con 0x7f299c0cc040 2024-09-16T01:50:37.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.061+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f299c156f10 con 0x7f299c0cc040 2024-09-16T01:50:37.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.062+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f298c00ac30 con 0x7f299c0cc040 2024-09-16T01:50:37.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.063+0000 7f2993400640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f297806c300 0x7f297806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.063+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f298c08b440 con 0x7f299c0cc040 2024-09-16T01:50:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.064+0000 7f2993400640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2978074640 0x7f2978076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.064+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2978077150 con 0x7f2978074640 2024-09-16T01:50:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.064+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f298c08b7f0 con 0x7f299c0cc040 2024-09-16T01:50:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.064+0000 7f299b400640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f297806c300 0x7f297806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.064+0000 7f299aa00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2978074640 0x7f2978076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.065+0000 7f299aa00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2978074640 0x7f2978076a80 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.065+0000 7f299b400640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f297806c300 0x7f297806e7c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2994000f60 tx=0x7f2994007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.065+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2978077150 con 0x7f2978074640 2024-09-16T01:50:37.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:37.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.093+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f299c06cd20 con 0x7f2978074640 2024-09-16T01:50:37.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.094+0000 7f2993400640 1 -- 172.21.15.146:0/3619162448 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f299c06cd20 con 0x7f2978074640 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2978074640 msgr2=0x7f2978076a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f2978074640 0x7f2978076a80 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f297806c300 msgr2=0x7f297806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f297806c300 0x7f297806e7c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2994000f60 tx=0x7f2994007ea0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f299c0cc040 msgr2=0x7f299c1f8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 --2- 172.21.15.146:0/3619162448 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f299c0cc040 0x7f299c1f8270 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f298c00c760 tx=0x7f298c00cc20 comp rx=0 tx=0).stop 2024-09-16T01:50:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f299be00640 1 -- 172.21.15.146:0/3619162448 reap_dead start 2024-09-16T01:50:37.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 shutdown_connections 2024-09-16T01:50:37.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.097+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 >> 172.21.15.146:0/3619162448 conn(0x7f299c0ccc20 msgr2=0x7f299c0c8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.098+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 shutdown_connections 2024-09-16T01:50:37.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.098+0000 7f29a0c00640 1 -- 172.21.15.146:0/3619162448 wait complete. 2024-09-16T01:50:37.113 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:37.111+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:37.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-16T01:50:37.216 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:37.214+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:37.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.257+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f75080c2d10 con 0x7f75080c75a0 2024-09-16T01:50:37.289 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:37.289 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:37.289 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:37.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:37.313 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:37.312+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.325+0000 7f45c6800640 1 Processor -- start 2024-09-16T01:50:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.325+0000 7f45c6800640 1 -- start start 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.327+0000 7f45c6800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 0x7f45c00c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.327+0000 7f45c6800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f45c00d0870 0x7f45c00c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.327+0000 7f45c6800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c015e220 con 0x7f45c0154690 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.328+0000 7f45c6800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f45c00c6160 con 0x7f45c00d0870 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.328+0000 7f45c6800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f45c015e500 con 0x7f45c00d0490 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.328+0000 7f45c5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 0x7f45c00c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.328+0000 7f45c4a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f45c00d0870 0x7f45c00c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.328+0000 7f45c5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 0x7f45c00c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50740/0 (socket says 172.21.15.146:50740) 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.328+0000 7f45c5400640 1 -- 172.21.15.146:0/316661427 learned_addr learned my addr 172.21.15.146:0/316661427 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.329+0000 7f45c4a00640 1 -- 172.21.15.146:0/316661427 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 msgr2=0x7f45c00c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.329+0000 7f45c4a00640 1 --2- 172.21.15.146:0/316661427 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 0x7f45c00c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.329+0000 7f45c4a00640 1 -- 172.21.15.146:0/316661427 >> v1:172.21.15.161:6789/0 conn(0x7f45c0154690 legacy=0x7f45c0154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.329+0000 7f45c4a00640 1 -- 172.21.15.146:0/316661427 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c015e7e0 con 0x7f45c00d0870 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.329+0000 7f45bfe00640 1 -- 172.21.15.146:0/316661427 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3905341277 0 0) 0x7f45c015e220 con 0x7f45c0154690 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.329+0000 7f45c5400640 1 --2- 172.21.15.146:0/316661427 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 0x7f45c00c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:37.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.331+0000 7f45c4a00640 1 --2- 172.21.15.146:0/316661427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0870 0x7f45c00c6b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009a00 tx=0x7f45ac02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857824bf0f8ab0a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.331+0000 7f45bfe00640 1 -- 172.21.15.146:0/316661427 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac036070 con 0x7f45c00d0870 2024-09-16T01:50:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.332+0000 7f45bfe00640 1 -- 172.21.15.146:0/316661427 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45ac031600 con 0x7f45c00d0870 2024-09-16T01:50:37.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.332+0000 7f45bfe00640 1 -- 172.21.15.146:0/316661427 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac0319d0 con 0x7f45c00d0870 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 -- 172.21.15.146:0/316661427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0870 msgr2=0x7f45c00c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 --2- 172.21.15.146:0/316661427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0870 0x7f45c00c6b30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009a00 tx=0x7f45ac02a480 comp rx=0 tx=0).stop 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 -- 172.21.15.146:0/316661427 shutdown_connections 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 --2- 172.21.15.146:0/316661427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0870 0x7f45c00c6b30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 --2- 172.21.15.146:0/316661427 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c00d0490 0x7f45c00c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 -- 172.21.15.146:0/316661427 >> 172.21.15.146:0/316661427 conn(0x7f45c006d6f0 msgr2=0x7f45c006e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 -- 172.21.15.146:0/316661427 shutdown_connections 2024-09-16T01:50:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.333+0000 7f45c6800640 1 -- 172.21.15.146:0/316661427 wait complete. 2024-09-16T01:50:37.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.336+0000 7f45c6800640 1 Processor -- start 2024-09-16T01:50:37.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.336+0000 7f45c6800640 1 -- start start 2024-09-16T01:50:37.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0490 0x7f45c01f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c6800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45c00d0870 0x7f45c01f06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c6800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c0154690 0x7f45c00b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c6800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f45c015dec0 con 0x7f45c00d0490 2024-09-16T01:50:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c6800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45c015eda0 con 0x7f45c00d0870 2024-09-16T01:50:37.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c6800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f45c00c51f0 con 0x7f45c0154690 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.338+0000 7f45c5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0490 0x7f45c01f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.339+0000 7f45c4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45c00d0870 0x7f45c01f06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.339+0000 7f45c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c0154690 0x7f45c00b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.339+0000 7f45c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c0154690 0x7f45c00b5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50754/0 (socket says 172.21.15.146:50754) 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.339+0000 7f45c5e00640 1 -- 172.21.15.146:0/4000504674 learned_addr learned my addr 172.21.15.146:0/4000504674 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.339+0000 7f45c5400640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c0154690 msgr2=0x7f45c00b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.340+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45c0154690 0x7f45c00b5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.340+0000 7f45c5400640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45c00d0870 msgr2=0x7f45c01f06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.340+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45c00d0870 0x7f45c01f06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.340+0000 7f45c5400640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c00b6300 con 0x7f45c00d0490 2024-09-16T01:50:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.340+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0490 0x7f45c01f47b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45b400e930 tx=0x7f45b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.340+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b400ca00 con 0x7f45c00d0490 2024-09-16T01:50:37.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.341+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b400cba0 con 0x7f45c00d0490 2024-09-16T01:50:37.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.341+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4004030 con 0x7f45c00d0490 2024-09-16T01:50:37.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.341+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c00b3520 con 0x7f45c00d0490 2024-09-16T01:50:37.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.341+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c00b37e0 con 0x7f45c00d0490 2024-09-16T01:50:37.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.345+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f45b40041d0 con 0x7f45c00d0490 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.345+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f45c0154ac0 con 0x7f45c00d0490 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.346+0000 7f45bd600640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f45a006c300 0x7f45a006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.346+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7f45b408b3b0 con 0x7f45c00d0490 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.346+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v48) ==== 54+0+16 (secure 0 0 0) 0x7f45b4017020 con 0x7f45c00d0490 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.347+0000 7f45c4a00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f45a006c300 0x7f45a006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.347+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f45c00c9d70 0x7f45c014eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.347+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45c0153490 con 0x7f45c00c9d70 2024-09-16T01:50:37.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.348+0000 7f45c5e00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f45c00c9d70 0x7f45c014eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.348+0000 7f45c4a00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f45a006c300 0x7f45a006e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f45ac033000 tx=0x7f45ac008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.349+0000 7f45c5e00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f45c00c9d70 0x7f45c014eba0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.350+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45b8001af0 con 0x7f45c00c9d70 2024-09-16T01:50:37.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.378+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c00c7a30 con 0x7f45c00c9d70 2024-09-16T01:50:37.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.381+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c00c7a30 con 0x7f45c00c9d70 2024-09-16T01:50:37.383 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:37.384 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:37.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.382+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f45c00b2090 0x7f45c00b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.382+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f45c0153cf0 con 0x7f45c00b2090 2024-09-16T01:50:37.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.382+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f45c00b2090 0x7f45c00b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.383+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f45c00b2090 0x7f45c00b2470 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.385+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c0153cf0 con 0x7f45c00b2090 2024-09-16T01:50:37.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.402+0000 7fb58b200640 1 Processor -- start 2024-09-16T01:50:37.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.402+0000 7fb58b200640 1 -- start start 2024-09-16T01:50:37.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.403+0000 7fb58b200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5840c75a0 0x7fb58415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.403+0000 7fb58b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840cc040 0x7fb5840cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.403+0000 7fb58b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58406d140 con 0x7fb5840c6b70 2024-09-16T01:50:37.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.403+0000 7fb58b200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb58415b880 con 0x7fb5840c75a0 2024-09-16T01:50:37.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.403+0000 7fb58b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb5840cab20 con 0x7fb5840cc040 2024-09-16T01:50:37.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.404+0000 7fb589e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb5840c6b70 0x7fb5840cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43830/0 (socket says 172.21.15.146:43830) 2024-09-16T01:50:37.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.404+0000 7fb58a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840cc040 0x7fb5840cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.404+0000 7fb589400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5840c75a0 0x7fb58415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.404+0000 7fb589400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5840c75a0 0x7fb58415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34632/0 (socket says 172.21.15.146:34632) 2024-09-16T01:50:37.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.404+0000 7fb589e00640 1 -- 172.21.15.146:0/1959820963 learned_addr learned my addr 172.21.15.146:0/1959820963 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:37.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.404+0000 7fb58a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840cc040 0x7fb5840cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50764/0 (socket says 172.21.15.146:50764) 2024-09-16T01:50:37.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.405+0000 7fb589400640 1 -- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840cc040 msgr2=0x7fb5840cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:37.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.405+0000 7fb589400640 1 --2- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840cc040 0x7fb5840cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.405+0000 7fb589400640 1 -- 172.21.15.146:0/1959820963 >> v1:172.21.15.146:6789/0 conn(0x7fb5840c6b70 legacy=0x7fb5840cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.405+0000 7fb589400640 1 -- 172.21.15.146:0/1959820963 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58406d420 con 0x7fb5840c75a0 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.406+0000 7fb589400640 1 --2- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c75a0 0x7fb58415b340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5740097a0 tx=0x7fb57402a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b4d513854eab7ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.406+0000 7fb588a00640 1 -- 172.21.15.146:0/1959820963 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574037070 con 0x7fb5840c75a0 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.406+0000 7fb588a00640 1 -- 172.21.15.146:0/1959820963 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57402ae40 con 0x7fb5840c75a0 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.406+0000 7fb588a00640 1 -- 172.21.15.146:0/1959820963 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574032660 con 0x7fb5840c75a0 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 -- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c75a0 msgr2=0x7fb58415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 --2- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c75a0 0x7fb58415b340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5740097a0 tx=0x7fb57402a0f0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 -- 172.21.15.146:0/1959820963 shutdown_connections 2024-09-16T01:50:37.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 --2- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840cc040 0x7fb5840cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 --2- 172.21.15.146:0/1959820963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840c75a0 0x7fb58415b340 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 -- 172.21.15.146:0/1959820963 >> 172.21.15.146:0/1959820963 conn(0x7fb5840ccc20 msgr2=0x7fb5840cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 -- 172.21.15.146:0/1959820963 shutdown_connections 2024-09-16T01:50:37.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.407+0000 7fb58b200640 1 -- 172.21.15.146:0/1959820963 wait complete. 2024-09-16T01:50:37.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.409+0000 7fb58b200640 1 Processor -- start 2024-09-16T01:50:37.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.409+0000 7fb58b200640 1 -- start start 2024-09-16T01:50:37.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.410+0000 7fb58b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840c6b70 0x7fb5840c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.410+0000 7fb58b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5840c75a0 0x7fb5840c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.410+0000 7fb58b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840cc040 0x7fb5841f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.410+0000 7fb58b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb5840cb050 con 0x7fb5840c75a0 2024-09-16T01:50:37.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.410+0000 7fb58b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb5840cad70 con 0x7fb5840cc040 2024-09-16T01:50:37.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.410+0000 7fb58b200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb5840cbc80 con 0x7fb5840c6b70 2024-09-16T01:50:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.411+0000 7fb58a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840cc040 0x7fb5841f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.411+0000 7fb58a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840cc040 0x7fb5841f3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:34648/0 (socket says 172.21.15.146:34648) 2024-09-16T01:50:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.411+0000 7fb58a800640 1 -- 172.21.15.146:0/3465438993 learned_addr learned my addr 172.21.15.146:0/3465438993 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.411+0000 7fb589e00640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840c6b70 0x7fb5840c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.411+0000 7fb589400640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5840c75a0 0x7fb5840c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.412+0000 7fb589400640 1 -- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840c6b70 msgr2=0x7fb5840c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.412+0000 7fb589400640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840c6b70 0x7fb5840c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.412+0000 7fb589400640 1 -- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840cc040 msgr2=0x7fb5841f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.412+0000 7fb589400640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5840cc040 0x7fb5841f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.412+0000 7fb589400640 1 -- 172.21.15.146:0/3465438993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5840c5920 con 0x7fb5840c75a0 2024-09-16T01:50:37.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb589400640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5840c75a0 0x7fb5840c5020 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb574031000 tx=0x7fb5740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574041020 con 0x7fb5840c75a0 2024-09-16T01:50:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb589e00640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb5840c6b70 0x7fb5840c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57403c670 con 0x7fb5840c75a0 2024-09-16T01:50:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb574037040 con 0x7fb5840c75a0 2024-09-16T01:50:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5841f08a0 con 0x7fb5840c75a0 2024-09-16T01:50:37.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.413+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5841f0e40 con 0x7fb5840c75a0 2024-09-16T01:50:37.417 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:37.412+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:37.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.415+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb5841f1500 con 0x7fb5840c75a0 2024-09-16T01:50:37.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.415+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c0152370 con 0x7f45c00b2090 2024-09-16T01:50:37.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.417+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c0152370 con 0x7f45c00b2090 2024-09-16T01:50:37.420 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:37.420 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:37.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.418+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb574044020 con 0x7fb5840c75a0 2024-09-16T01:50:37.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.420+0000 7fb57e000640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb56806c380 0x7fb56806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.420+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5770+0+0 (secure 0 0 0) 0x7fb5740ab570 con 0x7fb5840c75a0 2024-09-16T01:50:37.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.420+0000 7fb57e000640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fb5680746c0 0x7fb568076b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.420+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5680771d0 con 0x7fb5680746c0 2024-09-16T01:50:37.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.420+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fb5740b2050 con 0x7fb5840c75a0 2024-09-16T01:50:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.419+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f45c01e7460 0x7f45c01e78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.419+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f45c00ab9b0 con 0x7f45c01e7460 2024-09-16T01:50:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.419+0000 7f45c4a00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f45c01e7460 0x7f45c01e78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.421+0000 7f45c4a00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f45c01e7460 0x7f45c01e78a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.421+0000 7fb589e00640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb56806c380 0x7fb56806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.421+0000 7fb58a800640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fb5680746c0 0x7fb568076b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.422+0000 7fb589e00640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb56806c380 0x7fb56806e840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb578000f60 tx=0x7fb578009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.422+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c00ab9b0 con 0x7f45c01e7460 2024-09-16T01:50:37.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.423+0000 7fb58a800640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fb5680746c0 0x7fb568076b00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.424+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5680771d0 con 0x7fb5680746c0 2024-09-16T01:50:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.445+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fb5840cf320 con 0x7fb5680746c0 2024-09-16T01:50:37.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.446+0000 7fb57e000640 1 -- 172.21.15.146:0/3465438993 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fb5840cf320 con 0x7fb5680746c0 2024-09-16T01:50:37.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.446+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fb5680746c0 msgr2=0x7fb568076b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.446+0000 7fb58b200640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fb5680746c0 0x7fb568076b00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.446+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb56806c380 msgr2=0x7fb56806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.446+0000 7fb58b200640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb56806c380 0x7fb56806e840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb578000f60 tx=0x7fb578009040 comp rx=0 tx=0).stop 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5840c75a0 msgr2=0x7fb5840c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58b200640 1 --2- 172.21.15.146:0/3465438993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5840c75a0 0x7fb5840c5020 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb574031000 tx=0x7fb5740046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58a800640 1 -- 172.21.15.146:0/3465438993 reap_dead start 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 shutdown_connections 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 >> 172.21.15.146:0/3465438993 conn(0x7fb5840ccc20 msgr2=0x7fb5840cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 shutdown_connections 2024-09-16T01:50:37.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.447+0000 7fb58b200640 1 -- 172.21.15.146:0/3465438993 wait complete. 2024-09-16T01:50:37.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.449+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c0062630 con 0x7f45c01e7460 2024-09-16T01:50:37.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.451+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c0062630 con 0x7f45c01e7460 2024-09-16T01:50:37.454 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.452+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f45c00630b0 0x7f45c00ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.452+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f45c00634f0 con 0x7f45c00630b0 2024-09-16T01:50:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.454+0000 7f45c5e00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f45c00630b0 0x7f45c00ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.455+0000 7f45c5e00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f45c00630b0 0x7f45c00ce090 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.456+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c00634f0 con 0x7f45c00630b0 2024-09-16T01:50:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.482+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c00ce8a0 con 0x7f45c00630b0 2024-09-16T01:50:37.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.485+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c00ce8a0 con 0x7f45c00630b0 2024-09-16T01:50:37.487 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:37.487 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:37.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.485+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f45c002c640 0x7f45c002ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.485+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f45c00cefc0 con 0x7f45c002c640 2024-09-16T01:50:37.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.486+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f45c002c640 0x7f45c002ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.486+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f45c002c640 0x7f45c002ca80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.487+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c00cefc0 con 0x7f45c002c640 2024-09-16T01:50:37.515 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:37.515+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:37.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.514+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c002cfc0 con 0x7f45c002c640 2024-09-16T01:50:37.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.517+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c002cfc0 con 0x7f45c002c640 2024-09-16T01:50:37.519 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:37.519 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:37.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.517+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f45c002d640 0x7f45c0206fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.517+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f45c002da80 con 0x7f45c002d640 2024-09-16T01:50:37.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.517+0000 7f45c4a00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f45c002d640 0x7f45c0206fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.518+0000 7f45c4a00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f45c002d640 0x7f45c0206fc0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.519+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c002da80 con 0x7f45c002d640 2024-09-16T01:50:37.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.545+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c002da80 con 0x7f45c002d640 2024-09-16T01:50:37.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.547+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c002da80 con 0x7f45c002d640 2024-09-16T01:50:37.549 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.547+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f45c0207a10 0x7f45c0209e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.547+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f45c020cc50 con 0x7f45c0207a10 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.548+0000 7f45c5e00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f45c0207a10 0x7f45c0209e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:37.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.548+0000 7f45c5e00640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f45c0207a10 0x7f45c0209e00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.549+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c020cc50 con 0x7f45c0207a10 2024-09-16T01:50:37.564 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:37.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.576+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c020a3c0 con 0x7f45c0207a10 2024-09-16T01:50:37.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.578+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c020a3c0 con 0x7f45c0207a10 2024-09-16T01:50:37.580 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:37.581 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:37.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.578+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f45c020c240 0x7f45c020c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:37.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.578+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f45c020fff0 con 0x7f45c020c240 2024-09-16T01:50:37.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.579+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f45c020c240 0x7f45c020c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:37.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.579+0000 7f45c5400640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f45c020c240 0x7f45c020c680 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:37.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.580+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c020fff0 con 0x7f45c020c240 2024-09-16T01:50:37.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.607+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f45c020a5d0 con 0x7f45c020c240 2024-09-16T01:50:37.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45bd600640 1 -- 172.21.15.146:0/4000504674 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f45c020a5d0 con 0x7f45c020c240 2024-09-16T01:50:37.611 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f45c00c9d70 msgr2=0x7f45c014eba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f45c00c9d70 0x7f45c014eba0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f45c00b2090 msgr2=0x7f45c00b2470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f45c00b2090 0x7f45c00b2470 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f45c01e7460 msgr2=0x7f45c01e78a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f45c01e7460 0x7f45c01e78a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f45c00630b0 msgr2=0x7f45c00ce090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f45c00630b0 0x7f45c00ce090 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f45c002c640 msgr2=0x7f45c002ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f45c002c640 0x7f45c002ca80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f45c002d640 msgr2=0x7f45c0206fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f45c002d640 0x7f45c0206fc0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f45c0207a10 msgr2=0x7f45c0209e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f45c0207a10 0x7f45c0209e00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f45c020c240 msgr2=0x7f45c020c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f45c020c240 0x7f45c020c680 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c5e00640 1 -- 172.21.15.146:0/4000504674 reap_dead start 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c5e00640 1 -- 172.21.15.146:0/4000504674 reap_dead start 2024-09-16T01:50:37.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c5e00640 1 -- 172.21.15.146:0/4000504674 reap_dead start 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c5e00640 1 -- 172.21.15.146:0/4000504674 reap_dead start 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c5e00640 1 -- 172.21.15.146:0/4000504674 reap_dead start 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f45a006c300 msgr2=0x7f45a006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f45a006c300 0x7f45a006e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f45ac033000 tx=0x7f45ac008030 comp rx=0 tx=0).stop 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0490 msgr2=0x7f45c01f47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.609+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0490 0x7f45c01f47b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45b400e930 tx=0x7f45b400edf0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.610+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 shutdown_connections 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.610+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f45a006c300 0x7f45a006e7c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.610+0000 7f45c6800640 1 --2- 172.21.15.146:0/4000504674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45c00d0490 0x7f45c01f47b0 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45b400e930 tx=0x7f45b400edf0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.610+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 >> 172.21.15.146:0/4000504674 conn(0x7f45c006d6f0 msgr2=0x7f45c00ccfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.610+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 shutdown_connections 2024-09-16T01:50:37.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.610+0000 7f45c6800640 1 -- 172.21.15.146:0/4000504674 wait complete. 2024-09-16T01:50:37.617 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:37.615+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.651+0000 7f74ffe00640 1 -- 172.21.15.146:0/821226084 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v49) ==== 101+0+0 (secure 0 0 0) 0x7f74f4079bb0 con 0x7f75080c75a0 2024-09-16T01:50:37.653 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f74e806c2b0 msgr2=0x7f74e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f74e806c2b0 0x7f74e806e770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f74f8005880 tx=0x7f74f8009040 comp rx=0 tx=0).stop 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75080c75a0 msgr2=0x7f75081f41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75080c75a0 0x7f75081f41f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f75080ca7a0 tx=0x7f74f40046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 shutdown_connections 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f74e806c2b0 0x7f74e806e770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75080cc040 0x7f75081f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75080c75a0 0x7f75081f41f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 --2- 172.21.15.146:0/821226084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75080c6b70 0x7f75080c53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:37.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 >> 172.21.15.146:0/821226084 conn(0x7f75080ccc20 msgr2=0x7f75080cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:37.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 shutdown_connections 2024-09-16T01:50:37.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:37.654+0000 7f750d600640 1 -- 172.21.15.146:0/821226084 wait complete. 2024-09-16T01:50:37.686 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:37.686 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:37.695 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:37.723 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:37.721+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:37.799 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:37.800 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:37.808 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:37.819 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:37.817+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:37.912 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:37.912 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:37.919 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:37.917+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:37.921 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:38.020 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:38.019+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:38.025 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.025 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:38.122 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:38.121+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:38.179 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.179 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.188 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:38.223 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:38.221+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:38.324 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:38.323+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:38.331 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.331 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.340 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:38.425 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:38.424+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:38.484 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.484 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.492 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:38.526 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:38.524+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:38.627 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:38.625+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:38.636 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.636 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.645 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:38.728 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:38.726+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:38.750 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.750 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.758 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:38.829 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:38.827+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:38.863 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:38.863 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:38.872 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:38.929 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:38.927+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:39.016 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:39.017 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:39.025 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:39.069 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:39.068+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:39.129 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:39.129 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:39.130 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:39.129+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:39.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:39.232 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:39.230+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:39.332 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:39.331+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:39.406 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:39.406 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:39.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:39.466 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:39.464+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:39.535 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:39.533+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:39.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:39.635 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:39.634+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:39.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:50:39.687 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:39.687 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:39.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:39.737 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:39.736+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:39.838 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:39.836+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:39.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.888+0000 7f6a4b000640 1 Processor -- start 2024-09-16T01:50:39.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.889+0000 7f6a4b000640 1 -- start start 2024-09-16T01:50:39.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a4b000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a440d0870 0x7f6a44154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a4b000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a440c6d00 0x7f6a44154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a4b000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a4415e220 con 0x7f6a440d0490 2024-09-16T01:50:39.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a4b000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6a440c6160 con 0x7f6a440d0870 2024-09-16T01:50:39.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a4b000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4415e500 con 0x7f6a440c6d00 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a49200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a440d0870 0x7f6a44154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a49200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a440d0870 0x7f6a44154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47596/0 (socket says 172.21.15.146:47596) 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a49c00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6a440d0490 0x7f6a440c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:60016/0 (socket says 172.21.15.146:60016) 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a49200640 1 -- 172.21.15.146:0/1310157357 learned_addr learned my addr 172.21.15.146:0/1310157357 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.891+0000 7f6a4a600640 1 --2- 172.21.15.146:0/1310157357 >> v2:172.21.15.161:3300/0 conn(0x7f6a440c6d00 0x7f6a44154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.892+0000 7f6a4a600640 1 -- 172.21.15.146:0/1310157357 >> v2:172.21.15.146:3300/0 conn(0x7f6a440d0870 msgr2=0x7f6a44154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:39.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.892+0000 7f6a4a600640 1 --2- 172.21.15.146:0/1310157357 >> v2:172.21.15.146:3300/0 conn(0x7f6a440d0870 0x7f6a44154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.892+0000 7f6a4a600640 1 -- 172.21.15.146:0/1310157357 >> v1:172.21.15.161:6789/0 conn(0x7f6a440d0490 legacy=0x7f6a440c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.892+0000 7f6a4a600640 1 -- 172.21.15.146:0/1310157357 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4415e7e0 con 0x7f6a440c6d00 2024-09-16T01:50:39.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.894+0000 7f6a4a600640 1 --2- 172.21.15.146:0/1310157357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440c6d00 0x7f6a44154bd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a440b2f90 tx=0x7f6a3c02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41dc89a3642f7c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.894+0000 7f6a43e00640 1 -- 172.21.15.146:0/1310157357 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c007740 con 0x7f6a440c6d00 2024-09-16T01:50:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.894+0000 7f6a43e00640 1 -- 172.21.15.146:0/1310157357 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3c0078e0 con 0x7f6a440c6d00 2024-09-16T01:50:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.894+0000 7f6a43e00640 1 -- 172.21.15.146:0/1310157357 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c007cb0 con 0x7f6a440c6d00 2024-09-16T01:50:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.895+0000 7f6a4b000640 1 -- 172.21.15.146:0/1310157357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440c6d00 msgr2=0x7f6a44154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.895+0000 7f6a4b000640 1 --2- 172.21.15.146:0/1310157357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440c6d00 0x7f6a44154bd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a440b2f90 tx=0x7f6a3c02a710 comp rx=0 tx=0).stop 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.896+0000 7f6a4b000640 1 -- 172.21.15.146:0/1310157357 shutdown_connections 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.896+0000 7f6a4b000640 1 --2- 172.21.15.146:0/1310157357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440c6d00 0x7f6a44154bd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.896+0000 7f6a4b000640 1 --2- 172.21.15.146:0/1310157357 >> v2:172.21.15.146:3300/0 conn(0x7f6a440d0870 0x7f6a44154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.896+0000 7f6a4b000640 1 -- 172.21.15.146:0/1310157357 >> 172.21.15.146:0/1310157357 conn(0x7f6a4406d6f0 msgr2=0x7f6a4406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.896+0000 7f6a4b000640 1 -- 172.21.15.146:0/1310157357 shutdown_connections 2024-09-16T01:50:39.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.896+0000 7f6a4b000640 1 -- 172.21.15.146:0/1310157357 wait complete. 2024-09-16T01:50:39.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.899+0000 7f6a4b000640 1 Processor -- start 2024-09-16T01:50:39.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.899+0000 7f6a4b000640 1 -- start start 2024-09-16T01:50:39.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.900+0000 7f6a4b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a440c6d00 0x7f6a4415d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.900+0000 7f6a4b000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a440d0490 0x7f6a441f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.900+0000 7f6a4b000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440d0870 0x7f6a441f0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.900+0000 7f6a4b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4415dec0 con 0x7f6a440c6d00 2024-09-16T01:50:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.900+0000 7f6a4b000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4415eda0 con 0x7f6a440d0870 2024-09-16T01:50:39.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.900+0000 7f6a4b000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a440c51f0 con 0x7f6a440d0490 2024-09-16T01:50:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.901+0000 7f6a49c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a440c6d00 0x7f6a4415d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.901+0000 7f6a49c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a440c6d00 0x7f6a4415d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47602/0 (socket says 172.21.15.146:47602) 2024-09-16T01:50:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.901+0000 7f6a49c00640 1 -- 172.21.15.146:0/2280048399 learned_addr learned my addr 172.21.15.146:0/2280048399 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.901+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a440d0490 0x7f6a441f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.901+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440d0870 0x7f6a441f0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.902+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a440d0490 msgr2=0x7f6a441f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.902+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a440d0490 0x7f6a441f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.902+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a440c6d00 msgr2=0x7f6a4415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.902+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a440c6d00 0x7f6a4415d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.902+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a44153a10 con 0x7f6a440d0870 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.903+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a440c6d00 0x7f6a4415d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.903+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440d0870 0x7f6a441f0a80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c031000 tx=0x7f6a3c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.903+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c043dc0 con 0x7f6a440d0870 2024-09-16T01:50:39.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.903+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3c031630 con 0x7f6a440d0870 2024-09-16T01:50:39.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.903+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c0319e0 con 0x7f6a440d0870 2024-09-16T01:50:39.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.904+0000 7f6a4b000640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a44153c80 con 0x7f6a440d0870 2024-09-16T01:50:39.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.904+0000 7f6a4b000640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a441e3c30 con 0x7f6a440d0870 2024-09-16T01:50:39.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.905+0000 7f6a4b000640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6a440c65f0 con 0x7f6a440d0870 2024-09-16T01:50:39.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.907+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a3c039040 con 0x7f6a440d0870 2024-09-16T01:50:39.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.909+0000 7f6a41600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a2406c2b0 0x7f6a2406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.909+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5828+0+0 (secure 0 0 0) 0x7f6a3c0b5160 con 0x7f6a440d0870 2024-09-16T01:50:39.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.909+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v50) ==== 54+0+16 (secure 0 0 0) 0x7f6a3c0b7700 con 0x7f6a440d0870 2024-09-16T01:50:39.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.909+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a2406c2b0 0x7f6a2406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.910+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a441e42f0 0x7f6a441f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.910+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a44152620 con 0x7f6a441e42f0 2024-09-16T01:50:39.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.910+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a2406c2b0 0x7f6a2406e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6a38004e00 tx=0x7f6a38007f10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.911+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a441e42f0 0x7f6a441f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.912+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a441e42f0 0x7f6a441f3e50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.914+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a44152620 con 0x7f6a441e42f0 2024-09-16T01:50:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.937+0000 7f4598e00640 1 Processor -- start 2024-09-16T01:50:39.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.937+0000 7f4598e00640 1 -- start start 2024-09-16T01:50:39.940 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:39.937+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:39.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.938+0000 7f4598e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0bf0 0x7f45940c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.938+0000 7f4598e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f45940b3790 0x7f45940ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.938+0000 7f4598e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4594155430 con 0x7f45940d0fd0 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.938+0000 7f4598e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f459415aea0 con 0x7f45940b3790 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.938+0000 7f4598e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f459415b180 con 0x7f45940d0bf0 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4591400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f45940d0fd0 0x7f45940b3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43846/0 (socket says 172.21.15.146:43846) 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4591400640 1 -- 172.21.15.146:0/2432729781 learned_addr learned my addr 172.21.15.146:0/2432729781 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4592800640 1 --2- 172.21.15.146:0/2432729781 >> v2:172.21.15.161:3300/0 conn(0x7f45940b3790 0x7f45940ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4591e00640 1 --2- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0bf0 0x7f45940c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4592800640 1 -- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0bf0 msgr2=0x7f45940c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4592800640 1 --2- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0bf0 0x7f45940c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.939+0000 7f4592800640 1 -- 172.21.15.146:0/2432729781 >> v1:172.21.15.146:6789/0 conn(0x7f45940d0fd0 legacy=0x7f45940b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.941+0000 7f4592800640 1 -- 172.21.15.146:0/2432729781 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459415b460 con 0x7f45940b3790 2024-09-16T01:50:39.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.942+0000 7f4592800640 1 --2- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940b3790 0x7f45940ca520 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f458c009d90 tx=0x7f458c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97872f9efa7487ef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.942+0000 7f4590a00640 1 -- 172.21.15.146:0/2432729781 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c007740 con 0x7f45940b3790 2024-09-16T01:50:39.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.943+0000 7f4590a00640 1 -- 172.21.15.146:0/2432729781 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458c0078e0 con 0x7f45940b3790 2024-09-16T01:50:39.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.943+0000 7f4590a00640 1 -- 172.21.15.146:0/2432729781 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c007cb0 con 0x7f45940b3790 2024-09-16T01:50:39.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 -- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940b3790 msgr2=0x7f45940ca520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 --2- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940b3790 0x7f45940ca520 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f458c009d90 tx=0x7f458c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a440b2b50 con 0x7f6a441e42f0 2024-09-16T01:50:39.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 -- 172.21.15.146:0/2432729781 shutdown_connections 2024-09-16T01:50:39.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 --2- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940b3790 0x7f45940ca520 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 --2- 172.21.15.146:0/2432729781 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0bf0 0x7f45940c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 -- 172.21.15.146:0/2432729781 >> 172.21.15.146:0/2432729781 conn(0x7f459406c900 msgr2=0x7f459406cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:39.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 -- 172.21.15.146:0/2432729781 shutdown_connections 2024-09-16T01:50:39.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.944+0000 7f4598e00640 1 -- 172.21.15.146:0/2432729781 wait complete. 2024-09-16T01:50:39.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.946+0000 7f4598e00640 1 Processor -- start 2024-09-16T01:50:39.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.946+0000 7f4598e00640 1 -- start start 2024-09-16T01:50:39.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.947+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a440b2b50 con 0x7f6a441e42f0 2024-09-16T01:50:39.949 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:39.949 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:39.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.947+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a441532c0 0x7f6a44144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.947+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6a441e3e70 con 0x7f6a441532c0 2024-09-16T01:50:39.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.948+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a441532c0 0x7f6a44144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.948+0000 7f4598e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 0x7f45941f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.948+0000 7f4598e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940d0bf0 0x7f459414e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f4598e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0fd0 0x7f459414ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f4591e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 0x7f45941f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f4591e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 0x7f45941f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47610/0 (socket says 172.21.15.146:47610) 2024-09-16T01:50:39.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f4591e00640 1 -- 172.21.15.146:0/3870136786 learned_addr learned my addr 172.21.15.146:0/3870136786 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:39.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a441532c0 0x7f6a44144740 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f459415edc0 con 0x7f45940b3790 2024-09-16T01:50:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.949+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f459415ba20 con 0x7f45940d0bf0 2024-09-16T01:50:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f459415ef40 con 0x7f45940d0fd0 2024-09-16T01:50:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4591400640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940d0bf0 0x7f459414e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4592800640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0fd0 0x7f459414ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a441e3e70 con 0x7f6a441532c0 2024-09-16T01:50:39.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4591e00640 1 -- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0fd0 msgr2=0x7f459414ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4591e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0fd0 0x7f459414ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4591e00640 1 -- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940d0bf0 msgr2=0x7f459414e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4591e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940d0bf0 0x7f459414e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4591e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459414d360 con 0x7f45940b3790 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.950+0000 7f4592800640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0fd0 0x7f459414ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.951+0000 7f4591e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 0x7f45941f42a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f458400e9e0 tx=0x7f458400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.951+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584017070 con 0x7f45940b3790 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.952+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458400c9d0 con 0x7f45940b3790 2024-09-16T01:50:39.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.952+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458400cdc0 con 0x7f45940b3790 2024-09-16T01:50:39.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.952+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459415c6d0 con 0x7f45940b3790 2024-09-16T01:50:39.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.952+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459415c3f0 con 0x7f45940b3790 2024-09-16T01:50:39.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.955+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f458401a070 con 0x7f45940b3790 2024-09-16T01:50:39.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.955+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f459414e010 con 0x7f45940b3790 2024-09-16T01:50:39.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.956+0000 7f458a000640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f457406c2b0 0x7f457406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.956+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5828+0+0 (secure 0 0 0) 0x7f458408b540 con 0x7f45940b3790 2024-09-16T01:50:39.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.957+0000 7f4591400640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f457406c2b0 0x7f457406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.958+0000 7f4591400640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f457406c2b0 0x7f457406e770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f459415e320 tx=0x7f4580007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.962+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4584059540 con 0x7f45940b3790 2024-09-16T01:50:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.981+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a441e6440 con 0x7f6a441532c0 2024-09-16T01:50:39.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.984+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a441e6440 con 0x7f6a441532c0 2024-09-16T01:50:39.986 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.984+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a44145190 0x7f6a44062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.984+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6a441e65f0 con 0x7f6a44145190 2024-09-16T01:50:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.985+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a44145190 0x7f6a44062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.985+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a44145190 0x7f6a44062630 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:39.987+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a441e65f0 con 0x7f6a44145190 2024-09-16T01:50:40.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.015+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a44062e40 con 0x7f6a44145190 2024-09-16T01:50:40.020 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:40.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:16 2.1d 2:b871830b:::605.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.192025971999996, 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074611160000000003, 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825041+0000", 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825050+0000", 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5890000000000003e-06 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6470000000000004e-06 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825079+0000", 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0571999999999999e-05 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825092+0000", 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.324e-05 2024-09-16T01:50:40.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178271 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825393+0000", 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122652 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825625+0000", 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000231952 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827807+0000", 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002181695 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832395+0000", 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045881159999999997 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832456+0000", 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1178000000000003e-05 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832502+0000", 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6204e-05 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:17 2.1d 2:b871830b:::605.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:40.026 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.191919923, 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075256139999999999, 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825147+0000", 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825148+0000", 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825153+0000", 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1699999999999996e-06 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825181+0000", 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8274000000000001e-05 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825189+0000", 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8509999999999999e-06 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825991+0000", 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00080194900000000004 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826068+0000", 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6783e-05 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826181+0000", 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113251 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829054+0000", 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028730729999999999 2024-09-16T01:50:40.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832568+0000", 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035134010000000002 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1902999999999993e-05 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832673+0000", 2024-09-16T01:50:40.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3164000000000002e-05 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:20 2.1c 2:3ed09add:::607.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.191865297000007, 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073187670000000003, 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.032 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825202+0000", 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4000000000000001e-07 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825228+0000", 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5660000000000002e-05 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825235+0000", 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.43e-06 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825244+0000", 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4689999999999997e-06 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825343+0000", 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8968000000000005e-05 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825461+0000", 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011790099999999999 2024-09-16T01:50:40.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825642+0000", 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018077499999999999 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827990+0000", 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002348087 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832439+0000", 2024-09-16T01:50:40.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044494870000000002 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832482+0000", 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180000000000003e-05 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832520+0000", 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8170000000000002e-05 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.037 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:21 2.1c 2:3ed09add:::607.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.191796964000005, 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073969559999999997, 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:40.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825270+0000", 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825271+0000", 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.85e-07 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825275+0000", 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3159999999999998e-06 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825284+0000", 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5979999999999997e-06 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825291+0000", 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1799999999999999e-06 2024-09-16T01:50:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825959+0000", 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066839700000000004 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826036+0000", 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6607999999999996e-05 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826138+0000", 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102563 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829095+0000", 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029568260000000001 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832559+0000", 2024-09-16T01:50:40.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034634209999999999 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832629+0000", 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9888000000000006e-05 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832667+0000", 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8374000000000002e-05 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.043 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:11 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.157541761000004, 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017873126999999999, 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859525+0000", 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859527+0000", 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2790000000000001e-06 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859536+0000", 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7499999999999992e-06 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859575+0000", 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8705999999999998e-05 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859638+0000", 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3085999999999999e-05 2024-09-16T01:50:40.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860259+0000", 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062087899999999996 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860406+0000", 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014708599999999999 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860550+0000", 2024-09-16T01:50:40.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014363900000000001 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869067+0000", 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085171980000000001 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877287+0000", 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082201640000000003 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877347+0000", 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9660000000000001e-05 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877398+0000", 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.168e-05 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:15 2.8 2:1470b2f8:::613.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.157358563000003, 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015192597, 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859708+0000", 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859709+0000", 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8800000000000002e-07 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859715+0000", 2024-09-16T01:50:40.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1890000000000004e-06 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859730+0000", 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4399000000000001e-05 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859738+0000", 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863504+0000", 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037655190000000002 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863644+0000", 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014016199999999999 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863839+0000", 2024-09-16T01:50:40.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019467700000000001 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873421+0000", 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095824780000000002 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874779+0000", 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013575250000000001 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874850+0000", 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1779000000000005e-05 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874901+0000", 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0587000000000002e-05 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:28 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.157304554999996, 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019993687, 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859762+0000", 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859763+0000", 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.68e-07 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859767+0000", 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2810000000000002e-06 2024-09-16T01:50:40.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859779+0000", 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1803e-05 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859785+0000", 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.139e-06 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860772+0000", 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000986654 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860813+0000", 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0890999999999999e-05 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.860871+0000", 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8452000000000002e-05 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869073+0000", 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018239999999999 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879661+0000", 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010588235 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879716+0000", 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4660000000000002e-05 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879756+0000", 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0179999999999998e-05 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:32 2.8 2:1470b2f8:::613.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.157272114999998, 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023610079999999999, 2024-09-16T01:50:40.061 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859795+0000", 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7599999999999997e-07 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859798+0000", 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0809999999999998e-06 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859805+0000", 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9789999999999996e-06 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.859810+0000", 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1050000000000003e-06 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864189+0000", 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043782170000000002 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864258+0000", 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9212000000000001e-05 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864369+0000", 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111242 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873428+0000", 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090588370000000001 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883295+0000", 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098669389999999999 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883356+0000", 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883405+0000", 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9070000000000003e-05 2024-09-16T01:50:40.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:14 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.153904113999999, 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0082018330000000004, 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863163+0000", 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863164+0000", 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863177+0000", 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3142999999999999e-05 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863191+0000", 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3679999999999999e-05 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863201+0000", 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7079999999999992e-06 2024-09-16T01:50:40.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864429+0000", 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001228163 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864448+0000", 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432e-05 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871295+0000", 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068463789999999997 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871365+0000", 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9839000000000007e-05 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:31 2.9 e22/19 2:92e99a1d:::612.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:40.071 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.153833438000007, 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081623670000000002, 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:40.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863234+0000", 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5400000000000002e-07 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863239+0000", 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4060000000000004e-06 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863247+0000", 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863252+0000", 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1229999999999999e-06 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864636+0000", 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013843589999999999 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864652+0000", 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5897e-05 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871368+0000", 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067159050000000003 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871396+0000", 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.783e-05 2024-09-16T01:50:40.075 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:19 2.9 e22/19 2:91bd965c:::501.00000000:head v 22'3, mlcod=0'0)", 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.153797062999999, 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081559130000000007, 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863270+0000", 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9900000000000002e-07 2024-09-16T01:50:40.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863274+0000", 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2760000000000002e-06 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863281+0000", 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3960000000000003e-06 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863287+0000", 2024-09-16T01:50:40.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1750000000000002e-06 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864796+0000", 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015084639999999999 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864812+0000", 2024-09-16T01:50:40.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6396999999999999e-05 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871402+0000", 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065903209999999997 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871426+0000", 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3584999999999999e-05 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:13 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.153169320000003, 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095956289999999996, 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863898+0000", 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863899+0000", 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.78e-07 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863907+0000", 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9260000000000006e-06 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863917+0000", 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.893e-06 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863924+0000", 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.058e-06 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865775+0000", 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001850689 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865794+0000", 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8766999999999998e-05 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873442+0000", 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076481129999999998 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873493+0000", 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1304999999999998e-05 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:30 2.13 e22/19 2:ca4dd7f5:::611.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:40.085 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.153114712999994, 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095741130000000004, 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863952+0000", 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863953+0000", 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2e-07 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863984+0000", 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1158000000000002e-05 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863992+0000", 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863998+0000", 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.322e-06 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865997+0000", 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019997439999999999 2024-09-16T01:50:40.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866014+0000", 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6456e-05 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873498+0000", 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074839420000000004 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.873526+0000", 2024-09-16T01:50:40.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8651000000000001e-05 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:3 2.1c 2:3d0fce1b:::628.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080976304999993, 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006366165, 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936091+0000", 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936093+0000", 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2510000000000001e-06 2024-09-16T01:50:40.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936100+0000", 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9469999999999997e-06 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936115+0000", 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5603999999999998e-05 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936126+0000", 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.082e-05 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936205+0000", 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8267000000000001e-05 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936335+0000", 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130326 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936465+0000", 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013000899999999999 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939766+0000", 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003300908 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942372+0000", 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026063409999999999 2024-09-16T01:50:40.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942418+0000", 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5930000000000002e-05 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942457+0000", 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8761999999999999e-05 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:4 2.1c 2:3d0fce1b:::628.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080917993, 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064411370000000004, 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:40.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936149+0000", 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936153+0000", 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.732e-06 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936160+0000", 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4400000000000002e-06 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936163+0000", 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4460000000000001e-06 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936817+0000", 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065436400000000005 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1488000000000003e-05 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141326 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940704+0000", 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036634290000000002 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942500+0000", 2024-09-16T01:50:40.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001796707 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942546+0000", 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5413e-05 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942590+0000", 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4434000000000002e-05 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080843917999999, 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063328309999999997, 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936223+0000", 2024-09-16T01:50:40.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7300000000000002e-07 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936228+0000", 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3080000000000001e-06 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936232+0000", 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0239999999999999e-06 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936236+0000", 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5850000000000001e-06 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936870+0000", 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063394799999999996 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936940+0000", 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9282999999999994e-05 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937082+0000", 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014212799999999999 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940719+0000", 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036377190000000002 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942467+0000", 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001747951 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942522+0000", 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4518999999999998e-05 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942556+0000", 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3992999999999998e-05 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:16 2.1d 2:b9436051:::62e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080785297000006, 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061316870000000002, 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:40.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936282+0000", 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0800000000000004e-07 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936286+0000", 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3859999999999999e-06 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936293+0000", 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0870000000000001e-06 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936298+0000", 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9930000000000003e-06 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936819+0000", 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052160800000000001 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936899+0000", 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9950999999999994e-05 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937040+0000", 2024-09-16T01:50:40.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014082400000000001 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940546+0000", 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035054249999999999 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942321+0000", 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017759379999999999 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942371+0000", 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9453999999999998e-05 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.942413+0000", 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2413000000000002e-05 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:21 2.1d 2:be329c1b:::631.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.113 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080763363000003, 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091413369999999994, 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936304+0000", 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5400000000000002e-07 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936306+0000", 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5739999999999998e-06 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936310+0000", 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5779999999999999e-06 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936313+0000", 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1729999999999999e-06 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937341+0000", 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010279130000000001 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937433+0000", 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1481e-05 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937544+0000", 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011184300000000001 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940780+0000", 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032357250000000001 2024-09-16T01:50:40.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945374+0000", 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045941150000000002 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945414+0000", 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9746000000000002e-05 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945445+0000", 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0935000000000001e-05 2024-09-16T01:50:40.118 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:22 2.1d 2:be329c1b:::631.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080733003000006, 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092093190000000005, 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936334+0000", 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9400000000000001e-07 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936337+0000", 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7650000000000002e-06 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936344+0000", 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6680000000000004e-06 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936352+0000", 2024-09-16T01:50:40.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5569999999999996e-06 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939307+0000", 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029545209999999999 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939382+0000", 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5570999999999999e-05 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939534+0000", 2024-09-16T01:50:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151335 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944720+0000", 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0051861889999999999 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945482+0000", 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00076238800000000002 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945519+0000", 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6569999999999997e-05 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945543+0000", 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4360999999999999e-05 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:23 2.1c 2:399c8d31:::104.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:40.124 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.080706423999999, 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090781730000000001, 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:40.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936361+0000", 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4000000000000002e-07 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936364+0000", 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3299999999999999e-06 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936371+0000", 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5309999999999998e-06 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936376+0000", 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937272+0000", 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089679400000000002 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937372+0000", 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9839000000000004e-05 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937515+0000", 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014303699999999999 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940832+0000", 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033165059999999999 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945351+0000", 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045193819999999997 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945403+0000", 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2052000000000002e-05 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945439+0000", 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5432999999999998e-05 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.130 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.130 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.130 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:40.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.018+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a44062e40 con 0x7f6a44145190 2024-09-16T01:50:40.130 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:40.131 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:40.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.019+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a440c8d40 0x7f6a440c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:40.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.019+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6a44073120 con 0x7f6a440c8d40 2024-09-16T01:50:40.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.020+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a440c8d40 0x7f6a440c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:40.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.021+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a440c8d40 0x7f6a440c91c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:40.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.022+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a44073120 con 0x7f6a440c8d40 2024-09-16T01:50:40.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:40.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.051+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a44073120 con 0x7f6a440c8d40 2024-09-16T01:50:40.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.054+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a44073120 con 0x7f6a440c8d40 2024-09-16T01:50:40.134 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.054+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a441ff3b0 0x7f6a442017a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.054+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6a44073120 con 0x7f6a441ff3b0 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.055+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a441ff3b0 0x7f6a442017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.056+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a441ff3b0 0x7f6a442017a0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.057+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a44073120 con 0x7f6a441ff3b0 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.085+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a44073120 con 0x7f6a441ff3b0 2024-09-16T01:50:40.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.087+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a44073120 con 0x7f6a441ff3b0 2024-09-16T01:50:40.136 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:40.136 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.088+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a442021f0 0x7f6a442045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.088+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6a442073b0 con 0x7f6a442021f0 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.088+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a442021f0 0x7f6a442045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.089+0000 7f6a49200640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a442021f0 0x7f6a442045e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.090+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a442073b0 con 0x7f6a442021f0 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.119+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a44204ba0 con 0x7f6a442021f0 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.121+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a44204ba0 con 0x7f6a442021f0 2024-09-16T01:50:40.137 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:40.138 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:40.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.122+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a44206b30 0x7f6a44206f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:40.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.122+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6a4420a750 con 0x7f6a44206b30 2024-09-16T01:50:40.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.122+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a44206b30 0x7f6a44206f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:40.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.123+0000 7f6a4a600640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a44206b30 0x7f6a44206f90 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:40.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.124+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a4420a750 con 0x7f6a44206b30 2024-09-16T01:50:40.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.155+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a442095d0 con 0x7f6a44206b30 2024-09-16T01:50:40.158 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:40.156+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:40.163 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:40.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.158+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a442095d0 con 0x7f6a44206b30 2024-09-16T01:50:40.163 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:40.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.158+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a44209cf0 0x7f6a4420a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:40.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.158+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6a4420dac0 con 0x7f6a44209cf0 2024-09-16T01:50:40.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.158+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a44209cf0 0x7f6a4420a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:40.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.159+0000 7f6a49c00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a44209cf0 0x7f6a4420a130 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:40.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.160+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a4420dac0 con 0x7f6a44209cf0 2024-09-16T01:50:40.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.190+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a44204b70 con 0x7f6a44209cf0 2024-09-16T01:50:40.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a41600640 1 -- 172.21.15.146:0/2280048399 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a44204b70 con 0x7f6a44209cf0 2024-09-16T01:50:40.195 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:40.195 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:40.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a441e42f0 msgr2=0x7f6a441f3e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a441e42f0 0x7f6a441f3e50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a441532c0 msgr2=0x7f6a44144740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a441532c0 0x7f6a44144740 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a44145190 msgr2=0x7f6a44062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a44145190 0x7f6a44062630 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a440c8d40 msgr2=0x7f6a440c91c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a440c8d40 0x7f6a440c91c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a441ff3b0 msgr2=0x7f6a442017a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a441ff3b0 0x7f6a442017a0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a442021f0 msgr2=0x7f6a442045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a442021f0 0x7f6a442045e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a44206b30 msgr2=0x7f6a44206f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a44206b30 0x7f6a44206f90 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a44209cf0 msgr2=0x7f6a4420a130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.193+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a44209cf0 0x7f6a4420a130 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 reap_dead start 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 reap_dead start 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 reap_dead start 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 reap_dead start 2024-09-16T01:50:40.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a4a600640 1 -- 172.21.15.146:0/2280048399 reap_dead start 2024-09-16T01:50:40.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a2406c2b0 msgr2=0x7f6a2406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a2406c2b0 0x7f6a2406e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6a38004e00 tx=0x7f6a38007f10 comp rx=0 tx=0).stop 2024-09-16T01:50:40.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440d0870 msgr2=0x7f6a441f0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440d0870 0x7f6a441f0a80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c031000 tx=0x7f6a3c004a40 comp rx=0 tx=0).stop 2024-09-16T01:50:40.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.194+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 shutdown_connections 2024-09-16T01:50:40.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.195+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a2406c2b0 0x7f6a2406e770 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.195+0000 7f6a36a00640 1 --2- 172.21.15.146:0/2280048399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a440d0870 0x7f6a441f0a80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.195+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 >> 172.21.15.146:0/2280048399 conn(0x7f6a4406d6f0 msgr2=0x7f6a440cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:40.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.195+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 shutdown_connections 2024-09-16T01:50:40.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.195+0000 7f6a36a00640 1 -- 172.21.15.146:0/2280048399 wait complete. 2024-09-16T01:50:40.264 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:40.262+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.285+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f45941547d0 con 0x7f45940b3790 2024-09-16T01:50:40.337 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:40.335+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:40.426 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:40.426 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:40.426 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:40.426 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:40.438 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:40.437+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:40.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:40.539 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:40.537+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:40.640 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:40.639+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:40.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.679+0000 7f458a000640 1 -- 172.21.15.146:0/3870136786 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v51) ==== 87+0+0 (secure 0 0 0) 0x7f458405d840 con 0x7f45940b3790 2024-09-16T01:50:40.682 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:50:40.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.684+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f457406c2b0 msgr2=0x7f457406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.684+0000 7f4598e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f457406c2b0 0x7f457406e770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f459415e320 tx=0x7f4580007e80 comp rx=0 tx=0).stop 2024-09-16T01:50:40.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.684+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 msgr2=0x7f45941f42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:40.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.684+0000 7f4598e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 0x7f45941f42a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f458400e9e0 tx=0x7f458400eea0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 shutdown_connections 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f457406c2b0 0x7f457406e770 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45940d0fd0 0x7f459414ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45940d0bf0 0x7f459414e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 --2- 172.21.15.146:0/3870136786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45940b3790 0x7f45941f42a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 >> 172.21.15.146:0/3870136786 conn(0x7f459406c900 msgr2=0x7f45940c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 shutdown_connections 2024-09-16T01:50:40.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:40.686+0000 7f4598e00640 1 -- 172.21.15.146:0/3870136786 wait complete. 2024-09-16T01:50:40.740 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:40.740 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:40.740 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:40.740 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:40.740 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:40.741 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:40.738+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:40.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:40.841 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:40.839+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:40.942 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:40.941+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:40.971 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.199555498, 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.768001766, 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.748077544, 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.748057626, 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:40.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.74799704199999, 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:40.998 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:40.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:41.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.50171152199999, 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.004 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:41.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:41.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.501561008, 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:41.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:41.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.50154144, 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:41.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:41.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.931385226, 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.022 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:41.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.93121740799999, 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.029 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:41.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:41.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.93119842599999, 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.035 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:41.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.93117879499999, 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:41.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:41.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.046 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.93104488900001, 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:41.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.931033351, 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:41.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:41.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:41.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:41.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:41.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:41.061 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.93101494, 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:41.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.93090027, 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:41.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.920118542, 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:41.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:41.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:41.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:41.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 111.920001884, 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:41.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:41.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:41.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:41.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.104468644999997, 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:41.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:41.091 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.104407307000002, 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:41.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.096 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:41.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:41.098 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:41.097+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:41.199 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:41.197+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:41.300 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:41.299+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:41.315 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:41.315 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:41.315 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:41.315 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:41.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:41.401 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:41.400+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:41.502 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:41.501+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:41.535 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:41.535 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:41.536 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:41.536 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:41.536 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:41.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:41.603 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:41.601+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:41.705 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:41.703+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:41.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.892505627999995, 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:41.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:41.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.892390806999998, 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:41.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.732 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.892348936999994, 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:41.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:41.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:41.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:41.738 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.892304658, 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:41.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:41.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:41.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.892275780999995, 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:41.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.888568574000004, 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:41.748 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:41.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:41.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.888514592999996, 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.853099369000006, 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.759 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.850039219999999, 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:41.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:41.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:41.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.768 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.849179771999999, 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:41.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.849125878999999, 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:41.774 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:41.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:41.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.843101906000001, 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:41.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:41.784 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.843051778000003, 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:41.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.842975444999993, 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.842944090000003, 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:41.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.804858635000002, 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:41.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.802 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.804067103999998, 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:41.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:41.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 75.779985279000002, 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:41.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 75.779913367000006, 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:41.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:41.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.449270966999997, 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:41.819 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:41.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:41.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:41.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:41.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:41.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:41.825 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:41.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:41.827 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:41.825+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:41.927 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:41.925+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:42.028 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:42.026+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:42.034 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:42.034 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:42.034 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:42.034 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:42.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:42.136 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:42.134+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:42.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:42.229 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:42.228+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:42.267 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:42.267 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:42.267 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:42.267 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:42.268 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:42.280 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:42.369 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:42.367+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:42.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.416+0000 7ff3dd200640 1 Processor -- start 2024-09-16T01:50:42.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.416+0000 7ff3dd200640 1 -- start start 2024-09-16T01:50:42.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3dd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3d80d0490 0x7ff3d80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3dd200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d815e220 con 0x7ff3d8154da0 2024-09-16T01:50:42.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3dd200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d815e500 con 0x7ff3d80d0870 2024-09-16T01:50:42.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3dd200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff3d80c6160 con 0x7ff3d80d0490 2024-09-16T01:50:42.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3d7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3d80d0490 0x7ff3d80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3d7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3d80d0490 0x7ff3d80c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47612/0 (socket says 172.21.15.146:47612) 2024-09-16T01:50:42.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.418+0000 7ff3d7400640 1 -- 172.21.15.146:0/626222864 learned_addr learned my addr 172.21.15.146:0/626222864 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:42.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.419+0000 7ff3d7400640 1 -- 172.21.15.146:0/626222864 >> v1:172.21.15.161:6789/0 conn(0x7ff3d80d0870 legacy=0x7ff3d8154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.419+0000 7ff3d6a00640 1 -- 172.21.15.146:0/626222864 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 342984292 0 0) 0x7ff3d815e500 con 0x7ff3d80d0870 2024-09-16T01:50:42.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.419+0000 7ff3d7400640 1 -- 172.21.15.146:0/626222864 >> v1:172.21.15.146:6789/0 conn(0x7ff3d8154da0 legacy=0x7ff3d80c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.419+0000 7ff3d7400640 1 -- 172.21.15.146:0/626222864 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d815e7e0 con 0x7ff3d80d0490 2024-09-16T01:50:42.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.420+0000 7ff3d7400640 1 --2- 172.21.15.146:0/626222864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0490 0x7ff3d80c65f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009870 tx=0x7ff3c802a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba414f47429d91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.420+0000 7ff3d6a00640 1 -- 172.21.15.146:0/626222864 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8037070 con 0x7ff3d80d0490 2024-09-16T01:50:42.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.420+0000 7ff3d6a00640 1 -- 172.21.15.146:0/626222864 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c8032640 con 0x7ff3d80d0490 2024-09-16T01:50:42.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.421+0000 7ff3d6a00640 1 -- 172.21.15.146:0/626222864 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8032a10 con 0x7ff3d80d0490 2024-09-16T01:50:42.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.421+0000 7ff3dd200640 1 -- 172.21.15.146:0/626222864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0490 msgr2=0x7ff3d80c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.421+0000 7ff3dd200640 1 --2- 172.21.15.146:0/626222864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0490 0x7ff3d80c65f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009870 tx=0x7ff3c802a3c0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.422+0000 7ff3dd200640 1 -- 172.21.15.146:0/626222864 shutdown_connections 2024-09-16T01:50:42.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.422+0000 7ff3dd200640 1 --2- 172.21.15.146:0/626222864 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0490 0x7ff3d80c65f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.422+0000 7ff3dd200640 1 -- 172.21.15.146:0/626222864 >> 172.21.15.146:0/626222864 conn(0x7ff3d806d6f0 msgr2=0x7ff3d806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:42.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.422+0000 7ff3dd200640 1 -- 172.21.15.146:0/626222864 shutdown_connections 2024-09-16T01:50:42.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.422+0000 7ff3dd200640 1 -- 172.21.15.146:0/626222864 wait complete. 2024-09-16T01:50:42.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.425+0000 7ff3dd200640 1 Processor -- start 2024-09-16T01:50:42.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.425+0000 7ff3dd200640 1 -- start start 2024-09-16T01:50:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3dd200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff3d80d0490 0x7ff3d81f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3dd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0870 0x7ff3d815d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3dd200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3d8154da0 0x7ff3d81e4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3dd200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d815dec0 con 0x7ff3d80d0870 2024-09-16T01:50:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3dd200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d815eda0 con 0x7ff3d8154da0 2024-09-16T01:50:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3dd200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d80c5900 con 0x7ff3d80d0490 2024-09-16T01:50:42.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3d7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff3d80d0490 0x7ff3d81f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0870 0x7ff3d815d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3d7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff3d80d0490 0x7ff3d81f50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:50808/0 (socket says 172.21.15.146:50808) 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3d7400640 1 -- 172.21.15.146:0/301914433 learned_addr learned my addr 172.21.15.146:0/301914433 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.427+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3d8154da0 0x7ff3d81e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3cfe00640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff3d80d0490 msgr2=0x7ff3d81f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff3d80d0490 0x7ff3d81f50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3cfe00640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3d8154da0 msgr2=0x7ff3d81e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3d8154da0 0x7ff3d81e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3cfe00640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d814fb40 con 0x7ff3d80d0870 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff3d80d0490 0x7ff3d81f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.428+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3d8154da0 0x7ff3d81e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:42.432 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:42.428+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.429+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0870 0x7ff3d815d2f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000c400 tx=0x7ff3c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.429+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0017070 con 0x7ff3d80d0870 2024-09-16T01:50:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.429+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c00035f0 con 0x7ff3d80d0870 2024-09-16T01:50:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.429+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c00039e0 con 0x7ff3d80d0870 2024-09-16T01:50:42.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.429+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d81e6240 con 0x7ff3d80d0870 2024-09-16T01:50:42.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.429+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d81f48c0 con 0x7ff3d80d0870 2024-09-16T01:50:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.432+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff3c001a070 con 0x7ff3d80d0870 2024-09-16T01:50:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.433+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff3b0005270 con 0x7ff3d80d0870 2024-09-16T01:50:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.434+0000 7ff3cf400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff3b406c300 0x7ff3b406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.434+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5828+0+0 (secure 0 0 0) 0x7ff3c008a720 con 0x7ff3d80d0870 2024-09-16T01:50:42.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.434+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v51) ==== 54+0+16 (secure 0 0 0) 0x7ff3c0090170 con 0x7ff3d80d0870 2024-09-16T01:50:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.435+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff3b406c300 0x7ff3b406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.435+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff3b0000fa0 0x7ff3b00045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.435+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3b0004b20 con 0x7ff3b0000fa0 2024-09-16T01:50:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.435+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff3b0000fa0 0x7ff3b00045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.436+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff3b406c300 0x7ff3b406e7c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80090b0 tx=0x7ff3c802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.436+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff3b0000fa0 0x7ff3b00045e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.437+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b0004b20 con 0x7ff3b0000fa0 2024-09-16T01:50:42.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.453+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b0005090 con 0x7ff3b0000fa0 2024-09-16T01:50:42.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.455+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b0005090 con 0x7ff3b0000fa0 2024-09-16T01:50:42.457 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:42.458 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:42.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.455+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff3b0001c70 0x7ff3b0002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.455+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff3b00022b0 con 0x7ff3b0001c70 2024-09-16T01:50:42.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.459+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff3b0001c70 0x7ff3b0002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.459+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff3b0001c70 0x7ff3b0002990 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.460+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b00022b0 con 0x7ff3b0001c70 2024-09-16T01:50:42.468 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:42.468 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:42.469 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-16T01:50:42.469 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:50:42.469 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:50:42.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-16T01:50:42.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.479+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b00031a0 con 0x7ff3b0001c70 2024-09-16T01:50:42.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.482+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b00031a0 con 0x7ff3b0001c70 2024-09-16T01:50:42.484 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:42.484 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:42.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.482+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff3b0009cd0 0x7ff3b000c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.482+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff3b0003ec0 con 0x7ff3b0009cd0 2024-09-16T01:50:42.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.482+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff3b0009cd0 0x7ff3b000c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.483+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff3b0009cd0 0x7ff3b000c0c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.485+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b0003ec0 con 0x7ff3b0009cd0 2024-09-16T01:50:42.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.504+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b000c930 con 0x7ff3b0009cd0 2024-09-16T01:50:42.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.506+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b000c930 con 0x7ff3b0009cd0 2024-09-16T01:50:42.508 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:42.509 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:42.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.506+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff3b000d3b0 0x7ff3b000d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.507+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff3b000df80 con 0x7ff3b000d3b0 2024-09-16T01:50:42.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.507+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff3b000d3b0 0x7ff3b000d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.508+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff3b000d3b0 0x7ff3b000d810 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.509+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b000df80 con 0x7ff3b000d3b0 2024-09-16T01:50:42.531 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:42.530+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:42.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.532+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b0010580 con 0x7ff3b000d3b0 2024-09-16T01:50:42.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.535+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b0010580 con 0x7ff3b000d3b0 2024-09-16T01:50:42.537 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:42.538 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:42.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.536+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff3b0010db0 0x7ff3b00131a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.536+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff3b00161d0 con 0x7ff3b0010db0 2024-09-16T01:50:42.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.537+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff3b0010db0 0x7ff3b00131a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.537+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff3b0010db0 0x7ff3b00131a0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.538+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b00161d0 con 0x7ff3b0010db0 2024-09-16T01:50:42.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.564+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b0014ef0 con 0x7ff3b0010db0 2024-09-16T01:50:42.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.566+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b0014ef0 con 0x7ff3b0010db0 2024-09-16T01:50:42.569 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:42.570 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:42.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.567+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff3b0015610 0x7ff3b0015a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.567+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff3b00195a0 con 0x7ff3b0015610 2024-09-16T01:50:42.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.567+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff3b0015610 0x7ff3b0015a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.568+0000 7ff3cfe00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff3b0015610 0x7ff3b0015a50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.569+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b00195a0 con 0x7ff3b0015610 2024-09-16T01:50:42.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.584+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b00184a0 con 0x7ff3b0015610 2024-09-16T01:50:42.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.587+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b00184a0 con 0x7ff3b0015610 2024-09-16T01:50:42.589 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:42.589 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:42.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.587+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff3b0018bc0 0x7ff3b0019000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.587+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff3b001c970 con 0x7ff3b0018bc0 2024-09-16T01:50:42.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.587+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff3b0018bc0 0x7ff3b0019000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.588+0000 7ff3d7e00640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff3b0018bc0 0x7ff3b0019000 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.589+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b001c970 con 0x7ff3b0018bc0 2024-09-16T01:50:42.594 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:42.594 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:42.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.605+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b001b7f0 con 0x7ff3b0018bc0 2024-09-16T01:50:42.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.608+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b001b7f0 con 0x7ff3b0018bc0 2024-09-16T01:50:42.610 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:42.610 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:42.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.608+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff3b001be70 0x7ff3b001c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.608+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff3b001fce0 con 0x7ff3b001be70 2024-09-16T01:50:42.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.608+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff3b001be70 0x7ff3b001c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.609+0000 7ff3d7400640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff3b001be70 0x7ff3b001c2b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:42.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.610+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3b001fce0 con 0x7ff3b001be70 2024-09-16T01:50:42.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.629+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff3b00137b0 con 0x7ff3b001be70 2024-09-16T01:50:42.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.631+0000 7ff3cf400640 1 -- 172.21.15.146:0/301914433 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff3b00137b0 con 0x7ff3b001be70 2024-09-16T01:50:42.633 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.631+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff3b0000fa0 msgr2=0x7ff3b00045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff3b0000fa0 0x7ff3b00045e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff3b0001c70 msgr2=0x7ff3b0002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff3b0001c70 0x7ff3b0002990 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff3b0009cd0 msgr2=0x7ff3b000c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff3b0009cd0 0x7ff3b000c0c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff3b000d3b0 msgr2=0x7ff3b000d810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff3b000d3b0 0x7ff3b000d810 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff3b0010db0 msgr2=0x7ff3b00131a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff3b0010db0 0x7ff3b00131a0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff3b0015610 msgr2=0x7ff3b0015a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff3b0015610 0x7ff3b0015a50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff3b0018bc0 msgr2=0x7ff3b0019000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff3b0018bc0 0x7ff3b0019000 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff3b001be70 msgr2=0x7ff3b001c2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff3b001be70 0x7ff3b001c2b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3d7e00640 1 -- 172.21.15.146:0/301914433 reap_dead start 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3d7e00640 1 -- 172.21.15.146:0/301914433 reap_dead start 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3d7e00640 1 -- 172.21.15.146:0/301914433 reap_dead start 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3d7e00640 1 -- 172.21.15.146:0/301914433 reap_dead start 2024-09-16T01:50:42.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3d7e00640 1 -- 172.21.15.146:0/301914433 reap_dead start 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff3b406c300 msgr2=0x7ff3b406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff3b406c300 0x7ff3b406e7c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80090b0 tx=0x7ff3c802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0870 msgr2=0x7ff3d815d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0870 0x7ff3d815d2f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000c400 tx=0x7ff3c000c8c0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 shutdown_connections 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff3b406c300 0x7ff3b406e7c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 --2- 172.21.15.146:0/301914433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3d80d0870 0x7ff3d815d2f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 >> 172.21.15.146:0/301914433 conn(0x7ff3d806d6f0 msgr2=0x7ff3d80c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 shutdown_connections 2024-09-16T01:50:42.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.632+0000 7ff3dd200640 1 -- 172.21.15.146:0/301914433 wait complete. 2024-09-16T01:50:42.657 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:42.656+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:42.713 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T01:50:42.713 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:42.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:50:42.730 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:42.730 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:42.732 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:42.731+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:42.739 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:42.833 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:42.831+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:42.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.884+0000 7f6322c00640 1 Processor -- start 2024-09-16T01:50:42.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.884+0000 7f6322c00640 1 -- start start 2024-09-16T01:50:42.886 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:42.887 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.885+0000 7f6322c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0c6b70 0x7f631c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.885+0000 7f6322c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f631c0c75a0 0x7f631c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.885+0000 7f6322c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f631c06d140 con 0x7f631c0cbab0 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.885+0000 7f6322c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f631c15b880 con 0x7f631c0c75a0 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.885+0000 7f6322c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f631c0cab20 con 0x7f631c0c6b70 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6320e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f631c0c75a0 0x7f631c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6321800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0c6b70 0x7f631c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6322200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f631c0cbab0 0x7f631c0cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34182/0 (socket says 172.21.15.146:34182) 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6322200640 1 -- 172.21.15.146:0/3392215083 learned_addr learned my addr 172.21.15.146:0/3392215083 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6320e00640 1 -- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0c6b70 msgr2=0x7f631c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6320e00640 1 --2- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0c6b70 0x7f631c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6320e00640 1 -- 172.21.15.146:0/3392215083 >> v1:172.21.15.161:6789/0 conn(0x7f631c0cbab0 legacy=0x7f631c0cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f631be00640 1 -- 172.21.15.146:0/3392215083 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2782310496 0 0) 0x7f631c06d140 con 0x7f631c0cbab0 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6320e00640 1 -- 172.21.15.146:0/3392215083 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c06d420 con 0x7f631c0c75a0 2024-09-16T01:50:42.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.886+0000 7f6321800640 1 --2- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0c6b70 0x7f631c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.887+0000 7f6320e00640 1 --2- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c0cb570 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63080097a0 tx=0x7f630802a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fedb668bf0e85b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.887+0000 7f631be00640 1 -- 172.21.15.146:0/3392215083 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308037070 con 0x7f631c0c75a0 2024-09-16T01:50:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.887+0000 7f631be00640 1 -- 172.21.15.146:0/3392215083 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f630802adb0 con 0x7f631c0c75a0 2024-09-16T01:50:42.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.887+0000 7f631be00640 1 -- 172.21.15.146:0/3392215083 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308032660 con 0x7f631c0c75a0 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 -- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 msgr2=0x7f631c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 --2- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c0cb570 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63080097a0 tx=0x7f630802a060 comp rx=0 tx=0).stop 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 -- 172.21.15.146:0/3392215083 shutdown_connections 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 --2- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c0cb570 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 --2- 172.21.15.146:0/3392215083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0c6b70 0x7f631c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 -- 172.21.15.146:0/3392215083 >> 172.21.15.146:0/3392215083 conn(0x7f631c0ccc20 msgr2=0x7f631c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 -- 172.21.15.146:0/3392215083 shutdown_connections 2024-09-16T01:50:42.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.888+0000 7f6322c00640 1 -- 172.21.15.146:0/3392215083 wait complete. 2024-09-16T01:50:42.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.890+0000 7f6322c00640 1 Processor -- start 2024-09-16T01:50:42.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.890+0000 7f6322c00640 1 -- start start 2024-09-16T01:50:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f631c0c6b70 0x7f631c0c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 0x7f631c15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f631c0cb050 con 0x7f631c0c6b70 2024-09-16T01:50:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f631c0cad70 con 0x7f631c0c75a0 2024-09-16T01:50:42.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f631c06cd20 con 0x7f631c0cbab0 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6320e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6322200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 0x7f631c15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.891+0000 7f6321800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f631c0c6b70 0x7f631c0c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6322200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 0x7f631c15cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37270/0 (socket says 172.21.15.146:37270) 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6322200640 1 -- 172.21.15.146:0/3030070545 learned_addr learned my addr 172.21.15.146:0/3030070545 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6320e00640 1 -- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 msgr2=0x7f631c15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6320e00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 0x7f631c15cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6320e00640 1 -- 172.21.15.146:0/3030070545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f631c0c6b70 msgr2=0x7f631c0c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6320e00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f631c0c6b70 0x7f631c0c8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6320e00640 1 -- 172.21.15.146:0/3030070545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c155540 con 0x7f631c0c75a0 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6322200640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 0x7f631c15cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:42.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.892+0000 7f6320e00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c15dee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f631c0ca7a0 tx=0x7f63080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.897 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.893+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308037040 con 0x7f631c0c75a0 2024-09-16T01:50:42.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.893+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6308004030 con 0x7f631c0c75a0 2024-09-16T01:50:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.893+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308004400 con 0x7f631c0c75a0 2024-09-16T01:50:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.893+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631c155820 con 0x7f631c0c75a0 2024-09-16T01:50:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.893+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631c155de0 con 0x7f631c0c75a0 2024-09-16T01:50:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.894+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631c0ca280 con 0x7f631c0c75a0 2024-09-16T01:50:42.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.895+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6308003610 con 0x7f631c0c75a0 2024-09-16T01:50:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.896+0000 7f6319600640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62fc06c2e0 0x7f62fc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.896+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5828+0+0 (secure 0 0 0) 0x7f63080abfc0 con 0x7f631c0c75a0 2024-09-16T01:50:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.896+0000 7f6321800640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62fc06c2e0 0x7f62fc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.896+0000 7f6321800640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62fc06c2e0 0x7f62fc06e7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6310009560 tx=0x7f6310009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:42.900+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f630807a050 con 0x7f631c0c75a0 2024-09-16T01:50:42.934 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:42.932+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:43.001 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.001 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.009 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:43.035 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:43.033+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:43.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.093+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f631c0cda30 con 0x7f631c0c75a0 2024-09-16T01:50:43.113 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.113 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.122 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:43.136 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:43.134+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:43.226 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.226 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:43.236 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:43.235+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:43.337 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:43.335+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:43.339 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.339 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.348 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:43.438 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:43.437+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:43.452 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.453 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.461 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:43.539 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:43.538+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:43.606 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.606 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.614 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:43.640 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:43.639+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:43.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.699+0000 7f6319600640 1 -- 172.21.15.146:0/3030070545 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v52) ==== 97+0+0 (secure 0 0 0) 0x7f630807e350 con 0x7f631c0c75a0 2024-09-16T01:50:43.702 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.702+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62fc06c2e0 msgr2=0x7f62fc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.702+0000 7f6322c00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62fc06c2e0 0x7f62fc06e7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6310009560 tx=0x7f6310009190 comp rx=0 tx=0).stop 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.702+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 msgr2=0x7f631c15dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.702+0000 7f6322c00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c15dee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f631c0ca7a0 tx=0x7f63080046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 shutdown_connections 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62fc06c2e0 0x7f62fc06e7a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f631c0cbab0 0x7f631c15cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f631c0c75a0 0x7f631c15dee0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:43.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 --2- 172.21.15.146:0/3030070545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f631c0c6b70 0x7f631c0c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:43.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 >> 172.21.15.146:0/3030070545 conn(0x7f631c0ccc20 msgr2=0x7f631c0ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:43.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 shutdown_connections 2024-09-16T01:50:43.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:43.703+0000 7f6322c00640 1 -- 172.21.15.146:0/3030070545 wait complete. 2024-09-16T01:50:43.721 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.721 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.730 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:43.764 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:43.762+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:43.834 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.834 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.843 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:43.843 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:43.841+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:43.944 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:43.942+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:43.988 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:43.988 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:43.998 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:44.044 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:44.042+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:44.101 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:44.101 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:44.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:44.144 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:44.143+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:44.245 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:44.245+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:44.275 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:44.275 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:44.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:44.346 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:44.345+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:44.447 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:44.447 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:44.447 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:44.445+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:44.461 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:44.548 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:44.547+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:44.649 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:44.647+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:44.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:44.668 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:44.668 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:44.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:44.769 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:44.766+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:44.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.811+0000 7f30d9a00640 1 Processor -- start 2024-09-16T01:50:44.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.812+0000 7f30d9a00640 1 -- start start 2024-09-16T01:50:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d9a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f30d4071a10 0x7f30d4073280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d406ea70 con 0x7f30d40c7300 2024-09-16T01:50:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d9a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d41511c0 con 0x7f30d415a250 2024-09-16T01:50:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d9a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f30d40725c0 con 0x7f30d4071a10 2024-09-16T01:50:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d2a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f30d4071a10 0x7f30d4073280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d2000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f30d40c7300 0x7f30d40c7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53404/0 (socket says 172.21.15.146:53404) 2024-09-16T01:50:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d2000640 1 -- 172.21.15.146:0/1077119549 learned_addr learned my addr 172.21.15.146:0/1077119549 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.813+0000 7f30d3400640 1 --1- 172.21.15.146:0/1077119549 >> v1:172.21.15.161:6789/0 conn(0x7f30d415a250 0x7f30d415a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34194/0 (socket says 172.21.15.146:34194) 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.814+0000 7f30d2a00640 1 -- 172.21.15.146:0/1077119549 >> v1:172.21.15.161:6789/0 conn(0x7f30d415a250 legacy=0x7f30d415a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.814+0000 7f30d2a00640 1 -- 172.21.15.146:0/1077119549 >> v1:172.21.15.146:6789/0 conn(0x7f30d40c7300 legacy=0x7f30d40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.814+0000 7f30d2a00640 1 -- 172.21.15.146:0/1077119549 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d406ed10 con 0x7f30d4071a10 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.815+0000 7f30d2a00640 1 --2- 172.21.15.146:0/1077119549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d4071a10 0x7f30d4073280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f30c40097a0 tx=0x7f30c402a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56b3e2711ce0a732 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.815+0000 7f30d1600640 1 -- 172.21.15.146:0/1077119549 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4037070 con 0x7f30d4071a10 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.815+0000 7f30d1600640 1 -- 172.21.15.146:0/1077119549 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c4032640 con 0x7f30d4071a10 2024-09-16T01:50:44.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.815+0000 7f30d1600640 1 -- 172.21.15.146:0/1077119549 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4032a10 con 0x7f30d4071a10 2024-09-16T01:50:44.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 -- 172.21.15.146:0/1077119549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d4071a10 msgr2=0x7f30d4073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 --2- 172.21.15.146:0/1077119549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d4071a10 0x7f30d4073280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f30c40097a0 tx=0x7f30c402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 -- 172.21.15.146:0/1077119549 shutdown_connections 2024-09-16T01:50:44.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 --2- 172.21.15.146:0/1077119549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d4071a10 0x7f30d4073280 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 -- 172.21.15.146:0/1077119549 >> 172.21.15.146:0/1077119549 conn(0x7f30d406bc10 msgr2=0x7f30d406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:44.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 -- 172.21.15.146:0/1077119549 shutdown_connections 2024-09-16T01:50:44.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.816+0000 7f30d9a00640 1 -- 172.21.15.146:0/1077119549 wait complete. 2024-09-16T01:50:44.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.818+0000 7f30d9a00640 1 Processor -- start 2024-09-16T01:50:44.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.818+0000 7f30d9a00640 1 -- start start 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d4071a10 0x7f30d4153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d9a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d40c7300 0x7f30d41f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f30d415a250 0x7f30d41f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f30d406f2d0 con 0x7f30d4071a10 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d9a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f30d415adb0 con 0x7f30d40c7300 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d9a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f30d406f5b0 con 0x7f30d415a250 2024-09-16T01:50:44.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d4071a10 0x7f30d4153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d4071a10 0x7f30d4153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38806/0 (socket says 172.21.15.146:38806) 2024-09-16T01:50:44.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 -- 172.21.15.146:0/759142420 learned_addr learned my addr 172.21.15.146:0/759142420 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:44.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d40c7300 0x7f30d41f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f30d415a250 0x7f30d41f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f30d415a250 msgr2=0x7f30d41f4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f30d415a250 0x7f30d41f4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d40c7300 msgr2=0x7f30d41f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.819+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d40c7300 0x7f30d41f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30d2a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d41f07a0 con 0x7f30d4071a10 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f30d40c7300 0x7f30d41f3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d4071a10 0x7f30d4153e50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30c40098c0 tx=0x7f30c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4041030 con 0x7f30d4071a10 2024-09-16T01:50:44.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c4030750 con 0x7f30d4071a10 2024-09-16T01:50:44.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4037040 con 0x7f30d4071a10 2024-09-16T01:50:44.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d41f0a10 con 0x7f30d4071a10 2024-09-16T01:50:44.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.820+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d41f0fd0 con 0x7f30d4071a10 2024-09-16T01:50:44.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.822+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f30d406f890 con 0x7f30d4071a10 2024-09-16T01:50:44.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.824+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f30c4044070 con 0x7f30d4071a10 2024-09-16T01:50:44.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.824+0000 7f30caa00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f30b406c2b0 0x7f30b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.825+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5828+0+0 (secure 0 0 0) 0x7f30c40ab7c0 con 0x7f30d4071a10 2024-09-16T01:50:44.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.825+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v52) ==== 54+0+16 (secure 0 0 0) 0x7f30c40abba0 con 0x7f30d4071a10 2024-09-16T01:50:44.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.825+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f30b406c2b0 0x7f30b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.825+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f30d4148510 0x7f30d4142a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.825+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30d415aa90 con 0x7f30d4148510 2024-09-16T01:50:44.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.825+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f30d4148510 0x7f30d4142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.826+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f30b406c2b0 0x7f30b406e770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30d4151e80 tx=0x7f30c0007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.826+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f30d4148510 0x7f30d4142a80 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.827+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d415aa90 con 0x7f30d4148510 2024-09-16T01:50:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.842+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d415a7b0 con 0x7f30d4148510 2024-09-16T01:50:44.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.845+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d415a7b0 con 0x7f30d4148510 2024-09-16T01:50:44.847 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:44.847 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:44.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.845+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f30d4145b10 0x7f30d4145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.845+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f30d41434d0 con 0x7f30d4145b10 2024-09-16T01:50:44.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.845+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f30d4145b10 0x7f30d4145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.846+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f30d4145b10 0x7f30d4145f30 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.847+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d41434d0 con 0x7f30d4145b10 2024-09-16T01:50:44.850 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:44.848+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:44.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.870+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d41f1340 con 0x7f30d4145b10 2024-09-16T01:50:44.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.873+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d41f1340 con 0x7f30d4145b10 2024-09-16T01:50:44.875 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.874+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f30d4062630 0x7f30d4062a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.874+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f30d4148b70 con 0x7f30d4062630 2024-09-16T01:50:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.874+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f30d4062630 0x7f30d4062a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.875+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f30d4062630 0x7f30d4062a10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.876+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d4148b70 con 0x7f30d4062630 2024-09-16T01:50:44.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.890+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d4146980 con 0x7f30d4062630 2024-09-16T01:50:44.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.892+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d4146980 con 0x7f30d4062630 2024-09-16T01:50:44.895 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:44.895 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:44.895 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:44.896 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:44.896 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:44.896 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:44.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.893+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f30d4146e70 0x7f30d41472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.893+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f30d40635d0 con 0x7f30d4146e70 2024-09-16T01:50:44.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.893+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f30d4146e70 0x7f30d41472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.894+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f30d4146e70 0x7f30d41472d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.895+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d40635d0 con 0x7f30d4146e70 2024-09-16T01:50:44.910 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:44.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.909+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d40635d0 con 0x7f30d4146e70 2024-09-16T01:50:44.915 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.911+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d40635d0 con 0x7f30d4146e70 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.911+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f30d402c640 0x7f30d402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.911+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f30d402cfa0 con 0x7f30d402c640 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.912+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f30d402c640 0x7f30d402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.912+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f30d402c640 0x7f30d402ca60 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.913+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d402cfa0 con 0x7f30d402c640 2024-09-16T01:50:44.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.928+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d402cfa0 con 0x7f30d402c640 2024-09-16T01:50:44.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.930+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d402cfa0 con 0x7f30d402c640 2024-09-16T01:50:44.933 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:44.933 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:44.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.931+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f30d402d620 0x7f30d4206fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.931+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f30d402da40 con 0x7f30d402d620 2024-09-16T01:50:44.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.931+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f30d402d620 0x7f30d4206fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.932+0000 7f30d2000640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f30d402d620 0x7f30d4206fb0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.932+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d402da40 con 0x7f30d402d620 2024-09-16T01:50:44.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.947+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d402da40 con 0x7f30d402d620 2024-09-16T01:50:44.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.949+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d402da40 con 0x7f30d402d620 2024-09-16T01:50:44.951 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:44.951 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:44.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.949+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f30d4207a00 0x7f30d4209df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.949+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f30d420cc70 con 0x7f30d4207a00 2024-09-16T01:50:44.952 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:44.949+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.950+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f30d4207a00 0x7f30d4209df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.950+0000 7f30d3400640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f30d4207a00 0x7f30d4209df0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.951+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d420cc70 con 0x7f30d4207a00 2024-09-16T01:50:44.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.966+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d420a600 con 0x7f30d4207a00 2024-09-16T01:50:44.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.968+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d420a600 con 0x7f30d4207a00 2024-09-16T01:50:44.970 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:44.971 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:44.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.968+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f30d420ad20 0x7f30d420b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:44.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.968+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f30d4210040 con 0x7f30d420ad20 2024-09-16T01:50:44.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.969+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f30d420ad20 0x7f30d420b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:44.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.969+0000 7f30d2a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f30d420ad20 0x7f30d420b160 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:44.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.970+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30d4210040 con 0x7f30d420ad20 2024-09-16T01:50:44.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.983+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f30d420eec0 con 0x7f30d420ad20 2024-09-16T01:50:44.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.985+0000 7f30caa00640 1 -- 172.21.15.146:0/759142420 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f30d420eec0 con 0x7f30d420ad20 2024-09-16T01:50:44.988 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:44.989 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:44.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f30d4148510 msgr2=0x7f30d4142a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f30d4148510 0x7f30d4142a80 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f30d4145b10 msgr2=0x7f30d4145f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f30d4145b10 0x7f30d4145f30 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f30d4062630 msgr2=0x7f30d4062a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f30d4062630 0x7f30d4062a10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f30d4146e70 msgr2=0x7f30d41472d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f30d4146e70 0x7f30d41472d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f30d402c640 msgr2=0x7f30d402ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f30d402c640 0x7f30d402ca60 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f30d402d620 msgr2=0x7f30d4206fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f30d402d620 0x7f30d4206fb0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f30d4207a00 msgr2=0x7f30d4209df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f30d4207a00 0x7f30d4209df0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f30d420ad20 msgr2=0x7f30d420b160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f30d420ad20 0x7f30d420b160 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d3400640 1 -- 172.21.15.146:0/759142420 reap_dead start 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d3400640 1 -- 172.21.15.146:0/759142420 reap_dead start 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f30b406c2b0 msgr2=0x7f30b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f30b406c2b0 0x7f30b406e770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30d4151e80 tx=0x7f30c0007e80 comp rx=0 tx=0).stop 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d4071a10 msgr2=0x7f30d4153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 --2- 172.21.15.146:0/759142420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d4071a10 0x7f30d4153e50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f30c40098c0 tx=0x7f30c40046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d3400640 1 -- 172.21.15.146:0/759142420 reap_dead start 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d3400640 1 -- 172.21.15.146:0/759142420 reap_dead start 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d3400640 1 -- 172.21.15.146:0/759142420 reap_dead start 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 shutdown_connections 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 >> 172.21.15.146:0/759142420 conn(0x7f30d406bc10 msgr2=0x7f30d406fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 shutdown_connections 2024-09-16T01:50:44.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:44.986+0000 7f30d9a00640 1 -- 172.21.15.146:0/759142420 wait complete. 2024-09-16T01:50:45.051 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:45.051+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:45.077 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:45.077 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:45.078 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:45.078 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:45.078 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:45.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:45.153 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:45.152+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:45.253 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:45.252+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 117.486286359, 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 117.05473262700001, 2024-09-16T01:50:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 117.03480840500001, 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 117.034788487, 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 117.034727903, 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.788442383, 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.78829186900001, 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:45.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.78827230100001, 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:45.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:45.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:45.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:45.310 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.218116087, 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:45.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:45.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.217948269, 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:45.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:45.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.217929287, 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:45.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:45.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:45.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.217909656, 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.329 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:45.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:45.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.334 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21777575, 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:45.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:45.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.340 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21776421200001, 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:45.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.21774580100001, 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:45.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:45.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:45.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.217631131, 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.353 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:45.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:45.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:45.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:45.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.20684940300001, 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:45.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.206732745, 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.365 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:45.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:45.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.391199506000007, 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:45.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.391138167999998, 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:45.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:45.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:45.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:45.381 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:45.383 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:45.383 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:45.381+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:45.483 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:45.483+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:45.583 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:45.583 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:45.584 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:45.584 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:45.584 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:45.582+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:45.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:45.684 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:45.684+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:45.723 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:50:45.786 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:45.784+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:45.798 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:45.798 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:45.799 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:45.799 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:45.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:45.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.880+0000 7f5abc600640 1 Processor -- start 2024-09-16T01:50:45.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.880+0000 7f5abc600640 1 -- start start 2024-09-16T01:50:45.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.881+0000 7f5abc600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5ab40cc0c0 0x7f5ab415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:45.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.881+0000 7f5abc600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab415b1d0 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.881+0000 7f5abc600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab415b4b0 con 0x7f5ab40cc4a0 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.881+0000 7f5abc600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5ab40cbd00 con 0x7f5ab40cc0c0 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.881+0000 7f5ab9200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5ab40cb2f0 0x7f5ab415e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53416/0 (socket says 172.21.15.146:53416) 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.881+0000 7f5ab9200640 1 -- 172.21.15.146:0/2009891494 learned_addr learned my addr 172.21.15.146:0/2009891494 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.882+0000 7f5ab9c00640 1 --2- 172.21.15.146:0/2009891494 >> v2:172.21.15.161:3300/0 conn(0x7f5ab40cc0c0 0x7f5ab415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.882+0000 7f5ab9c00640 1 -- 172.21.15.146:0/2009891494 >> v1:172.21.15.161:6789/0 conn(0x7f5ab40cc4a0 legacy=0x7f5ab4165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.882+0000 7f5ab9c00640 1 -- 172.21.15.146:0/2009891494 >> v1:172.21.15.146:6789/0 conn(0x7f5ab40cb2f0 legacy=0x7f5ab415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:45.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.882+0000 7f5ab3e00640 1 -- 172.21.15.146:0/2009891494 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3173407982 0 0) 0x7f5ab415b1d0 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.882+0000 7f5ab9c00640 1 -- 172.21.15.146:0/2009891494 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab415b790 con 0x7f5ab40cc0c0 2024-09-16T01:50:45.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.883+0000 7f5ab9c00640 1 --2- 172.21.15.146:0/2009891494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc0c0 0x7f5ab415aab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80097a0 tx=0x7f5aa802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af74c97bfadf7302 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:45.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5ab3e00640 1 -- 172.21.15.146:0/2009891494 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8037070 con 0x7f5ab40cc0c0 2024-09-16T01:50:45.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5ab3e00640 1 -- 172.21.15.146:0/2009891494 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa8032440 con 0x7f5ab40cc0c0 2024-09-16T01:50:45.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5ab3e00640 1 -- 172.21.15.146:0/2009891494 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8032810 con 0x7f5ab40cc0c0 2024-09-16T01:50:45.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 -- 172.21.15.146:0/2009891494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc0c0 msgr2=0x7f5ab415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:45.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 --2- 172.21.15.146:0/2009891494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc0c0 0x7f5ab415aab0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80097a0 tx=0x7f5aa802a230 comp rx=0 tx=0).stop 2024-09-16T01:50:45.887 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:45.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 -- 172.21.15.146:0/2009891494 shutdown_connections 2024-09-16T01:50:45.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 --2- 172.21.15.146:0/2009891494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc0c0 0x7f5ab415aab0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:45.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 -- 172.21.15.146:0/2009891494 >> 172.21.15.146:0/2009891494 conn(0x7f5ab406d560 msgr2=0x7f5ab406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:45.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 -- 172.21.15.146:0/2009891494 shutdown_connections 2024-09-16T01:50:45.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.884+0000 7f5abc600640 1 -- 172.21.15.146:0/2009891494 wait complete. 2024-09-16T01:50:45.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.886+0000 7f5abc600640 1 Processor -- start 2024-09-16T01:50:45.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.886+0000 7f5abc600640 1 -- start start 2024-09-16T01:50:45.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5abc600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 0x7f5ab414f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:45.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5abc600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ab40cc0c0 0x7f5ab40c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:45.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5abc600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc4a0 0x7f5ab40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5abc600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab415e540 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5abc600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab415ba70 con 0x7f5ab40cc4a0 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5abc600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab40cc8b0 con 0x7f5ab40cc0c0 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5ab9c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 0x7f5ab414f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5ab9c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 0x7f5ab414f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38820/0 (socket says 172.21.15.146:38820) 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5ab9c00640 1 -- 172.21.15.146:0/740610140 learned_addr learned my addr 172.21.15.146:0/740610140 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5ab9200640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ab40cc0c0 0x7f5ab40c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:45.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.887+0000 7f5aba600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc4a0 0x7f5ab40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:45.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5ab9c00640 1 -- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ab40cc0c0 msgr2=0x7f5ab40c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:45.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5ab9c00640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ab40cc0c0 0x7f5ab40c8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:45.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5ab9c00640 1 -- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc4a0 msgr2=0x7f5ab40c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:45.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5ab9c00640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc4a0 0x7f5ab40c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:45.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5ab9c00640 1 -- 172.21.15.146:0/740610140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab40c8180 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5aba600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc4a0 0x7f5ab40c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.888+0000 7f5ab9c00640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 0x7f5ab414f8b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80098c0 tx=0x7f5aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.889+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8037040 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.889+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa8042070 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.889+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8030bd0 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.889+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab414f310 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.889+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41f1860 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.890+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab40cf100 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.891+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5aa8047460 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.892+0000 7f5ab1600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a9406c2b0 0x7f5a9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:45.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.892+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5828+0+0 (secure 0 0 0) 0x7f5aa80abfc0 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.893+0000 7f5ab9200640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a9406c2b0 0x7f5a9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:45.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.894+0000 7f5ab9200640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a9406c2b0 0x7f5a9406e770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4152e20 tx=0x7f5aa0007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:45.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:45.897+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5ab40cf100 con 0x7f5ab40cb2f0 2024-09-16T01:50:45.988 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:45.987+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.020 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.19240005499999, 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054498039999999999, 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824720+0000", 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824728+0000", 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2400000000000007e-06 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824738+0000", 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0346999999999999e-05 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824757+0000", 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8195000000000001e-05 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824769+0000", 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2369e-05 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824972+0000", 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203248 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825164+0000", 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019228499999999999 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825370+0000", 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000205906 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829171+0000", 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038009799999999998 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830048+0000", 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087659499999999998 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830121+0000", 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2797000000000001e-05 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830170+0000", 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8841999999999998e-05 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:46.026 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.192285234, 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:46.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:46.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:46.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.19224336400001, 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:46.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:46.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:46.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.039 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.192199085, 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:46.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:46.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.045 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.19217020799999, 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.188463001, 2024-09-16T01:50:46.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:46.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:46.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "pg_update_log_missing(2.b epoch 22/19 rep_tid 4 entries 22'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.18840901999999, 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005476084, 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828711+0000", 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1199999999999998e-07 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828718+0000", 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1960000000000002e-06 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828735+0000", 2024-09-16T01:50:46.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6552999999999999e-05 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828747+0000", 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1821000000000001e-05 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829880+0000", 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001133362 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834187+0000", 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043067399999999999 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.152993796000004, 2024-09-16T01:50:46.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:46.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:46.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:46.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.149933646999997, 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.064 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:46.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:46.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.068 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.149074198999998, 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:46.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.149020305999997, 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:46.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:46.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:46.080 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.142996332999999, 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:46.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.142946205000001, 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:46.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:46.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.142869872000006, 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:46.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.142838517000001, 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:46.095 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:46.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.099 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:9 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'4, mlcod=22'2)", 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 85.104753062, 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035782140000000001, 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912367+0000", 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912368+0000", 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.181e-06 2024-09-16T01:50:46.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912378+0000", 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0029e-05 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912402+0000", 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3821000000000001e-05 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912412+0000", 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0059e-05 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912567+0000", 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015538999999999999 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912608+0000", 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0884999999999999e-05 2024-09-16T01:50:46.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915900+0000", 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032915240000000001 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915945+0000", 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5324999999999997e-05 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.3.13:10 2.3 e22/19 2:c04be1b2:::621.00000000:head v 22'5, mlcod=22'2)", 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 85.103961530999996, 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038561530000000002, 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913158+0000", 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913159+0000", 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6100000000000001e-07 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6500000000000001e-06 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913182+0000", 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7028e-05 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.913189+0000", 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1520000000000003e-06 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915486+0000", 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022975449999999998 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915505+0000", 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8346e-05 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.916976+0000", 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014711699999999999 2024-09-16T01:50:46.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.917014+0000", 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8701000000000001e-05 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:17 2.14 2:2eebc8f4:::62f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.079879706, 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003426424, 2024-09-16T01:50:46.109 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937240+0000", 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937242+0000", 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.874e-06 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937250+0000", 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9750000000000007e-06 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937271+0000", 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1444e-05 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937281+0000", 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0064000000000001e-05 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937407+0000", 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126031 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937568+0000", 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016096 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937719+0000", 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015070900000000001 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940327+0000", 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026074029999999999 2024-09-16T01:50:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940540+0000", 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021321400000000001 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940614+0000", 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4691999999999996e-05 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940667+0000", 2024-09-16T01:50:46.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2058000000000002e-05 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:18 2.14 2:2eebc8f4:::62f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.079807794000004, 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038065210000000002, 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937312+0000", 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937313+0000", 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0999999999999998e-07 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937318+0000", 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7459999999999997e-06 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937330+0000", 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1234e-05 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.937336+0000", 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4840000000000001e-06 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938018+0000", 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068215699999999997 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938088+0000", 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9908999999999998e-05 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938222+0000", 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134177 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.940677+0000", 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002454482 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941011+0000", 2024-09-16T01:50:46.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033437400000000002 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941069+0000", 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8106000000000002e-05 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941119+0000", 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9242000000000001e-05 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.749165394000002, 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:46.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:46.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.127 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:46.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:46.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.088+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5ab40cb7d0 con 0x7f5ab40cb2f0 2024-09-16T01:50:46.129 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:46.127+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:46.229 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:46.228+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:46.301 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:46.301 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:46.301 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:46.301 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:46.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:46.330 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:46.328+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:46.431 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:46.429+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:46.492 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:46.492 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:46.492 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:46.505 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:46.531 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:46.530+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:46.632 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:46.631+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.712 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.88393831, 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063070110000000004, 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825666+0000", 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825667+0000", 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825672+0000", 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6920000000000001e-06 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825689+0000", 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7039999999999999e-05 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825696+0000", 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5909999999999998e-06 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826617+0000", 2024-09-16T01:50:46.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00092117800000000004 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826697+0000", 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9598999999999995e-05 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826832+0000", 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013548 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.829938+0000", 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031059059999999999 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831894+0000", 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019552240000000002 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831942+0000", 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8389e-05 2024-09-16T01:50:46.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.831973+0000", 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1043000000000001e-05 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:29 2.b 2:d0630e4c:::mds0_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.88389676200001, 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087271810000000005, 2024-09-16T01:50:46.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825708+0000", 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4300000000000003e-07 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825734+0000", 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5638000000000001e-05 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825743+0000", 2024-09-16T01:50:46.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7350000000000002e-06 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825750+0000", 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827333+0000", 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015834689999999999 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827472+0000", 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138796 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827572+0000", 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9910999999999999e-05 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832328+0000", 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047559079999999997 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834353+0000", 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002025654 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834392+0000", 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8099000000000002e-05 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834435+0000", 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3169e-05 2024-09-16T01:50:46.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:34 2.b 2:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.883828501, 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088227679999999999, 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825776+0000", 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825777+0000", 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9500000000000001e-07 2024-09-16T01:50:46.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825781+0000", 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0350000000000003e-06 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825788+0000", 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5089999999999996e-06 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.825794+0000", 2024-09-16T01:50:46.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4240000000000001e-06 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827917+0000", 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021233570000000002 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828052+0000", 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013528300000000001 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834599+0000", 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065463650000000002 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843330636000005, 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012518068, 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.728 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866274+0000", 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866276+0000", 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866303+0000", 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7229e-05 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866329+0000", 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.563e-05 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866343+0000", 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4287999999999999e-05 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867782+0000", 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0014384510000000001 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867911+0000", 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012957999999999999 2024-09-16T01:50:46.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868028+0000", 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117025 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877878+0000", 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098499209999999993 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878708+0000", 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00082986900000000001 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878752+0000", 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3735000000000001e-05 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878792+0000", 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0025999999999999e-05 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843230356000007, 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014487969, 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866374+0000", 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866375+0000", 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9899999999999999e-07 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866379+0000", 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6519999999999997e-06 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866388+0000", 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9579999999999996e-06 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866395+0000", 2024-09-16T01:50:46.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7750000000000004e-06 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868287+0000", 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001892344 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868360+0000", 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3023999999999997e-05 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:46.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868473+0000", 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112321 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878461+0000", 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099884610000000006 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880760+0000", 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022984479999999998 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880823+0000", 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3276999999999997e-05 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880862+0000", 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.739 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:7 2.b 2:d70a01c6:::60c.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843200318000001, 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020806099000000002, 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866404+0000", 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.82e-07 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866408+0000", 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0110000000000002e-06 2024-09-16T01:50:46.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866415+0000", 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5830000000000003e-06 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866421+0000", 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6659999999999996e-06 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:46.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022638100000000002 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868842+0000", 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015727699999999999 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868942+0000", 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100182 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878519+0000", 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095771399999999996 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887083+0000", 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085641750000000003 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887155+0000", 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2117000000000001e-05 2024-09-16T01:50:46.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887210+0000", 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4656e-05 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:8 2.b 2:d70a01c6:::60c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843161926999997, 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020917458, 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866442+0000", 2024-09-16T01:50:46.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866443+0000", 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866447+0000", 2024-09-16T01:50:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8340000000000001e-06 2024-09-16T01:50:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866455+0000", 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866460+0000", 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5629999999999999e-06 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869260+0000", 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027996039999999998 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869328+0000", 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8730000000000001e-05 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869461+0000", 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264099999999999 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878525+0000", 2024-09-16T01:50:46.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0090635630000000002 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887259+0000", 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087339310000000003 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887305+0000", 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6538000000000001e-05 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887360+0000", 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4706000000000001e-05 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:10 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843135344999993, 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01479957, 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.754 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866469+0000", 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8100000000000003e-07 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866473+0000", 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1089999999999998e-06 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866481+0000", 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5299999999999996e-06 2024-09-16T01:50:46.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866486+0000", 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.566e-06 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870818+0000", 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004332519 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870937+0000", 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011880899999999999 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871093+0000", 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015636400000000001 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877793+0000", 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066997560000000003 2024-09-16T01:50:46.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881167+0000", 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033735029999999999 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881242+0000", 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5139e-05 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881269+0000", 2024-09-16T01:50:46.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6794000000000001e-05 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:27 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843097572000005, 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014854865, 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.760 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866507+0000", 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866511+0000", 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7479999999999999e-06 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866518+0000", 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:50:46.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866525+0000", 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2750000000000001e-06 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875004+0000", 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084799100000000002 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875082+0000", 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7698e-05 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875190+0000", 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010785000000000001 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.877830+0000", 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026396480000000001 2024-09-16T01:50:46.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881273+0000", 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034436839999999998 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881341+0000", 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7359000000000003e-05 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881362+0000", 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0928e-05 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:33 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843072133999996, 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020942512, 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.766 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866532+0000", 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866533+0000", 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.27e-07 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866552+0000", 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8921999999999999e-05 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866559+0000", 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2570000000000001e-06 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866564+0000", 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0540000000000002e-06 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869714+0000", 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031497920000000002 2024-09-16T01:50:46.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869828+0000", 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114331 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869968+0000", 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013971799999999999 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880969+0000", 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011001031 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887401+0000", 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064320319999999999 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887441+0000", 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9937000000000001e-05 2024-09-16T01:50:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887475+0000", 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4010999999999997e-05 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:2 2.5 2:a6e36b3f:::401.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.772 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.843033149999997, 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014499972999999999, 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866571+0000", 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866572+0000", 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8400000000000005e-07 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866575+0000", 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0929999999999999e-06 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866592+0000", 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7765999999999999e-05 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866598+0000", 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4219999999999999e-06 2024-09-16T01:50:46.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871180+0000", 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045822939999999998 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871272+0000", 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1308000000000001e-05 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:46.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.871419+0000", 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014714 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875993+0000", 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004574486 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881013+0000", 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050202240000000002 2024-09-16T01:50:46.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881046+0000", 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2944000000000002e-05 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881071+0000", 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4811999999999999e-05 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.778 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:18 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.842985433999999, 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020977030000000001, 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866619+0000", 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1799999999999995e-07 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866623+0000", 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4709999999999999e-06 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866630+0000", 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1590000000000004e-06 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866635+0000", 2024-09-16T01:50:46.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3079999999999998e-06 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870303+0000", 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036679210000000002 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870398+0000", 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4464999999999998e-05 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:46.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870501+0000", 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010291800000000001 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.880977+0000", 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010476014000000001 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887510+0000", 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065331759999999999 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887553+0000", 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2737000000000001e-05 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.887596+0000", 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3343000000000002e-05 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.784 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:22 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.842961054, 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01481146, 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866643+0000", 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866644+0000", 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-07 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866646+0000", 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2139999999999999e-06 2024-09-16T01:50:46.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866652+0000", 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2160000000000003e-06 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866677+0000", 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4663000000000001e-05 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876491+0000", 2024-09-16T01:50:46.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098146110000000009 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.876616+0000", 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012430699999999999 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.881455+0000", 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048389840000000002 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:24 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.842919765999994, 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016598702999999999, 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866685+0000", 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5400000000000002e-07 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866688+0000", 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9909999999999998e-06 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866695+0000", 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.883e-06 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866700+0000", 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7280000000000003e-06 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868221+0000", 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015209620000000001 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868334+0000", 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113584 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883283+0000", 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014949100999999999 2024-09-16T01:50:46.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:25 2.0 2:0250a5a6:::201.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.842885140000007, 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011987815000000001, 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866719+0000", 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866720+0000", 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6300000000000001e-07 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866738+0000", 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8218000000000001e-05 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866745+0000", 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7279999999999995e-06 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.866751+0000", 2024-09-16T01:50:46.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.66e-06 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868589+0000", 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018382730000000001 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868709+0000", 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119571 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.878707+0000", 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099980020000000006 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.2.11:5 2.0 2:03625d61:::614.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 91.814175650999999, 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081102819999999999, 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.797 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895429+0000", 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895434+0000", 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4909999999999999e-06 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895444+0000", 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0560000000000001e-05 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.895451+0000", 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5590000000000002e-06 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896385+0000", 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093368200000000002 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896454+0000", 2024-09-16T01:50:46.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9567000000000007e-05 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.896559+0000", 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104609 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.899222+0000", 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026632000000000001 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903485+0000", 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042627999999999997 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903517+0000", 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2345999999999998e-05 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:14.903539+0000", 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1838e-05 2024-09-16T01:50:46.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:14 2.1a 2:594e21d0:::623.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 85.797769411999994, 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068542239999999999, 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.803 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911835+0000", 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911838+0000", 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.435e-06 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911844+0000", 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.122e-06 2024-09-16T01:50:46.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911849+0000", 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6979999999999997e-06 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914481+0000", 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026327249999999998 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914555+0000", 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3886000000000004e-05 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.914697+0000", 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014193800000000001 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918088+0000", 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033908559999999998 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918610+0000", 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052154100000000004 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918656+0000", 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6499999999999999e-05 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918689+0000", 2024-09-16T01:50:46.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3052000000000001e-05 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.3.13:23 2.18 2:1c1db442:::103.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 85.797635865999993, 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067043830000000004, 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911968+0000", 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911969+0000", 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7100000000000002e-07 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911980+0000", 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1090999999999999e-05 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911989+0000", 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6270000000000006e-06 2024-09-16T01:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.911994+0000", 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3639999999999998e-06 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912099+0000", 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010514900000000001 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912210+0000", 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000111346 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.912342+0000", 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000131197 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.915846+0000", 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035043269999999998 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918573+0000", 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027272289999999999 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918632+0000", 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8854000000000003e-05 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:20.918673+0000", 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0728e-05 2024-09-16T01:50:46.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.773036536000006, 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084942400000000001, 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936568+0000", 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5300000000000004e-07 2024-09-16T01:50:46.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936594+0000", 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5137999999999999e-05 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936602+0000", 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8999999999999995e-06 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936609+0000", 2024-09-16T01:50:46.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9480000000000002e-06 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938373+0000", 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001763857 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938479+0000", 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105816 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:46.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938622+0000", 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014314699999999999 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941905+0000", 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032832 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.944943+0000", 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030378670000000001 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945019+0000", 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5532999999999997e-05 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945062+0000", 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3180999999999999e-05 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.4.15:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 80.772971037000005, 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085927369999999996, 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:46.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936633+0000", 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936634+0000", 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.44e-07 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936637+0000", 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.287e-06 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936647+0000", 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7189999999999996e-06 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.936653+0000", 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.319e-06 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938867+0000", 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022140079999999999 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.938935+0000", 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7644999999999994e-05 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.939072+0000", 2024-09-16T01:50:46.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013753600000000001 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.941912+0000", 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028393350000000001 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945119+0000", 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032077160000000002 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945183+0000", 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3723e-05 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:25.945226+0000", 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2805000000000001e-05 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:46.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:46.827 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:46.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.711+0000 7f5ab1600640 1 -- 172.21.15.146:0/740610140 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v53) ==== 91+0+0 (secure 0 0 0) 0x7f5ab40cb7d0 con 0x7f5ab40cb2f0 2024-09-16T01:50:46.828 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:50:46.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a9406c2b0 msgr2=0x7f5a9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:46.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a9406c2b0 0x7f5a9406e770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4152e20 tx=0x7f5aa0007e40 comp rx=0 tx=0).stop 2024-09-16T01:50:46.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 msgr2=0x7f5ab414f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 0x7f5ab414f8b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80098c0 tx=0x7f5aa80046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 shutdown_connections 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5a9406c2b0 0x7f5a9406e770 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ab40cc4a0 0x7f5ab40c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5ab40cc0c0 0x7f5ab40c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 --2- 172.21.15.146:0/740610140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40cb2f0 0x7f5ab414f8b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 >> 172.21.15.146:0/740610140 conn(0x7f5ab406d560 msgr2=0x7f5ab40d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 shutdown_connections 2024-09-16T01:50:46.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:46.714+0000 7f5abc600640 1 -- 172.21.15.146:0/740610140 wait complete. 2024-09-16T01:50:46.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:46.832 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:46.826+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:46.928 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:46.927+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:46.948 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:46.948 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:46.949 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:46.949 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:46.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:47.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:47.029 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:47.028+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:47.103 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:47.103 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:47.103 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:47.103 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:47.103 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:47.112 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:47.130 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:47.129+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:47.139 INFO:tasks.workunit.client.0.smithi146.stderr:Updating files: 63% (8128/12889) Updating files: 64% (8249/12889) Updating files: 65% (8378/12889) Updating files: 66% (8507/12889) Updating files: 67% (8636/12889) Updating files: 68% (8765/12889) Updating files: 69% (8894/12889) Updating files: 70% (9023/12889) Updating files: 71% (9152/12889) Updating files: 72% (9281/12889) Updating files: 73% (9409/12889) Updating files: 74% (9538/12889) Updating files: 75% (9667/12889) Updating files: 76% (9796/12889) Updating files: 77% (9925/12889) Updating files: 78% (10054/12889) Updating files: 79% (10183/12889) Updating files: 80% (10312/12889) Updating files: 81% (10441/12889) Updating files: 82% (10569/12889) Updating files: 83% (10698/12889) Updating files: 84% (10827/12889) Updating files: 85% (10956/12889) Updating files: 86% (11085/12889) Updating files: 87% (11214/12889) Updating files: 88% (11343/12889) Updating files: 89% (11472/12889) Updating files: 90% (11601/12889) Updating files: 91% (11729/12889) Updating files: 92% (11858/12889) Updating files: 93% (11987/12889) Updating files: 94% (12116/12889) Updating files: 95% (12245/12889) Updating files: 96% (12374/12889) Updating files: 97% (12503/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-16T01:50:47.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.156+0000 7f420a000640 1 Processor -- start 2024-09-16T01:50:47.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.156+0000 7f420a000640 1 -- start start 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f420a000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4204152c30 0x7f42040c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f420a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42040c4240 con 0x7f42040c6e90 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f420a000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42040c4500 con 0x7f42040c6ab0 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f420a000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f42040c4da0 con 0x7f4204152c30 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4203400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4204152c30 0x7f42040c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4203e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f42040c6e90 0x7f42040c7ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53432/0 (socket says 172.21.15.146:53432) 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4203e00640 1 -- 172.21.15.146:0/894038981 learned_addr learned my addr 172.21.15.146:0/894038981 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:47.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4202a00640 1 --1- 172.21.15.146:0/894038981 >> v1:172.21.15.161:6789/0 conn(0x7f42040c6ab0 0x7f42040c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34214/0 (socket says 172.21.15.146:34214) 2024-09-16T01:50:47.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4203400640 1 -- 172.21.15.146:0/894038981 >> v1:172.21.15.161:6789/0 conn(0x7f42040c6ab0 legacy=0x7f42040c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4203400640 1 -- 172.21.15.146:0/894038981 >> v1:172.21.15.146:6789/0 conn(0x7f42040c6e90 legacy=0x7f42040c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.158+0000 7f4202000640 1 -- 172.21.15.146:0/894038981 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1555535452 0 0) 0x7f42040c4240 con 0x7f42040c6e90 2024-09-16T01:50:47.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.159+0000 7f4203400640 1 -- 172.21.15.146:0/894038981 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42040c47e0 con 0x7f4204152c30 2024-09-16T01:50:47.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f4203400640 1 --2- 172.21.15.146:0/894038981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4204152c30 0x7f42040c36b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097d0 tx=0x7f41f402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a39534e9a3377cfb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f4202000640 1 -- 172.21.15.146:0/894038981 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4037070 con 0x7f4204152c30 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f4202000640 1 -- 172.21.15.146:0/894038981 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f4032440 con 0x7f4204152c30 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f4202000640 1 -- 172.21.15.146:0/894038981 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4032810 con 0x7f4204152c30 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f420a000640 1 -- 172.21.15.146:0/894038981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4204152c30 msgr2=0x7f42040c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f420a000640 1 --2- 172.21.15.146:0/894038981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4204152c30 0x7f42040c36b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41f40097d0 tx=0x7f41f402a210 comp rx=0 tx=0).stop 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f420a000640 1 -- 172.21.15.146:0/894038981 shutdown_connections 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f420a000640 1 --2- 172.21.15.146:0/894038981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4204152c30 0x7f42040c36b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.160+0000 7f420a000640 1 -- 172.21.15.146:0/894038981 >> 172.21.15.146:0/894038981 conn(0x7f420406bc10 msgr2=0x7f420406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.161+0000 7f420a000640 1 -- 172.21.15.146:0/894038981 shutdown_connections 2024-09-16T01:50:47.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.161+0000 7f420a000640 1 -- 172.21.15.146:0/894038981 wait complete. 2024-09-16T01:50:47.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.162+0000 7f420a000640 1 Processor -- start 2024-09-16T01:50:47.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.162+0000 7f420a000640 1 -- start start 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f420a000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 0x7f42041f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f420a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42040c6e90 0x7f42041f1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f420a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4204152c30 0x7f42041f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f420a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42040c9350 con 0x7f42040c6e90 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f420a000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42040c6630 con 0x7f4204152c30 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f420a000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f42040c94d0 con 0x7f42040c6ab0 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f4202a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42040c6e90 0x7f42041f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f4203400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 0x7f42041f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f4203e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4204152c30 0x7f42041f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f4203400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 0x7f42041f0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37316/0 (socket says 172.21.15.146:37316) 2024-09-16T01:50:47.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f4202a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42040c6e90 0x7f42041f1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38826/0 (socket says 172.21.15.146:38826) 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.163+0000 7f4203400640 1 -- 172.21.15.146:0/3971894321 learned_addr learned my addr 172.21.15.146:0/3971894321 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f4203400640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4204152c30 msgr2=0x7f42041f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4204152c30 0x7f42041f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f4203400640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42040c6e90 msgr2=0x7f42041f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42040c6e90 0x7f42041f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f4203400640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42040cb9a0 con 0x7f42040c6ab0 2024-09-16T01:50:47.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 0x7f42041f0c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41f40098f0 tx=0x7f41f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4037040 con 0x7f42040c6ab0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f4042070 con 0x7f42040c6ab0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42041f2a00 con 0x7f42040c6ab0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4030bb0 con 0x7f42040c6ab0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.164+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4204143290 con 0x7f42040c6ab0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.165+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f42040c36b0 con 0x7f42040c6ab0 2024-09-16T01:50:47.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.166+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f41f4047460 con 0x7f42040c6ab0 2024-09-16T01:50:47.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.167+0000 7f41fb400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41e406c2e0 0x7f41e406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.167+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(53..53 src has 1..53) ==== 5828+0+0 (secure 0 0 0) 0x7f41f40abfe0 con 0x7f42040c6ab0 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.168+0000 7f4202a00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41e406c2e0 0x7f41e406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.167+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v53) ==== 54+0+16 (secure 0 0 0) 0x7f41f4032e20 con 0x7f42040c6ab0 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.168+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f42041f22e0 0x7f42040cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.168+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4204073a20 con 0x7f42041f22e0 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.168+0000 7f4203e00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f42041f22e0 0x7f42040cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.168+0000 7f4202a00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41e406c2e0 0x7f41e406e7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f42041e8f00 tx=0x7f41f0007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.169+0000 7f4203e00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f42041f22e0 0x7f42040cb0c0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.169+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4204073a20 con 0x7f42041f22e0 2024-09-16T01:50:47.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.184+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f42040c7530 con 0x7f42041f22e0 2024-09-16T01:50:47.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.186+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f42040c7530 con 0x7f42041f22e0 2024-09-16T01:50:47.188 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:47.188 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:47.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.186+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f420413ef20 0x7f4204062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.186+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f42041491f0 con 0x7f420413ef20 2024-09-16T01:50:47.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.186+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f420413ef20 0x7f4204062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.187+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f420413ef20 0x7f4204062630 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.187+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42041491f0 con 0x7f420413ef20 2024-09-16T01:50:47.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.210+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f42040c9050 con 0x7f420413ef20 2024-09-16T01:50:47.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.212+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f42040c9050 con 0x7f420413ef20 2024-09-16T01:50:47.215 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:47.215 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:47.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.213+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4204063080 0x7f42040c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.213+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f42041f0440 con 0x7f4204063080 2024-09-16T01:50:47.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.213+0000 7f4202a00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4204063080 0x7f42040c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.213+0000 7f4202a00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4204063080 0x7f42040c00b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.214+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42041f0440 con 0x7f4204063080 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.218 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.390748843, 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0073505619999999997, 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825018+0000", 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825023+0000", 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1150000000000004e-06 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825036+0000", 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3022e-05 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825049+0000", 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3779e-05 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825058+0000", 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3620000000000006e-06 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825161+0000", 2024-09-16T01:50:47.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010298600000000001 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825279+0000", 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118087 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825418+0000", 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013899299999999999 2024-09-16T01:50:47.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829035+0000", 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036175130000000002 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832264+0000", 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003228908 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832322+0000", 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7751000000000003e-05 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.832369+0000", 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7046000000000003e-05 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.390661838, 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0091091320000000007, 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.225 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825106+0000", 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4900000000000005e-07 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825113+0000", 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.229e-06 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825125+0000", 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2493e-05 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825133+0000", 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6340000000000001e-06 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825778+0000", 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00064556499999999996 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825887+0000", 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108416 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826017+0000", 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013044 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829740+0000", 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037227200000000001 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834119+0000", 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043795930000000002 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834175+0000", 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5492999999999999e-05 2024-09-16T01:50:47.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834215+0000", 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9700000000000003e-05 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:47.232 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.390467626, 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061249809999999998, 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825300+0000", 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.34e-07 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825303+0000", 2024-09-16T01:50:47.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.96e-06 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825311+0000", 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6000000000000001e-06 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825317+0000", 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6139999999999997e-06 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.828779+0000", 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034614210000000001 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829036+0000", 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00025713999999999999 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829173+0000", 2024-09-16T01:50:47.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000136797 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831180+0000", 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020068859999999998 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831366+0000", 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018597399999999999 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831398+0000", 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2518999999999998e-05 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831425+0000", 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6636e-05 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:47.238 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.390441833, 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006225062, 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:47.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825326+0000", 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2e-07 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825335+0000", 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4339999999999998e-06 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825342+0000", 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6900000000000003e-06 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825348+0000", 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5470000000000004e-06 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829429+0000", 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040815720000000003 2024-09-16T01:50:47.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829541+0000", 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112285 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829647+0000", 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105592 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831435+0000", 2024-09-16T01:50:47.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017880330000000001 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831474+0000", 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8909999999999998e-05 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831518+0000", 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3516999999999997e-05 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831551+0000", 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3062000000000003e-05 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.390412118, 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062900040000000001, 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.244 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825355+0000", 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825356+0000", 2024-09-16T01:50:47.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9399999999999995e-07 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825359+0000", 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0620000000000001e-06 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825366+0000", 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6899999999999992e-06 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.825380+0000", 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3193000000000001e-05 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829978+0000", 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045982879999999999 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830056+0000", 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8466000000000004e-05 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.830226+0000", 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000169944 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831440+0000", 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012132790000000001 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831590+0000", 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015076400000000001 2024-09-16T01:50:47.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831621+0000", 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0440999999999999e-05 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.831645+0000", 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4383e-05 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.38945864, 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080128709999999995, 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:47.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826309+0000", 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826310+0000", 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3200000000000003e-07 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826315+0000", 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6980000000000003e-06 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826328+0000", 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2568999999999999e-05 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826336+0000", 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9670000000000001e-06 2024-09-16T01:50:47.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826916+0000", 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00058005899999999996 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827058+0000", 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000142533 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:47.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827196+0000", 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013736299999999999 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.829940+0000", 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027445600000000001 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834254+0000", 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0043136210000000001 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834292+0000", 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8266000000000003e-05 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834322+0000", 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9303e-05 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.255 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 103.38941273099999, 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080982150000000006, 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "started", 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826355+0000", 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5799999999999999e-07 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826368+0000", 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3152999999999999e-05 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826378+0000", 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.826385+0000", 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5440000000000004e-06 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827629+0000", 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001243886 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.827855+0000", 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000226615 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:03.834453+0000", 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006597548 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.259 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:14 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 98.355215643999998, 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013360132, 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860552+0000", 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6600000000000002e-07 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860560+0000", 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7919999999999993e-06 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860569+0000", 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3999999999999998e-06 2024-09-16T01:50:47.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860576+0000", 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8589999999999999e-06 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860646+0000", 2024-09-16T01:50:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9676000000000002e-05 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143559 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860917+0000", 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012731900000000001 2024-09-16T01:50:47.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862638+0000", 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017212670000000001 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873802+0000", 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011164074 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873866+0000", 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4381999999999994e-05 2024-09-16T01:50:47.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873912+0000", 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5337999999999999e-05 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:31 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:47.267 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 98.355027422000006, 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013335217, 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860740+0000", 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860741+0000", 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9899999999999999e-07 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860748+0000", 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3250000000000002e-06 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860761+0000", 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2958e-05 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860769+0000", 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2230000000000007e-06 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861199+0000", 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000429178 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861286+0000", 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7436999999999996e-05 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861415+0000", 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012907000000000001 2024-09-16T01:50:47.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863136+0000", 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001720627 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873957+0000", 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010821809999999999 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874029+0000", 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1641000000000001e-05 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874075+0000", 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6048999999999999e-05 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:19 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.273 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 98.354983250999993, 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013415967000000001, 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860784+0000", 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860785+0000", 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6300000000000005e-07 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860789+0000", 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5279999999999997e-06 2024-09-16T01:50:47.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860797+0000", 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.418e-06 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860802+0000", 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.344e-06 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861655+0000", 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085269000000000002 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861767+0000", 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011184900000000001 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861914+0000", 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014756700000000001 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.863730+0000", 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001815564 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874127+0000", 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010396924 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874163+0000", 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6285999999999998e-05 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874200+0000", 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7134000000000001e-05 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:6 2.1f 2:fa28013d:::615.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.279 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.319128798999998, 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006133548, 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896639+0000", 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896640+0000", 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6299999999999993e-07 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896645+0000", 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6540000000000004e-06 2024-09-16T01:50:47.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896655+0000", 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0316000000000001e-05 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.896664+0000", 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2090000000000005e-06 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897019+0000", 2024-09-16T01:50:47.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035541199999999999 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897088+0000", 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9388999999999996e-05 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897195+0000", 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010608299999999999 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.900354+0000", 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031591919999999999 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902602+0000", 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002248468 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902659+0000", 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6889999999999999e-05 2024-09-16T01:50:47.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.902772+0000", 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112972 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:9 2.6 2:62978f0a:::617.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.318337568999993, 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063594389999999997, 2024-09-16T01:50:47.285 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897430+0000", 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897431+0000", 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8899999999999998e-07 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897437+0000", 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3060000000000003e-06 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897447+0000", 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9920000000000002e-06 2024-09-16T01:50:47.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897455+0000", 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.627e-06 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897523+0000", 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7912999999999998e-05 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897652+0000", 2024-09-16T01:50:47.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000129259 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897777+0000", 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012468599999999999 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901149+0000", 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033725389999999999 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903683+0000", 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025335539999999999 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903730+0000", 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7339999999999997e-05 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903789+0000", 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9234000000000002e-05 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.290 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:10 2.6 2:62978f0a:::617.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.318044537999995, 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0061735380000000001, 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897723+0000", 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897724+0000", 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0139999999999999e-06 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897730+0000", 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7629999999999997e-06 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897740+0000", 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0499999999999999e-05 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897748+0000", 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1519999999999992e-06 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898076+0000", 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00032781800000000001 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898143+0000", 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6905999999999999e-05 2024-09-16T01:50:47.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898267+0000", 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123979 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.901274+0000", 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00300716 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903796+0000", 2024-09-16T01:50:47.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025223210000000001 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903859+0000", 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2638000000000006e-05 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903896+0000", 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7286999999999997e-05 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.2.11:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.317952822999999, 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060644100000000001, 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897815+0000", 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5099999999999999e-07 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897831+0000", 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5534000000000001e-05 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897850+0000", 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8825999999999999e-05 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.897858+0000", 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3939999999999996e-06 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898448+0000", 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059040600000000003 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898526+0000", 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7600000000000002e-05 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.898646+0000", 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011967400000000001 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903214+0000", 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045682589999999999 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903787+0000", 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00057291199999999997 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903842+0000", 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4747999999999998e-05 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:14.903879+0000", 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7305999999999998e-05 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:8 2.6 2:6642f7a8:::620.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 86.303952154000001, 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0071307610000000002, 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.302 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911815+0000", 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911816+0000", 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2299999999999999e-07 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911822+0000", 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0460000000000002e-06 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911846+0000", 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4193999999999999e-05 2024-09-16T01:50:47.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.911857+0000", 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.042e-05 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912714+0000", 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00085704200000000003 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912790+0000", 2024-09-16T01:50:47.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6019999999999994e-05 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912934+0000", 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014454 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916495+0000", 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035605770000000001 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918865+0000", 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002370451 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918906+0000", 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0123999999999997e-05 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918946+0000", 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.307 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:12 2.1f 2:ffe2d39d:::622.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 86.303509707000003, 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060451970000000004, 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912258+0000", 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912259+0000", 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0729999999999999e-06 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912265+0000", 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4409999999999998e-06 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912276+0000", 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0798000000000001e-05 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912286+0000", 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0176999999999999e-05 2024-09-16T01:50:47.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912581+0000", 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00029479199999999999 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912692+0000", 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110874 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:47.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913326+0000", 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00063410300000000001 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916590+0000", 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003264543 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918212+0000", 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0016214440000000001 2024-09-16T01:50:47.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918252+0000", 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0562999999999998e-05 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918303+0000", 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0389000000000002e-05 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:21 2.6 2:62eebe7c:::627.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 86.303467306000002, 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0067672629999999999, 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912300+0000", 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912301+0000", 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1999999999999999e-07 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912305+0000", 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9770000000000002e-06 2024-09-16T01:50:47.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912312+0000", 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9920000000000001e-06 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912318+0000", 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9089999999999997e-06 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913164+0000", 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084597199999999996 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913267+0000", 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103433 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913378+0000", 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011124300000000001 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.916553+0000", 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031750900000000002 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.918992+0000", 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00243837 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919023+0000", 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1498999999999998e-05 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919067+0000", 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4158e-05 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.3.13:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 86.303441417000002, 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0068352660000000004, 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.319 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912326+0000", 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8299999999999997e-07 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912336+0000", 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0079e-05 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912345+0000", 2024-09-16T01:50:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0639999999999994e-06 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.912351+0000", 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.686e-06 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913855+0000", 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0015034829999999999 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.913971+0000", 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011639 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.914107+0000", 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013549999999999999 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.917094+0000", 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002987132 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919106+0000", 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0020125389999999998 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919139+0000", 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2670999999999999e-05 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:20.919161+0000", 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2238999999999998e-05 2024-09-16T01:50:47.324 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:9 2.9 2:954a9a2b:::62b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 81.278197722000002, 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074502429999999996, 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937570+0000", 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937571+0000", 2024-09-16T01:50:47.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1480000000000001e-06 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937578+0000", 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1890000000000001e-06 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937593+0000", 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4613999999999999e-05 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937602+0000", 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937662+0000", 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9778000000000003e-05 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937799+0000", 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013690299999999999 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937967+0000", 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016826100000000001 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940308+0000", 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0023405750000000001 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944953+0000", 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046456860000000004 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.944987+0000", 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3997e-05 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945020+0000", 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2653000000000001e-05 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.330 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.4.15:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 81.278142776999999, 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080442459999999997, 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:47.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937625+0000", 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8999999999999996e-07 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937632+0000", 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7399999999999998e-06 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937640+0000", 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.354e-06 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.937647+0000", 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5050000000000004e-06 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939098+0000", 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001451473 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939454+0000", 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035574400000000001 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.939603+0000", 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000149228 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.940646+0000", 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001042821 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945559+0000", 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049129070000000002 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945623+0000", 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4072999999999995e-05 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:25.945669+0000", 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5711000000000003e-05 2024-09-16T01:50:47.335 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.336 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.336 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.336 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:47.336 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:47.336 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:47.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.228+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f42041434d0 con 0x7f4204063080 2024-09-16T01:50:47.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.231+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f42041434d0 con 0x7f4204063080 2024-09-16T01:50:47.337 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:47.338 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:47.339 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:47.336+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:47.339 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.231+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f42040c0d50 0x7f4204143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.231+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f42041434d0 con 0x7f42040c0d50 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.231+0000 7f4203e00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f42040c0d50 0x7f4204143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.232+0000 7f4203e00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f42040c0d50 0x7f4204143dc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.232+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42041434d0 con 0x7f42040c0d50 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.246+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f42041434d0 con 0x7f42040c0d50 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.248+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f42041434d0 con 0x7f42040c0d50 2024-09-16T01:50:47.340 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:47.341 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:47.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.248+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4204144ac0 0x7f420402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.248+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4204144f20 con 0x7f4204144ac0 2024-09-16T01:50:47.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.248+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4204144ac0 0x7f420402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.249+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4204144ac0 0x7f420402c640 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.249+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4204144f20 con 0x7f4204144ac0 2024-09-16T01:50:47.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.263+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4204144f20 con 0x7f4204144ac0 2024-09-16T01:50:47.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.265+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4204144f20 con 0x7f4204144ac0 2024-09-16T01:50:47.342 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.265+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f420402d090 0x7f420402d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.265+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f420402d9f0 con 0x7f420402d090 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.265+0000 7f4202a00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f420402d090 0x7f420402d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.266+0000 7f4202a00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f420402d090 0x7f420402d4b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.267+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f420402d9f0 con 0x7f420402d090 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.281+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f420402d9f0 con 0x7f420402d090 2024-09-16T01:50:47.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.283+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f420402d9f0 con 0x7f420402d090 2024-09-16T01:50:47.344 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.283+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42042033e0 0x7f42042057d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.283+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f42042085a0 con 0x7f42042033e0 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.283+0000 7f4203e00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42042033e0 0x7f42042057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.284+0000 7f4203e00640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42042033e0 0x7f42042057d0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.285+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42042085a0 con 0x7f42042033e0 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.298+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4204205d90 con 0x7f42042033e0 2024-09-16T01:50:47.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.300+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4204205d90 con 0x7f42042033e0 2024-09-16T01:50:47.346 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:47.346 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.300+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4204206390 0x7f42042087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.300+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f420420b840 con 0x7f4204206390 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.301+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4204206390 0x7f42042087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.301+0000 7f4203400640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4204206390 0x7f42042087a0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.302+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f420420b840 con 0x7f4204206390 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.315+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f420420b810 con 0x7f4204206390 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.317+0000 7f41fb400640 1 -- 172.21.15.146:0/3971894321 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f420420b810 con 0x7f4204206390 2024-09-16T01:50:47.347 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f42041f22e0 msgr2=0x7f42040cb0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f42041f22e0 0x7f42040cb0c0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f420413ef20 msgr2=0x7f4204062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f420413ef20 0x7f4204062630 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4204063080 msgr2=0x7f42040c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4204063080 0x7f42040c00b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f42040c0d50 msgr2=0x7f4204143dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f42040c0d50 0x7f4204143dc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4204144ac0 msgr2=0x7f420402c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f4204144ac0 0x7f420402c640 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f420402d090 msgr2=0x7f420402d4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f420402d090 0x7f420402d4b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42042033e0 msgr2=0x7f42042057d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42042033e0 0x7f42042057d0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4204206390 msgr2=0x7f42042087a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4204206390 0x7f42042087a0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f4203e00640 1 -- 172.21.15.146:0/3971894321 reap_dead start 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f4203e00640 1 -- 172.21.15.146:0/3971894321 reap_dead start 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f4203e00640 1 -- 172.21.15.146:0/3971894321 reap_dead start 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f4203e00640 1 -- 172.21.15.146:0/3971894321 reap_dead start 2024-09-16T01:50:47.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f4203e00640 1 -- 172.21.15.146:0/3971894321 reap_dead start 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41e406c2e0 msgr2=0x7f41e406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41e406c2e0 0x7f41e406e7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f42041e8f00 tx=0x7f41f0007e40 comp rx=0 tx=0).stop 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 msgr2=0x7f42041f0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 0x7f42041f0c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41f40098f0 tx=0x7f41f40046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 shutdown_connections 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41e406c2e0 0x7f41e406e7a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 --2- 172.21.15.146:0/3971894321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42040c6ab0 0x7f42041f0c40 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41f40098f0 tx=0x7f41f40046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 >> 172.21.15.146:0/3971894321 conn(0x7f420406bc10 msgr2=0x7f4204150450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 shutdown_connections 2024-09-16T01:50:47.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.318+0000 7f420a000640 1 -- 172.21.15.146:0/3971894321 wait complete. 2024-09-16T01:50:47.438 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:47.436+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:47.483 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:47.483 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:47.492 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:47.538 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:47.537+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:47.596 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:47.597 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:47.605 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:47.640 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:47.638+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:47.730 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:47.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:47.730 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T01:50:47.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T01:50:47.750 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:47.750 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:47.759 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:47.777 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:47.775+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:47.826 INFO:tasks.workunit.client.0.smithi146.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr: git switch -c 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.827 INFO:tasks.workunit.client.0.smithi146.stderr:Or undo this operation with: 2024-09-16T01:50:47.828 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.828 INFO:tasks.workunit.client.0.smithi146.stderr: git switch - 2024-09-16T01:50:47.828 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.828 INFO:tasks.workunit.client.0.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T01:50:47.828 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-16T01:50:47.828 INFO:tasks.workunit.client.0.smithi146.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T01:50:47.842 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T01:50:47.842 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:47.840+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:47.864 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:47.864 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:47.873 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:47.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.885+0000 7f85e4e00640 1 Processor -- start 2024-09-16T01:50:47.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.885+0000 7f85e4e00640 1 -- start start 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.886+0000 7f85e4e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f85e0152850 0x7f85e0158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.886+0000 7f85e4e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f85e00c6bb0 0x7f85e00c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.886+0000 7f85e4e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e0072f50 con 0x7f85e0158bb0 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.886+0000 7f85e4e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f85e0073210 con 0x7f85e0152850 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.886+0000 7f85e4e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f85e015cb50 con 0x7f85e00c6bb0 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.886+0000 7f85dde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f85e0152850 0x7f85e0158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85dde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f85e0152850 0x7f85e0158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38830/0 (socket says 172.21.15.146:38830) 2024-09-16T01:50:47.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85dde00640 1 -- 172.21.15.146:0/2915660907 learned_addr learned my addr 172.21.15.146:0/2915660907 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:47.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85de800640 1 --2- 172.21.15.146:0/2915660907 >> v2:172.21.15.161:3300/0 conn(0x7f85e00c6bb0 0x7f85e00c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85dde00640 1 -- 172.21.15.146:0/2915660907 >> v2:172.21.15.161:3300/0 conn(0x7f85e00c6bb0 msgr2=0x7f85e00c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:47.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85dde00640 1 --2- 172.21.15.146:0/2915660907 >> v2:172.21.15.161:3300/0 conn(0x7f85e00c6bb0 0x7f85e00c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85dde00640 1 -- 172.21.15.146:0/2915660907 >> v1:172.21.15.161:6789/0 conn(0x7f85e0158bb0 legacy=0x7f85e00c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85dde00640 1 -- 172.21.15.146:0/2915660907 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e015bd90 con 0x7f85e0152850 2024-09-16T01:50:47.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.887+0000 7f85de800640 1 --2- 172.21.15.146:0/2915660907 >> v2:172.21.15.161:3300/0 conn(0x7f85e00c6bb0 0x7f85e00c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:50:47.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.888+0000 7f85dde00640 1 --2- 172.21.15.146:0/2915660907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e0152850 0x7f85e0158670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85e006f350 tx=0x7f85d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303acca407b7c82d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.888+0000 7f85dd400640 1 -- 172.21.15.146:0/2915660907 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0037070 con 0x7f85e0152850 2024-09-16T01:50:47.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.888+0000 7f85dd400640 1 -- 172.21.15.146:0/2915660907 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d0032440 con 0x7f85e0152850 2024-09-16T01:50:47.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.888+0000 7f85dd400640 1 -- 172.21.15.146:0/2915660907 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0032810 con 0x7f85e0152850 2024-09-16T01:50:47.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 -- 172.21.15.146:0/2915660907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e0152850 msgr2=0x7f85e0158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 --2- 172.21.15.146:0/2915660907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e0152850 0x7f85e0158670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85e006f350 tx=0x7f85d002a270 comp rx=0 tx=0).stop 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 -- 172.21.15.146:0/2915660907 shutdown_connections 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 --2- 172.21.15.146:0/2915660907 >> v2:172.21.15.161:3300/0 conn(0x7f85e00c6bb0 0x7f85e00c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 --2- 172.21.15.146:0/2915660907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e0152850 0x7f85e0158670 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 -- 172.21.15.146:0/2915660907 >> 172.21.15.146:0/2915660907 conn(0x7f85e006f3c0 msgr2=0x7f85e006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 -- 172.21.15.146:0/2915660907 shutdown_connections 2024-09-16T01:50:47.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.889+0000 7f85e4e00640 1 -- 172.21.15.146:0/2915660907 wait complete. 2024-09-16T01:50:47.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.890+0000 7f85e4e00640 1 Processor -- start 2024-09-16T01:50:47.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 -- start start 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e00c6bb0 0x7f85e014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85e0152850 0x7f85e00c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 0x7f85e01e1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f85e015c630 con 0x7f85e00c6bb0 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f85e015c350 con 0x7f85e0152850 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.891+0000 7f85e4e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f85e0070200 con 0x7f85e0158bb0 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85dde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e00c6bb0 0x7f85e014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85de800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 0x7f85e01e1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85d5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85e0152850 0x7f85e00c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85de800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 0x7f85e01e1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37330/0 (socket says 172.21.15.146:37330) 2024-09-16T01:50:47.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85de800640 1 -- 172.21.15.146:0/1623412935 learned_addr learned my addr 172.21.15.146:0/1623412935 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85dde00640 1 -- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 msgr2=0x7f85e01e1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85dde00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 0x7f85e01e1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85dde00640 1 -- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85e0152850 msgr2=0x7f85e00c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85dde00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85e0152850 0x7f85e00c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85dde00640 1 -- 172.21.15.146:0/1623412935 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e01f0770 con 0x7f85e00c6bb0 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85de800640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 0x7f85e01e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.892+0000 7f85d5400640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85e0152850 0x7f85e00c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:47.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.893+0000 7f85dde00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e00c6bb0 0x7f85e014c210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f85d0031000 tx=0x7f85d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.893+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0041020 con 0x7f85e00c6bb0 2024-09-16T01:50:47.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.893+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d0030700 con 0x7f85e00c6bb0 2024-09-16T01:50:47.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.893+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0037040 con 0x7f85e00c6bb0 2024-09-16T01:50:47.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.893+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e01e4680 con 0x7f85e00c6bb0 2024-09-16T01:50:47.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.893+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e01f0d10 con 0x7f85e00c6bb0 2024-09-16T01:50:47.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.894+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e01507f0 con 0x7f85e00c6bb0 2024-09-16T01:50:47.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.895+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f85d0044020 con 0x7f85e00c6bb0 2024-09-16T01:50:47.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.896+0000 7f85d6a00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f85bc06c2b0 0x7f85bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:47.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.896+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5828+0+0 (secure 0 0 0) 0x7f85d00aba50 con 0x7f85e00c6bb0 2024-09-16T01:50:47.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.896+0000 7f85d5400640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f85bc06c2b0 0x7f85bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:47.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.897+0000 7f85d5400640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f85bc06c2b0 0x7f85bc06e770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f85e00c45d0 tx=0x7f85c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:47.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:47.901+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85d004b030 con 0x7f85e00c6bb0 2024-09-16T01:50:47.927 INFO:tasks.workunit.client.0.smithi146.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T01:50:47.933 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T01:50:47.933 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T01:50:47.942 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:47.940+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:48.017 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.017 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.026 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:48.044 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:48.042+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:48.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.077+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f85e00718a0 con 0x7f85e00c6bb0 2024-09-16T01:50:48.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.078+0000 7f85d6a00640 1 -- 172.21.15.146:0/1623412935 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v53) ==== 81+0+35044 (secure 0 0 0) 0x7f85d0079b80 con 0x7f85e00c6bb0 2024-09-16T01:50:48.081 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:50:48.081 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:48.081 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 53, 2024-09-16T01:50:48.081 INFO:teuthology.orchestra.run.smithi146.stdout: "fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "created": "2024-09-16T01:48:34.399638+0000", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "modified": "2024-09-16T01:50:46.705136+0000", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "last_up_change": "2024-09-16T01:48:45.748664+0000", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "last_in_change": "2024-09-16T01:50:04.358386+0000", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_num": 5804032, 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_set": [ 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "nodeep-scrub", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "pglog_hardlimit", 2024-09-16T01:50:48.082 INFO:teuthology.orchestra.run.smithi146.stdout: "purged_snapdirs", 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_deletes", 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "sortbitwise" 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_version": 6, 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_snapshot": "", 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_max": 3, 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd": 8, 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "require_min_compat_client": "luminous", 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "min_compat_client": "luminous", 2024-09-16T01:50:48.083 INFO:teuthology.orchestra.run.smithi146.stdout: "require_osd_release": "squid", 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_crimson": false, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "pools": [ 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 1, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": ".mgr", 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:46.570355+0000", 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:50:48.084 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 1, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 1, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 1, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 1, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 1, 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:50:48.085 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "18", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:50:48.088 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:50:48.089 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:50:48.090 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_max": 32, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 1 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": {} 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:50:48.091 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 6.9000000953674316, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 6.9000000953674316, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.28999999165534973, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 2, 2024-09-16T01:50:48.092 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_metadata", 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:56.008811+0000", 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:50:48.093 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 32, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 32, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 32, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 32, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 32, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:50:48.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "21", 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:50:48.095 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:50:48.096 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:50:48.097 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_bias": 4, 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 16, 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_priority": 5 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": "cephfs" 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:50:48.098 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.9700000286102295, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.9700000286102295, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2.25, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2.25, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 3, 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_data", 2024-09-16T01:50:48.099 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:57.207121+0000", 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:50:48.100 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 64, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 64, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 64, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 64, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 64, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:50:48.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "22", 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:50:48.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.103 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 64, 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: "target_size_ratio": 0.89999997615814209 2024-09-16T01:50:48.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "data": "cephfs" 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.3200000524520874, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.3200000524520874, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 1.5, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 1.5, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:50:48.105 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "osds": [ 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.106 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6824", 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.107 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6825", 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6826", 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6827", 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.109 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6830", 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6831", 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6828", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6829", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6825/61440064", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6827/61440064", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6831/61440064", 2024-09-16T01:50:48.111 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6829/61440064", 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.112 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6808", 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6809", 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6810", 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6811", 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6814", 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6815", 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6812", 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6813", 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6809/2375201615", 2024-09-16T01:50:48.117 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6811/2375201615", 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6815/2375201615", 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6813/2375201615", 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.118 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6800", 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6801", 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6802", 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6803", 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6806", 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6807", 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6804", 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6805", 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6801/1818492127", 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6803/1818492127", 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6807/1818492127", 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6805/1818492127", 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.123 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:50:48.124 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 50, 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6816", 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6817", 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6818", 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6819", 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6822", 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6823", 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6820", 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6821", 2024-09-16T01:50:48.128 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6817/1633804394", 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6819/1633804394", 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1633804394", 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6821/1633804394", 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.129 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0.8634185791015625, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 13, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.130 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6800", 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6801", 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.131 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6802", 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6803", 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.132 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6806", 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6807", 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6804", 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6805", 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.134 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6801/154390265", 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6803/154390265", 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6807/154390265", 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6805/154390265", 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.135 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 41, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6824", 2024-09-16T01:50:48.136 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6825", 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6826", 2024-09-16T01:50:48.137 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6827", 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.138 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6830", 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6831", 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.139 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6828", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6829", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6825/356357279", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6827/356357279", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6831/356357279", 2024-09-16T01:50:48.140 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6829/356357279", 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.141 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 50, 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6808", 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6809", 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6810", 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6811", 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6814", 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.144 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6815", 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6812", 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6813", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6809/3661001615", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6811/3661001615", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6815/3661001615", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6813/3661001615", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 0, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 35, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:50:48.147 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6816", 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6817", 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6818", 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6819", 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6822", 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6823", 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6820", 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6821", 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6817/1279166573", 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6819/1279166573", 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6823/1279166573", 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6821/1279166573", 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_xinfo": [ 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:50:48.152 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.135308+0000", 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.153 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.290750+0000", 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.712744+0000", 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.988678+0000", 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.155 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.198114+0000", 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.156 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.085332+0000", 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.394321+0000", 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:50:48.157 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.236983+0000", 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap": [], 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_items": [ 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.13", 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:50:48.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 0, 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 7 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.20", 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 6, 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 5 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.33", 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 0, 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 7 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_primaries": [], 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_temp": [], 2024-09-16T01:50:48.160 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_temp": [], 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklist": {}, 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "range_blocklist": {}, 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profiles": { 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "default": { 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-failure-domain": "osd", 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "k": "2", 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "m": "1", 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin": "jerasure", 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "technique": "reed_sol_van" 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps_queue": [], 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "new_removed_snaps": [], 2024-09-16T01:50:48.161 INFO:teuthology.orchestra.run.smithi146.stdout: "new_purged_snaps": [], 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_node_flags": {}, 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "device_class_flags": {}, 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode": { 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_enabled": false, 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_bucket_count": 0, 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "degraded_stretch_mode": 0, 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "recovering_stretch_mode": 0, 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_bucket": 0 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:48.162 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:48.164 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.164 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:48.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f85bc06c2b0 msgr2=0x7f85bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f85bc06c2b0 0x7f85bc06e770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f85e00c45d0 tx=0x7f85c8009040 comp rx=0 tx=0).stop 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e00c6bb0 msgr2=0x7f85e014c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e00c6bb0 0x7f85e014c210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f85d0031000 tx=0x7f85d00046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 shutdown_connections 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f85bc06c2b0 0x7f85bc06e770 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85e0158bb0 0x7f85e01e1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f85e0152850 0x7f85e00c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 --2- 172.21.15.146:0/1623412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85e00c6bb0 0x7f85e014c210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 >> 172.21.15.146:0/1623412935 conn(0x7f85e006f3c0 msgr2=0x7f85e006e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 shutdown_connections 2024-09-16T01:50:48.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.081+0000 7f85e4e00640 1 -- 172.21.15.146:0/1623412935 wait complete. 2024-09-16T01:50:48.167 INFO:tasks.thrashosds.thrasher:j is {'epoch': 53, 'fsid': '4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d', 'created': '2024-09-16T01:48:34.399638+0000', 'modified': '2024-09-16T01:50:46.705136+0000', 'last_up_change': '2024-09-16T01:48:45.748664+0000', 'last_in_change': '2024-09-16T01:50:04.358386+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:48:46.570355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:48:56.008811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:48:57.207121+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e7758ddf-75ff-43c7-abae-3647b170f7c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 61440064}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 61440064}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 61440064}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 61440064}]}, 'public_addr': '172.21.15.146:6825/61440064', 'cluster_addr': '172.21.15.146:6827/61440064', 'heartbeat_back_addr': '172.21.15.146:6831/61440064', 'heartbeat_front_addr': '172.21.15.146:6829/61440064', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a065e77e-26f0-4245-abe8-e36c2fb4436b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 2375201615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 2375201615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 2375201615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 2375201615}]}, 'public_addr': '172.21.15.146:6809/2375201615', 'cluster_addr': '172.21.15.146:6811/2375201615', 'heartbeat_back_addr': '172.21.15.146:6815/2375201615', 'heartbeat_front_addr': '172.21.15.146:6813/2375201615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fefb6d8f-6799-4344-a977-85ab983ec0cb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 1818492127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 1818492127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 1818492127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 1818492127}]}, 'public_addr': '172.21.15.146:6801/1818492127', 'cluster_addr': '172.21.15.146:6803/1818492127', 'heartbeat_back_addr': '172.21.15.146:6807/1818492127', 'heartbeat_front_addr': '172.21.15.146:6805/1818492127', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9d1a2f96-efcb-4a73-b5db-ec5e2a63122a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 50, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 1633804394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 1633804394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1633804394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 1633804394}]}, 'public_addr': '172.21.15.146:6817/1633804394', 'cluster_addr': '172.21.15.146:6819/1633804394', 'heartbeat_back_addr': '172.21.15.146:6823/1633804394', 'heartbeat_front_addr': '172.21.15.146:6821/1633804394', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c', 'up': 1, 'in': 1, 'weight': 0.8634185791015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6800', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6801', 'nonce': 154390265}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6802', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6803', 'nonce': 154390265}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6806', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6807', 'nonce': 154390265}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6804', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6805', 'nonce': 154390265}]}, 'public_addr': '172.21.15.161:6801/154390265', 'cluster_addr': '172.21.15.161:6803/154390265', 'heartbeat_back_addr': '172.21.15.161:6807/154390265', 'heartbeat_front_addr': '172.21.15.161:6805/154390265', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '11562dd5-d537-4d88-a282-e8687caebad6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6824', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6825', 'nonce': 356357279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6826', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6827', 'nonce': 356357279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6830', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6831', 'nonce': 356357279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6828', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6829', 'nonce': 356357279}]}, 'public_addr': '172.21.15.161:6825/356357279', 'cluster_addr': '172.21.15.161:6827/356357279', 'heartbeat_back_addr': '172.21.15.161:6831/356357279', 'heartbeat_front_addr': '172.21.15.161:6829/356357279', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b31adf01-6768-49cf-a232-127b9d0d2dc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 50, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6808', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6809', 'nonce': 3661001615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6810', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6811', 'nonce': 3661001615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6814', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6815', 'nonce': 3661001615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6812', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6813', 'nonce': 3661001615}]}, 'public_addr': '172.21.15.161:6809/3661001615', 'cluster_addr': '172.21.15.161:6811/3661001615', 'heartbeat_back_addr': '172.21.15.161:6815/3661001615', 'heartbeat_front_addr': '172.21.15.161:6813/3661001615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '67c95c28-b600-430c-9474-a902cf46a73b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6816', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6817', 'nonce': 1279166573}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6818', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6819', 'nonce': 1279166573}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6822', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6823', 'nonce': 1279166573}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6820', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6821', 'nonce': 1279166573}]}, 'public_addr': '172.21.15.161:6817/1279166573', 'cluster_addr': '172.21.15.161:6819/1279166573', 'heartbeat_back_addr': '172.21.15.161:6823/1279166573', 'heartbeat_front_addr': '172.21.15.161:6821/1279166573', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.135308+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.290750+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.712744+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.988678+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.198114+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.085332+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.394321+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.236983+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.13', 'mappings': [{'from': 0, 'to': 7}]}, {'pgid': '3.20', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '3.33', 'mappings': [{'from': 0, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T01:50:48.167 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.20 2024-09-16T01:50:48.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.20 2024-09-16T01:50:48.168 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:48.162+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:48.264 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:48.263+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:48.272 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.272 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.281 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:48.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.360+0000 7f925a400640 1 Processor -- start 2024-09-16T01:50:48.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.360+0000 7f925a400640 1 -- start start 2024-09-16T01:50:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f925a400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92540c6b70 0x7f925415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f925a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925406d140 con 0x7f92540c75a0 2024-09-16T01:50:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f925a400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925406d420 con 0x7f92540cc040 2024-09-16T01:50:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f925a400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f925415b880 con 0x7f92540c6b70 2024-09-16T01:50:48.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f9259000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92540c6b70 0x7f925415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f924be00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f92540c75a0 0x7f92540cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53446/0 (socket says 172.21.15.146:53446) 2024-09-16T01:50:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f924be00640 1 -- 172.21.15.146:0/3104506180 learned_addr learned my addr 172.21.15.146:0/3104506180 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:48.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.361+0000 7f9259a00640 1 --1- 172.21.15.146:0/3104506180 >> v1:172.21.15.161:6789/0 conn(0x7f92540cc040 0x7f9254165160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34228/0 (socket says 172.21.15.146:34228) 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.362+0000 7f9253e00640 1 -- 172.21.15.146:0/3104506180 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1551987903 0 0) 0x7f925406d420 con 0x7f92540cc040 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.362+0000 7f9253e00640 1 -- 172.21.15.146:0/3104506180 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9238003520 con 0x7f92540cc040 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.362+0000 7f9259000640 1 -- 172.21.15.146:0/3104506180 >> v1:172.21.15.161:6789/0 conn(0x7f92540cc040 legacy=0x7f9254165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.362+0000 7f9259000640 1 -- 172.21.15.146:0/3104506180 >> v1:172.21.15.146:6789/0 conn(0x7f92540c75a0 legacy=0x7f92540cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.362+0000 7f9259000640 1 -- 172.21.15.146:0/3104506180 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925406d700 con 0x7f92540c6b70 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.363+0000 7f9259000640 1 --2- 172.21.15.146:0/3104506180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415b340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f92440097a0 tx=0x7f924402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fc50c922488b1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:48.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.363+0000 7f9253e00640 1 -- 172.21.15.146:0/3104506180 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244037070 con 0x7f92540c6b70 2024-09-16T01:50:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.363+0000 7f9253e00640 1 -- 172.21.15.146:0/3104506180 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9244032440 con 0x7f92540c6b70 2024-09-16T01:50:48.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.363+0000 7f9253e00640 1 -- 172.21.15.146:0/3104506180 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440327f0 con 0x7f92540c6b70 2024-09-16T01:50:48.366 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:48.364+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 -- 172.21.15.146:0/3104506180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 msgr2=0x7f925415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 --2- 172.21.15.146:0/3104506180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415b340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f92440097a0 tx=0x7f924402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 -- 172.21.15.146:0/3104506180 shutdown_connections 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 --2- 172.21.15.146:0/3104506180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415b340 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 -- 172.21.15.146:0/3104506180 >> 172.21.15.146:0/3104506180 conn(0x7f92540ccc20 msgr2=0x7f92540cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 -- 172.21.15.146:0/3104506180 shutdown_connections 2024-09-16T01:50:48.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.364+0000 7f925a400640 1 -- 172.21.15.146:0/3104506180 wait complete. 2024-09-16T01:50:48.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.365+0000 7f925a400640 1 Processor -- start 2024-09-16T01:50:48.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 -- start start 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92540c75a0 0x7f925415a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 0x7f92541f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f92540cb050 con 0x7f92540c6b70 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92540cab20 con 0x7f92540cc040 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.366+0000 7f925a400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f92540cbc80 con 0x7f92540c75a0 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f924be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92540c75a0 0x7f925415a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 0x7f92541f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 0x7f92541f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:52452/0 (socket says 172.21.15.146:52452) 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 -- 172.21.15.146:0/2423555249 learned_addr learned my addr 172.21.15.146:0/2423555249 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:48.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 -- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92540c75a0 msgr2=0x7f925415a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92540c75a0 0x7f925415a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 -- 172.21.15.146:0/2423555249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 msgr2=0x7f925415a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 -- 172.21.15.146:0/2423555249 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9254159f80 con 0x7f92540cc040 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f924be00640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92540c75a0 0x7f925415a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9259a00640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 0x7f92541f3ed0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f924c00c7c0 tx=0x7f924c00cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.367+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c017070 con 0x7f92540cc040 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.368+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924c00a640 con 0x7f92540cc040 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.368+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92541552d0 con 0x7f92540cc040 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.368+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c00aa10 con 0x7f92540cc040 2024-09-16T01:50:48.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.368+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92541e7ad0 con 0x7f92540cc040 2024-09-16T01:50:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.369+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925406cd20 con 0x7f92540cc040 2024-09-16T01:50:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.369+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f924c01a070 con 0x7f92540cc040 2024-09-16T01:50:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.370+0000 7f9251600640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f923006c2e0 0x7f923006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:48.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.370+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5828+0+0 (secure 0 0 0) 0x7f924c08ab20 con 0x7f92540cc040 2024-09-16T01:50:48.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.371+0000 7f9259000640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f923006c2e0 0x7f923006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:48.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.371+0000 7f9259000640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f923006c2e0 0x7f923006e7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9244034000 tx=0x7f924402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:48.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.375+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f924c058b60 con 0x7f92540cc040 2024-09-16T01:50:48.385 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.385 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.394 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:48.466 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:48.464+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:48.539 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.539 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:48.553 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T01:50:48.566 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:48.565+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:48.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.569+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "3.20"} v 0) -- 0x7f92540cda30 con 0x7f92540cc040 2024-09-16T01:50:48.658 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T01:50:48.668 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:48.667+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:48.712 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.712 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.718+0000 7f9251600640 1 -- 172.21.15.146:0/2423555249 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "3.20"}]=0 clear 3.20 pg_upmap_items mapping v54) ==== 118+0+0 (secure 0 0 0) 0x7f924c05ce60 con 0x7f92540cc040 2024-09-16T01:50:48.720 INFO:teuthology.orchestra.run.smithi146.stderr:clear 3.20 pg_upmap_items mapping 2024-09-16T01:50:48.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f923006c2e0 msgr2=0x7f923006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f923006c2e0 0x7f923006e7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9244034000 tx=0x7f924402e000 comp rx=0 tx=0).stop 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 msgr2=0x7f92541f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 0x7f92541f3ed0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f924c00c7c0 tx=0x7f924c00cc80 comp rx=0 tx=0).stop 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 shutdown_connections 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f923006c2e0 0x7f923006e7a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92540cc040 0x7f92541f3ed0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92540c75a0 0x7f925415a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 --2- 172.21.15.146:0/2423555249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540c6b70 0x7f925415a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 >> 172.21.15.146:0/2423555249 conn(0x7f92540ccc20 msgr2=0x7f92540c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 shutdown_connections 2024-09-16T01:50:48.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:48.721+0000 7f925a400640 1 -- 172.21.15.146:0/2423555249 wait complete. 2024-09-16T01:50:48.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:48.730 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T01:50:48.734 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T01:50:48.734 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T01:50:48.784 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:48.782+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:48.802 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T01:50:48.810 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:50:48.810 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T01:50:48.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:50:48.870 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:48.869+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:48.888 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-09-16T01:50:48.889 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-09-16T01:50:48.889 DEBUG:teuthology.orchestra.run.smithi146:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-09-16T01:50:48.894 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:48.894 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:48.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:48.948 INFO:tasks.workunit.client.0.smithi146.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-09-16T01:50:48.971 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:48.970+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.012+0000 7f4909a00640 1 Processor -- start 2024-09-16T01:50:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.013+0000 7f4909a00640 1 -- start start 2024-09-16T01:50:49.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4909a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4904152850 0x7f4904158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4909a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4904158bb0 0x7f49040c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4909a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4904072f50 con 0x7f49040c6bb0 2024-09-16T01:50:49.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4909a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4904073210 con 0x7f4904158bb0 2024-09-16T01:50:49.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4909a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f490415cb50 con 0x7f4904152850 2024-09-16T01:50:49.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4902a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4904152850 0x7f4904158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f48fa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4904158bb0 0x7f49040c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4903400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f49040c6bb0 0x7f49040c86f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53450/0 (socket says 172.21.15.146:53450) 2024-09-16T01:50:49.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.014+0000 7f4903400640 1 -- 172.21.15.146:0/3902154284 learned_addr learned my addr 172.21.15.146:0/3902154284 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:49.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.015+0000 7f48fa000640 1 -- 172.21.15.146:0/3902154284 >> v2:172.21.15.161:3300/0 conn(0x7f4904152850 msgr2=0x7f4904158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:49.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.015+0000 7f48fa000640 1 --2- 172.21.15.146:0/3902154284 >> v2:172.21.15.161:3300/0 conn(0x7f4904152850 0x7f4904158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.015+0000 7f48fa000640 1 -- 172.21.15.146:0/3902154284 >> v1:172.21.15.146:6789/0 conn(0x7f49040c6bb0 legacy=0x7f49040c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.015+0000 7f48fa000640 1 -- 172.21.15.146:0/3902154284 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490415bd90 con 0x7f4904158bb0 2024-09-16T01:50:49.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.015+0000 7f4902000640 1 -- 172.21.15.146:0/3902154284 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001307713 0 0) 0x7f4904072f50 con 0x7f49040c6bb0 2024-09-16T01:50:49.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.015+0000 7f4902a00640 1 --2- 172.21.15.146:0/3902154284 >> v2:172.21.15.161:3300/0 conn(0x7f4904152850 0x7f4904158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:49.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.016+0000 7f48fa000640 1 --2- 172.21.15.146:0/3902154284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f49040c58c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4904073680 tx=0x7f48ec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a2406cc9e17cff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.016+0000 7f4902000640 1 -- 172.21.15.146:0/3902154284 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec037070 con 0x7f4904158bb0 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.016+0000 7f4902000640 1 -- 172.21.15.146:0/3902154284 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48ec032440 con 0x7f4904158bb0 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.016+0000 7f4902000640 1 -- 172.21.15.146:0/3902154284 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec032810 con 0x7f4904158bb0 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 -- 172.21.15.146:0/3902154284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 msgr2=0x7f49040c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 --2- 172.21.15.146:0/3902154284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f49040c58c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4904073680 tx=0x7f48ec02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 -- 172.21.15.146:0/3902154284 shutdown_connections 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 --2- 172.21.15.146:0/3902154284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f49040c58c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 --2- 172.21.15.146:0/3902154284 >> v2:172.21.15.161:3300/0 conn(0x7f4904152850 0x7f4904158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 -- 172.21.15.146:0/3902154284 >> 172.21.15.146:0/3902154284 conn(0x7f490406f3c0 msgr2=0x7f490406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:49.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 -- 172.21.15.146:0/3902154284 shutdown_connections 2024-09-16T01:50:49.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.017+0000 7f4909a00640 1 -- 172.21.15.146:0/3902154284 wait complete. 2024-09-16T01:50:49.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.018+0000 7f4909a00640 1 Processor -- start 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.018+0000 7f4909a00640 1 -- start start 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.019+0000 7f4909a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49040c6bb0 0x7f49041f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.019+0000 7f4909a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4904152850 0x7f49040c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.019+0000 7f4909a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f4904156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.019+0000 7f4909a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f490415c630 con 0x7f4904158bb0 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.019+0000 7f4909a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f490415c350 con 0x7f4904152850 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.019+0000 7f4909a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4904070200 con 0x7f49040c6bb0 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f4904156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f4904156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38884/0 (socket says 172.21.15.146:38884) 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 -- 172.21.15.146:0/1246836099 learned_addr learned my addr 172.21.15.146:0/1246836099 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4902a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49040c6bb0 0x7f49041f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f48fa000640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4904152850 0x7f49040c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 -- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49040c6bb0 msgr2=0x7f49041f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49040c6bb0 0x7f49041f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 -- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4904152850 msgr2=0x7f49040c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4904152850 0x7f49040c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 -- 172.21.15.146:0/1246836099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49041f12d0 con 0x7f4904158bb0 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f48fa000640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4904152850 0x7f49040c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.020+0000 7f4903400640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f4904156d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c7d0 tx=0x7f48fc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.021+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc018070 con 0x7f4904158bb0 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.021+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48fc00a640 con 0x7f4904158bb0 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.021+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48fc00aa70 con 0x7f4904158bb0 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.021+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49040ca620 con 0x7f4904158bb0 2024-09-16T01:50:49.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.021+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4904157d10 con 0x7f4904158bb0 2024-09-16T01:50:49.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.022+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49040718c0 con 0x7f4904158bb0 2024-09-16T01:50:49.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.024+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f48fc00ac10 con 0x7f4904158bb0 2024-09-16T01:50:49.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.025+0000 7f48fb400640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f48e006c2e0 0x7f48e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.025+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5799+0+0 (secure 0 0 0) 0x7f48fc08a9c0 con 0x7f4904158bb0 2024-09-16T01:50:49.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.025+0000 7f4902a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f48e006c2e0 0x7f48e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.026+0000 7f4902a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f48e006c2e0 0x7f48e006e7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009560 tx=0x7f48f4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.029+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f48fc0589e0 con 0x7f4904158bb0 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Iozone: Performance Test of File I/O 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Version $Revision: 3.494 $ 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Compiled for 64 bit mode. 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Build: linux-AMD64 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-16T01:50:49.041 INFO:tasks.workunit.client.0.smithi146.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Run began: Mon Sep 16 01:50:48 2024 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Include close in write timing 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Include fsync in write timing 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: File size set to 1048576 kB 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Record Size 16 kB 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Output is in kBytes/sec 2024-09-16T01:50:49.042 INFO:tasks.workunit.client.0.smithi146.stdout: Time Resolution = 0.000001 seconds. 2024-09-16T01:50:49.043 INFO:tasks.workunit.client.0.smithi146.stdout: Processor cache size set to 1024 kBytes. 2024-09-16T01:50:49.043 INFO:tasks.workunit.client.0.smithi146.stdout: Processor cache line size set to 32 bytes. 2024-09-16T01:50:49.043 INFO:tasks.workunit.client.0.smithi146.stdout: File stride size set to 17 * record size. 2024-09-16T01:50:49.043 INFO:tasks.workunit.client.0.smithi146.stdout: Throughput test with 1 process 2024-09-16T01:50:49.043 INFO:tasks.workunit.client.0.smithi146.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-09-16T01:50:49.079 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:49.078+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:49.135 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:49.136 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:49.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:49.192 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:49.190+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.245+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f49040c93d0 con 0x7f4904158bb0 2024-09-16T01:50:49.275 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:49.273+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:49.352 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:49.360 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:49.360 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:49.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:49.376 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:49.375+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:49.477 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:49.475+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.510+0000 7f705fc00640 1 Processor -- start 2024-09-16T01:50:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.510+0000 7f705fc00640 1 -- start start 2024-09-16T01:50:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705fc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f70580c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705fc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70580c9180 0x7f70580c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705fc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70580c4240 con 0x7f7058152c30 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705fc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f70580c71c0 con 0x7f70580c9180 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705fc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70580c3a90 con 0x7f70580c36b0 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705d200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7058152c30 0x7f70580c6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53454/0 (socket says 172.21.15.146:53454) 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.511+0000 7f705d200640 1 -- 172.21.15.146:0/1403343701 learned_addr learned my addr 172.21.15.146:0/1403343701 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f705dc00640 1 --2- 172.21.15.146:0/1403343701 >> v2:172.21.15.161:3300/0 conn(0x7f70580c9180 0x7f70580c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f7057e00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f70580c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f705dc00640 1 -- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 msgr2=0x7f70580c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f705dc00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f70580c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f7057400640 1 -- 172.21.15.146:0/1403343701 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128525021 0 0) 0x7f70580c4240 con 0x7f7058152c30 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f705dc00640 1 -- 172.21.15.146:0/1403343701 >> v1:172.21.15.146:6789/0 conn(0x7f7058152c30 legacy=0x7f70580c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f705dc00640 1 -- 172.21.15.146:0/1403343701 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70580c4500 con 0x7f70580c9180 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.512+0000 7f7057e00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f70580c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.514+0000 7f705dc00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70580c9180 0x7f70580c7ff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7050009d90 tx=0x7f705002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=483f0dfc26634f8c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.514+0000 7f7057400640 1 -- 172.21.15.146:0/1403343701 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050007740 con 0x7f70580c9180 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.514+0000 7f7057400640 1 -- 172.21.15.146:0/1403343701 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70500078e0 con 0x7f70580c9180 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.514+0000 7f7057400640 1 -- 172.21.15.146:0/1403343701 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050007cb0 con 0x7f70580c9180 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 -- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70580c9180 msgr2=0x7f70580c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70580c9180 0x7f70580c7ff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7050009d90 tx=0x7f705002a660 comp rx=0 tx=0).stop 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 -- 172.21.15.146:0/1403343701 shutdown_connections 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f70580c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 --2- 172.21.15.146:0/1403343701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70580c9180 0x7f70580c7ff0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 -- 172.21.15.146:0/1403343701 >> 172.21.15.146:0/1403343701 conn(0x7f705806bc10 msgr2=0x7f705806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 -- 172.21.15.146:0/1403343701 shutdown_connections 2024-09-16T01:50:49.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.515+0000 7f705fc00640 1 -- 172.21.15.146:0/1403343701 wait complete. 2024-09-16T01:50:49.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.516+0000 7f705fc00640 1 Processor -- start 2024-09-16T01:50:49.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 -- start start 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f7058154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 0x7f70581532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7058152c30 0x7f7058072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f70580c4ac0 con 0x7f70580c9180 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70580c4da0 con 0x7f7058152c30 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.517+0000 7f705fc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70580c6630 con 0x7f70580c36b0 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 0x7f70581532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f705d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f7058154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 0x7f70581532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38886/0 (socket says 172.21.15.146:38886) 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 -- 172.21.15.146:0/2993306257 learned_addr learned my addr 172.21.15.146:0/2993306257 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7058152c30 0x7f7058072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 msgr2=0x7f7058154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f7058154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7058152c30 msgr2=0x7f7058072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7058152c30 0x7f7058072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f7057e00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70581e6a30 con 0x7f70580c9180 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70580c36b0 0x7f7058154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.518+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7058152c30 0x7f7058072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.519+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 0x7f70581532b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f704400e990 tx=0x7f704400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.519+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044017070 con 0x7f70580c9180 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.519+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f704400c9e0 con 0x7f70580c9180 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.519+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7058071bb0 con 0x7f70580c9180 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.519+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704400cdb0 con 0x7f70580c9180 2024-09-16T01:50:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.519+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7058072170 con 0x7f70580c9180 2024-09-16T01:50:49.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.520+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f70580c8c40 con 0x7f70580c9180 2024-09-16T01:50:49.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.521+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f704401a070 con 0x7f70580c9180 2024-09-16T01:50:49.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.522+0000 7f7054c00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.522+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5799+0+0 (secure 0 0 0) 0x7f704408b6c0 con 0x7f70580c9180 2024-09-16T01:50:49.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.523+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.523+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v54) ==== 54+0+16 (secure 0 0 0) 0x7f7044091170 con 0x7f70580c9180 2024-09-16T01:50:49.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.523+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7058150450 0x7f70581f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.523+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7058156140 con 0x7f7058150450 2024-09-16T01:50:49.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.523+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f704c007fd0 tx=0x7f704c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.523+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7058150450 0x7f70581f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.524+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7058150450 0x7f70581f39c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.525+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7058156140 con 0x7f7058150450 2024-09-16T01:50:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.540+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f70580c7530 con 0x7f7058150450 2024-09-16T01:50:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.541+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70580c7530 con 0x7f7058150450 2024-09-16T01:50:49.544 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:49.544 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:49.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.542+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7058150830 0x7f70581f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.542+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f705806ff10 con 0x7f7058150830 2024-09-16T01:50:49.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.542+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7058150830 0x7f70581f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.543+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7058150830 0x7f70581f4410 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.543+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705806ff10 con 0x7f7058150830 2024-09-16T01:50:49.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.559+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f70581537f0 con 0x7f7058150830 2024-09-16T01:50:49.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.561+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70581537f0 con 0x7f7058150830 2024-09-16T01:50:49.563 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:49.564 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:49.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.561+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7058148a70 0x7f7058148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.561+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f70581493b0 con 0x7f7058148a70 2024-09-16T01:50:49.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.561+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7058148a70 0x7f7058148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.562+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7058148a70 0x7f7058148e70 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.562+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70440596f0 con 0x7f7058148a70 2024-09-16T01:50:49.578 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:49.577+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:49.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.577+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7058062900 con 0x7f7058148a70 2024-09-16T01:50:49.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.580+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7058062900 con 0x7f7058148a70 2024-09-16T01:50:49.582 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:49.583 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:49.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.580+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f70580631d0 0x7f70581e7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.580+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f70581e3cf0 con 0x7f70580631d0 2024-09-16T01:50:49.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.580+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f70580631d0 0x7f70581e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.581+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f70580631d0 0x7f70581e7080 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.582+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70581e3cf0 con 0x7f70580631d0 2024-09-16T01:50:49.590 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:49.590 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:49.590 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:49.591 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:49.591 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:49.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.596+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f70581e3cf0 con 0x7f70580631d0 2024-09-16T01:50:49.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.599+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70581e3cf0 con 0x7f70580631d0 2024-09-16T01:50:49.601 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:49.602 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:49.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.599+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f70581e7d20 0x7f705802c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.599+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f70581e3cf0 con 0x7f70581e7d20 2024-09-16T01:50:49.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.600+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f70581e7d20 0x7f705802c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.600+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f70581e7d20 0x7f705802c640 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.601+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70581e3cf0 con 0x7f70581e7d20 2024-09-16T01:50:49.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:49.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.615+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f70581e3cf0 con 0x7f70581e7d20 2024-09-16T01:50:49.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.617+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f70581e3cf0 con 0x7f70581e7d20 2024-09-16T01:50:49.619 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.617+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f705802d090 0x7f705802d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.617+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f705802da50 con 0x7f705802d090 2024-09-16T01:50:49.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.617+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f705802d090 0x7f705802d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.618+0000 7f7057e00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f705802d090 0x7f705802d510 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.619+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705802da50 con 0x7f705802d090 2024-09-16T01:50:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.632+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f705802da50 con 0x7f705802d090 2024-09-16T01:50:49.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.634+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f705802da50 con 0x7f705802d090 2024-09-16T01:50:49.637 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.634+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f70582074e0 0x7f70582098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.636+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f705820c800 con 0x7f70582074e0 2024-09-16T01:50:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.636+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f70582074e0 0x7f70582098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.636+0000 7f705dc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f70582074e0 0x7f70582098d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.636+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705820c800 con 0x7f70582074e0 2024-09-16T01:50:49.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.650+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7058209e10 con 0x7f70582074e0 2024-09-16T01:50:49.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.652+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7058209e10 con 0x7f70582074e0 2024-09-16T01:50:49.654 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.652+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f705820a530 0x7f705820a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.652+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f705820fbd0 con 0x7f705820a530 2024-09-16T01:50:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.653+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f705820a530 0x7f705820a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.653+0000 7f705d200640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f705820a530 0x7f705820a970 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:49.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.654+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705820fbd0 con 0x7f705820a530 2024-09-16T01:50:49.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.668+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f705820ea50 con 0x7f705820a530 2024-09-16T01:50:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.669+0000 7f7054c00640 1 -- 172.21.15.146:0/2993306257 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f705820ea50 con 0x7f705820a530 2024-09-16T01:50:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7058150450 msgr2=0x7f70581f39c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7058150450 0x7f70581f39c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7058150830 msgr2=0x7f70581f4410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7058150830 0x7f70581f4410 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7058148a70 msgr2=0x7f7058148e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7058148a70 0x7f7058148e70 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f70580631d0 msgr2=0x7f70581e7080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f70580631d0 0x7f70581e7080 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f70581e7d20 msgr2=0x7f705802c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f70581e7d20 0x7f705802c640 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f705802d090 msgr2=0x7f705802d510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f705802d090 0x7f705802d510 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f70582074e0 msgr2=0x7f70582098d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f70582074e0 0x7f70582098d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705dc00640 1 -- 172.21.15.146:0/2993306257 reap_dead start 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f705820a530 msgr2=0x7f705820a970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f705820a530 0x7f705820a970 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705dc00640 1 -- 172.21.15.146:0/2993306257 reap_dead start 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705dc00640 1 -- 172.21.15.146:0/2993306257 reap_dead start 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705dc00640 1 -- 172.21.15.146:0/2993306257 reap_dead start 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 msgr2=0x7f703806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f704c007fd0 tx=0x7f704c007de0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 msgr2=0x7f70581532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 0x7f70581532b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f704400e990 tx=0x7f704400ee50 comp rx=0 tx=0).stop 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 shutdown_connections 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703806c2b0 0x7f703806e770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.670+0000 7f705fc00640 1 --2- 172.21.15.146:0/2993306257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70580c9180 0x7f70581532b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.671+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 >> 172.21.15.146:0/2993306257 conn(0x7f705806bc10 msgr2=0x7f70580c2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:49.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.671+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 shutdown_connections 2024-09-16T01:50:49.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.671+0000 7f705fc00640 1 -- 172.21.15.146:0/2993306257 wait complete. 2024-09-16T01:50:49.678 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:49.676+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:49.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.718+0000 7f48fb400640 1 -- 172.21.15.146:0/1246836099 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v55) ==== 101+0+0 (secure 0 0 0) 0x7f48fc05cce0 con 0x7f4904158bb0 2024-09-16T01:50:49.720 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f48e006c2e0 msgr2=0x7f48e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f48e006c2e0 0x7f48e006e7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009560 tx=0x7f48f4009190 comp rx=0 tx=0).stop 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 msgr2=0x7f4904156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f4904156d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c7d0 tx=0x7f48fc00cc90 comp rx=0 tx=0).stop 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 shutdown_connections 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f48e006c2e0 0x7f48e006e7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4904158bb0 0x7f4904156d80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4904152850 0x7f49040c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 --2- 172.21.15.146:0/1246836099 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49040c6bb0 0x7f49041f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:49.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.721+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 >> 172.21.15.146:0/1246836099 conn(0x7f490406f3c0 msgr2=0x7f490406f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:49.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.722+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 shutdown_connections 2024-09-16T01:50:49.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:49.722+0000 7f4909a00640 1 -- 172.21.15.146:0/1246836099 wait complete. 2024-09-16T01:50:49.783 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:49.783 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:49.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 122.012374139, 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:49.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:49.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:49.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.580820407, 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076494839999999998, 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 74 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.791 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199908+0000", 2024-09-16T01:50:49.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5600000000000001e-07 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199910+0000", 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3530000000000002e-06 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199914+0000", 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.507e-06 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.199917+0000", 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201181+0000", 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012636909999999999 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201226+0000", 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4946e-05 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.201294+0000", 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8229e-05 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.202150+0000", 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085571099999999999 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207499+0000", 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053494340000000001 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207534+0000", 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4798999999999997e-05 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.207557+0000", 2024-09-16T01:50:49.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3390000000000001e-05 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.560896185, 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095345970000000006, 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 86 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219832+0000", 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219834+0000", 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219837+0000", 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3739999999999998e-06 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219840+0000", 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6359999999999998e-06 2024-09-16T01:50:49.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221228+0000", 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001388317 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221268+0000", 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9796000000000003e-05 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221334+0000", 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6029000000000001e-05 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222486+0000", 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001151766 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229319+0000", 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068331779999999996 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229347+0000", 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8425e-05 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229366+0000", 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9131e-05 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.803 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.560876267, 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096066580000000006, 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219852+0000", 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.22e-07 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219899+0000", 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7114000000000003e-05 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219904+0000", 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7350000000000004e-06 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219907+0000", 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.993e-06 2024-09-16T01:50:49.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221525+0000", 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001617807 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221564+0000", 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.221641+0000", 2024-09-16T01:50:49.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7390000000000003e-05 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222890+0000", 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001248627 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229399+0000", 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065092529999999996 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229434+0000", 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4743999999999998e-05 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229458+0000", 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4533e-05 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.809 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.560815683, 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096361740000000008, 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 88 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219912+0000", 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219913+0000", 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.27e-07 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219915+0000", 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219918+0000", 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3790000000000002e-06 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.219924+0000", 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2360000000000004e-06 2024-09-16T01:50:49.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222187+0000", 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002263064 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222230+0000", 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3859999999999997e-05 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.222298+0000", 2024-09-16T01:50:49.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7441999999999996e-05 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.223301+0000", 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002925 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229497+0000", 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006195809 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229528+0000", 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1143999999999999e-05 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.229549+0000", 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0789e-05 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.815 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.314530163, 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083678369999999995, 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 115 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466198+0000", 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2100000000000002e-07 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466329+0000", 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130455 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466338+0000", 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0690000000000008e-06 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466342+0000", 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999996e-06 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470359+0000", 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040172460000000004 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470400+0000", 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1545999999999998e-05 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470501+0000", 2024-09-16T01:50:49.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010046 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471882+0000", 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013805709999999999 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474515+0000", 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026338849999999999 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474545+0000", 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9989999999999999e-05 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.474566+0000", 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0366000000000001e-05 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:49.821 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.314379649, 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013500404000000001, 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 116 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:49.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466348+0000", 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466349+0000", 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5999999999999999e-07 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466351+0000", 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.458e-06 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466357+0000", 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4140000000000002e-06 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466361+0000", 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7709999999999997e-06 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470887+0000", 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045256920000000004 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470926+0000", 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8674999999999998e-05 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.470991+0000", 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5063999999999999e-05 2024-09-16T01:50:49.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472064+0000", 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010735499999999999 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479790+0000", 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007725951 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479826+0000", 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5395999999999998e-05 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479849+0000", 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3073000000000001e-05 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.827 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.314360081, 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013567350000000001, 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 117 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466368+0000", 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8599999999999999e-07 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466373+0000", 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2250000000000002e-06 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466379+0000", 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0650000000000004e-06 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.466383+0000", 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2980000000000002e-06 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471231+0000", 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048483609999999998 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471274+0000", 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2416999999999997e-05 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.471342+0000", 2024-09-16T01:50:49.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8502000000000002e-05 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.472206+0000", 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086405100000000001 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479879+0000", 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0076726199999999998 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479913+0000", 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3902000000000002e-05 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.479935+0000", 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2523000000000001e-05 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:49.833 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.744203867, 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080440679999999997, 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 134 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036524+0000", 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036525+0000", 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036648+0000", 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012339 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036657+0000", 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0950000000000002e-06 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036661+0000", 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2910000000000001e-06 2024-09-16T01:50:49.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038693+0000", 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002031775 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038732+0000", 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8819999999999998e-05 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.038795+0000", 2024-09-16T01:50:49.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2969e-05 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041638+0000", 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002843319 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044419+0000", 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027808730000000001 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044512+0000", 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2793000000000004e-05 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044568+0000", 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6301999999999997e-05 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.839 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.744036049, 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079840499999999995, 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 135 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036692+0000", 2024-09-16T01:50:49.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7899999999999999e-07 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036695+0000", 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.712e-06 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036703+0000", 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3799999999999996e-06 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036706+0000", 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3069999999999998e-06 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039236+0000", 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00252988 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039275+0000", 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9419000000000003e-05 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039338+0000", 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3268999999999993e-05 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041719+0000", 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023803819999999999 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044619+0000", 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029005900000000002 2024-09-16T01:50:49.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044652+0000", 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2916999999999997e-05 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044676+0000", 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3714999999999999e-05 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.744017067, 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080387640000000003, 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 136 2024-09-16T01:50:49.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036711+0000", 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.41e-07 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036716+0000", 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0950000000000003e-06 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036720+0000", 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.053e-06 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036725+0000", 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.515e-06 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039545+0000", 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028196850000000002 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039614+0000", 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9159000000000006e-05 2024-09-16T01:50:49.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039701+0000", 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7545000000000003e-05 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042080+0000", 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023788669999999998 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044703+0000", 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0026225390000000001 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044732+0000", 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8894000000000002e-05 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044750+0000", 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8170999999999999e-05 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.743997436, 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080933120000000001, 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 137 2024-09-16T01:50:49.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036731+0000", 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2999999999999999e-07 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036849+0000", 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011776499999999999 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036854+0000", 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:50:49.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036858+0000", 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.535e-06 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.039985+0000", 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031270170000000002 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040023+0000", 2024-09-16T01:50:49.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8195000000000003e-05 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040094+0000", 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1316000000000006e-05 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042086+0000", 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019912810000000001 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044775+0000", 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002689482 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044803+0000", 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7983e-05 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044824+0000", 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0767e-05 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.74386353, 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080282270000000006, 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036865+0000", 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.58e-07 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036867+0000", 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036870+0000", 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3579999999999999e-06 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036873+0000", 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5500000000000001e-06 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040429+0000", 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035560520000000001 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040468+0000", 2024-09-16T01:50:49.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9546999999999999e-05 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040530+0000", 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2059999999999999e-05 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042091+0000", 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015606890000000001 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044847+0000", 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002756072 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044874+0000", 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6701000000000001e-05 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.044893+0000", 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8839999999999999e-05 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.743851992, 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010113788, 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036876+0000", 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0100000000000001e-07 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036883+0000", 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.097e-06 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036887+0000", 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3809999999999999e-06 2024-09-16T01:50:49.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036891+0000", 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8410000000000002e-06 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040858+0000", 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039678359999999998 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040900+0000", 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1442000000000001e-05 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.040969+0000", 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9457999999999998e-05 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042461+0000", 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001491474 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046934+0000", 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0044727070000000002 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046967+0000", 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3079e-05 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.046990+0000", 2024-09-16T01:50:49.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3272e-05 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.743833581, 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01019538, 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036894+0000", 2024-09-16T01:50:49.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036895+0000", 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0800000000000001e-07 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036995+0000", 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010009900000000001 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.036999+0000", 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3499999999999999e-06 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037004+0000", 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0100000000000003e-06 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041191+0000", 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041866100000000003 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041229+0000", 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8093999999999998e-05 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041290+0000", 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1558999999999998e-05 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.042465+0000", 2024-09-16T01:50:49.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011749950000000001 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047033+0000", 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045672739999999996 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047065+0000", 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.269e-05 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047090+0000", 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4491e-05 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.743718911, 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010148492, 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.876 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 141 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037009+0000", 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7099999999999998e-07 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037014+0000", 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.758e-06 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037019+0000", 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999997e-06 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.037024+0000", 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.101e-06 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041729+0000", 2024-09-16T01:50:49.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047053040000000004 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041768+0000", 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8506e-05 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.041882+0000", 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011406 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.043332+0000", 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001450663 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047114+0000", 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037813870000000002 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047140+0000", 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6840000000000001e-05 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047158+0000", 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7164e-05 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:143 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.732937183, 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073554190000000002, 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047791+0000", 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.37e-07 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047887+0000", 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5592000000000002e-05 2024-09-16T01:50:49.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047893+0000", 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0979999999999999e-06 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047897+0000", 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.534e-06 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048194+0000", 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029762099999999997 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048233+0000", 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8451999999999997e-05 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048294+0000", 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1538000000000006e-05 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.049325+0000", 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010304190000000001 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055091+0000", 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057666870000000004 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055125+0000", 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3593999999999997e-05 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055146+0000", 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1347000000000001e-05 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:144 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 120.73282052499999, 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073339970000000001, 2024-09-16T01:50:49.888 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 144 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047908+0000", 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5699999999999998e-07 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.047994+0000", 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6358000000000002e-05 2024-09-16T01:50:49.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048002+0000", 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.123e-06 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048008+0000", 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1880000000000001e-06 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048528+0000", 2024-09-16T01:50:49.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051995299999999997 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048570+0000", 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2324000000000003e-05 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.048644+0000", 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3961999999999994e-05 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:49.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.052274+0000", 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036303590000000001 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055184+0000", 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029098399999999999 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055217+0000", 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2492e-05 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:49.055242+0000", 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4941000000000001e-05 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:11 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.917287286, 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011648319000000001, 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.894 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863441+0000", 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863442+0000", 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4410000000000001e-06 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863452+0000", 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6870000000000006e-06 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863464+0000", 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2493999999999999e-05 2024-09-16T01:50:49.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863473+0000", 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3000000000000002e-06 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867681+0000", 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042079090000000001 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867709+0000", 2024-09-16T01:50:49.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.809e-05 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874532+0000", 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0068230239999999996 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875089+0000", 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055737400000000002 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:28 2.16 e22/19 2:6f2c300e:::60f.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 100.917225948, 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01581428, 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863502+0000", 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863503+0000", 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2899999999999999e-07 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:49.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863507+0000", 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7600000000000002e-06 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863515+0000", 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0360000000000006e-06 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.863521+0000", 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872259+0000", 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087379050000000007 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872281+0000", 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2072000000000001e-05 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879273+0000", 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006991611 2024-09-16T01:50:49.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.879316+0000", 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3390999999999998e-05 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:49.903 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:49.904 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:49.781+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:49.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:49.946 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:49.944+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:50.007 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:50.005+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:50.108 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:50.106+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:50.108 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:50.108 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:50.108 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:50.108 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:50.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:50.209 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:50.207+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:50.309 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:50.307+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:50.314 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:50.314 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:50.314 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:50.314 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:50.314 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:50.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:50.423 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:50.421+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:50.512 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:50.510+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:50.624 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:50.620+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:50.666 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.83779192, 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056789170000000003, 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824835+0000", 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6899999999999996e-07 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824840+0000", 2024-09-16T01:50:50.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1760000000000003e-06 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824848+0000", 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8149999999999995e-06 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824855+0000", 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3969999999999999e-06 2024-09-16T01:50:50.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826072+0000", 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012175860000000001 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826182+0000", 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011003299999999999 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:50.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826327+0000", 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014442099999999999 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830198+0000", 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038715429999999999 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830398+0000", 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199664 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830473+0000", 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5043000000000006e-05 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830513+0000", 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.83775005, 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073307229999999999, 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824876+0000", 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824877+0000", 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7100000000000002e-07 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824882+0000", 2024-09-16T01:50:50.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.729e-06 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824890+0000", 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4200000000000007e-06 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824898+0000", 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9000000000000006e-06 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826709+0000", 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018104550000000001 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.826862+0000", 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015355300000000001 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827015+0000", 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015280900000000001 2024-09-16T01:50:50.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830898+0000", 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038829139999999999 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832116+0000", 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001217713 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832166+0000", 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0414999999999998e-05 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832207+0000", 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1244e-05 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:50.678 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.837705771, 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.007373704, 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:50.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824921+0000", 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1099999999999995e-07 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824926+0000", 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3549999999999998e-06 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824934+0000", 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3590000000000008e-06 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824940+0000", 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.964e-06 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.827963+0000", 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030234229999999999 2024-09-16T01:50:50.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828131+0000", 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016780199999999999 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828236+0000", 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010496400000000001 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.830945+0000", 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027091889999999999 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832216+0000", 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001270291 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832259+0000", 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3547999999999997e-05 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.832294+0000", 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5197999999999997e-05 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e21)", 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.837676894, 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091880980000000004, 2024-09-16T01:50:50.684 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824950+0000", 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.13e-07 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824977+0000", 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6580999999999998e-05 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824983+0000", 2024-09-16T01:50:50.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1879999999999999e-06 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.824989+0000", 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828764+0000", 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037748289999999999 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828903+0000", 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139386 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834138+0000", 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052348059999999998 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:29 2.b e22/19 2:d0630e4c:::mds0_sessionmap:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 106.83396968700001, 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054985809999999998, 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828657+0000", 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828658+0000", 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2900000000000002e-07 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828663+0000", 2024-09-16T01:50:50.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5489999999999997e-06 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828672+0000", 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7730000000000005e-06 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828679+0000", 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3139999999999998e-06 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828861+0000", 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018128600000000001 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.828893+0000", 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1980999999999998e-05 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834096+0000", 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0052031999999999998 2024-09-16T01:50:50.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:03.834155+0000", 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9549000000000002e-05 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:15 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.79850048199999, 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0084377040000000007, 2024-09-16T01:50:50.693 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864126+0000", 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864127+0000", 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.113e-06 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864135+0000", 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1990000000000006e-06 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864155+0000", 2024-09-16T01:50:50.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0058e-05 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.864166+0000", 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.023e-05 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865139+0000", 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000973708 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.865159+0000", 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9279e-05 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872509+0000", 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073505419999999998 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.872564+0000", 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4574999999999998e-05 2024-09-16T01:50:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:32 2.8 e22/19 2:1470b2f8:::613.00000000:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.795440333, 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015898095000000001, 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867186+0000", 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867187+0000", 2024-09-16T01:50:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8199999999999998e-07 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867197+0000", 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4790000000000002e-06 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867948+0000", 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075172699999999999 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.867959+0000", 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1001e-05 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869927+0000", 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019674779999999999 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.869948+0000", 2024-09-16T01:50:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1719e-05 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883026+0000", 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013078021 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.883084+0000", 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7788000000000003e-05 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:12 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.794580885, 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014447523, 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868046+0000", 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868047+0000", 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.798e-06 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868054+0000", 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3369999999999996e-06 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868065+0000", 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1474000000000001e-05 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868074+0000", 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3850000000000002e-06 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868170+0000", 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6328000000000003e-05 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868685+0000", 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00051544099999999995 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868817+0000", 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013117900000000001 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870872+0000", 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002055702 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882377+0000", 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011505246 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882446+0000", 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8702999999999996e-05 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882493+0000", 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.693e-05 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.1.9:29 2.14 2:2e2b5214:::610.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.794526992, 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014505634999999999, 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868099+0000", 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868100+0000", 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2399999999999998e-07 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868104+0000", 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9160000000000003e-06 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868112+0000", 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4309999999999994e-06 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.868119+0000", 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7649999999999996e-06 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870086+0000", 2024-09-16T01:50:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019668929999999999 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870160+0000", 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3398000000000004e-05 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.870285+0000", 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012504700000000001 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874563+0000", 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004278263 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882509+0000", 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079456660000000005 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882556+0000", 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7448999999999999e-05 2024-09-16T01:50:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.882605+0000", 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9183e-05 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:7 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'3, mlcod=22'1)", 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.788503019, 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012577031000000001, 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874123+0000", 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874124+0000", 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5000000000000002e-07 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874132+0000", 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9740000000000002e-06 2024-09-16T01:50:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874142+0000", 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9529999999999994e-06 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874148+0000", 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-06 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874616+0000", 2024-09-16T01:50:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00046764300000000001 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874634+0000", 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7949999999999999e-05 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886637+0000", 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012003381 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886701+0000", 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3084999999999997e-05 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:8 2.b e22/19 2:d70a01c6:::60c.00000000:head v 22'4, mlcod=22'1)", 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.78845289100001, 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012554974, 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874174+0000", 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8299999999999998e-07 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874205+0000", 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1350999999999998e-05 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874213+0000", 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869999999999992e-06 2024-09-16T01:50:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874233+0000", 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9656999999999998e-05 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874848+0000", 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061471799999999999 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874864+0000", 2024-09-16T01:50:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6565000000000001e-05 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886704+0000", 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011839989 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886729+0000", 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4423999999999999e-05 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:33 2.3 e22/19 2:c112adae:::101.00000000:head v 22'1, mlcod=0'0)", 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.788376558, 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012504431999999999, 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874250+0000", 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4099999999999999e-07 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874261+0000", 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0693000000000001e-05 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874268+0000", 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2099999999999996e-06 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874274+0000", 2024-09-16T01:50:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7450000000000001e-06 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875031+0000", 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00075732900000000005 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875046+0000", 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4548e-05 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886731+0000", 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011685009 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886754+0000", 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3456999999999999e-05 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.1.9:18 2.3 e22/19 2:c27f454c:::mds1_sessionmap:head v 22'2, mlcod=0'0)", 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.78834520300001, 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012496195999999999, 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874281+0000", 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874282+0000", 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.03e-07 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874294+0000", 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1936000000000001e-05 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874300+0000", 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.874320+0000", 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9539999999999999e-05 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875259+0000", 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093904600000000002 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.875275+0000", 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6430999999999999e-05 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886756+0000", 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01148075 2024-09-16T01:50:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:08.886777+0000", 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1365e-05 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:43 2.14 2:2e2fa760:::200.00000001:head [write 2014~2077 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e23)", 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 71.394672080000007, 2024-09-16T01:50:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057191869999999997, 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267954+0000", 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267957+0000", 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7209999999999999e-06 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.267974+0000", 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6379e-05 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268002+0000", 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8560000000000001e-05 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268016+0000", 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.358e-05 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268092+0000", 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6069999999999995e-05 2024-09-16T01:50:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op must wait for map", 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.268099+0000", 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6249999999999998e-06 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.271964+0000", 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038650540000000001 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272037+0000", 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2229000000000003e-05 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.272113+0000", 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6215000000000001e-05 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273222+0000", 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108709 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273548+0000", 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00032667699999999999 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273630+0000", 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1652999999999999e-05 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:49:39.273674+0000", 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3714999999999997e-05 2024-09-16T01:50:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.407728127, 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:50:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:50:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:11 3.14 e55/20 3:29a5f800:::10000000002.00000008:head v 55'1, mlcod=0'0)", 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.347803+0000", 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.314823931, 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019103017, 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347803+0000", 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347803+0000", 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347803+0000", 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1300000000000001e-07 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.360655+0000", 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012852555 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.360685+0000", 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.921e-05 2024-09-16T01:50:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.360697+0000", 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2386e-05 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.360880+0000", 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018255100000000001 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.360902+0000", 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2691e-05 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.366835+0000", 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005933191 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.366906+0000", 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0119999999999999e-05 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:13 3.2 e55/20 3:438a76aa:::10000000002.0000000a:head v 55'1, mlcod=0'0)", 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.389206+0000", 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.27342092099999998, 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010509777999999999, 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.389206+0000", 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.389206+0000", 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.389207+0000", 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4529999999999999e-06 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393465+0000", 2024-09-16T01:50:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042575180000000001 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393492+0000", 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7509e-05 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393503+0000", 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0985000000000001e-05 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393570+0000", 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7118000000000004e-05 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393599+0000", 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8988000000000001e-05 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.399666+0000", 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060673250000000001 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.399715+0000", 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8881999999999997e-05 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:18 3.30 e55/20 3:0ca36612:::10000000002.0000000f:head v 55'1, mlcod=0'0)", 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.415309+0000", 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.247317483, 2024-09-16T01:50:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057316379999999998, 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.415309+0000", 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.415309+0000", 2024-09-16T01:50:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.415311+0000", 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3020000000000002e-06 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.416689+0000", 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013775879999999999 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.416717+0000", 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7699999999999999e-05 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.416730+0000", 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3395e-05 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.416815+0000", 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4948000000000001e-05 2024-09-16T01:50:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.416839+0000", 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4048000000000001e-05 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.420994+0000", 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004154713 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.421041+0000", 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6944000000000003e-05 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:28 3.36 e55/20 3:6d346ec1:::10000000002.00000019:head v 55'1, mlcod=0'0)", 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.479723+0000", 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.182903758, 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015804854, 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.479723+0000", 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.479723+0000", 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.479725+0000", 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9350000000000001e-06 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.487119+0000", 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073947329999999997 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.487150+0000", 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0988000000000002e-05 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.487164+0000", 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3677e-05 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.487998+0000", 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083378200000000003 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.488027+0000", 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8900000000000001e-05 2024-09-16T01:50:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.495463+0000", 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074363459999999999 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.495528+0000", 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4492999999999994e-05 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:33 3.5 3:a31f5a4b:::10000000002.0000001e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.526676+0000", 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.135950232, 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032362625999999999, 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 33 2024-09-16T01:50:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.526676+0000", 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.526676+0000", 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.526678+0000", 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6309999999999999e-06 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.548885+0000", 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022207286999999999 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.548916+0000", 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0815999999999998e-05 2024-09-16T01:50:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.548930+0000", 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4031e-05 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.549021+0000", 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0828000000000005e-05 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.549185+0000", 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016452400000000001 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.549337+0000", 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015207299999999999 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.558030+0000", 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086927170000000008 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.558920+0000", 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00088993399999999999 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.558988+0000", 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7992999999999998e-05 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.559039+0000", 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0791999999999998e-05 2024-09-16T01:50:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:50.771 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:50.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:50.773 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:50.770+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:50.873 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:50.871+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:50.973 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:50.971+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:51.074 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:51.072+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:51.099 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:90 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.099 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.047492360999999997, 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047523768000000001, 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 90 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048233+0000", 2024-09-16T01:50:51.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7070000000000001e-06 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072652+0000", 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024419021999999999 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072684+0000", 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1900000000000003e-05 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072697+0000", 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3227999999999999e-05 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072918+0000", 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022140899999999999 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.073060+0000", 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014170900000000001 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.073209+0000", 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149133 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.079317+0000", 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061078970000000002 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.104 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop_reply(client.4619.0:90 3.11 e55/20)", 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.095452+0000", 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.00027017099999999999, 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038773900000000001, 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "queued for pg", 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095452+0000", 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095452+0000", 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095454+0000", 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5430000000000001e-06 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095461+0000", 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4379999999999998e-06 2024-09-16T01:50:51.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095484+0000", 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2719000000000001e-05 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095497+0000", 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2944e-05 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.107 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:50:51.108 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 2 2024-09-16T01:50:51.108 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:51.118 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:51.180 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:51.176+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:51.277 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:51.275+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:51.377 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:51.375+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:51.383 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:51.383 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:51.384 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:51.384 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:51.384 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:51.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:51.495 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:51.494+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:51.578 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:51.576+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:51.681 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:51.680+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:51.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:51.720 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4602799120000001, 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:51.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:11 3.14 3:29a5f800:::10000000002.00000008:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.328881+0000", 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.3866784320000001, 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038380428000000001, 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 11 2024-09-16T01:50:51.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328881+0000", 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328881+0000", 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328883+0000", 2024-09-16T01:50:51.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.069e-06 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347064+0000", 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018180939 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347078+0000", 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3278999999999999e-05 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347083+0000", 2024-09-16T01:50:51.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1560000000000003e-06 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347134+0000", 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0794999999999997e-05 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347222+0000", 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8837000000000003e-05 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.347313+0000", 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0278999999999994e-05 2024-09-16T01:50:51.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.361003+0000", 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013690357 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.367155+0000", 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061516349999999999 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.367214+0000", 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9120999999999999e-05 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.367262+0000", 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7960999999999997e-05 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:14 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.3629595999999999, 2024-09-16T01:50:51.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051400593000000001, 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 14 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:50:51.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352602+0000", 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393382+0000", 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040779871000000002 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393422+0000", 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9765000000000003e-05 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393436+0000", 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3633e-05 2024-09-16T01:50:51.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393509+0000", 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3847e-05 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393696+0000", 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186911 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:51.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393839+0000", 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014274700000000001 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.401310+0000", 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074705040000000002 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403876+0000", 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025662020000000001 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403939+0000", 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3423999999999994e-05 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.404001+0000", 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1649000000000006e-05 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.261907042, 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:50:51.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.746 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1531044109999999, 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:50:51.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:50:51.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.753 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:37 3.30 3:0d7ddb42:::10000000002.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.597266+0000", 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.1182939279999999, 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049636995000000003, 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.754 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 37 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597266+0000", 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597266+0000", 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597268+0000", 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8020000000000001e-06 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.631882+0000", 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034614170999999999 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.631901+0000", 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9601000000000001e-05 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.631914+0000", 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-05 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.632014+0000", 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9798999999999998e-05 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.632166+0000", 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152032 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.632317+0000", 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015093999999999999 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.645621+0000", 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013304261 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.646789+0000", 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001168056 2024-09-16T01:50:51.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.646848+0000", 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8539999999999999e-05 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.646903+0000", 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5192999999999998e-05 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:53 3.9 3:9387ba5e:::10000000002.00000032:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.727663+0000", 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.98789675300000002, 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048406738999999997, 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 53 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.727663+0000", 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.727663+0000", 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.727665+0000", 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.244e-06 2024-09-16T01:50:51.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753019+0000", 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025353704000000001 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753046+0000", 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7200999999999999e-05 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753059+0000", 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2741e-05 2024-09-16T01:50:51.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753160+0000", 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101446 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753321+0000", 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016072800000000001 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:51.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753482+0000", 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016113799999999999 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.771934+0000", 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018452110000000001 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775957+0000", 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040221429999999997 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.776016+0000", 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9511e-05 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.776070+0000", 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3773000000000002e-05 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:61 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.766 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.797869+0000", 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.917690848, 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037545544, 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 61 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.797869+0000", 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.797869+0000", 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.797871+0000", 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2230000000000001e-06 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.814901+0000", 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017029289 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.814932+0000", 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1084000000000003e-05 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.814944+0000", 2024-09-16T01:50:51.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2490999999999999e-05 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.815052+0000", 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010787 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.815198+0000", 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014595999999999999 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.815334+0000", 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136315 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.834617+0000", 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019282938999999999 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835137+0000", 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00052005599999999997 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835336+0000", 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019874899999999999 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835415+0000", 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8567999999999997e-05 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.772 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:66 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.89387491500000005, 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053741734999999999, 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821687+0000", 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3750000000000001e-06 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862658+0000", 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040970816 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862692+0000", 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3420000000000002e-05 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862706+0000", 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4017e-05 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862879+0000", 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173015 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863040+0000", 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016103800000000001 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863192+0000", 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152738 2024-09-16T01:50:51.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.867722+0000", 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045298689999999997 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875297+0000", 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075744860000000001 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875372+0000", 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5049999999999995e-05 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875427+0000", 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4911000000000003e-05 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.778 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.833535778, 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.779 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:50:51.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:50:51.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:50:51.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:50:51.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:50:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.786 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:73 3.13 3:ca5a6998:::10000000002.00000046:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.925484+0000", 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.79007632299999997, 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050407227999999998, 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 73 2024-09-16T01:50:51.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925484+0000", 2024-09-16T01:50:51.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925484+0000", 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925486+0000", 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.125e-06 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946032+0000", 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020546420999999999 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946049+0000", 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6677e-05 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946059+0000", 2024-09-16T01:50:51.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9979999999999998e-06 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946147+0000", 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8294000000000005e-05 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946274+0000", 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127007 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946407+0000", 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013287100000000001 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.955245+0000", 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088381459999999998 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.975775+0000", 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020529830999999998 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.975836+0000", 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1328000000000006e-05 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.975891+0000", 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4530000000000001e-05 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:75 3.30 3:0d2dbd17:::10000000002.00000048:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.76945435100000004, 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050908518999999999, 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.794 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946107+0000", 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7710000000000001e-06 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960806+0000", 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014698386000000001 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960821+0000", 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960832+0000", 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0698e-05 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960964+0000", 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013269700000000001 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961122+0000", 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157765 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961267+0000", 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014486100000000001 2024-09-16T01:50:51.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981986+0000", 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020719158000000001 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996886+0000", 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014899654999999999 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996958+0000", 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2095999999999995e-05 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997014+0000", 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6258999999999998e-05 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:90 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.800 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.66732984299999998, 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048363178, 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 90 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048233+0000", 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7070000000000001e-06 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072652+0000", 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024419021999999999 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072684+0000", 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1900000000000003e-05 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072697+0000", 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3227999999999999e-05 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072918+0000", 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022140899999999999 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.073060+0000", 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014170900000000001 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.073209+0000", 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149133 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.079317+0000", 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061078970000000002 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.096463+0000", 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017145512000000002 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.096532+0000", 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9739000000000004e-05 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.096593+0000", 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:103 3.d 3:b1e2826f:::10000000002.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.159691+0000", 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.55586851699999995, 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048938638999999999, 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.807 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 103 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.159691+0000", 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.159691+0000", 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.159694+0000", 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.847e-06 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200343+0000", 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040648534 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200370+0000", 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6913000000000001e-05 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200382+0000", 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2218000000000001e-05 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200485+0000", 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103402 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200654+0000", 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016836100000000001 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200806+0000", 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015213 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.207774+0000", 2024-09-16T01:50:51.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0069686119999999999 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.208519+0000", 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074514399999999999 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.208562+0000", 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2327999999999999e-05 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.208630+0000", 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8150000000000003e-05 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:105 3.30 3:0eedefab:::10000000002.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.200418+0000", 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.51514157299999996, 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032969156999999999, 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 105 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200418+0000", 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200418+0000", 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200420+0000", 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4610000000000001e-06 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.222688+0000", 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022268295 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.222712+0000", 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3674999999999999e-05 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.222724+0000", 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2498e-05 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.222819+0000", 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4834999999999996e-05 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.222990+0000", 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017128900000000001 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.223169+0000", 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017897099999999999 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.230448+0000", 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072782070000000001 2024-09-16T01:50:51.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.233303+0000", 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002855598 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.233351+0000", 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7598000000000001e-05 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.233387+0000", 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6730000000000002e-05 2024-09-16T01:50:51.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:110 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.47919806399999998, 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053794485000000003, 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 110 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236365+0000", 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1590000000000002e-06 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278463+0000", 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042097744999999999 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278494+0000", 2024-09-16T01:50:51.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1009000000000001e-05 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278508+0000", 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3901e-05 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278639+0000", 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013188100000000001 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278806+0000", 2024-09-16T01:50:51.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016689799999999999 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278974+0000", 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016719799999999999 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.286538+0000", 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075647910000000004 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290050+0000", 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035116969999999998 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290114+0000", 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4298e-05 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290156+0000", 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1907999999999999e-05 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:112 3.14 3:280f29a3:::10000000002.0000006d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.278544+0000", 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.43701628100000001, 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040090057999999998, 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.825 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 112 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278544+0000", 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278544+0000", 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278545+0000", 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.578e-06 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.306923+0000", 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028377775000000001 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.306945+0000", 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2048999999999998e-05 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.306955+0000", 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.307015+0000", 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0479000000000002e-05 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.307113+0000", 2024-09-16T01:50:51.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8039000000000001e-05 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.307202+0000", 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9067000000000006e-05 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.312168+0000", 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049660590000000001 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.318466+0000", 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062973259999999998 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.318564+0000", 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8549999999999997e-05 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.318634+0000", 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9337000000000004e-05 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.33487303800000001, 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.831 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:50:51.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:50:51.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:147 3.9 3:91a8729a:::10000000002.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.621256+0000", 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.094303475999999997, 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036961630000000002, 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 147 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.837 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621256+0000", 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621256+0000", 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621259+0000", 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3010000000000001e-06 2024-09-16T01:50:51.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.645714+0000", 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024454900000000002 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.645734+0000", 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0798e-05 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.645746+0000", 2024-09-16T01:50:51.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1739000000000001e-05 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.645842+0000", 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5421999999999995e-05 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.646003+0000", 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161609 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.646165+0000", 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016216700000000001 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654125+0000", 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079598410000000005 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.658122+0000", 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039965529999999999 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.658177+0000", 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5318000000000001e-05 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.658218+0000", 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0982000000000002e-05 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.842 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:149 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.646642+0000", 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.068918038000000001, 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035986297, 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 149 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.646642+0000", 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.646642+0000", 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.646644+0000", 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2730000000000001e-06 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.674481+0000", 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027837147999999999 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.674507+0000", 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6129999999999999e-05 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.674520+0000", 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2638e-05 2024-09-16T01:50:51.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.674626+0000", 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010625500000000001 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.674789+0000", 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162423 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.674943+0000", 2024-09-16T01:50:51.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154668 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.679652+0000", 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047082749999999996 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.682502+0000", 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0028501020000000002 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.682568+0000", 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5914000000000006e-05 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.682628+0000", 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0470999999999998e-05 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:51.848 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:51.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:50:51.850 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:51.851 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:51.848+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:51.951 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:51.950+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:51.986 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:51.987 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:51.987 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:51.987 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:51.998 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:52.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.037+0000 7f6af1800640 1 Processor -- start 2024-09-16T01:50:52.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.037+0000 7f6af1800640 1 -- start start 2024-09-16T01:50:52.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.039+0000 7f6af1800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0c7770 0x7f6aec0d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.039+0000 7f6af1800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aec0cb600 con 0x7f6aec0d0810 2024-09-16T01:50:52.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.039+0000 7f6af1800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aec0cb8e0 con 0x7f6aec154f40 2024-09-16T01:50:52.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.039+0000 7f6af1800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6aec0cbbc0 con 0x7f6aec0c7770 2024-09-16T01:50:52.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.040+0000 7f6aea000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0c7770 0x7f6aec0d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.040+0000 7f6aeb400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6aec0d0810 0x7f6aec15ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53462/0 (socket says 172.21.15.146:53462) 2024-09-16T01:50:52.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.040+0000 7f6aeb400640 1 -- 172.21.15.146:0/2970038718 learned_addr learned my addr 172.21.15.146:0/2970038718 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:52.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.041+0000 7f6ae9600640 1 -- 172.21.15.146:0/2970038718 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2075575530 0 0) 0x7f6aec0cb600 con 0x7f6aec0d0810 2024-09-16T01:50:52.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.041+0000 7f6ae9600640 1 -- 172.21.15.146:0/2970038718 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ad0003520 con 0x7f6aec0d0810 2024-09-16T01:50:52.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.042+0000 7f6aea000640 1 -- 172.21.15.146:0/2970038718 >> v1:172.21.15.161:6789/0 conn(0x7f6aec154f40 legacy=0x7f6aec15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:52.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.042+0000 7f6ae9600640 1 -- 172.21.15.146:0/2970038718 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3681529448 0 0) 0x7f6ad0003520 con 0x7f6aec0d0810 2024-09-16T01:50:52.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.042+0000 7f6aea000640 1 -- 172.21.15.146:0/2970038718 >> v1:172.21.15.146:6789/0 conn(0x7f6aec0d0810 legacy=0x7f6aec15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.043+0000 7f6aea000640 1 -- 172.21.15.146:0/2970038718 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec0cc0c0 con 0x7f6aec0c7770 2024-09-16T01:50:52.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.045+0000 7f6aea000640 1 --2- 172.21.15.146:0/2970038718 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0c7770 0x7f6aec0d02d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0097d0 tx=0x7f6adc02a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62a59c6d42ecabf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.046+0000 7f6ae9600640 1 -- 172.21.15.146:0/2970038718 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc037070 con 0x7f6aec0c7770 2024-09-16T01:50:52.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.046+0000 7f6ae9600640 1 -- 172.21.15.146:0/2970038718 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc02abd0 con 0x7f6aec0c7770 2024-09-16T01:50:52.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.046+0000 7f6ae9600640 1 -- 172.21.15.146:0/2970038718 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40019f0 con 0x7f6aec0c7770 2024-09-16T01:50:52.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.047+0000 7f6af1800640 1 -- 172.21.15.146:0/2970038718 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0c7770 msgr2=0x7f6aec0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.047+0000 7f6af1800640 1 --2- 172.21.15.146:0/2970038718 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0c7770 0x7f6aec0d02d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0097d0 tx=0x7f6adc02a140 comp rx=0 tx=0).stop 2024-09-16T01:50:52.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.047+0000 7f6af1800640 1 -- 172.21.15.146:0/2970038718 shutdown_connections 2024-09-16T01:50:52.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.047+0000 7f6af1800640 1 --2- 172.21.15.146:0/2970038718 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0c7770 0x7f6aec0d02d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.047+0000 7f6af1800640 1 -- 172.21.15.146:0/2970038718 >> 172.21.15.146:0/2970038718 conn(0x7f6aec06cb50 msgr2=0x7f6aec06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:52.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.048+0000 7f6af1800640 1 -- 172.21.15.146:0/2970038718 shutdown_connections 2024-09-16T01:50:52.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.048+0000 7f6af1800640 1 -- 172.21.15.146:0/2970038718 wait complete. 2024-09-16T01:50:52.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.054+0000 7f6af1800640 1 Processor -- start 2024-09-16T01:50:52.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.055+0000 7f6af1800640 1 -- start start 2024-09-16T01:50:52.056 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:52.055+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:52.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.057+0000 7f6af1800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0d0810 0x7f6aec0c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.057+0000 7f6af1800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aec154f40 0x7f6aec1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.058+0000 7f6af1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aec1f0e00 0x7f6aec0c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.058+0000 7f6af1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec0b6210 con 0x7f6aec1f0e00 2024-09-16T01:50:52.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.058+0000 7f6af1800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec155320 con 0x7f6aec154f40 2024-09-16T01:50:52.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.058+0000 7f6af1800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6aec15b280 con 0x7f6aec0d0810 2024-09-16T01:50:52.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.058+0000 7f6aeaa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0d0810 0x7f6aec0c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.059+0000 7f6aeaa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0d0810 0x7f6aec0c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37376/0 (socket says 172.21.15.146:37376) 2024-09-16T01:50:52.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.059+0000 7f6aeaa00640 1 -- 172.21.15.146:0/4058873404 learned_addr learned my addr 172.21.15.146:0/4058873404 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:52.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.059+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aec154f40 0x7f6aec1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.059+0000 7f6aea000640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0d0810 msgr2=0x7f6aec0c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.060+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aec0d0810 0x7f6aec0c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.060+0000 7f6aea000640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aec1f0e00 msgr2=0x7f6aec0c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.060+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aec1f0e00 0x7f6aec0c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.061+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aec1f0e00 0x7f6aec0c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.061+0000 7f6aea000640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec0c70a0 con 0x7f6aec154f40 2024-09-16T01:50:52.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.061+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aec1f0e00 0x7f6aec0c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:52.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.063+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aec154f40 0x7f6aec1515e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6adc031000 tx=0x7f6adc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.065+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc042070 con 0x7f6aec154f40 2024-09-16T01:50:52.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.065+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc004260 con 0x7f6aec154f40 2024-09-16T01:50:52.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.065+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc037040 con 0x7f6aec154f40 2024-09-16T01:50:52.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.065+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec1f12b0 con 0x7f6aec154f40 2024-09-16T01:50:52.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.065+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1f1870 con 0x7f6aec154f40 2024-09-16T01:50:52.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.070+0000 7f6ad7e00640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6ac0005270 con 0x7f6aec154f40 2024-09-16T01:50:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.073+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6adc003610 con 0x7f6aec154f40 2024-09-16T01:50:52.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.074+0000 7f6ae2a00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6acc06c300 0x7f6acc06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.075+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5799+0+0 (secure 0 0 0) 0x7f6adc076510 con 0x7f6aec154f40 2024-09-16T01:50:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.075+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v55) ==== 54+0+16 (secure 0 0 0) 0x7f6adc07a5a0 con 0x7f6aec154f40 2024-09-16T01:50:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.075+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6acc06c300 0x7f6acc06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.075+0000 7f6ad7e00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6ac0000fa0 0x7f6ac00045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.075+0000 7f6ad7e00640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ac0005550 con 0x7f6ac0000fa0 2024-09-16T01:50:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.076+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6ac0000fa0 0x7f6ac00045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.077+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6ac0000fa0 0x7f6ac00045e0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.078+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6acc06c300 0x7f6acc06e7c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8006fd0 tx=0x7f6ad8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.078+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ac0005550 con 0x7f6ac0000fa0 2024-09-16T01:50:52.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.099+0000 7f2cefe00640 1 Processor -- start 2024-09-16T01:50:52.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.099+0000 7f2cefe00640 1 -- start start 2024-09-16T01:50:52.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2cefe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2cefe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cf00c75a0 0x7f2cf00cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2cefe00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf006d140 con 0x7f2cf00cbab0 2024-09-16T01:50:52.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2cefe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf015b880 con 0x7f2cf00c75a0 2024-09-16T01:50:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2cefe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00cab20 con 0x7f2cf00c6b70 2024-09-16T01:50:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2ceea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2ceea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37392/0 (socket says 172.21.15.146:37392) 2024-09-16T01:50:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2ceea00640 1 -- 172.21.15.146:0/1008523362 learned_addr learned my addr 172.21.15.146:0/1008523362 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.100+0000 7f2cee000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2cf00c75a0 0x7f2cf00cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.101+0000 7f2ceea00640 1 -- 172.21.15.146:0/1008523362 >> v2:172.21.15.161:3300/0 conn(0x7f2cf00c75a0 msgr2=0x7f2cf00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.101+0000 7f2ceea00640 1 --2- 172.21.15.146:0/1008523362 >> v2:172.21.15.161:3300/0 conn(0x7f2cf00c75a0 0x7f2cf00cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.101+0000 7f2ceea00640 1 -- 172.21.15.146:0/1008523362 >> v1:172.21.15.161:6789/0 conn(0x7f2cf00cbab0 legacy=0x7f2cf00cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.101+0000 7f2ceea00640 1 -- 172.21.15.146:0/1008523362 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf006d420 con 0x7f2cf00c6b70 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.102+0000 7f2ceea00640 1 --2- 172.21.15.146:0/1008523362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015b340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009870 tx=0x7f2ce002a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=411cc91c8d3dc62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.102+0000 7f2ced600640 1 -- 172.21.15.146:0/1008523362 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0037070 con 0x7f2cf00c6b70 2024-09-16T01:50:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.102+0000 7f2ced600640 1 -- 172.21.15.146:0/1008523362 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce002ae00 con 0x7f2cf00c6b70 2024-09-16T01:50:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.102+0000 7f2ced600640 1 -- 172.21.15.146:0/1008523362 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0032660 con 0x7f2cf00c6b70 2024-09-16T01:50:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.103+0000 7f2cefe00640 1 -- 172.21.15.146:0/1008523362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 msgr2=0x7f2cf015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.103+0000 7f2cefe00640 1 --2- 172.21.15.146:0/1008523362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015b340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009870 tx=0x7f2ce002a0f0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.105+0000 7f2cefe00640 1 -- 172.21.15.146:0/1008523362 shutdown_connections 2024-09-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.105+0000 7f2cefe00640 1 --2- 172.21.15.146:0/1008523362 >> v2:172.21.15.161:3300/0 conn(0x7f2cf00c75a0 0x7f2cf00cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.105+0000 7f2cefe00640 1 --2- 172.21.15.146:0/1008523362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015b340 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.105+0000 7f2cefe00640 1 -- 172.21.15.146:0/1008523362 >> 172.21.15.146:0/1008523362 conn(0x7f2cf00ccc20 msgr2=0x7f2cf00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.106+0000 7f2cefe00640 1 -- 172.21.15.146:0/1008523362 shutdown_connections 2024-09-16T01:50:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.106+0000 7f2cefe00640 1 -- 172.21.15.146:0/1008523362 wait complete. 2024-09-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.108+0000 7f2cefe00640 1 Processor -- start 2024-09-16T01:50:52.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.109+0000 7f2cefe00640 1 -- start start 2024-09-16T01:50:52.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.109+0000 7f6ad7e00640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ac0004f70 con 0x7f6ac0000fa0 2024-09-16T01:50:52.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.111+0000 7f2cefe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.111+0000 7f2cefe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 0x7f2cf0157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.111+0000 7f2cefe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf00cbab0 0x7f2cf00729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.111+0000 7f2cefe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00cb050 con 0x7f2cf00cbab0 2024-09-16T01:50:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.111+0000 7f2cefe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00cad70 con 0x7f2cf00c75a0 2024-09-16T01:50:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.111+0000 7f2cefe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf006cd20 con 0x7f2cf00c6b70 2024-09-16T01:50:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ac0004f70 con 0x7f6ac0000fa0 2024-09-16T01:50:52.113 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f6ad7e00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6ac0001cb0 0x7f6ac0002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f2cee000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 0x7f2cf0157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f2cee000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 0x7f2cf0157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:52520/0 (socket says 172.21.15.146:52520) 2024-09-16T01:50:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f2cee000640 1 -- 172.21.15.146:0/2902530746 learned_addr learned my addr 172.21.15.146:0/2902530746 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:52.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f6ad7e00640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6ac0002290 con 0x7f6ac0001cb0 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.112+0000 7f2ceea00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cef400640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf00cbab0 0x7f2cf00729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cee000640 1 -- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 msgr2=0x7f2cf015c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cee000640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cee000640 1 -- 172.21.15.146:0/2902530746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf00cbab0 msgr2=0x7f2cf00729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cee000640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf00cbab0 0x7f2cf00729a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cee000640 1 -- 172.21.15.146:0/2902530746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf0157240 con 0x7f2cf00c75a0 2024-09-16T01:50:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.113+0000 7f2cee000640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 0x7f2cf0157cb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc00d5d0 tx=0x7f2cdc00da90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.114+0000 7f2cef400640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf00cbab0 0x7f2cf00729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:52.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.114+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc004240 con 0x7f2cf00c75a0 2024-09-16T01:50:52.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.114+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0043e0 con 0x7f2cf00c75a0 2024-09-16T01:50:52.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.114+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc003830 con 0x7f2cf00c75a0 2024-09-16T01:50:52.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.114+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf00ceb00 con 0x7f2cf00c75a0 2024-09-16T01:50:52.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.115+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf00cf2e0 con 0x7f2cf00c75a0 2024-09-16T01:50:52.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.118+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6ac0001cb0 0x7f6ac0002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.118+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6ac0001cb0 0x7f6ac0002990 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.117+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf01577e0 con 0x7f2cf00c75a0 2024-09-16T01:50:52.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.119+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2cdc0039d0 con 0x7f2cf00c75a0 2024-09-16T01:50:52.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.120+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ac0002290 con 0x7f6ac0001cb0 2024-09-16T01:50:52.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.120+0000 7f2ce6a00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cd006c2b0 0x7f2cd006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.120+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5799+0+0 (secure 0 0 0) 0x7f2cdc08bac0 con 0x7f2cf00c75a0 2024-09-16T01:50:52.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.120+0000 7f2ceea00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cd006c2b0 0x7f2cd006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.121+0000 7f2ceea00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cd006c2b0 0x7f2cd006e770 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0034000 tx=0x7f2ce002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.122+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2cdc05a120 con 0x7f2cf00c75a0 2024-09-16T01:50:52.134 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:52.134 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:52.134 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:52.134 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:52.134 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:52.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.142+0000 7f6ad7e00640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6ac00031d0 con 0x7f6ac0001cb0 2024-09-16T01:50:52.145 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.144+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6ac00031d0 con 0x7f6ac0001cb0 2024-09-16T01:50:52.146 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:52.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.145+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6aec1f1ab0 0x7f6aec153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.145+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6aec0c44f0 con 0x7f6aec1f1ab0 2024-09-16T01:50:52.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.146+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6aec1f1ab0 0x7f6aec153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.147+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6aec1f1ab0 0x7f6aec153860 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.148+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6aec0c44f0 con 0x7f6aec1f1ab0 2024-09-16T01:50:52.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.185+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6aec0d02d0 con 0x7f6aec1f1ab0 2024-09-16T01:50:52.190 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:52.188+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:52.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.193+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6aec0d02d0 con 0x7f6aec1f1ab0 2024-09-16T01:50:52.195 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:52.195 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:52.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.194+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6aec0c5f00 0x7f6aec150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.194+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6aec150f60 con 0x7f6aec0c5f00 2024-09-16T01:50:52.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.195+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6aec0c5f00 0x7f6aec150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.196+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6aec0c5f00 0x7f6aec150a20 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.197+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ae40071b0 con 0x7f6aec0c5f00 2024-09-16T01:50:52.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.222+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6aec0cbd70 con 0x7f6aec0c5f00 2024-09-16T01:50:52.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.224+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6aec0cbd70 con 0x7f6aec0c5f00 2024-09-16T01:50:52.225 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:52.225 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:52.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.224+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6aec1f4880 0x7f6aec1f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.224+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6aec1f51a0 con 0x7f6aec1f4880 2024-09-16T01:50:52.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.225+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6aec1f4880 0x7f6aec1f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.225+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6aec1f4880 0x7f6aec1f4c60 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.226+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6aec1f51a0 con 0x7f6aec1f4880 2024-09-16T01:50:52.252 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5 3.7 3:e001da36:::10000000002.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.9900098769999999, 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044893833000000001, 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.253 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:50:52.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260358+0000", 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.672e-06 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297299+0000", 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036941017999999999 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297321+0000", 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2223000000000001e-05 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297332+0000", 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.058e-05 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297462+0000", 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013043399999999999 2024-09-16T01:50:52.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297624+0000", 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000162263 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297768+0000", 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014397800000000001 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.303867+0000", 2024-09-16T01:50:52.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060983820000000003 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305178+0000", 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013110330000000001 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305214+0000", 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6380000000000001e-05 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305250+0000", 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.587e-05 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:9 3.8 3:106f9e3b:::10000000002.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.297359+0000", 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.9530065350000001, 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034407954999999997, 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.259 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 9 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297359+0000", 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297359+0000", 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297361+0000", 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5260000000000001e-06 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324776+0000", 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027414984999999999 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324795+0000", 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8760999999999999e-05 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324821+0000", 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5839000000000001e-05 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324886+0000", 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4920999999999997e-05 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324991+0000", 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105048 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.325088+0000", 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7378000000000002e-05 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331340+0000", 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062516919999999997 2024-09-16T01:50:52.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331694+0000", 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035455699999999997 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331729+0000", 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4588000000000002e-05 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331767+0000", 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8659999999999999e-05 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:13 3.2 3:438a76aa:::10000000002.0000000a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.8961575480000001, 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045857943999999998, 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.265 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 13 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:50:52.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354210+0000", 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.203e-06 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388636+0000", 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034425859000000003 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388673+0000", 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7004e-05 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388693+0000", 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0194999999999999e-05 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388771+0000", 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8737000000000002e-05 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388904+0000", 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013215899999999999 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.389016+0000", 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112642 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.392245+0000", 2024-09-16T01:50:52.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032287370000000002 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.399956+0000", 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0077109190000000001 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400015+0000", 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9119000000000001e-05 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400066+0000", 2024-09-16T01:50:52.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1369999999999998e-05 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:21 3.8 3:129cb038:::10000000002.00000012:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.405307+0000", 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.845059059, 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026753832000000002, 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.271 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 21 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.405307+0000", 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.405307+0000", 2024-09-16T01:50:52.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.405308+0000", 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.37e-06 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427405+0000", 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022096525999999998 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427420+0000", 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4732999999999999e-05 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427445+0000", 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5222999999999999e-05 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427503+0000", 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7701000000000002e-05 2024-09-16T01:50:52.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427606+0000", 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010304799999999999 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427703+0000", 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7345999999999999e-05 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.431471+0000", 2024-09-16T01:50:52.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037677499999999998 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.431987+0000", 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00051606500000000001 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.432023+0000", 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6678999999999999e-05 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.432061+0000", 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7391000000000002e-05 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:29 3.7 3:e177cc1c:::10000000002.0000001a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.458164+0000", 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.792202327, 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.03214736, 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.277 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 29 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458164+0000", 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458164+0000", 2024-09-16T01:50:52.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458166+0000", 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.317e-06 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484169+0000", 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026003025999999999 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484188+0000", 2024-09-16T01:50:52.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9412e-05 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484197+0000", 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1089999999999998e-06 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484258+0000", 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1696e-05 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484390+0000", 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000131421 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484514+0000", 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012385 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.489955+0000", 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0054417959999999996 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.490201+0000", 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000245837 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.490259+0000", 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8168000000000003e-05 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.490311+0000", 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1727999999999997e-05 2024-09-16T01:50:52.282 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:30 3.7 3:e33cd625:::10000000002.0000001b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.766148622, 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035807259000000001, 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.283 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 30 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484219+0000", 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512447+0000", 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028228652 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512476+0000", 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8283e-05 2024-09-16T01:50:52.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512489+0000", 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3049e-05 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512566+0000", 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7420000000000001e-05 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512720+0000", 2024-09-16T01:50:52.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015377000000000001 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512891+0000", 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017147199999999999 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.516819+0000", 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003927458 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519938+0000", 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031196489999999999 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519989+0000", 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0380000000000002e-05 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.520025+0000", 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5839e-05 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.288 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:31 3.6 3:628433a0:::10000000002.0000001c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.517094+0000", 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.7332723720000001, 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029773069999999999, 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 31 2024-09-16T01:50:52.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.517094+0000", 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.517094+0000", 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.517096+0000", 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3640000000000002e-06 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542778+0000", 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025682304 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542797+0000", 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8357999999999999e-05 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542806+0000", 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5540000000000003e-06 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542871+0000", 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4879e-05 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542966+0000", 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4513999999999997e-05 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.543049+0000", 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3792e-05 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546391+0000", 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00334184 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546762+0000", 2024-09-16T01:50:52.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000370727 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546813+0000", 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1140000000000002e-05 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546867+0000", 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3597999999999997e-05 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.7007450849999999, 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:50:52.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:50:52.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.4537115039999999, 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.301 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:50:52.306 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.394552418, 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.307 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:50:52.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:50:52.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:50:52.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:82 3.2b 3:d50470f2:::10000000002.0000004f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.2876005799999999, 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047169015000000002, 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.313 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 82 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962768+0000", 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3269999999999999e-06 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002536+0000", 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039768464000000003 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002551+0000", 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002558+0000", 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8730000000000001e-06 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002630+0000", 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2025000000000002e-05 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002751+0000", 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121197 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002871+0000", 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011995 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.006288+0000", 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003417032 2024-09-16T01:50:52.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009828+0000", 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00353996 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009879+0000", 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0473999999999998e-05 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009934+0000", 2024-09-16T01:50:52.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5872999999999999e-05 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:114 3.8 3:10128d86:::10000000002.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.98171222899999999, 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038796303999999997, 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.319 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 114 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268656+0000", 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.858e-06 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303790+0000", 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035134060000000002 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303806+0000", 2024-09-16T01:50:52.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6611000000000001e-05 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303814+0000", 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4329999999999998e-06 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303869+0000", 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5639e-05 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303979+0000", 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110043 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.304078+0000", 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9006999999999996e-05 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.306722+0000", 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002643074 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307380+0000", 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00065801399999999995 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307414+0000", 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4082000000000003e-05 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307450+0000", 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6483000000000003e-05 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.324 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:125 3.7 3:e320baf1:::10000000002.0000007a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.375300+0000", 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.87506577399999996, 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032541303000000001, 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.325 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 125 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.375300+0000", 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.375300+0000", 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.375302+0000", 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.131e-06 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402165+0000", 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026862832999999999 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402184+0000", 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8352e-05 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402196+0000", 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2818e-05 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402261+0000", 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4658000000000003e-05 2024-09-16T01:50:52.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402396+0000", 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000135029 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402520+0000", 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123607 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407361+0000", 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004841756 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407733+0000", 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00037123000000000001 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407786+0000", 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3112999999999998e-05 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407842+0000", 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5776000000000003e-05 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:132 3.2b 3:d52309a9:::10000000002.00000081:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.78067510500000004, 2024-09-16T01:50:52.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034693794, 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 132 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469692+0000", 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.516e-06 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496852+0000", 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027159651999999999 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496868+0000", 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5869999999999999e-05 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496875+0000", 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.391e-06 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496952+0000", 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6320000000000001e-05 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497109+0000", 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015718700000000001 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497263+0000", 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015406799999999999 2024-09-16T01:50:52.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.500555+0000", 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032925720000000001 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504317+0000", 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037611150000000002 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504351+0000", 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4214000000000001e-05 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504385+0000", 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3889e-05 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.337 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.48297410600000001, 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.340 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.438713876, 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:50:52.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.346 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:50:52.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.349 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.40821405599999999, 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:50:52.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:50:52.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.352 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:50:52.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.36665853399999998, 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.356 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:50:52.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:183 3.1b 3:db82751e:::10000000002.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.27927191800000001, 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040945861, 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 183 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.362 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971104+0000", 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8719999999999997e-06 2024-09-16T01:50:52.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999810+0000", 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028705819 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999832+0000", 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2560000000000001e-05 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999847+0000", 2024-09-16T01:50:52.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4956e-05 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999905+0000", 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8084000000000002e-05 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000017+0000", 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111222 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:52.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000113+0000", 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6131999999999994e-05 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.002657+0000", 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025443089999999998 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011949+0000", 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0092920350000000006 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011989+0000", 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9987999999999997e-05 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.012040+0000", 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0884000000000003e-05 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:52.367 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.22861184000000001, 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:50:52.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:50:52.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:52.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:52.373 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.252+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6aec1f51a0 con 0x7f6aec1f4880 2024-09-16T01:50:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.253+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6aec1f51a0 con 0x7f6aec1f4880 2024-09-16T01:50:52.375 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:52.376 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:52.376 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:52.372+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.254+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6aec062b40 0x7f6aec062f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.254+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6aec0cbd70 con 0x7f6aec062b40 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.257+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6aec062b40 0x7f6aec062f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.258+0000 7f6aea000640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6aec062b40 0x7f6aec062f80 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.258+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6aec0cbd70 con 0x7f6aec062b40 2024-09-16T01:50:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.288+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6aec0cbd70 con 0x7f6aec062b40 2024-09-16T01:50:52.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.292+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6aec0cbd70 con 0x7f6aec062b40 2024-09-16T01:50:52.378 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.292+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6aec15bbd0 0x7f6aec15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.292+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6aec0c4050 con 0x7f6aec15bbd0 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.294+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6aec15bbd0 0x7f6aec15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.296+0000 7f6aeb400640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6aec15bbd0 0x7f6aec15bfd0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.297+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6aec0c4050 con 0x7f6aec15bbd0 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.313+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6aec0c4050 con 0x7f6aec15bbd0 2024-09-16T01:50:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.316+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6aec0c4050 con 0x7f6aec15bbd0 2024-09-16T01:50:52.380 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:52.380 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.316+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6aec02cb50 0x7f6aec02cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.316+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6aec0c8fd0 con 0x7f6aec02cb50 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.317+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6aec02cb50 0x7f6aec02cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.317+0000 7f6aeaa00640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6aec02cb50 0x7f6aec02cfb0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.318+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6aec0c8fd0 con 0x7f6aec02cb50 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.337+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6aec0c8fd0 con 0x7f6aec02cb50 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6ae2a00640 1 -- 172.21.15.146:0/4058873404 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6aec0c8fd0 con 0x7f6aec02cb50 2024-09-16T01:50:52.381 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:52.383 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:52.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6ac0000fa0 msgr2=0x7f6ac00045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6ac0000fa0 0x7f6ac00045e0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6ac0001cb0 msgr2=0x7f6ac0002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6ac0001cb0 0x7f6ac0002990 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6aec1f1ab0 msgr2=0x7f6aec153860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6aec1f1ab0 0x7f6aec153860 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6aec0c5f00 msgr2=0x7f6aec150a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6aec0c5f00 0x7f6aec150a20 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6aeb400640 1 -- 172.21.15.146:0/4058873404 reap_dead start 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6aeb400640 1 -- 172.21.15.146:0/4058873404 reap_dead start 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6aec1f4880 msgr2=0x7f6aec1f4c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.339+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6aec1f4880 0x7f6aec1f4c60 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6aec062b40 msgr2=0x7f6aec062f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6aec062b40 0x7f6aec062f80 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6aec15bbd0 msgr2=0x7f6aec15bfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6aec15bbd0 0x7f6aec15bfd0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6aec02cb50 msgr2=0x7f6aec02cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6aec02cb50 0x7f6aec02cfb0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6acc06c300 msgr2=0x7f6acc06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6acc06c300 0x7f6acc06e7c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8006fd0 tx=0x7f6ad8008040 comp rx=0 tx=0).stop 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aec154f40 msgr2=0x7f6aec1515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 --2- 172.21.15.146:0/4058873404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aec154f40 0x7f6aec1515e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6adc031000 tx=0x7f6adc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6aeb400640 1 -- 172.21.15.146:0/4058873404 reap_dead start 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6aeb400640 1 -- 172.21.15.146:0/4058873404 reap_dead start 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 shutdown_connections 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.340+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 >> 172.21.15.146:0/4058873404 conn(0x7f6aec06cb50 msgr2=0x7f6aec06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.341+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 shutdown_connections 2024-09-16T01:50:52.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.341+0000 7f6af1800640 1 -- 172.21.15.146:0/4058873404 wait complete. 2024-09-16T01:50:52.476 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:52.475+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:52.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.499+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f2cf006da90 con 0x7f2cf00c75a0 2024-09-16T01:50:52.556 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:52.556 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:52.571 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:52.576 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:52.575+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:52.677 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:52.675+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:52.694 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:52.694 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:52.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:52.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.749+0000 7f2ce6a00640 1 -- 172.21.15.146:0/2902530746 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v56) ==== 87+0+0 (secure 0 0 0) 0x7f2cdc059a20 con 0x7f2cf00c75a0 2024-09-16T01:50:52.750 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:50:52.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.752+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cd006c2b0 msgr2=0x7f2cd006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.752+0000 7f2cefe00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cd006c2b0 0x7f2cd006e770 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0034000 tx=0x7f2ce002e000 comp rx=0 tx=0).stop 2024-09-16T01:50:52.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.752+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 msgr2=0x7f2cf0157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:52.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.752+0000 7f2cefe00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 0x7f2cf0157cb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc00d5d0 tx=0x7f2cdc00da90 comp rx=0 tx=0).stop 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.753+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 shutdown_connections 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.753+0000 7f2cefe00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2cd006c2b0 0x7f2cd006e770 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.753+0000 7f2cefe00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cf00cbab0 0x7f2cf00729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.753+0000 7f2cefe00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2cf00c75a0 0x7f2cf0157cb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.753+0000 7f2cefe00640 1 --2- 172.21.15.146:0/2902530746 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2cf00c6b70 0x7f2cf015c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.753+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 >> 172.21.15.146:0/2902530746 conn(0x7f2cf00ccc20 msgr2=0x7f2cf00cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.754+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 shutdown_connections 2024-09-16T01:50:52.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:52.754+0000 7f2cefe00640 1 -- 172.21.15.146:0/2902530746 wait complete. 2024-09-16T01:50:52.770 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:50:52.770 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 initial writers = 281412.06 kB/sec 2024-09-16T01:50:52.771 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 initial writers = 281360.87 kB/sec 2024-09-16T01:50:52.771 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 281412.06 kB/sec 2024-09-16T01:50:52.771 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 281412.06 kB/sec 2024-09-16T01:50:52.771 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 281412.06 kB/sec 2024-09-16T01:50:52.771 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:50:52.816 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:52.815+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:52.852 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:52.852 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:52.860 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:52.904 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:52.902+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:52.965 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:52.965 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:52.973 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:53.014 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:53.012+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:53.081 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:53.080+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:53.115 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:53.115 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:53.124 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:53.182 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:53.180+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:53.227 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:53.228 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:53.236 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:53.284 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:53.283+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:53.381 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:53.381 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:53.385 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:53.382+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:53.391 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:53.486 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:53.484+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:53.537 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:53.537 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:53.545 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:53.586 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:53.586+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:53.687 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:53.685+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:53.689 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:53.690 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:53.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:53.743 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:53.743 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:53.744 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T01:50:53.744 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:50:53.744 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:50:53.744 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-16T01:50:53.788 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:53.786+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:53.847 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:53.847 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T01:50:53.847 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:53.858 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:53.859 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:53.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:53.917 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:53.916+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:53.990 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:53.989+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:54.030 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:54.031 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:54.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:54.091 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:54.090+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:54.192 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:54.191+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:54.243 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:54.243 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:54.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:54.293 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:54.291+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:54.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:54.394 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:54.392+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:54.472 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:54.472 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:54.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:54.494 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:54.493+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:54.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.540+0000 7f27efe00640 1 Processor -- start 2024-09-16T01:50:54.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.540+0000 7f27efe00640 1 -- start start 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27efe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f27f006d4a0 0x7f27f00c8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27f00c81d0 0x7f27f0159380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27efe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f01588c0 con 0x7f27f006d880 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27efe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f27f0158ef0 con 0x7f27f00c81d0 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27efe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f27f015c600 con 0x7f27f006d4a0 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27eea00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f27f006d4a0 0x7f27f00c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27ee000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27f00c81d0 0x7f27f0159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27eea00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f27f006d4a0 0x7f27f00c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48212/0 (socket says 172.21.15.146:48212) 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27eea00640 1 -- 172.21.15.146:0/2776028849 learned_addr learned my addr 172.21.15.146:0/2776028849 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.541+0000 7f27ef400640 1 --1- 172.21.15.146:0/2776028849 >> v1:172.21.15.146:6789/0 conn(0x7f27f006d880 0x7f27f015be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43332/0 (socket says 172.21.15.146:43332) 2024-09-16T01:50:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.542+0000 7f27eea00640 1 -- 172.21.15.146:0/2776028849 >> v2:172.21.15.146:3300/0 conn(0x7f27f00c81d0 msgr2=0x7f27f0159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:54.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.542+0000 7f27eea00640 1 --2- 172.21.15.146:0/2776028849 >> v2:172.21.15.146:3300/0 conn(0x7f27f00c81d0 0x7f27f0159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.542+0000 7f27eea00640 1 -- 172.21.15.146:0/2776028849 >> v1:172.21.15.146:6789/0 conn(0x7f27f006d880 legacy=0x7f27f015be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.542+0000 7f27eea00640 1 -- 172.21.15.146:0/2776028849 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f015c8e0 con 0x7f27f006d4a0 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.543+0000 7f27eea00640 1 --2- 172.21.15.146:0/2776028849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d4a0 0x7f27f00c8640 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27f00715b0 tx=0x7f27e002a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf4de8fc2ac15fe9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.543+0000 7f27ed600640 1 -- 172.21.15.146:0/2776028849 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0037070 con 0x7f27f006d4a0 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.543+0000 7f27ed600640 1 -- 172.21.15.146:0/2776028849 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e002ae50 con 0x7f27f006d4a0 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.543+0000 7f27ed600640 1 -- 172.21.15.146:0/2776028849 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0032860 con 0x7f27f006d4a0 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 -- 172.21.15.146:0/2776028849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d4a0 msgr2=0x7f27f00c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 --2- 172.21.15.146:0/2776028849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d4a0 0x7f27f00c8640 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27f00715b0 tx=0x7f27e002a300 comp rx=0 tx=0).stop 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 -- 172.21.15.146:0/2776028849 shutdown_connections 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 --2- 172.21.15.146:0/2776028849 >> v2:172.21.15.146:3300/0 conn(0x7f27f00c81d0 0x7f27f0159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 --2- 172.21.15.146:0/2776028849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d4a0 0x7f27f00c8640 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 -- 172.21.15.146:0/2776028849 >> 172.21.15.146:0/2776028849 conn(0x7f27f00cd010 msgr2=0x7f27f00cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 -- 172.21.15.146:0/2776028849 shutdown_connections 2024-09-16T01:50:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.544+0000 7f27efe00640 1 -- 172.21.15.146:0/2776028849 wait complete. 2024-09-16T01:50:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.546+0000 7f27efe00640 1 Processor -- start 2024-09-16T01:50:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.546+0000 7f27efe00640 1 -- start start 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27efe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 0x7f27f01527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27efe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d880 0x7f27f01f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f00c81d0 0x7f27f0155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27efe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f27f00c7ba0 con 0x7f27f00c81d0 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27efe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0158b80 con 0x7f27f006d880 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27efe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f27f00c7d20 con 0x7f27f006d4a0 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 0x7f27f01527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27ee000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d880 0x7f27f01f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27ef400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f00c81d0 0x7f27f0155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 0x7f27f01527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:51336/0 (socket says 172.21.15.146:51336) 2024-09-16T01:50:54.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 -- 172.21.15.146:0/2142703893 learned_addr learned my addr 172.21.15.146:0/2142703893 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:54.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d880 0x7f27f01f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48218/0 (socket says 172.21.15.146:48218) 2024-09-16T01:50:54.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d880 msgr2=0x7f27f01f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d880 0x7f27f01f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f00c81d0 msgr2=0x7f27f0155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f00c81d0 0x7f27f0155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27eea00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00be330 con 0x7f27f006d4a0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f27f006d880 0x7f27f01f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.547+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f00c81d0 0x7f27f0155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.548+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 0x7f27f01527b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27e002a340 tx=0x7f27e00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.548+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0041070 con 0x7f27f006d4a0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.548+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e003c680 con 0x7f27f006d4a0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.548+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0037040 con 0x7f27f006d4a0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.548+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f00bf8f0 con 0x7f27f006d4a0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.548+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f00c6420 con 0x7f27f006d4a0 2024-09-16T01:50:54.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.549+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f27f0076300 con 0x7f27f006d4a0 2024-09-16T01:50:54.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.551+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f27e0044070 con 0x7f27f006d4a0 2024-09-16T01:50:54.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.552+0000 7f27e6a00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f27d006c2b0 0x7f27d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.552+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 5799+0+0 (secure 0 0 0) 0x7f27e00ac1b0 con 0x7f27f006d4a0 2024-09-16T01:50:54.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.552+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f27d006c2b0 0x7f27d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.552+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v56) ==== 54+0+16 (secure 0 0 0) 0x7f27f0076300 con 0x7f27f006d4a0 2024-09-16T01:50:54.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.553+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f27f00be500 0x7f27f0155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.553+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27f01f40f0 con 0x7f27f00be500 2024-09-16T01:50:54.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.553+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f27d006c2b0 0x7f27d006e770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f27f00cd980 tx=0x7f27dc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.553+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f27f00be500 0x7f27f0155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.554+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f27f00be500 0x7f27f0155940 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.554+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f01f40f0 con 0x7f27f00be500 2024-09-16T01:50:54.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.569+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f00c8640 con 0x7f27f00be500 2024-09-16T01:50:54.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.571+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f00c8640 con 0x7f27f00be500 2024-09-16T01:50:54.572 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:54.572 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:54.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.571+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f27f00c0140 0x7f27f00cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.571+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f27f00cd6d0 con 0x7f27f00c0140 2024-09-16T01:50:54.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.571+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f27f00c0140 0x7f27f00cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.572+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f27f00c0140 0x7f27f00cb860 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.573+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f00cd6d0 con 0x7f27f00c0140 2024-09-16T01:50:54.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.587+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f00cd3f0 con 0x7f27f00c0140 2024-09-16T01:50:54.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.589+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f00cd3f0 con 0x7f27f00c0140 2024-09-16T01:50:54.590 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:54.591 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:54.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.590+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f27f00b1b70 0x7f27f00b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.590+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f27f0153230 con 0x7f27f00b1b70 2024-09-16T01:50:54.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.590+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f27f00b1b70 0x7f27f00b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.590+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f27f00b1b70 0x7f27f00b1f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.591+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f0153230 con 0x7f27f00b1b70 2024-09-16T01:50:54.595 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:54.594+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:54.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.605+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f0153630 con 0x7f27f00b1b70 2024-09-16T01:50:54.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.607+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f0153630 con 0x7f27f00b1b70 2024-09-16T01:50:54.608 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:54.608 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:54.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.607+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f27f0062630 0x7f27f0062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.607+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f27f0153630 con 0x7f27f0062630 2024-09-16T01:50:54.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.607+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f27f0062630 0x7f27f0062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.608+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f27f0062630 0x7f27f0062a50 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.609+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f0153630 con 0x7f27f0062630 2024-09-16T01:50:54.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.622+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f0153630 con 0x7f27f0062630 2024-09-16T01:50:54.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.624+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f0153630 con 0x7f27f0062630 2024-09-16T01:50:54.625 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:54.626 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:54.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.624+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f27f002c640 0x7f27f002caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.624+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f27f00634a0 con 0x7f27f002c640 2024-09-16T01:50:54.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.625+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f27f002c640 0x7f27f002caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.625+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f27f002c640 0x7f27f002caa0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.626+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f00634a0 con 0x7f27f002c640 2024-09-16T01:50:54.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.640+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f00634a0 con 0x7f27f002c640 2024-09-16T01:50:54.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.642+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f00634a0 con 0x7f27f002c640 2024-09-16T01:50:54.644 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:54.645 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:54.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.642+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f27f002d4f0 0x7f27f0206f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.642+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f27f002d930 con 0x7f27f002d4f0 2024-09-16T01:50:54.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.643+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f27f002d4f0 0x7f27f0206f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.643+0000 7f27ee000640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f27f002d4f0 0x7f27f0206f40 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.644+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f002d930 con 0x7f27f002d4f0 2024-09-16T01:50:54.650 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:54.651 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:54.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.658+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f002db70 con 0x7f27f002d4f0 2024-09-16T01:50:54.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.660+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f002db70 con 0x7f27f002d4f0 2024-09-16T01:50:54.661 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:54.661 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:54.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.660+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f27f002d930 0x7f27f02099a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.660+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f27f020c7f0 con 0x7f27f002d930 2024-09-16T01:50:54.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.660+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f27f002d930 0x7f27f02099a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.661+0000 7f27ef400640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f27f002d930 0x7f27f02099a0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.662+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f020c7f0 con 0x7f27f002d930 2024-09-16T01:50:54.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:54.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.675+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f0209ee0 con 0x7f27f002d930 2024-09-16T01:50:54.680 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.677+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f0209ee0 con 0x7f27f002d930 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.677+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f27f020a600 0x7f27f020aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.677+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f27f020fbc0 con 0x7f27f020a600 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.678+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f27f020a600 0x7f27f020aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.678+0000 7f27eea00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f27f020a600 0x7f27f020aa40 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.679+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27f020fbc0 con 0x7f27f020a600 2024-09-16T01:50:54.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.693+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f27f020b040 con 0x7f27f020a600 2024-09-16T01:50:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27e6a00640 1 -- 172.21.15.146:0/2142703893 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f27f020b040 con 0x7f27f020a600 2024-09-16T01:50:54.696 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f27f00be500 msgr2=0x7f27f0155940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f27f00be500 0x7f27f0155940 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f27f00c0140 msgr2=0x7f27f00cb860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f27f00c0140 0x7f27f00cb860 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f27f00b1b70 msgr2=0x7f27f00b1f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f27f00b1b70 0x7f27f00b1f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f27f0062630 msgr2=0x7f27f0062a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f27f0062630 0x7f27f0062a50 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f27f002c640 msgr2=0x7f27f002caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f27f002c640 0x7f27f002caa0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f27f002d4f0 msgr2=0x7f27f0206f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f27f002d4f0 0x7f27f0206f40 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f27f002d930 msgr2=0x7f27f02099a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f27f002d930 0x7f27f02099a0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f27f020a600 msgr2=0x7f27f020aa40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27ef400640 1 -- 172.21.15.146:0/2142703893 reap_dead start 2024-09-16T01:50:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f27f020a600 0x7f27f020aa40 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27ef400640 1 -- 172.21.15.146:0/2142703893 reap_dead start 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27ef400640 1 -- 172.21.15.146:0/2142703893 reap_dead start 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27ef400640 1 -- 172.21.15.146:0/2142703893 reap_dead start 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f27d006c2b0 msgr2=0x7f27d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f27d006c2b0 0x7f27d006e770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f27f00cd980 tx=0x7f27dc009190 comp rx=0 tx=0).stop 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 msgr2=0x7f27f01527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.695+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 0x7f27f01527b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27e002a340 tx=0x7f27e00046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.696+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 shutdown_connections 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.696+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f27d006c2b0 0x7f27d006e770 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.696+0000 7f27efe00640 1 --2- 172.21.15.146:0/2142703893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27f006d4a0 0x7f27f01527b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.696+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 >> 172.21.15.146:0/2142703893 conn(0x7f27f00cd010 msgr2=0x7f27f00cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.696+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 shutdown_connections 2024-09-16T01:50:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.696+0000 7f27efe00640 1 -- 172.21.15.146:0/2142703893 wait complete. 2024-09-16T01:50:54.700 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:54.694+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:54.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:50:54.797 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:54.795+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 127.07076713399999, 2024-09-16T01:50:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:7 3.2a 3:557f550e:::10000000002.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:50:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5804267999999997, 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081303988999999993, 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 7 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:50:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258696+0000", 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7320000000000001e-06 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305572+0000", 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046875737000000001 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305609+0000", 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7704999999999999e-05 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305623+0000", 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310196+0000", 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045734269999999997 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310344+0000", 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147181 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310523+0000", 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017908400000000001 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.327195+0000", 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016672724999999999 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339893+0000", 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012697559000000001 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339953+0000", 2024-09-16T01:50:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0084999999999998e-05 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339998+0000", 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5218e-05 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:8 3.1c 3:3ae0713d:::10000000002.00000005:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.306290+0000", 2024-09-16T01:50:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5328307749999999, 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037928180999999998, 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 8 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306290+0000", 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306290+0000", 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306292+0000", 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.424e-06 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.316009+0000", 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0097174970000000003 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.316040+0000", 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1038000000000003e-05 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.316055+0000", 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5145e-05 2024-09-16T01:50:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.316153+0000", 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7915e-05 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.316311+0000", 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157823 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.316458+0000", 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014658100000000001 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.327533+0000", 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011074993999999999 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344133+0000", 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016600136000000001 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344173+0000", 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9878e-05 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344218+0000", 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5750000000000001e-05 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:20 3.21 3:85380590:::10000000002.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4274197419999997, 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055783419000000001, 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 20 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411703+0000", 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.84e-06 2024-09-16T01:50:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441653+0000", 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029949640999999999 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441685+0000", 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2452e-05 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441702+0000", 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441798+0000", 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6501000000000003e-05 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441964+0000", 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165635 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.442314+0000", 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035073800000000002 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.455571+0000", 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013256273000000001 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467368+0000", 2024-09-16T01:50:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011797656 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467432+0000", 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3585999999999997e-05 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467485+0000", 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2729999999999998e-05 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:22 3.10 3:0a7b425d:::10000000002.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.3973814319999995, 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050802171, 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 22 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441741+0000", 2024-09-16T01:50:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.641e-06 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459091+0000", 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017350175999999998 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459122+0000", 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1041000000000003e-05 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459136+0000", 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.384e-05 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461076+0000", 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001939213 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461219+0000", 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143101 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461360+0000", 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014149900000000001 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492196+0000", 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030836319000000001 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492406+0000", 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020962800000000001 2024-09-16T01:50:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492461+0000", 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5251999999999999e-05 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492542+0000", 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0461000000000001e-05 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:23 3.21 3:87fa6b7a:::10000000002.00000014:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.3799412469999996, 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049069672000000002, 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 23 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459181+0000", 2024-09-16T01:50:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.356e-06 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499609+0000", 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040428104999999999 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499638+0000", 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8640999999999999e-05 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499650+0000", 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1989e-05 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499729+0000", 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869000000000007e-05 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499916+0000", 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018738500000000001 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.500068+0000", 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015149099999999999 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.506758+0000", 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066901900000000004 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508128+0000", 2024-09-16T01:50:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013700330000000001 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508197+0000", 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9077000000000003e-05 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508249+0000", 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2536e-05 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:40 3.f e55/20 3:f045ad74:::10000000002.00000025:head v 55'1, mlcod=0'0)", 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.588742+0000", 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2503786799999999, 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038996202000000001, 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.588742+0000", 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.588742+0000", 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.588744+0000", 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.327e-06 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.595287+0000", 2024-09-16T01:50:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065433069999999999 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.595319+0000", 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1850999999999997e-05 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.595333+0000", 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4104000000000001e-05 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.595422+0000", 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8832999999999995e-05 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.595448+0000", 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6143e-05 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627681+0000", 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032232971999999999 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627739+0000", 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7664999999999998e-05 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:43 3.1a 3:5917aad8:::10000000002.00000028:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2384762370000004, 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041950973000000003, 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 43 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600647+0000", 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.136e-06 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623155+0000", 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022507547999999999 2024-09-16T01:50:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623186+0000", 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1078000000000003e-05 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623196+0000", 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0825999999999999e-05 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624399+0000", 2024-09-16T01:50:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012021969999999999 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624531+0000", 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013200899999999999 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624681+0000", 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015020999999999999 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.639604+0000", 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014922962 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642466+0000", 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002862028 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642534+0000", 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8503999999999993e-05 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642596+0000", 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1475000000000003e-05 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:44 3.19 3:9961cbc1:::10000000002.00000029:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.215427021, 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044190814000000002, 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 44 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623696+0000", 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000002e-06 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660636+0000", 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036939840000000002 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660662+0000", 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5891e-05 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660672+0000", 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0362e-05 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660758+0000", 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5834999999999994e-05 2024-09-16T01:50:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660916+0000", 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158572 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.661072+0000", 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015595600000000001 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667770+0000", 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066975170000000001 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667785+0000", 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4882000000000001e-05 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667837+0000", 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2343999999999999e-05 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667885+0000", 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7679000000000002e-05 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:59 3.1c 3:394f8a26:::10000000002.00000038:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.0463854330000002, 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051871274000000002, 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 59 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792738+0000", 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1100000000000001e-06 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824239+0000", 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031501159000000001 2024-09-16T01:50:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824261+0000", 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2515e-05 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824272+0000", 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0696999999999999e-05 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824397+0000", 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012514399999999999 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824552+0000", 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015457599999999999 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824715+0000", 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163657 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.832638+0000", 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079224959999999994 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844470+0000", 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011831734999999999 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844542+0000", 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2194999999999996e-05 2024-09-16T01:50:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844607+0000", 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4989999999999999e-05 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:91 3.2a 3:5747858d:::10000000002.00000058:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.7785321440000001, 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045690485000000003, 2024-09-16T01:50:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 91 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060591+0000", 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1009999999999999e-06 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094735+0000", 2024-09-16T01:50:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034143896999999999 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094758+0000", 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3485999999999998e-05 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094769+0000", 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0844e-05 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094852+0000", 2024-09-16T01:50:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2627999999999996e-05 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095011+0000", 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015938000000000001 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095160+0000", 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014846799999999999 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.105436+0000", 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010275986000000001 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106140+0000", 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000703905 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106219+0000", 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9672999999999994e-05 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106279+0000", 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0117000000000001e-05 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:92 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.7443152509999997, 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040038826999999999, 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 92 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094807+0000", 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1319999999999999e-06 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126953+0000", 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032146474000000001 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126978+0000", 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4737e-05 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126992+0000", 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3543000000000001e-05 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127074+0000", 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1889000000000002e-05 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127230+0000", 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156238 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127387+0000", 2024-09-16T01:50:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157207 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.131783+0000", 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004395805 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134755+0000", 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029720520000000002 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134795+0000", 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9909e-05 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134845+0000", 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9840999999999999e-05 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:97 3.1a 3:5b190c62:::10000000002.0000005e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.7118073950000001, 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040400441000000002, 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 97 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127315+0000", 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.626e-06 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156590+0000", 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029275042000000001 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156619+0000", 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8200000000000001e-05 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156631+0000", 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.252e-05 2024-09-16T01:50:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156742+0000", 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011125599999999999 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156905+0000", 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016292499999999999 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.157055+0000", 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014968699999999999 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.162589+0000", 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055339530000000003 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167534+0000", 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049449699999999999 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167659+0000", 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125575 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167714+0000", 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4687e-05 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:104 3.2a 3:563615cd:::10000000002.00000065:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.160763+0000", 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6783580850000002, 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039133845, 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 104 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.160763+0000", 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.160763+0000", 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.160765+0000", 2024-09-16T01:50:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.182036+0000", 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021271173000000001 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.182065+0000", 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9033000000000001e-05 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.182078+0000", 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2286999999999999e-05 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.182181+0000", 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010322799999999999 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.182351+0000", 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017013600000000001 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.182518+0000", 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167114 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.194279+0000", 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011760985999999999 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.199759+0000", 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054799799999999997 2024-09-16T01:50:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.199844+0000", 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4788000000000002e-05 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.199897+0000", 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2960000000000001e-05 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:124 3.21 3:844ac05a:::10000000002.00000079:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.4394342359999999, 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052276376999999999, 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 124 2024-09-16T01:50:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399689+0000", 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5100000000000001e-06 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443104+0000", 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043414547999999997 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443122+0000", 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8071e-05 2024-09-16T01:50:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443132+0000", 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0492999999999999e-05 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443200+0000", 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7248000000000004e-05 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443353+0000", 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015338799999999999 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443504+0000", 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150681 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447533+0000", 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040287889999999996 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451892+0000", 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043593520000000004 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451932+0000", 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0104000000000001e-05 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451963+0000", 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1192999999999998e-05 2024-09-16T01:50:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:139 3.1c 3:384ad1d9:::10000000002.00000088:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.298419118, 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061931110999999997, 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540704+0000", 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.221e-06 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586535+0000", 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045830901 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586563+0000", 2024-09-16T01:50:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8388e-05 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586591+0000", 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7424999999999999e-05 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586694+0000", 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103269 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586849+0000", 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154857 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.587008+0000", 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015895699999999999 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591909+0000", 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049009609999999997 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602502+0000", 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010592773 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602564+0000", 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2154000000000002e-05 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602633+0000", 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9204999999999999e-05 2024-09-16T01:50:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:148 3.1c 3:3afba7d1:::10000000002.00000091:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.2148469550000001, 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047360674999999998, 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 148 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624276+0000", 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1639999999999999e-06 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653877+0000", 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029600346 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653900+0000", 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3399000000000001e-05 2024-09-16T01:50:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653912+0000", 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2232000000000001e-05 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654079+0000", 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016647999999999999 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654232+0000", 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015308100000000001 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654384+0000", 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152573 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.662170+0000", 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077856990000000001 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671503+0000", 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093331279999999996 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671572+0000", 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8417999999999994e-05 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671635+0000", 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3155000000000001e-05 2024-09-16T01:50:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:156 3.2a 3:577cc1b8:::10000000002.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.706007+0000", 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.13311435, 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039065259999999997, 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 156 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.706007+0000", 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.706007+0000", 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.706009+0000", 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2589999999999999e-06 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.733502+0000", 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027492604 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.733529+0000", 2024-09-16T01:50:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7293000000000001e-05 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.733540+0000", 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1542000000000001e-05 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.733623+0000", 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2876999999999999e-05 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.733803+0000", 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180166 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.733941+0000", 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013757900000000001 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.744597+0000", 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010656287 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.744964+0000", 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036712000000000002 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.745023+0000", 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8449999999999998e-05 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.745072+0000", 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9082999999999997e-05 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:159 3.37 3:edaba8ff:::10000000002.0000009c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.736762+0000", 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.1023586239999998, 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037295452999999999, 2024-09-16T01:50:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 159 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.736762+0000", 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.736762+0000", 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.736765+0000", 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909999999999999e-06 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.757491+0000", 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020725881000000002 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.757516+0000", 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5063999999999999e-05 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.757528+0000", 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2438e-05 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.757676+0000", 2024-09-16T01:50:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014836999999999999 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.757840+0000", 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163167 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.757978+0000", 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138159 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.772056+0000", 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014078350999999999 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.773904+0000", 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018481470000000001 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.773972+0000", 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7875999999999999e-05 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.774058+0000", 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5809000000000005e-05 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:191 3.1a 3:5ab92411:::10000000002.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.8024557919999999, 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045215946999999999, 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 191 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036668+0000", 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4880000000000001e-06 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066884+0000", 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030216601999999999 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066908+0000", 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3555999999999999e-05 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066918+0000", 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0443e-05 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069411+0000", 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024928149999999998 2024-09-16T01:50:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069698+0000", 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287025 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069886+0000", 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018816899999999999 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.075677+0000", 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057910540000000003 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081769+0000", 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060919659999999999 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081834+0000", 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4797999999999999e-05 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081881+0000", 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7030999999999997e-05 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:54.965 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:54.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.932+0000 7f0851c00640 1 Processor -- start 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.932+0000 7f0851c00640 1 -- start start 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.933+0000 7f0851c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 0x7f084c158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.933+0000 7f0851c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f084c158bb0 0x7f084c0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.933+0000 7f0851c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084c072f50 con 0x7f084c0c6bb0 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.933+0000 7f0851c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f084c073210 con 0x7f084c152850 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.933+0000 7f0851c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f084c15cb50 con 0x7f084c158bb0 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084aa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 0x7f084c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084a000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f084c158bb0 0x7f084c0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084b400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f084c0c6bb0 0x7f084c0c86f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56320/0 (socket says 172.21.15.146:56320) 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084aa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 0x7f084c158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50858/0 (socket says 172.21.15.146:50858) 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084a000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f084c158bb0 0x7f084c0c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48228/0 (socket says 172.21.15.146:48228) 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084b400640 1 -- 172.21.15.146:0/2430733328 learned_addr learned my addr 172.21.15.146:0/2430733328 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084a000640 1 -- 172.21.15.146:0/2430733328 >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 msgr2=0x7f084c158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084a000640 1 --2- 172.21.15.146:0/2430733328 >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 0x7f084c158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084a000640 1 -- 172.21.15.146:0/2430733328 >> v1:172.21.15.161:6789/0 conn(0x7f084c0c6bb0 legacy=0x7f084c0c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084a000640 1 -- 172.21.15.146:0/2430733328 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c15bd90 con 0x7f084c158bb0 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.934+0000 7f084aa00640 1 --2- 172.21.15.146:0/2430733328 >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 0x7f084c158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.935+0000 7f084a000640 1 --2- 172.21.15.146:0/2430733328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c158bb0 0x7f084c0c58c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f084c0c67f0 tx=0x7f083802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99e5d5a518c5dbbf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.935+0000 7f0849600640 1 -- 172.21.15.146:0/2430733328 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838037070 con 0x7f084c158bb0 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.935+0000 7f0849600640 1 -- 172.21.15.146:0/2430733328 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0838032440 con 0x7f084c158bb0 2024-09-16T01:50:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.935+0000 7f0849600640 1 -- 172.21.15.146:0/2430733328 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838032810 con 0x7f084c158bb0 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 -- 172.21.15.146:0/2430733328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c158bb0 msgr2=0x7f084c0c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 --2- 172.21.15.146:0/2430733328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c158bb0 0x7f084c0c58c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f084c0c67f0 tx=0x7f083802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 -- 172.21.15.146:0/2430733328 shutdown_connections 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 --2- 172.21.15.146:0/2430733328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c158bb0 0x7f084c0c58c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 --2- 172.21.15.146:0/2430733328 >> v2:172.21.15.146:3300/0 conn(0x7f084c152850 0x7f084c158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 -- 172.21.15.146:0/2430733328 >> 172.21.15.146:0/2430733328 conn(0x7f084c06f3c0 msgr2=0x7f084c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 -- 172.21.15.146:0/2430733328 shutdown_connections 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.936+0000 7f0851c00640 1 -- 172.21.15.146:0/2430733328 wait complete. 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.938+0000 7f0851c00640 1 Processor -- start 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.938+0000 7f0851c00640 1 -- start start 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f0851c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 0x7f084c0c3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f0851c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c152850 0x7f084c0c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f0851c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 0x7f084c0cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f0851c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f084c15c630 con 0x7f084c158bb0 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f0851c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f084c15c350 con 0x7f084c152850 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f0851c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f084c070200 con 0x7f084c0c6bb0 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 0x7f084c0cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 0x7f084c0c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c152850 0x7f084c0c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 0x7f084c0c3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:51342/0 (socket says 172.21.15.146:51342) 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 0x7f084c0cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50862/0 (socket says 172.21.15.146:50862) 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 -- 172.21.15.146:0/4117997815 learned_addr learned my addr 172.21.15.146:0/4117997815 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 -- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c152850 msgr2=0x7f084c0c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c152850 0x7f084c0c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 -- 172.21.15.146:0/4117997815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 msgr2=0x7f084c0cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:54.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.939+0000 7f084aa00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 0x7f084c0cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f084aa00640 1 -- 172.21.15.146:0/4117997815 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c155c60 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f084b400640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 0x7f084c0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f084aa00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 0x7f084c0c3940 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f083c00ec10 tx=0x7f083c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c017070 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083c00cc30 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c013470 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c1f4200 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.940+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c1f4490 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.941+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c14ce90 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.942+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f083c01a070 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.943+0000 7f0842a00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f082c06c2b0 0x7f082c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.943+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 5799+0+0 (secure 0 0 0) 0x7f083c08b480 con 0x7f084c0c6bb0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.943+0000 7f084a000640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f082c06c2b0 0x7f082c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:54.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.944+0000 7f084a000640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f082c06c2b0 0x7f082c06e770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0838034000 tx=0x7f083802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:54.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:54.948+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f084c14ce90 con 0x7f084c0c6bb0 2024-09-16T01:50:54.976 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:54.966+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:55.067 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:55.067+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:55.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.139+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f084c0718a0 con 0x7f084c0c6bb0 2024-09-16T01:50:55.168 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:55.167+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:55.169 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:55.169 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:55.169 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:55.169 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:55.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:50:55.269 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:55.268+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:55.370 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:55.368+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:55.380 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:55.380 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:55.380 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:55.380 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:55.380 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:55.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:50:55.473 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:55.472+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:55.552 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.295042359, 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.553 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:50:55.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9820965839999998, 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:50:55.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:50:55.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:42 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9410141159999998, 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051426843, 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 42 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608928+0000", 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5689999999999999e-06 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648858+0000", 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039929485000000001 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648879+0000", 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1172e-05 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648889+0000", 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0512e-05 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648975+0000", 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5473e-05 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649103+0000", 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012862999999999999 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649230+0000", 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126561 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.655882+0000", 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066518819999999996 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660228+0000", 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043460699999999996 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660293+0000", 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5050999999999998e-05 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660353+0000", 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0438e-05 2024-09-16T01:50:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8285509019999999, 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.571 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:50:55.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:50:55.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.576 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:67 3.16 3:6a386151:::10000000002.00000040:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.7140330700000002, 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046477360000000002, 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 67 2024-09-16T01:50:55.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835910+0000", 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.872997+0000", 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03708761 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873029+0000", 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1911e-05 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873046+0000", 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6702999999999999e-05 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873132+0000", 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6034999999999999e-05 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873288+0000", 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015656000000000001 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873419+0000", 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130381 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.880727+0000", 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073081919999999998 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882276+0000", 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015495330000000001 2024-09-16T01:50:55.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882336+0000", 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9964999999999999e-05 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882385+0000", 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8588999999999998e-05 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:77 3.3c 3:3db0268f:::10000000002.0000004a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.6402419159999999, 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051390551, 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 77 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909702+0000", 2024-09-16T01:50:55.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9699999999999999e-06 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927863+0000", 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018160824999999998 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927887+0000", 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4802000000000001e-05 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927900+0000", 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928016+0000", 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116123 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928158+0000", 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141977 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928294+0000", 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135733 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.938330+0000", 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010036092999999999 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960987+0000", 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022656493 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961043+0000", 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6416999999999999e-05 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961089+0000", 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6152000000000001e-05 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.6076912219999997, 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.589 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:50:55.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:50:55.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.529560633, 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:50:55.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:50:55.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:85 3.36 e55/20 3:6fe880e3:::10000000002.00000052:head v 55'2, mlcod=0'0)", 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.5071743069999997, 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052325050999999997, 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042769+0000", 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3520000000000001e-06 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051520+0000", 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087513980000000005 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051666+0000", 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014630900000000001 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051711+0000", 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4768999999999998e-05 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052370+0000", 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065890100000000004 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052478+0000", 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010805 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095028+0000", 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042550194 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095091+0000", 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3077999999999995e-05 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.605 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:93 3.20 3:0788c2ef:::10000000002.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.075661+0000", 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4742795590000002, 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043486008999999999, 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 93 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.075661+0000", 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.075661+0000", 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.075664+0000", 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5150000000000001e-06 2024-09-16T01:50:55.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109488+0000", 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033824568999999999 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109523+0000", 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4887e-05 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109535+0000", 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1895e-05 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109637+0000", 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010240599999999999 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109798+0000", 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016033200000000001 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109957+0000", 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159634 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.118536+0000", 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085781139999999995 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.119075+0000", 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00053984200000000001 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.119115+0000", 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9884e-05 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.119147+0000", 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1931000000000003e-05 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:94 3.5 3:a282a134:::10000000002.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.109594+0000", 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4403463240000001, 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034197038999999999, 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 94 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.612 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109594+0000", 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109594+0000", 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.109596+0000", 2024-09-16T01:50:55.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.657e-06 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.136917+0000", 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0273212 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.136937+0000", 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9660000000000002e-05 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.136945+0000", 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.137014+0000", 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9408999999999999e-05 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.137168+0000", 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015427600000000001 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.137316+0000", 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147398 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.141801+0000", 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004485078 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.143679+0000", 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00187861 2024-09-16T01:50:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.143740+0000", 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1029000000000001e-05 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.143791+0000", 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1034e-05 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:98 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4113334289999999, 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052298391, 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 98 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138610+0000", 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-09-16T01:50:55.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166283+0000", 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027672980999999999 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166310+0000", 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7712000000000002e-05 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166324+0000", 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3239e-05 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166414+0000", 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0062000000000006e-05 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166567+0000", 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015352700000000001 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166726+0000", 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015874500000000001 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.187851+0000", 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021125117999999998 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190785+0000", 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029334970000000002 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190850+0000", 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5560000000000002e-05 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190906+0000", 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5451000000000002e-05 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.623 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:138 3.2e 3:7768043d:::10000000002.00000087:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.530968+0000", 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.0189724680000003, 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033728275000000002, 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 138 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.530968+0000", 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.530968+0000", 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.530970+0000", 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1050000000000002e-06 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555275+0000", 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024305011000000001 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555297+0000", 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1178999999999998e-05 2024-09-16T01:50:55.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555311+0000", 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4504999999999999e-05 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555423+0000", 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011233700000000001 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555589+0000", 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016574800000000001 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555755+0000", 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016627600000000001 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.561165+0000", 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0054099029999999998 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.564590+0000", 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003424365 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.564640+0000", 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0158000000000003e-05 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.564697+0000", 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6687999999999997e-05 2024-09-16T01:50:55.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:140 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9945855039999998, 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050007939000000001, 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555356+0000", 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1039999999999999e-06 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591189+0000", 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035832756 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591217+0000", 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8263e-05 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591230+0000", 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591354+0000", 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012385200000000001 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591571+0000", 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217099 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591739+0000", 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016809299999999999 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.598744+0000", 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070049939999999996 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605250+0000", 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065054079999999999 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605310+0000", 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0263000000000002e-05 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605363+0000", 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3140999999999998e-05 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.957347747, 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:50:55.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:50:55.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:145 3.16 3:6bf10b49:::10000000002.0000008e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.619755+0000", 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9301857689999999, 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038726400000000001, 2024-09-16T01:50:55.642 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 145 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.619755+0000", 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.619755+0000", 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.619757+0000", 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9759999999999998e-06 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.632949+0000", 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013191616 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.632973+0000", 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4250999999999999e-05 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.632985+0000", 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2414e-05 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.633092+0000", 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106405 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.633256+0000", 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164747 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.633415+0000", 2024-09-16T01:50:55.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158228 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649775+0000", 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016360204999999999 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.658364+0000", 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008589546 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.658435+0000", 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0555000000000005e-05 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.658481+0000", 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6456999999999999e-05 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.648 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.733191213, 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:50:55.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:50:55.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:175 3.26 3:66bf65cf:::10000000002.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.859955+0000", 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6899859309999998, 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036693007999999999, 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 175 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.655 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859955+0000", 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859955+0000", 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859957+0000", 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8619999999999999e-06 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872619+0000", 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012662242000000001 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872632+0000", 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2673999999999999e-05 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872640+0000", 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1370000000000002e-06 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.873650+0000", 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0010101039999999999 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.873825+0000", 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017534000000000001 2024-09-16T01:50:55.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.873972+0000", 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014647400000000001 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.886042+0000", 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012070317000000001 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896525+0000", 2024-09-16T01:50:55.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010483302999999999 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896600+0000", 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5124999999999996e-05 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896648+0000", 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7429999999999998e-05 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:181 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6532114920000001, 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046020995000000002, 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 181 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896731+0000", 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-06 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931107+0000", 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034375920999999997 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931130+0000", 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3668000000000001e-05 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931141+0000", 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0366999999999999e-05 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931281+0000", 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140622 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931432+0000", 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150469 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931599+0000", 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167338 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.940280+0000", 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086811649999999994 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942632+0000", 2024-09-16T01:50:55.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002352224 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942694+0000", 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2014999999999995e-05 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942750+0000", 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5763999999999997e-05 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.4359141900000001, 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:50:55.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:50:55.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:50:55.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:55.673 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:55.673 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:55.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:50:55.674 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:55.673+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.770+0000 7f0842a00640 1 -- 172.21.15.146:0/4117997815 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v57) ==== 97+0+0 (secure 0 0 0) 0x7f084c155c60 con 0x7f084c0c6bb0 2024-09-16T01:50:55.771 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:50:55.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f082c06c2b0 msgr2=0x7f082c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:55.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f082c06c2b0 0x7f082c06e770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0838034000 tx=0x7f083802e000 comp rx=0 tx=0).stop 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 msgr2=0x7f084c0c3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 0x7f084c0c3940 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f083c00ec10 tx=0x7f083c00c520 comp rx=0 tx=0).stop 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 shutdown_connections 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f082c06c2b0 0x7f082c06e770 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f084c158bb0 0x7f084c0cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f084c152850 0x7f084c0c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 --2- 172.21.15.146:0/4117997815 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f084c0c6bb0 0x7f084c0c3940 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 >> 172.21.15.146:0/4117997815 conn(0x7f084c06f3c0 msgr2=0x7f084c071ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 shutdown_connections 2024-09-16T01:50:55.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:55.772+0000 7f0851c00640 1 -- 172.21.15.146:0/4117997815 wait complete. 2024-09-16T01:50:55.775 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:55.773+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:55.838 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:55.838 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:55.838 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:50:55.838 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:55.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:50:55.876 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:55.875+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:55.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:50:55.943 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:50:55.987 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:55.985+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:56.014 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:56.015 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:56.015 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:56.015 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:56.015 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:56.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:50:56.078 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:56.077+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:56.178 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:56.178+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:56.186 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:56.186 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:56.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:56.186 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:56.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.186 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.9284901530000003, 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:50:56.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:14 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.8311698410000004, 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051400593000000001, 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 14 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.193 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352602+0000", 2024-09-16T01:50:56.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393382+0000", 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040779871000000002 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393422+0000", 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9765000000000003e-05 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393436+0000", 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3633e-05 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393509+0000", 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3847e-05 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393696+0000", 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186911 2024-09-16T01:50:56.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393839+0000", 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014274700000000001 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.401310+0000", 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074705040000000002 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403876+0000", 2024-09-16T01:50:56.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025662020000000001 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403939+0000", 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3423999999999994e-05 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.404001+0000", 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1649000000000006e-05 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7301172830000002, 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.199 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:50:56.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.204 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6213146519999997, 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.205 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:50:56.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:50:56.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:37 3.30 3:0d7ddb42:::10000000002.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.597266+0000", 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5865041690000004, 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049636995000000003, 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.211 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 37 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597266+0000", 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597266+0000", 2024-09-16T01:50:56.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597268+0000", 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8020000000000001e-06 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.631882+0000", 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034614170999999999 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.631901+0000", 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9601000000000001e-05 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.631914+0000", 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.26e-05 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.632014+0000", 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9798999999999998e-05 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.632166+0000", 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152032 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.632317+0000", 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015093999999999999 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.645621+0000", 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013304261 2024-09-16T01:50:56.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.646789+0000", 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001168056 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.646848+0000", 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8539999999999999e-05 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.646903+0000", 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5192999999999998e-05 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:53 3.9 3:9387ba5e:::10000000002.00000032:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.727663+0000", 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4561069939999998, 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048406738999999997, 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.217 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 53 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.727663+0000", 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.727663+0000", 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.727665+0000", 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.244e-06 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753019+0000", 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025353704000000001 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753046+0000", 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7200999999999999e-05 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753059+0000", 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2741e-05 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753160+0000", 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101446 2024-09-16T01:50:56.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753321+0000", 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016072800000000001 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.753482+0000", 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016113799999999999 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.771934+0000", 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018452110000000001 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775957+0000", 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040221429999999997 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.776016+0000", 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9511e-05 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.776070+0000", 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3773000000000002e-05 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:66 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.362085156, 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053741734999999999, 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.223 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821687+0000", 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3750000000000001e-06 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862658+0000", 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040970816 2024-09-16T01:50:56.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862692+0000", 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3420000000000002e-05 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862706+0000", 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4017e-05 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862879+0000", 2024-09-16T01:50:56.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173015 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863040+0000", 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016103800000000001 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863192+0000", 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152738 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.867722+0000", 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045298689999999997 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875297+0000", 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075744860000000001 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875372+0000", 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5049999999999995e-05 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875427+0000", 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4911000000000003e-05 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.3017460190000003, 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:50:56.229 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:50:56.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:50:56.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:73 3.13 3:ca5a6998:::10000000002.00000046:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.925484+0000", 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2582865639999996, 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050407227999999998, 2024-09-16T01:50:56.235 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 73 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925484+0000", 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925484+0000", 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925486+0000", 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.125e-06 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946032+0000", 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020546420999999999 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946049+0000", 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6677e-05 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946059+0000", 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9979999999999998e-06 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946147+0000", 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8294000000000005e-05 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946274+0000", 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127007 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946407+0000", 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013287100000000001 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.955245+0000", 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088381459999999998 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.975775+0000", 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020529830999999998 2024-09-16T01:50:56.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.975836+0000", 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1328000000000006e-05 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.975891+0000", 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4530000000000001e-05 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.242 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:75 3.30 3:0d2dbd17:::10000000002.00000048:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.242 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:50:56.242 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2376645919999998, 2024-09-16T01:50:56.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050908518999999999, 2024-09-16T01:50:56.244 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946107+0000", 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7710000000000001e-06 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960806+0000", 2024-09-16T01:50:56.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014698386000000001 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960821+0000", 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960832+0000", 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0698e-05 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960964+0000", 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013269700000000001 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961122+0000", 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157765 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961267+0000", 2024-09-16T01:50:56.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014486100000000001 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981986+0000", 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020719158000000001 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996886+0000", 2024-09-16T01:50:56.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014899654999999999 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996958+0000", 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2095999999999995e-05 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997014+0000", 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6258999999999998e-05 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:90 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1355400839999996, 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048363178, 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 90 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.251 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048230+0000", 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.048233+0000", 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7070000000000001e-06 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072652+0000", 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024419021999999999 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072684+0000", 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1900000000000003e-05 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072697+0000", 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3227999999999999e-05 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.072918+0000", 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022140899999999999 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.073060+0000", 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014170900000000001 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.073209+0000", 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149133 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.079317+0000", 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061078970000000002 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.096463+0000", 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017145512000000002 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.096532+0000", 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9739000000000004e-05 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.096593+0000", 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0921999999999997e-05 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:103 3.d 3:b1e2826f:::10000000002.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.159691+0000", 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0240787579999999, 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048938638999999999, 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 103 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.159691+0000", 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.159691+0000", 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.159694+0000", 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.847e-06 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200343+0000", 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040648534 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200370+0000", 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6913000000000001e-05 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200382+0000", 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2218000000000001e-05 2024-09-16T01:50:56.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200485+0000", 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103402 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200654+0000", 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016836100000000001 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:56.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.200806+0000", 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015213 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.207774+0000", 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0069686119999999999 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.208519+0000", 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074514399999999999 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.208562+0000", 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2327999999999999e-05 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.208630+0000", 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8150000000000003e-05 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:110 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9474083049999997, 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053794485000000003, 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 110 2024-09-16T01:50:56.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236365+0000", 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1590000000000002e-06 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278463+0000", 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042097744999999999 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278494+0000", 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1009000000000001e-05 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278508+0000", 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3901e-05 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278639+0000", 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013188100000000001 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278806+0000", 2024-09-16T01:50:56.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016689799999999999 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278974+0000", 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016719799999999999 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.286538+0000", 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075647910000000004 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290050+0000", 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035116969999999998 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290114+0000", 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4298e-05 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290156+0000", 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1907999999999999e-05 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:112 3.14 3:280f29a3:::10000000002.0000006d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.278544+0000", 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9052265219999995, 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040090057999999998, 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 112 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278544+0000", 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278544+0000", 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278545+0000", 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.578e-06 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.306923+0000", 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028377775000000001 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.306945+0000", 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2048999999999998e-05 2024-09-16T01:50:56.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.306955+0000", 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.307015+0000", 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0479000000000002e-05 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.307113+0000", 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8039000000000001e-05 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.307202+0000", 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9067000000000006e-05 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.312168+0000", 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049660590000000001 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.318466+0000", 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062973259999999998 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.318564+0000", 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8549999999999997e-05 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.318634+0000", 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9337000000000004e-05 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.274 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.803083279, 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:50:56.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:50:56.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:160 3.9 3:93988fc9:::10000000002.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4340932019999997, 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056552534000000002, 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 160 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749679+0000", 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.956e-06 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794169+0000", 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04449061 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794202+0000", 2024-09-16T01:50:56.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2626999999999998e-05 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794217+0000", 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794305+0000", 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8158999999999994e-05 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794460+0000", 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155277 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794625+0000", 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164854 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.801888+0000", 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072630239999999999 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806094+0000", 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042063919999999998 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806178+0000", 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3287000000000004e-05 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806229+0000", 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765000000000003e-05 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.286 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.389510595, 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.287 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:50:56.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:50:56.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:164 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.361638578, 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053828972000000003, 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 164 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.293 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822134+0000", 2024-09-16T01:50:56.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841247+0000", 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019113208 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841279+0000", 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2265999999999999e-05 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841291+0000", 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2305e-05 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841376+0000", 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4507999999999998e-05 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841535+0000", 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159216 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841687+0000", 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151846 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859703+0000", 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018016197000000001 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875876+0000", 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016172628000000001 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875929+0000", 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3108e-05 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875960+0000", 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1491000000000001e-05 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:193 3.28 3:144fb21c:::10000000002.000000be:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.298 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.040693+0000", 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.1430773260000002, 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040830065999999998, 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 193 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.040693+0000", 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.040693+0000", 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.040695+0000", 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3640000000000002e-06 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066783+0000", 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026087894 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066826+0000", 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2821999999999998e-05 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066843+0000", 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7227e-05 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.067037+0000", 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193494 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.067139+0000", 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101964 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.067232+0000", 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3424999999999994e-05 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.070925+0000", 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036927040000000002 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081436+0000", 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511629999999999 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081478+0000", 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1647999999999998e-05 2024-09-16T01:50:56.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081523+0000", 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4894000000000001e-05 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:197 3.1d 3:bb6dfbbf:::10000000002.000000c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.0766315210000004, 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057555969999999998, 2024-09-16T01:50:56.304 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 197 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107141+0000", 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148704+0000", 2024-09-16T01:50:56.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041563044 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148738+0000", 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4480999999999997e-05 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148754+0000", 2024-09-16T01:50:56.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6021000000000001e-05 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148909+0000", 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154735 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149071+0000", 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016181600000000001 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149226+0000", 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015550000000000001 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.158210+0000", 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089836529999999994 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164352+0000", 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061416810000000004 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164426+0000", 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4192999999999999e-05 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164695+0000", 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026864800000000002 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:56.310 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:56.312 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:50:56.312 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:56.311+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:56.413 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:56.412+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:56.428 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:56.428 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:56.428 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:56.428 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:56.437 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:50:56.514 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:56.512+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:56.541 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:56.541 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:56.542 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:56.542 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:56.542 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:56.551 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:50:56.615 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:56.614+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:56.696 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:56.696 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5 3.7 3:e001da36:::10000000002.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.4337396179999997, 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044893833000000001, 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260358+0000", 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.672e-06 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297299+0000", 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036941017999999999 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297321+0000", 2024-09-16T01:50:56.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2223000000000001e-05 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297332+0000", 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.058e-05 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297462+0000", 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013043399999999999 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297624+0000", 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000162263 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297768+0000", 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014397800000000001 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.303867+0000", 2024-09-16T01:50:56.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060983820000000003 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305178+0000", 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013110330000000001 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305214+0000", 2024-09-16T01:50:56.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6380000000000001e-05 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305250+0000", 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.587e-05 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:9 3.8 3:106f9e3b:::10000000002.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.297359+0000", 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.3967362760000004, 2024-09-16T01:50:56.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034407954999999997, 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 9 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297359+0000", 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297359+0000", 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297361+0000", 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5260000000000001e-06 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324776+0000", 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027414984999999999 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324795+0000", 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8760999999999999e-05 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324821+0000", 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5839000000000001e-05 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324886+0000", 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4920999999999997e-05 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.324991+0000", 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105048 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.325088+0000", 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7378000000000002e-05 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331340+0000", 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0062516919999999997 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331694+0000", 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035455699999999997 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331729+0000", 2024-09-16T01:50:56.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4588000000000002e-05 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.331767+0000", 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8659999999999999e-05 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:13 3.2 3:438a76aa:::10000000002.0000000a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.339887289, 2024-09-16T01:50:56.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045857943999999998, 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 13 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354210+0000", 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.203e-06 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388636+0000", 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034425859000000003 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388673+0000", 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7004e-05 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388693+0000", 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0194999999999999e-05 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388771+0000", 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8737000000000002e-05 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388904+0000", 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013215899999999999 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.389016+0000", 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112642 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.392245+0000", 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032287370000000002 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.399956+0000", 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0077109190000000001 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400015+0000", 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9119000000000001e-05 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400066+0000", 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1369999999999998e-05 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:21 3.8 3:129cb038:::10000000002.00000012:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.405307+0000", 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.2887887999999998, 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026753832000000002, 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 21 2024-09-16T01:50:56.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.405307+0000", 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.405307+0000", 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.405308+0000", 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.37e-06 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427405+0000", 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022096525999999998 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427420+0000", 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4732999999999999e-05 2024-09-16T01:50:56.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427445+0000", 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5222999999999999e-05 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427503+0000", 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7701000000000002e-05 2024-09-16T01:50:56.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427606+0000", 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010304799999999999 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.427703+0000", 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7345999999999999e-05 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.431471+0000", 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037677499999999998 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.431987+0000", 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00051606500000000001 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.432023+0000", 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6678999999999999e-05 2024-09-16T01:50:56.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.432061+0000", 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7391000000000002e-05 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:29 3.7 3:e177cc1c:::10000000002.0000001a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.458164+0000", 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.2359320680000003, 2024-09-16T01:50:56.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.03214736, 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 29 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458164+0000", 2024-09-16T01:50:56.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458164+0000", 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458166+0000", 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.317e-06 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484169+0000", 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026003025999999999 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484188+0000", 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9412e-05 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484197+0000", 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1089999999999998e-06 2024-09-16T01:50:56.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484258+0000", 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1696e-05 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484390+0000", 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000131421 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484514+0000", 2024-09-16T01:50:56.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012385 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.489955+0000", 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0054417959999999996 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.490201+0000", 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000245837 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.490259+0000", 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8168000000000003e-05 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.490311+0000", 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1727999999999997e-05 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:30 3.7 3:e33cd625:::10000000002.0000001b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.728 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.2098783629999996, 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035807259000000001, 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 30 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484219+0000", 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512447+0000", 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028228652 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512476+0000", 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8283e-05 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512489+0000", 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3049e-05 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512566+0000", 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7420000000000001e-05 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512720+0000", 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015377000000000001 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512891+0000", 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017147199999999999 2024-09-16T01:50:56.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.516819+0000", 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003927458 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519938+0000", 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031196489999999999 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519989+0000", 2024-09-16T01:50:56.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0380000000000002e-05 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.520025+0000", 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5839e-05 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:31 3.6 3:628433a0:::10000000002.0000001c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.517094+0000", 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.1770021130000003, 2024-09-16T01:50:56.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029773069999999999, 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 31 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.517094+0000", 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.517094+0000", 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.517096+0000", 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3640000000000002e-06 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542778+0000", 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025682304 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542797+0000", 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8357999999999999e-05 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542806+0000", 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5540000000000003e-06 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542871+0000", 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4879e-05 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.542966+0000", 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4513999999999997e-05 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.543049+0000", 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3792e-05 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546391+0000", 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00334184 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546762+0000", 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000370727 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546813+0000", 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1140000000000002e-05 2024-09-16T01:50:56.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.546867+0000", 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3597999999999997e-05 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:50:56.740 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.1444748259999997, 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:50:56.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.8974412449999996, 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:50:56.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:50:56.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:50:56.752 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.8382821590000002, 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:50:56.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:50:56.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:82 3.2b 3:d50470f2:::10000000002.0000004f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:50:56.758 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.7313303209999997, 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047169015000000002, 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 82 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962768+0000", 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3269999999999999e-06 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002536+0000", 2024-09-16T01:50:56.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039768464000000003 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002551+0000", 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002558+0000", 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8730000000000001e-06 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002630+0000", 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2025000000000002e-05 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002751+0000", 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121197 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002871+0000", 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011995 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.006288+0000", 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003417032 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009828+0000", 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00353996 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009879+0000", 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0473999999999998e-05 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009934+0000", 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5872999999999999e-05 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:114 3.8 3:10128d86:::10000000002.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.4254419699999996, 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038796303999999997, 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.764 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 114 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268656+0000", 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.858e-06 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303790+0000", 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035134060000000002 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303806+0000", 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6611000000000001e-05 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303814+0000", 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4329999999999998e-06 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303869+0000", 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5639e-05 2024-09-16T01:50:56.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303979+0000", 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110043 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.304078+0000", 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9006999999999996e-05 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.306722+0000", 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002643074 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307380+0000", 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00065801399999999995 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307414+0000", 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4082000000000003e-05 2024-09-16T01:50:56.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307450+0000", 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6483000000000003e-05 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:125 3.7 3:e320baf1:::10000000002.0000007a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.375300+0000", 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.3187955149999997, 2024-09-16T01:50:56.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032541303000000001, 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 125 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.375300+0000", 2024-09-16T01:50:56.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.375300+0000", 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.375302+0000", 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.131e-06 2024-09-16T01:50:56.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402165+0000", 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026862832999999999 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402184+0000", 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8352e-05 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402196+0000", 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2818e-05 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402261+0000", 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4658000000000003e-05 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402396+0000", 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000135029 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.402520+0000", 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123607 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407361+0000", 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004841756 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407733+0000", 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00037123000000000001 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407786+0000", 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3112999999999998e-05 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.407842+0000", 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5776000000000003e-05 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.776 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:132 3.2b 3:d52309a9:::10000000002.00000081:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.2244048459999997, 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034693794, 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.777 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 132 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469692+0000", 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.516e-06 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496852+0000", 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027159651999999999 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496868+0000", 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5869999999999999e-05 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496875+0000", 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.391e-06 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496952+0000", 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6320000000000001e-05 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497109+0000", 2024-09-16T01:50:56.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015718700000000001 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497263+0000", 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015406799999999999 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.500555+0000", 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032925720000000001 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504317+0000", 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037611150000000002 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504351+0000", 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4214000000000001e-05 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504385+0000", 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3889e-05 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.782 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.9267038469999997, 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.783 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:50:56.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:50:56.788 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.8824436169999998, 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.789 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:50:56.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:50:56.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.794 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.8519437969999997, 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.795 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:50:56.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:50:56.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.8103882750000002, 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:50:56.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:50:56.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:50:56.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.806 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:183 3.1b 3:db82751e:::10000000002.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.7230016590000004, 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040945861, 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 183 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971104+0000", 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8719999999999997e-06 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999810+0000", 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028705819 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999832+0000", 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2560000000000001e-05 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999847+0000", 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4956e-05 2024-09-16T01:50:56.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999905+0000", 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8084000000000002e-05 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000017+0000", 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111222 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000113+0000", 2024-09-16T01:50:56.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6131999999999994e-05 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.002657+0000", 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025443089999999998 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011949+0000", 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0092920350000000006 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.811 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011989+0000", 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9987999999999997e-05 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.012040+0000", 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0884000000000003e-05 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:56.812 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.6723415809999995, 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:50:56.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:50:56.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:50:56.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:50:56.818 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:56.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:50:56.820 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:50:56.821 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:56.819+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:56.921 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:56.920+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:56.925 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:56.925 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:56.925 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:50:56.925 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:56.934 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:50:56.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.971+0000 7f8212200640 1 Processor -- start 2024-09-16T01:50:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.971+0000 7f8212200640 1 -- start start 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8212200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c152c30 0x7f820c0c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8212200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f820c0c6e90 0x7f820c0c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8212200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c0c4240 con 0x7f820c0c6ab0 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8212200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f820c0c4da0 con 0x7f820c0c6e90 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8212200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f820c0c4500 con 0x7f820c152c30 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820b400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c152c30 0x7f820c0c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8202a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f820c0c6ab0 0x7f820c0c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56328/0 (socket says 172.21.15.146:56328) 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f8202a00640 1 -- 172.21.15.146:0/3069864410 learned_addr learned my addr 172.21.15.146:0/3069864410 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:56.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820be00640 1 --2- 172.21.15.146:0/3069864410 >> v2:172.21.15.146:3300/0 conn(0x7f820c0c6e90 0x7f820c0c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820b400640 1 -- 172.21.15.146:0/3069864410 >> v2:172.21.15.146:3300/0 conn(0x7f820c0c6e90 msgr2=0x7f820c0c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820b400640 1 --2- 172.21.15.146:0/3069864410 >> v2:172.21.15.146:3300/0 conn(0x7f820c0c6e90 0x7f820c0c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820b400640 1 -- 172.21.15.146:0/3069864410 >> v1:172.21.15.161:6789/0 conn(0x7f820c0c6ab0 legacy=0x7f820c0c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820aa00640 1 -- 172.21.15.146:0/3069864410 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22558131 0 0) 0x7f820c0c4240 con 0x7f820c0c6ab0 2024-09-16T01:50:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.973+0000 7f820b400640 1 -- 172.21.15.146:0/3069864410 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c0c47e0 con 0x7f820c152c30 2024-09-16T01:50:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.974+0000 7f820b400640 1 --2- 172.21.15.146:0/3069864410 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c152c30 0x7f820c0c36b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0097a0 tx=0x7f81fc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24fed8cb1a9b34ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f820aa00640 1 -- 172.21.15.146:0/3069864410 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc037070 con 0x7f820c152c30 2024-09-16T01:50:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f820aa00640 1 -- 172.21.15.146:0/3069864410 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81fc032440 con 0x7f820c152c30 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f820aa00640 1 -- 172.21.15.146:0/3069864410 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc032810 con 0x7f820c152c30 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 -- 172.21.15.146:0/3069864410 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c152c30 msgr2=0x7f820c0c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 --2- 172.21.15.146:0/3069864410 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c152c30 0x7f820c0c36b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0097a0 tx=0x7f81fc02a180 comp rx=0 tx=0).stop 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 -- 172.21.15.146:0/3069864410 shutdown_connections 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 --2- 172.21.15.146:0/3069864410 >> v2:172.21.15.146:3300/0 conn(0x7f820c0c6e90 0x7f820c0c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 --2- 172.21.15.146:0/3069864410 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c152c30 0x7f820c0c36b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 -- 172.21.15.146:0/3069864410 >> 172.21.15.146:0/3069864410 conn(0x7f820c06bc10 msgr2=0x7f820c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 -- 172.21.15.146:0/3069864410 shutdown_connections 2024-09-16T01:50:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.975+0000 7f8212200640 1 -- 172.21.15.146:0/3069864410 wait complete. 2024-09-16T01:50:56.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.977+0000 7f8212200640 1 Processor -- start 2024-09-16T01:50:56.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.977+0000 7f8212200640 1 -- start start 2024-09-16T01:50:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8212200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c0c6ab0 0x7f820c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8212200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 0x7f820c1f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8212200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f820c152c30 0x7f820c1e3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8212200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0c9350 con 0x7f820c0c6ab0 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8212200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0c6630 con 0x7f820c152c30 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8212200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f820c0c94d0 con 0x7f820c0c6e90 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f820b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c0c6ab0 0x7f820c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8202a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 0x7f820c1f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f820be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f820c152c30 0x7f820c1e3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f820be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f820c152c30 0x7f820c1e3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48254/0 (socket says 172.21.15.146:48254) 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f8202a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 0x7f820c1f4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:51366/0 (socket says 172.21.15.146:51366) 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f820be00640 1 -- 172.21.15.146:0/464432278 learned_addr learned my addr 172.21.15.146:0/464432278 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.978+0000 7f820b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c0c6ab0 0x7f820c1510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50888/0 (socket says 172.21.15.146:50888) 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8202a00640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f820c152c30 msgr2=0x7f820c1e3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f820c152c30 0x7f820c1e3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8202a00640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c0c6ab0 msgr2=0x7f820c1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c0c6ab0 0x7f820c1510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8202a00640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c0c0710 con 0x7f820c0c6e90 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c0c6ab0 0x7f820c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 0x7f820c1f4240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c410 tx=0x7f81f400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f40074e0 con 0x7f820c0c6e90 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f40047e0 con 0x7f820c0c6e90 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4004bd0 con 0x7f820c0c6e90 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c0c1500 con 0x7f820c0c6e90 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.979+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c0c2190 con 0x7f820c0c6e90 2024-09-16T01:50:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.980+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f820c0c36b0 con 0x7f820c0c6e90 2024-09-16T01:50:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.982+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f81f4011070 con 0x7f820c0c6e90 2024-09-16T01:50:56.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.982+0000 7f8203e00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81e806c2e0 0x7f81e806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.982+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 5799+0+0 (secure 0 0 0) 0x7f81f408b120 con 0x7f820c0c6e90 2024-09-16T01:50:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.983+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81e806c2e0 0x7f81e806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.983+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v57) ==== 54+0+16 (secure 0 0 0) 0x7f81f408b4d0 con 0x7f820c0c6e90 2024-09-16T01:50:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.983+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f820c1f3e20 0x7f820c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.983+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f820c072c70 con 0x7f820c1f3e20 2024-09-16T01:50:56.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.983+0000 7f820be00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f820c1f3e20 0x7f820c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.983+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81e806c2e0 0x7f81e806e7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81fc034000 tx=0x7f81fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.984+0000 7f820be00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f820c1f3e20 0x7f820c0c2930 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:56.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.984+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c072c70 con 0x7f820c1f3e20 2024-09-16T01:50:57.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:56.999+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c1f3ba0 con 0x7f820c1f3e20 2024-09-16T01:50:57.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.001+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c1f3ba0 con 0x7f820c1f3e20 2024-09-16T01:50:57.002 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:57.002 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:57.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.001+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f820c1e48e0 0x7f820c1484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.001+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f820c1e4cc0 con 0x7f820c1e48e0 2024-09-16T01:50:57.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.001+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f820c1e48e0 0x7f820c1484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.002+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f820c1e48e0 0x7f820c1484b0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.003+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c1e4cc0 con 0x7f820c1e48e0 2024-09-16T01:50:57.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.017+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c0c7530 con 0x7f820c1e48e0 2024-09-16T01:50:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.019+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c0c7530 con 0x7f820c1e48e0 2024-09-16T01:50:57.020 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.019+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f820c148f00 0x7f820c062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.019+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f820c072ec0 con 0x7f820c148f00 2024-09-16T01:50:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.019+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f820c148f00 0x7f820c062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.020+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f820c148f00 0x7f820c062630 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.021+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c072ec0 con 0x7f820c148f00 2024-09-16T01:50:57.022 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:57.021+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:57.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.034+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c151610 con 0x7f820c148f00 2024-09-16T01:50:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.036+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c151610 con 0x7f820c148f00 2024-09-16T01:50:57.037 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.036+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f820c063080 0x7f820c1e52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.036+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f820c063500 con 0x7f820c063080 2024-09-16T01:50:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.036+0000 7f820be00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f820c063080 0x7f820c1e52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.037+0000 7f820be00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f820c063080 0x7f820c1e52b0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.038+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c063500 con 0x7f820c063080 2024-09-16T01:50:57.039 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:50:57.039 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:50:57.039 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:50:57.039 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:50:57.039 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:50:57.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:50:57.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.051+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c1e5ac0 con 0x7f820c063080 2024-09-16T01:50:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.052+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c1e5ac0 con 0x7f820c063080 2024-09-16T01:50:57.054 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.053+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f820c02c640 0x7f820c02caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.053+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f820c1e61e0 con 0x7f820c02c640 2024-09-16T01:50:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.053+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f820c02c640 0x7f820c02caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.053+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f820c02c640 0x7f820c02caa0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.054+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c1e61e0 con 0x7f820c02c640 2024-09-16T01:50:57.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.067+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c1517c0 con 0x7f820c02c640 2024-09-16T01:50:57.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.069+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f81fc048520 con 0x7f820c02c640 2024-09-16T01:50:57.071 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.070+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f820c02d4f0 0x7f820c207140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.070+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f820c02d970 con 0x7f820c02d4f0 2024-09-16T01:50:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.070+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f820c02d4f0 0x7f820c207140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.070+0000 7f8202a00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f820c02d4f0 0x7f820c207140 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.071+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c02d970 con 0x7f820c02d4f0 2024-09-16T01:50:57.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.084+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c02d970 con 0x7f820c02d4f0 2024-09-16T01:50:57.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.087+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c02d970 con 0x7f820c02d4f0 2024-09-16T01:50:57.088 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:57.089 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:57.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.088+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f820c207b90 0x7f820c209f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.088+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f820c20cdd0 con 0x7f820c207b90 2024-09-16T01:50:57.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.088+0000 7f820be00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f820c207b90 0x7f820c209f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.088+0000 7f820be00640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f820c207b90 0x7f820c209f80 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.089+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c20cdd0 con 0x7f820c207b90 2024-09-16T01:50:57.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.103+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c20a540 con 0x7f820c207b90 2024-09-16T01:50:57.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.104+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c20a540 con 0x7f820c207b90 2024-09-16T01:50:57.106 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:57.106 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:57.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.105+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f820c20ac60 0x7f820c20b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:57.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.105+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f820c210140 con 0x7f820c20ac60 2024-09-16T01:50:57.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.105+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f820c20ac60 0x7f820c20b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:57.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.106+0000 7f820b400640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f820c20ac60 0x7f820c20b0e0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:57.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.106+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820c210140 con 0x7f820c20ac60 2024-09-16T01:50:57.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.121+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f820c20efc0 con 0x7f820c20ac60 2024-09-16T01:50:57.122 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:57.122+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:57.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8203e00640 1 -- 172.21.15.146:0/464432278 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f820c20efc0 con 0x7f820c20ac60 2024-09-16T01:50:57.124 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f820c1f3e20 msgr2=0x7f820c0c2930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f820c1f3e20 0x7f820c0c2930 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f820c1e48e0 msgr2=0x7f820c1484b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f820c1e48e0 0x7f820c1484b0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f820c148f00 msgr2=0x7f820c062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f820c148f00 0x7f820c062630 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f820c063080 msgr2=0x7f820c1e52b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f820c063080 0x7f820c1e52b0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f820c02c640 msgr2=0x7f820c02caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f820c02c640 0x7f820c02caa0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f820c02d4f0 msgr2=0x7f820c207140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f820c02d4f0 0x7f820c207140 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f820c207b90 msgr2=0x7f820c209f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f820c207b90 0x7f820c209f80 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f820c20ac60 msgr2=0x7f820c20b0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f820c20ac60 0x7f820c20b0e0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f820be00640 1 -- 172.21.15.146:0/464432278 reap_dead start 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f820be00640 1 -- 172.21.15.146:0/464432278 reap_dead start 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f820be00640 1 -- 172.21.15.146:0/464432278 reap_dead start 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f820be00640 1 -- 172.21.15.146:0/464432278 reap_dead start 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.123+0000 7f820be00640 1 -- 172.21.15.146:0/464432278 reap_dead start 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81e806c2e0 msgr2=0x7f81e806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81e806c2e0 0x7f81e806e7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f81fc034000 tx=0x7f81fc02e000 comp rx=0 tx=0).stop 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 msgr2=0x7f820c1f4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 0x7f820c1f4240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c410 tx=0x7f81f400c8d0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 shutdown_connections 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f81e806c2e0 0x7f81e806e7a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 --2- 172.21.15.146:0/464432278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f820c0c6e90 0x7f820c1f4240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 >> 172.21.15.146:0/464432278 conn(0x7f820c06bc10 msgr2=0x7f820c14dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:57.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 shutdown_connections 2024-09-16T01:50:57.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:57.124+0000 7f8212200640 1 -- 172.21.15.146:0/464432278 wait complete. 2024-09-16T01:50:57.154 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.154 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:57.163 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:50:57.224 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:57.223+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:57.266 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.267 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:57.275 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:50:57.325 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:57.323+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:57.378 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.378 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:57.387 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:50:57.428 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:57.427+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:57.526 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:57.524+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:57.531 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.531 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:57.540 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:50:57.628 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:57.627+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:57.687 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.687 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:57.698 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:50:57.742 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:57.741+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:57.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:50:57.820 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.820 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:57.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:50:57.838 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:57.837+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:57.931 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:57.930+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:57.993 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:57.993 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:58.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:50:58.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.006+0000 7fa35ea00640 1 Processor -- start 2024-09-16T01:50:58.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.007+0000 7fa35ea00640 1 -- start start 2024-09-16T01:50:58.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.008+0000 7fa35ea00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3580c6b70 0x7fa35815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.008+0000 7fa35ea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580c75a0 0x7fa3580cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.008+0000 7fa35ea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa35806d140 con 0x7fa3580cbab0 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.008+0000 7fa35ea00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa35815b880 con 0x7fa3580c6b70 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.008+0000 7fa35ea00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa3580cab20 con 0x7fa3580c75a0 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.009+0000 7fa35d600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3580c6b70 0x7fa35815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.009+0000 7fa35cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580c75a0 0x7fa3580cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.009+0000 7fa35d600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa3580c6b70 0x7fa35815b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48262/0 (socket says 172.21.15.146:48262) 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.009+0000 7fa35d600640 1 -- 172.21.15.146:0/2931009065 learned_addr learned my addr 172.21.15.146:0/2931009065 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.010+0000 7fa35d600640 1 -- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580c75a0 msgr2=0x7fa3580cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.010+0000 7fa35d600640 1 --2- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580c75a0 0x7fa3580cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.010+0000 7fa35d600640 1 -- 172.21.15.146:0/2931009065 >> v1:172.21.15.146:6789/0 conn(0x7fa3580cbab0 legacy=0x7fa3580cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:58.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.010+0000 7fa35d600640 1 -- 172.21.15.146:0/2931009065 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35806d420 con 0x7fa3580c6b70 2024-09-16T01:50:58.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.012+0000 7fa35d600640 1 --2- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c6b70 0x7fa35815b340 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009870 tx=0x7fa34c02a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f387954cc6ae96e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:58.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.012+0000 7fa357e00640 1 -- 172.21.15.146:0/2931009065 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c037070 con 0x7fa3580c6b70 2024-09-16T01:50:58.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.012+0000 7fa357e00640 1 -- 172.21.15.146:0/2931009065 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c02ae20 con 0x7fa3580c6b70 2024-09-16T01:50:58.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.012+0000 7fa357e00640 1 -- 172.21.15.146:0/2931009065 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c032640 con 0x7fa3580c6b70 2024-09-16T01:50:58.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.013+0000 7fa35ea00640 1 -- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c6b70 msgr2=0x7fa35815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:58.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.013+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c6b70 0x7fa35815b340 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa34c009870 tx=0x7fa34c02a110 comp rx=0 tx=0).stop 2024-09-16T01:50:58.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.013+0000 7fa35ea00640 1 -- 172.21.15.146:0/2931009065 shutdown_connections 2024-09-16T01:50:58.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.013+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580c75a0 0x7fa3580cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.013+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2931009065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c6b70 0x7fa35815b340 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.013+0000 7fa35ea00640 1 -- 172.21.15.146:0/2931009065 >> 172.21.15.146:0/2931009065 conn(0x7fa3580ccc20 msgr2=0x7fa3580cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:58.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.019+0000 7fa35ea00640 1 -- 172.21.15.146:0/2931009065 shutdown_connections 2024-09-16T01:50:58.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.019+0000 7fa35ea00640 1 -- 172.21.15.146:0/2931009065 wait complete. 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.022+0000 7fa35ea00640 1 Processor -- start 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.022+0000 7fa35ea00640 1 -- start start 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3580c6b70 0x7fa3581f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35ea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c75a0 0x7fa3581f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35ea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580cbab0 0x7fa3580c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa3580cb050 con 0x7fa3580c6b70 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35ea00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3580cad70 con 0x7fa3580c75a0 2024-09-16T01:50:58.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35ea00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa35806cd20 con 0x7fa3580cbab0 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c75a0 0x7fa3581f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c75a0 0x7fa3581f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48266/0 (socket says 172.21.15.146:48266) 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35cc00640 1 -- 172.21.15.146:0/2968372489 learned_addr learned my addr 172.21.15.146:0/2968372489 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.023+0000 7fa35d600640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3580c6b70 0x7fa3581f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.024+0000 7fa35e000640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580cbab0 0x7fa3580c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.024+0000 7fa35d600640 1 -- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580cbab0 msgr2=0x7fa3580c18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.024+0000 7fa35d600640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580cbab0 0x7fa3580c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.024+0000 7fa35d600640 1 -- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c75a0 msgr2=0x7fa3581f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.024+0000 7fa35d600640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c75a0 0x7fa3581f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.024+0000 7fa35d600640 1 -- 172.21.15.146:0/2968372489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa358157410 con 0x7fa3580c6b70 2024-09-16T01:50:58.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.025+0000 7fa35d600640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3580c6b70 0x7fa3581f2be0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3580ca7a0 tx=0x7fa34c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:58.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.029+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c041020 con 0x7fa3580c6b70 2024-09-16T01:50:58.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.029+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c004490 con 0x7fa3580c6b70 2024-09-16T01:50:58.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.029+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c037040 con 0x7fa3580c6b70 2024-09-16T01:50:58.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.029+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3581576f0 con 0x7fa3580c6b70 2024-09-16T01:50:58.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.030+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3581f23e0 con 0x7fa3580c6b70 2024-09-16T01:50:58.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.032+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3580c66d0 con 0x7fa3580c6b70 2024-09-16T01:50:58.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.033+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa34c044020 con 0x7fa3580c6b70 2024-09-16T01:50:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.035+0000 7fa355600640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa33806c330 0x7fa33806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:58.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.035+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5799+0+0 (secure 0 0 0) 0x7fa34c0ab710 con 0x7fa3580c6b70 2024-09-16T01:50:58.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.036+0000 7fa35cc00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa33806c330 0x7fa33806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:58.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.037+0000 7fa35cc00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa33806c330 0x7fa33806e7f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa3580c9330 tx=0x7fa344008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:58.045 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:58.044+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.044+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa3580c66d0 con 0x7fa3580c6b70 2024-09-16T01:50:58.133 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:58.132+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:58.233 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:58.232+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:58.242 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:58.242 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:58.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:50:58.350 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:58.349+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:58.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.365+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fa3580ce650 con 0x7fa3580c6b70 2024-09-16T01:50:58.435 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:58.434+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:58.536 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:58.535+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:58.539 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:58.539 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:58.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:50:58.637 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:50:58.636+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:50:58.742 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:58.741+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:58.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.789+0000 7fa355600640 1 -- 172.21.15.146:0/2968372489 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v58) ==== 91+0+0 (secure 0 0 0) 0x7fa3580ce650 con 0x7fa3580c6b70 2024-09-16T01:50:58.790 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:50:58.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.792+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa33806c330 msgr2=0x7fa33806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:58.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.792+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa33806c330 0x7fa33806e7f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa3580c9330 tx=0x7fa344008040 comp rx=0 tx=0).stop 2024-09-16T01:50:58.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.792+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3580c6b70 msgr2=0x7fa3581f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:58.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.792+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3580c6b70 0x7fa3581f2be0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3580ca7a0 tx=0x7fa34c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.794+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 shutdown_connections 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.794+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa33806c330 0x7fa33806e7f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.794+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa3580cbab0 0x7fa3580c18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.794+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3580c75a0 0x7fa3581f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.794+0000 7fa35ea00640 1 --2- 172.21.15.146:0/2968372489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3580c6b70 0x7fa3581f2be0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.794+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 >> 172.21.15.146:0/2968372489 conn(0x7fa3580ccc20 msgr2=0x7fa3580cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.795+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 shutdown_connections 2024-09-16T01:50:58.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:58.795+0000 7fa35ea00640 1 -- 172.21.15.146:0/2968372489 wait complete. 2024-09-16T01:50:58.835 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:58.835 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:58.844 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:58.841+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:58.856 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:58.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:50:58.857 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:50:58.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-09-16T01:50:58.859 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:50:58.954 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:58.953+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:59.045 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:50:59.044+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:50:59.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:50:59.162 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:59.161+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:59.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.180+0000 7f04f2600640 1 Processor -- start 2024-09-16T01:50:59.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.183+0000 7f04f2600640 1 -- start start 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.185+0000 7f04f2600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f04ec0c5eb0 0x7f04ec0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.185+0000 7f04f2600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ec15af00 con 0x7f04ec0c6290 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.185+0000 7f04f2600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ec15b1e0 con 0x7f04ec15aa70 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.185+0000 7f04f2600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f04ec06dbd0 con 0x7f04ec0c5eb0 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.186+0000 7f04f1200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f04ec0c5eb0 0x7f04ec0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.186+0000 7f04f1200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f04ec0c5eb0 0x7f04ec0cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48276/0 (socket says 172.21.15.146:48276) 2024-09-16T01:50:59.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.186+0000 7f04f1200640 1 -- 172.21.15.146:0/3674677894 learned_addr learned my addr 172.21.15.146:0/3674677894 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:59.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.188+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290217977 0 0) 0x7f04ec15af00 con 0x7f04ec0c6290 2024-09-16T01:50:59.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.188+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04d4003520 con 0x7f04ec0c6290 2024-09-16T01:50:59.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4219381904 0 0) 0x7f04d4003520 con 0x7f04ec0c6290 2024-09-16T01:50:59.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 >> v2:172.21.15.161:3300/0 conn(0x7f04ec0c5eb0 msgr2=0x7f04ec0cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:59.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 --2- 172.21.15.146:0/3674677894 >> v2:172.21.15.161:3300/0 conn(0x7f04ec0c5eb0 0x7f04ec0cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 >> v1:172.21.15.161:6789/0 conn(0x7f04ec15aa70 legacy=0x7f04ec154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec15b4c0 con 0x7f04ec0c6290 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f04e40019d0 con 0x7f04ec0c6290 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 >> v1:172.21.15.146:6789/0 conn(0x7f04ec0c6290 legacy=0x7f04ec06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d40039b0 0x7f04d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04d4004800 0x7f04d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04d40051c0 0x7f04d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec06dbd0 con 0x7f04d40039b0 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04d400c0f0 con 0x7f04d4004800 2024-09-16T01:50:59.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.190+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f04d400c2f0 con 0x7f04d40051c0 2024-09-16T01:50:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.191+0000 7f04ebe00640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04d4004800 0x7f04d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.192+0000 7f04f1c00640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04d40051c0 0x7f04d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.192+0000 7f04ebe00640 1 -- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04d40051c0 msgr2=0x7f04d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.192+0000 7f04ebe00640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04d40051c0 0x7f04d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.192+0000 7f04ebe00640 1 -- 172.21.15.146:0/3674677894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d40039b0 msgr2=0x7f04d4003dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:50:59.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.192+0000 7f04ebe00640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d40039b0 0x7f04d4003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.192+0000 7f04ebe00640 1 -- 172.21.15.146:0/3674677894 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec15b4c0 con 0x7f04d4004800 2024-09-16T01:50:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.197+0000 7f04f1c00640 1 -- 172.21.15.146:0/3674677894 reap_dead start 2024-09-16T01:50:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.197+0000 7f04ebe00640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04d4004800 0x7f04d4004c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04d8001790 tx=0x7f04d4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0f9c541582700b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.197+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8001360 con 0x7f04d4004800 2024-09-16T01:50:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.197+0000 7f04eb400640 1 -- 172.21.15.146:0/3674677894 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d8037070 con 0x7f04d4004800 2024-09-16T01:50:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.198+0000 7f04f2600640 1 -- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04d4004800 msgr2=0x7f04d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.198+0000 7f04f2600640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04d4004800 0x7f04d4004c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04d8001790 tx=0x7f04d4005830 comp rx=0 tx=0).stop 2024-09-16T01:50:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.199+0000 7f04f2600640 1 -- 172.21.15.146:0/3674677894 shutdown_connections 2024-09-16T01:50:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.199+0000 7f04f2600640 1 --2- 172.21.15.146:0/3674677894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04d4004800 0x7f04d4004c80 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04d8001790 tx=0x7f04d4005830 comp rx=0 tx=0).stop 2024-09-16T01:50:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.199+0000 7f04f2600640 1 -- 172.21.15.146:0/3674677894 >> 172.21.15.146:0/3674677894 conn(0x7f04ec067b30 msgr2=0x7f04ec067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:59.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.199+0000 7f04f2600640 1 -- 172.21.15.146:0/3674677894 shutdown_connections 2024-09-16T01:50:59.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.199+0000 7f04f2600640 1 -- 172.21.15.146:0/3674677894 wait complete. 2024-09-16T01:50:59.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.202+0000 7f04f2600640 1 Processor -- start 2024-09-16T01:50:59.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.203+0000 7f04f2600640 1 -- start start 2024-09-16T01:50:59.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.205+0000 7f04f2600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04ec0c7730 0x7f04ec14e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f2600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04ec0c7b10 0x7f04ec14ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f2600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 0x7f04ec06f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f2600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec0c56f0 con 0x7f04ec0c7730 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f2600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec06c370 con 0x7f04ec06f3e0 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f2600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec0c5870 con 0x7f04ec0c7b10 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 0x7f04ec06f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 0x7f04ec06f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48296/0 (socket says 172.21.15.146:48296) 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.206+0000 7f04f1c00640 1 -- 172.21.15.146:0/1604551706 learned_addr learned my addr 172.21.15.146:0/1604551706 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:59.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.207+0000 7f04f1c00640 1 -- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04ec0c7b10 msgr2=0x7f04ec14ea60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:50:59.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.207+0000 7f04f1c00640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04ec0c7b10 0x7f04ec14ea60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.207+0000 7f04f1c00640 1 -- 172.21.15.146:0/1604551706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04ec0c7730 msgr2=0x7f04ec14e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.207+0000 7f04f1200640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04ec0c7730 0x7f04ec14e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.207+0000 7f04f1c00640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04ec0c7730 0x7f04ec14e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.207+0000 7f04f1c00640 1 -- 172.21.15.146:0/1604551706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec06efd0 con 0x7f04ec06f3e0 2024-09-16T01:50:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.208+0000 7f04f1c00640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 0x7f04ec06f890 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0d1350 tx=0x7f04e400ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.208+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4003c40 con 0x7f04ec06f3e0 2024-09-16T01:50:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.208+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e4018070 con 0x7f04ec06f3e0 2024-09-16T01:50:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.208+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e400da40 con 0x7f04ec06f3e0 2024-09-16T01:50:59.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.209+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec14da80 con 0x7f04ec06f3e0 2024-09-16T01:50:59.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.209+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec14dd20 con 0x7f04ec06f3e0 2024-09-16T01:50:59.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.210+0000 7f04f1200640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04ec0c7730 0x7f04ec14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:50:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.213+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec0cac30 con 0x7f04ec06f3e0 2024-09-16T01:50:59.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.213+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f04e400dbe0 con 0x7f04ec06f3e0 2024-09-16T01:50:59.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.215+0000 7f04e8c00640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f04cc06c3b0 0x7f04cc06e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.215+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5799+0+0 (secure 0 0 0) 0x7f04e408b5b0 con 0x7f04ec06f3e0 2024-09-16T01:50:59.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.216+0000 7f04f1200640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f04cc06c3b0 0x7f04cc06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.217+0000 7f04f1200640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f04cc06c3b0 0x7f04cc06e870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04e000d940 tx=0x7f04e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.220+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f04e4059ca0 con 0x7f04ec06f3e0 2024-09-16T01:50:59.234 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:50:59.235 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:50:59.248 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:50:59.248+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:50:59.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:50:59.346 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:59.345+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:59.447 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:50:59.446+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:50:59.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.477+0000 7ff302e00640 1 Processor -- start 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.477+0000 7ff302e00640 1 -- start start 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff302e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff2fc162a30 0x7ff2fc162e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff302e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc0c6cd0 0x7ff2fc0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff302e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2fc0c89b0 con 0x7ff2fc159300 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff302e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff2fc163370 con 0x7ff2fc162a30 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff302e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc0c76a0 con 0x7ff2fc0c6cd0 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff2fbe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff2fc159300 0x7ff2fc0c9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:56348/0 (socket says 172.21.15.146:56348) 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.480+0000 7ff2fbe00640 1 -- 172.21.15.146:0/3293844527 learned_addr learned my addr 172.21.15.146:0/3293844527 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.481+0000 7ff2fb400640 1 --2- 172.21.15.146:0/3293844527 >> v2:172.21.15.161:3300/0 conn(0x7ff2fc162a30 0x7ff2fc162e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.481+0000 7ff300e00640 1 --2- 172.21.15.146:0/3293844527 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc0c6cd0 0x7ff2fc0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.481+0000 7ff300e00640 1 -- 172.21.15.146:0/3293844527 >> v2:172.21.15.161:3300/0 conn(0x7ff2fc162a30 msgr2=0x7ff2fc162e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:50:59.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.481+0000 7ff300e00640 1 --2- 172.21.15.146:0/3293844527 >> v2:172.21.15.161:3300/0 conn(0x7ff2fc162a30 0x7ff2fc162e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.481+0000 7ff300e00640 1 -- 172.21.15.146:0/3293844527 >> v1:172.21.15.161:6789/0 conn(0x7ff2fc159300 legacy=0x7ff2fc0c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.481+0000 7ff300e00640 1 -- 172.21.15.146:0/3293844527 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc15f230 con 0x7ff2fc0c6cd0 2024-09-16T01:50:59.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.483+0000 7ff300e00640 1 --2- 172.21.15.146:0/3293844527 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc0c6cd0 0x7ff2fc0c7160 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009d90 tx=0x7ff2f402a550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb87bc92aafb916 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.483+0000 7ff2faa00640 1 -- 172.21.15.146:0/3293844527 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4007740 con 0x7ff2fc0c6cd0 2024-09-16T01:50:59.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.484+0000 7ff2faa00640 1 -- 172.21.15.146:0/3293844527 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f40078e0 con 0x7ff2fc0c6cd0 2024-09-16T01:50:59.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.484+0000 7ff2faa00640 1 -- 172.21.15.146:0/3293844527 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4007cb0 con 0x7ff2fc0c6cd0 2024-09-16T01:50:59.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.485+0000 7ff302e00640 1 -- 172.21.15.146:0/3293844527 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc0c6cd0 msgr2=0x7ff2fc0c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.485+0000 7ff302e00640 1 --2- 172.21.15.146:0/3293844527 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc0c6cd0 0x7ff2fc0c7160 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009d90 tx=0x7ff2f402a550 comp rx=0 tx=0).stop 2024-09-16T01:50:59.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.485+0000 7ff302e00640 1 -- 172.21.15.146:0/3293844527 shutdown_connections 2024-09-16T01:50:59.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.486+0000 7ff302e00640 1 --2- 172.21.15.146:0/3293844527 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc0c6cd0 0x7ff2fc0c7160 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.486+0000 7ff302e00640 1 --2- 172.21.15.146:0/3293844527 >> v2:172.21.15.161:3300/0 conn(0x7ff2fc162a30 0x7ff2fc162e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.486+0000 7ff302e00640 1 -- 172.21.15.146:0/3293844527 >> 172.21.15.146:0/3293844527 conn(0x7ff2fc06bc10 msgr2=0x7ff2fc06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:59.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.486+0000 7ff302e00640 1 -- 172.21.15.146:0/3293844527 shutdown_connections 2024-09-16T01:50:59.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.486+0000 7ff302e00640 1 -- 172.21.15.146:0/3293844527 wait complete. 2024-09-16T01:50:59.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.489+0000 7ff302e00640 1 Processor -- start 2024-09-16T01:50:59.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.489+0000 7ff302e00640 1 -- start start 2024-09-16T01:50:59.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.490+0000 7ff302e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2fc0c6cd0 0x7ff2fc159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.491+0000 7ff302e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc159300 0x7ff2fc15b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.491+0000 7ff302e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2fc162a30 0x7ff2fc15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.491+0000 7ff2fb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc159300 0x7ff2fc15b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.491+0000 7ff2fbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2fc0c6cd0 0x7ff2fc159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.491+0000 7ff2fbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2fc0c6cd0 0x7ff2fc159b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50918/0 (socket says 172.21.15.146:50918) 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.492+0000 7ff300e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2fc162a30 0x7ff2fc15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.492+0000 7ff300e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2fc162a30 0x7ff2fc15b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:48314/0 (socket says 172.21.15.146:48314) 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.492+0000 7ff302e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc15f7f0 con 0x7ff2fc0c6cd0 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.492+0000 7ff302e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc15fdb0 con 0x7ff2fc162a30 2024-09-16T01:50:59.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.492+0000 7ff302e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc15fad0 con 0x7ff2fc159300 2024-09-16T01:50:59.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.492+0000 7ff2fbe00640 1 -- 172.21.15.146:0/1755279818 learned_addr learned my addr 172.21.15.146:0/1755279818 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:50:59.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2fb400640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2fc162a30 msgr2=0x7ff2fc15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff2fc162a30 0x7ff2fc15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2fb400640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2fc0c6cd0 msgr2=0x7ff2fc159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2fc0c6cd0 0x7ff2fc159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2fb400640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc142210 con 0x7ff2fc159300 2024-09-16T01:50:59.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc159300 0x7ff2fc15b3b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e870 tx=0x7ff2e800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.493+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8017070 con 0x7ff2fc159300 2024-09-16T01:50:59.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.494+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e800ee60 con 0x7ff2fc159300 2024-09-16T01:50:59.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.494+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800cb40 con 0x7ff2fc159300 2024-09-16T01:50:59.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.494+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2fc155110 con 0x7ff2fc159300 2024-09-16T01:50:59.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.494+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2fc142be0 con 0x7ff2fc159300 2024-09-16T01:50:59.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.496+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff2fc163190 con 0x7ff2fc159300 2024-09-16T01:50:59.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.498+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff2e801a070 con 0x7ff2fc159300 2024-09-16T01:50:59.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.499+0000 7ff2f3e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff2dc06c2b0 0x7ff2dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.499+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 5799+0+0 (secure 0 0 0) 0x7ff2e808b350 con 0x7ff2fc159300 2024-09-16T01:50:59.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.500+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v58) ==== 54+0+16 (secure 0 0 0) 0x7ff2fc163190 con 0x7ff2fc159300 2024-09-16T01:50:59.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.500+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff2fc156b20 0x7ff2fc141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.500+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff2dc06c2b0 0x7ff2dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.500+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc162510 con 0x7ff2fc156b20 2024-09-16T01:50:59.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.501+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff2dc06c2b0 0x7ff2dc06e770 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec000f60 tx=0x7ff2ec000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.501+0000 7ff300e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff2fc156b20 0x7ff2fc141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.502+0000 7ff300e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff2fc156b20 0x7ff2fc141440 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.503+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc162510 con 0x7ff2fc156b20 2024-09-16T01:50:59.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.534+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc0c7830 con 0x7ff2fc156b20 2024-09-16T01:50:59.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.536+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc0c7830 con 0x7ff2fc156b20 2024-09-16T01:50:59.537 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:50:59.537 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:50:59.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.537+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff2fc156f00 0x7ff2fc13ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.537+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc141e90 con 0x7ff2fc156f00 2024-09-16T01:50:59.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.537+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff2fc156f00 0x7ff2fc13ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.539+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff2fc156f00 0x7ff2fc13ff70 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.540+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc141e90 con 0x7ff2fc156f00 2024-09-16T01:50:59.548 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:59.547+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:59.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.572+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc0ccef0 con 0x7ff2fc156f00 2024-09-16T01:50:59.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.578+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc0ccef0 con 0x7ff2fc156f00 2024-09-16T01:50:59.579 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:50:59.579 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:50:59.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.578+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff2fc062630 0x7ff2fc062a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.578+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc140d80 con 0x7ff2fc062630 2024-09-16T01:50:59.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.581+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff2fc062630 0x7ff2fc062a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.584+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff2fc062630 0x7ff2fc062a70 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.586+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc140d80 con 0x7ff2fc062630 2024-09-16T01:50:59.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.614+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc063280 con 0x7ff2fc062630 2024-09-16T01:50:59.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.616+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc063280 con 0x7ff2fc062630 2024-09-16T01:50:59.618 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:50:59.618 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:50:59.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.617+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff2fc1604c0 0x7ff2fc160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.617+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc160e60 con 0x7ff2fc1604c0 2024-09-16T01:50:59.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.618+0000 7ff300e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff2fc1604c0 0x7ff2fc160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.620+0000 7ff300e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff2fc1604c0 0x7ff2fc160920 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.622+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc160e60 con 0x7ff2fc1604c0 2024-09-16T01:50:59.649 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:50:59.648+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:50:59.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.658+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc02c910 con 0x7ff2fc1604c0 2024-09-16T01:50:59.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.665+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc02c910 con 0x7ff2fc1604c0 2024-09-16T01:50:59.670 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:50:59.671 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:50:59.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.670+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff2fc02d030 0x7ff2fc02d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.670+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc02d9d0 con 0x7ff2fc02d030 2024-09-16T01:50:59.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.670+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff2fc02d030 0x7ff2fc02d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.674 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:59.674 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:50:59.675 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:50:59.675 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:50:59.675 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:50:59.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.675+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff2fc02d030 0x7ff2fc02d490 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.676+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc02d9d0 con 0x7ff2fc02d030 2024-09-16T01:50:59.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:50:59.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.714+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc02d9d0 con 0x7ff2fc02d030 2024-09-16T01:50:59.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.714+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd out", "ids": ["7"]} v 0) -- 0x7f04ec0c68d0 con 0x7f04ec06f3e0 2024-09-16T01:50:59.718 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:50:59.717+0000 7f49bee00640 -1 mon.a@0(leader).osd e58 definitely_dead 0 2024-09-16T01:50:59.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.717+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc02d9d0 con 0x7ff2fc02d030 2024-09-16T01:50:59.718 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:50:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:50:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.718+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff2fc211a40 0x7ff2fc213e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.718+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc216c00 con 0x7ff2fc211a40 2024-09-16T01:50:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.718+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff2fc211a40 0x7ff2fc213e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.718+0000 7ff2fb400640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff2fc211a40 0x7ff2fc213e30 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.719+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc216c00 con 0x7ff2fc211a40 2024-09-16T01:50:59.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.746+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc214370 con 0x7ff2fc211a40 2024-09-16T01:50:59.750 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:50:59.749+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:50:59.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.750+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc214370 con 0x7ff2fc211a40 2024-09-16T01:50:59.751 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:50:59.751 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:50:59.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.750+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff2fc2149f0 0x7ff2fc216de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.750+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc219bb0 con 0x7ff2fc2149f0 2024-09-16T01:50:59.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.751+0000 7ff300e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff2fc2149f0 0x7ff2fc216de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.752+0000 7ff300e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff2fc2149f0 0x7ff2fc216de0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.754+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc219bb0 con 0x7ff2fc2149f0 2024-09-16T01:50:59.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.798+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc02db40 con 0x7ff2fc2149f0 2024-09-16T01:50:59.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.800+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc02db40 con 0x7ff2fc2149f0 2024-09-16T01:50:59.801 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:50:59.801 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:50:59.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.800+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff2fc217830 0x7ff2fc219c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:50:59.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.800+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff2fc21cbb0 con 0x7ff2fc217830 2024-09-16T01:50:59.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.800+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff2fc217830 0x7ff2fc219c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:50:59.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.802+0000 7ff2fbe00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff2fc217830 0x7ff2fc219c20 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:50:59.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.802+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2fc21cbb0 con 0x7ff2fc217830 2024-09-16T01:50:59.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.807+0000 7f04e8c00640 1 -- 172.21.15.146:0/1604551706 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["7"]}]=0 marked out osd.7. v59) ==== 87+0+0 (secure 0 0 0) 0x7f04e4003e20 con 0x7f04ec06f3e0 2024-09-16T01:50:59.809 INFO:teuthology.orchestra.run.smithi146.stderr:marked out osd.7. 2024-09-16T01:50:59.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.816+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f04cc06c3b0 msgr2=0x7f04cc06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.816+0000 7f04f2600640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f04cc06c3b0 0x7f04cc06e870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04e000d940 tx=0x7f04e0008040 comp rx=0 tx=0).stop 2024-09-16T01:50:59.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.816+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 msgr2=0x7f04ec06f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.816+0000 7f04f2600640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 0x7f04ec06f890 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0d1350 tx=0x7f04e400ff70 comp rx=0 tx=0).stop 2024-09-16T01:50:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.819+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 shutdown_connections 2024-09-16T01:50:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.819+0000 7f04f2600640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f04cc06c3b0 0x7f04cc06e870 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.819+0000 7f04f2600640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04ec06f3e0 0x7f04ec06f890 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.819+0000 7f04f2600640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04ec0c7b10 0x7f04ec14ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.819+0000 7f04f2600640 1 --2- 172.21.15.146:0/1604551706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04ec0c7730 0x7f04ec14e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.819+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 >> 172.21.15.146:0/1604551706 conn(0x7f04ec067b30 msgr2=0x7f04ec0cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:59.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.824+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 shutdown_connections 2024-09-16T01:50:59.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.825+0000 7f04f2600640 1 -- 172.21.15.146:0/1604551706 wait complete. 2024-09-16T01:50:59.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.831+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff2fc21b8f0 con 0x7ff2fc217830 2024-09-16T01:50:59.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.838+0000 7ff2f3e00640 1 -- 172.21.15.146:0/1755279818 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2fc21b8f0 con 0x7ff2fc217830 2024-09-16T01:50:59.841 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:50:59.841 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff2fc156b20 msgr2=0x7ff2fc141440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff2fc156b20 0x7ff2fc141440 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff2fc156f00 msgr2=0x7ff2fc13ff70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff2fc156f00 0x7ff2fc13ff70 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff2fc062630 msgr2=0x7ff2fc062a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff2fc062630 0x7ff2fc062a70 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff2fc1604c0 msgr2=0x7ff2fc160920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff2fc1604c0 0x7ff2fc160920 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff2fc02d030 msgr2=0x7ff2fc02d490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff2fc02d030 0x7ff2fc02d490 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff2fc211a40 msgr2=0x7ff2fc213e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff2fc211a40 0x7ff2fc213e30 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff2fc2149f0 msgr2=0x7ff2fc216de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.840+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff2fc2149f0 0x7ff2fc216de0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.841+0000 7ff300e00640 1 -- 172.21.15.146:0/1755279818 reap_dead start 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.841+0000 7ff300e00640 1 -- 172.21.15.146:0/1755279818 reap_dead start 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.841+0000 7ff300e00640 1 -- 172.21.15.146:0/1755279818 reap_dead start 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.841+0000 7ff300e00640 1 -- 172.21.15.146:0/1755279818 reap_dead start 2024-09-16T01:50:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.841+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff2fc217830 msgr2=0x7ff2fc219c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.841+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff2fc217830 0x7ff2fc219c20 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff2dc06c2b0 msgr2=0x7ff2dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff2dc06c2b0 0x7ff2dc06e770 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec000f60 tx=0x7ff2ec000ed0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc159300 msgr2=0x7ff2fc15b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:50:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc159300 0x7ff2fc15b3b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800e870 tx=0x7ff2e800ed30 comp rx=0 tx=0).stop 2024-09-16T01:50:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 shutdown_connections 2024-09-16T01:50:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff2fc217830 0x7ff2fc219c20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff2dc06c2b0 0x7ff2dc06e770 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 --2- 172.21.15.146:0/1755279818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff2fc159300 0x7ff2fc15b3b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:50:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.842+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 >> 172.21.15.146:0/1755279818 conn(0x7ff2fc06bc10 msgr2=0x7ff2fc06f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:50:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.843+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 shutdown_connections 2024-09-16T01:50:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:50:59.843+0000 7ff302e00640 1 -- 172.21.15.146:0/1755279818 wait complete. 2024-09-16T01:50:59.850 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:50:59.849+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:50:59.952 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:50:59.951+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:50:59.985 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:50:59.985 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 132.21436044999999, 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:50:59.986 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:59.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:59.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:50:59.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:59.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:59.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:7 3.2a 3:557f550e:::10000000002.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7240201160000002, 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081303988999999993, 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 7 2024-09-16T01:50:59.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:50:59.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258696+0000", 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7320000000000001e-06 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305572+0000", 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046875737000000001 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305609+0000", 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7704999999999999e-05 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305623+0000", 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310196+0000", 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045734269999999997 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310344+0000", 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147181 2024-09-16T01:50:59.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310523+0000", 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017908400000000001 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.327195+0000", 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016672724999999999 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339893+0000", 2024-09-16T01:50:59.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012697559000000001 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339953+0000", 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0084999999999998e-05 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339998+0000", 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5218e-05 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:50:59.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:20 3.21 3:85380590:::10000000002.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5710130580000001, 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055783419000000001, 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:50:59.999 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 20 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411703+0000", 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.84e-06 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441653+0000", 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029949640999999999 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441685+0000", 2024-09-16T01:51:00.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2452e-05 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441702+0000", 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441798+0000", 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6501000000000003e-05 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441964+0000", 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165635 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.442314+0000", 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035073800000000002 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.455571+0000", 2024-09-16T01:51:00.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013256273000000001 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467368+0000", 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011797656 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467432+0000", 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3585999999999997e-05 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467485+0000", 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2729999999999998e-05 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:22 3.10 3:0a7b425d:::10000000002.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.540974748, 2024-09-16T01:51:00.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050802171, 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 22 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441741+0000", 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.641e-06 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459091+0000", 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017350175999999998 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459122+0000", 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1041000000000003e-05 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459136+0000", 2024-09-16T01:51:00.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.384e-05 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461076+0000", 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001939213 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461219+0000", 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143101 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461360+0000", 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014149900000000001 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492196+0000", 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030836319000000001 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492406+0000", 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020962800000000001 2024-09-16T01:51:00.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492461+0000", 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5251999999999999e-05 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492542+0000", 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0461000000000001e-05 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:23 3.21 3:87fa6b7a:::10000000002.00000014:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5235345630000001, 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049069672000000002, 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 23 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:51:00.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459181+0000", 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.356e-06 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499609+0000", 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040428104999999999 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499638+0000", 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8640999999999999e-05 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499650+0000", 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1989e-05 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499729+0000", 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869000000000007e-05 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499916+0000", 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018738500000000001 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.500068+0000", 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015149099999999999 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.506758+0000", 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066901900000000004 2024-09-16T01:51:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508128+0000", 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013700330000000001 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508197+0000", 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9077000000000003e-05 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508249+0000", 2024-09-16T01:51:00.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2536e-05 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:43 3.1a 3:5917aad8:::10000000002.00000028:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3820695530000009, 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041950973000000003, 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.018 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 43 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600647+0000", 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.136e-06 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623155+0000", 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022507547999999999 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623186+0000", 2024-09-16T01:51:00.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1078000000000003e-05 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623196+0000", 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0825999999999999e-05 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624399+0000", 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012021969999999999 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624531+0000", 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013200899999999999 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624681+0000", 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015020999999999999 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.639604+0000", 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014922962 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642466+0000", 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002862028 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642534+0000", 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8503999999999993e-05 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642596+0000", 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1475000000000003e-05 2024-09-16T01:51:00.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:44 3.19 3:9961cbc1:::10000000002.00000029:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3590203370000005, 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044190814000000002, 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.024 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 44 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623696+0000", 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000002e-06 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660636+0000", 2024-09-16T01:51:00.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036939840000000002 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660662+0000", 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5891e-05 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660672+0000", 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0362e-05 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660758+0000", 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5834999999999994e-05 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660916+0000", 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158572 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.661072+0000", 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015595600000000001 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667770+0000", 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066975170000000001 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667785+0000", 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4882000000000001e-05 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667837+0000", 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2343999999999999e-05 2024-09-16T01:51:00.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667885+0000", 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7679000000000002e-05 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:59 3.1c 3:394f8a26:::10000000002.00000038:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.1899787489999998, 2024-09-16T01:51:00.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051871274000000002, 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 59 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792738+0000", 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1100000000000001e-06 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824239+0000", 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031501159000000001 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824261+0000", 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2515e-05 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824272+0000", 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0696999999999999e-05 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824397+0000", 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012514399999999999 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824552+0000", 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015457599999999999 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824715+0000", 2024-09-16T01:51:00.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163657 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.832638+0000", 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079224959999999994 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844470+0000", 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011831734999999999 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844542+0000", 2024-09-16T01:51:00.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2194999999999996e-05 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844607+0000", 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4989999999999999e-05 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:91 3.2a 3:5747858d:::10000000002.00000058:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.9221254600000002, 2024-09-16T01:51:00.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045690485000000003, 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 91 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060591+0000", 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1009999999999999e-06 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094735+0000", 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034143896999999999 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094758+0000", 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3485999999999998e-05 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094769+0000", 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0844e-05 2024-09-16T01:51:00.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094852+0000", 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2627999999999996e-05 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095011+0000", 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015938000000000001 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095160+0000", 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014846799999999999 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.105436+0000", 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010275986000000001 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106140+0000", 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000703905 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106219+0000", 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9672999999999994e-05 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106279+0000", 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0117000000000001e-05 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:92 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.042 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.8879085670000002, 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040038826999999999, 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 92 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.043 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094807+0000", 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1319999999999999e-06 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126953+0000", 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032146474000000001 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126978+0000", 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4737e-05 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126992+0000", 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3543000000000001e-05 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127074+0000", 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1889000000000002e-05 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127230+0000", 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156238 2024-09-16T01:51:00.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127387+0000", 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157207 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.131783+0000", 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004395805 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134755+0000", 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029720520000000002 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134795+0000", 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9909e-05 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134845+0000", 2024-09-16T01:51:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9840999999999999e-05 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:97 3.1a 3:5b190c62:::10000000002.0000005e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.8554007109999997, 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040400441000000002, 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.049 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 97 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:51:00.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127315+0000", 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.626e-06 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156590+0000", 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029275042000000001 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156619+0000", 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8200000000000001e-05 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156631+0000", 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.252e-05 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156742+0000", 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011125599999999999 2024-09-16T01:51:00.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156905+0000", 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016292499999999999 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.157055+0000", 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014968699999999999 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.162589+0000", 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055339530000000003 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167534+0000", 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049449699999999999 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167659+0000", 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125575 2024-09-16T01:51:00.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167714+0000", 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4687e-05 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:124 3.21 3:844ac05a:::10000000002.00000079:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5830275520000008, 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052276376999999999, 2024-09-16T01:51:00.055 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 124 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399689+0000", 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5100000000000001e-06 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443104+0000", 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043414547999999997 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443122+0000", 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8071e-05 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443132+0000", 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0492999999999999e-05 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443200+0000", 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7248000000000004e-05 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443353+0000", 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015338799999999999 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443504+0000", 2024-09-16T01:51:00.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150681 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447533+0000", 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040287889999999996 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451892+0000", 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043593520000000004 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451932+0000", 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0104000000000001e-05 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451963+0000", 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1192999999999998e-05 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.061 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:139 3.1c 3:384ad1d9:::10000000002.00000088:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.4420124340000005, 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061931110999999997, 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540704+0000", 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.221e-06 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586535+0000", 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045830901 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586563+0000", 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8388e-05 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586591+0000", 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7424999999999999e-05 2024-09-16T01:51:00.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586694+0000", 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103269 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586849+0000", 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154857 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.587008+0000", 2024-09-16T01:51:00.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015895699999999999 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591909+0000", 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049009609999999997 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602502+0000", 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010592773 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602564+0000", 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2154000000000002e-05 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602633+0000", 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9204999999999999e-05 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:148 3.1c 3:3afba7d1:::10000000002.00000091:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.067 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3584402709999992, 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047360674999999998, 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 148 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624276+0000", 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1639999999999999e-06 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653877+0000", 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029600346 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653900+0000", 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3399000000000001e-05 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653912+0000", 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2232000000000001e-05 2024-09-16T01:51:00.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654079+0000", 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016647999999999999 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654232+0000", 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015308100000000001 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654384+0000", 2024-09-16T01:51:00.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152573 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.662170+0000", 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077856990000000001 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671503+0000", 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093331279999999996 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671572+0000", 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8417999999999994e-05 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671635+0000", 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3155000000000001e-05 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:191 3.1a 3:5ab92411:::10000000002.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.073 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.9460491080000004, 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045215946999999999, 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 191 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036668+0000", 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4880000000000001e-06 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066884+0000", 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030216601999999999 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066908+0000", 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3555999999999999e-05 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066918+0000", 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0443e-05 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069411+0000", 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024928149999999998 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069698+0000", 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287025 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069886+0000", 2024-09-16T01:51:00.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018816899999999999 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.075677+0000", 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057910540000000003 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081769+0000", 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060919659999999999 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081834+0000", 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4797999999999999e-05 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081881+0000", 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7030999999999997e-05 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:441 3.1c 3:3913eadd:::10000000002.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.374032382, 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072104032999999998, 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 441 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608684+0000", 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4959999999999999e-06 2024-09-16T01:51:00.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650015+0000", 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041330193000000001 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650045+0000", 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0335e-05 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650057+0000", 2024-09-16T01:51:00.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650158+0000", 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100837 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650266+0000", 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107975 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650426+0000", 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016051800000000001 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.663669+0000", 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013242594 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680672+0000", 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017003041999999999 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680737+0000", 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4750000000000002e-05 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680786+0000", 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9217e-05 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:442 3.27 e58/20 3:e61532d3:::10000000002.000000b7:head v 58'3, mlcod=55'1)", 2024-09-16T01:51:00.085 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.651840+0000", 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.33087477199999998, 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039355339000000003, 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.651840+0000", 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.651840+0000", 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.651841+0000", 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.409e-06 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.670498+0000", 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018656810999999999 2024-09-16T01:51:00.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.670535+0000", 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6757000000000001e-05 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.670549+0000", 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4778e-05 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.671663+0000", 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011134389999999999 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.671689+0000", 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5907000000000001e-05 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691116+0000", 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019427120999999999 2024-09-16T01:51:00.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691195+0000", 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9116999999999995e-05 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:447 3.1a 3:5ab92411:::10000000002.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.664227+0000", 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.31848779900000002, 2024-09-16T01:51:00.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043974170999999999, 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 447 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.664227+0000", 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.664227+0000", 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.664229+0000", 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9230000000000001e-06 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680449+0000", 2024-09-16T01:51:00.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016220535000000001 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680479+0000", 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9935000000000001e-05 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680493+0000", 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3539e-05 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680615+0000", 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122423 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680713+0000", 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7816e-05 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680856+0000", 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142917 2024-09-16T01:51:00.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.701876+0000", 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021019880000000001 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.708073+0000", 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061970219999999999 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.708147+0000", 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4072000000000005e-05 2024-09-16T01:51:00.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.708201+0000", 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4109e-05 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:454 3.37 3:ec162ecd:::10000000002.000000c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.732176+0000", 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.25053872500000002, 2024-09-16T01:51:00.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050739933000000001, 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 454 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.732176+0000", 2024-09-16T01:51:00.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.732176+0000", 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.732178+0000", 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4219999999999999e-06 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.761814+0000", 2024-09-16T01:51:00.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029635946 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.761961+0000", 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147083 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.762014+0000", 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2902999999999998e-05 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.765825+0000", 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038106080000000001 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.766230+0000", 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040490300000000002 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.766806+0000", 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00057623199999999998 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.776801+0000", 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099953569999999999 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.782797+0000", 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059956180000000003 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.782865+0000", 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8300000000000007e-05 2024-09-16T01:51:00.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.782916+0000", 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0560999999999999e-05 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:461 3.19 3:9a784cd8:::10000000002.000000ca:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.835939+0000", 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.14677580200000001, 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046963036, 2024-09-16T01:51:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 461 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.835939+0000", 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.835939+0000", 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.835941+0000", 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.526e-06 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.868859+0000", 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032917517 2024-09-16T01:51:00.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.868890+0000", 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0944000000000001e-05 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.868903+0000", 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3004e-05 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.869041+0000", 2024-09-16T01:51:00.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013840400000000001 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.869137+0000", 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6471000000000005e-05 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.869278+0000", 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014032000000000001 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.874951+0000", 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056731580000000002 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.882790+0000", 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078386399999999992 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.882853+0000", 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-05 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.882902+0000", 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9101999999999998e-05 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.108 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:00.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:00.110 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:00.108+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:00.210 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:00.209+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:00.315 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:00.314+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:00.385 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:00.386 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:00.386 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:00.386 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:00.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:00.411 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:00.410+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:00.512 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:00.512+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:00.601 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:00.601 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:00.601 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:00.601 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:00.601 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:00.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:00.614 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:00.612+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:00.714 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:00.713+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:00.815 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:00.814+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.564791469999999, 2024-09-16T01:51:00.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:51:00.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:51:00.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.251845695, 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:00.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:00.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:42 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.210763226999999, 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051426843, 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.837 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 42 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608928+0000", 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5689999999999999e-06 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648858+0000", 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039929485000000001 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648879+0000", 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1172e-05 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648889+0000", 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0512e-05 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648975+0000", 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5473e-05 2024-09-16T01:51:00.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649103+0000", 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012862999999999999 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649230+0000", 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126561 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.655882+0000", 2024-09-16T01:51:00.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066518819999999996 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660228+0000", 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043460699999999996 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660293+0000", 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5050999999999998e-05 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660353+0000", 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0438e-05 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.098300012999999, 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:00.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:00.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.849 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:67 3.16 3:6a386151:::10000000002.00000040:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.9837821810000005, 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046477360000000002, 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 67 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835910+0000", 2024-09-16T01:51:00.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.872997+0000", 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03708761 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873029+0000", 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1911e-05 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873046+0000", 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6702999999999999e-05 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873132+0000", 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6034999999999999e-05 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873288+0000", 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015656000000000001 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873419+0000", 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130381 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.880727+0000", 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073081919999999998 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882276+0000", 2024-09-16T01:51:00.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015495330000000001 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882336+0000", 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9964999999999999e-05 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882385+0000", 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8588999999999998e-05 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:77 3.3c 3:3db0268f:::10000000002.0000004a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.9099910270000002, 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051390551, 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 77 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909702+0000", 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9699999999999999e-06 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927863+0000", 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018160824999999998 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927887+0000", 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4802000000000001e-05 2024-09-16T01:51:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927900+0000", 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928016+0000", 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116123 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928158+0000", 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141977 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928294+0000", 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135733 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.938330+0000", 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010036092999999999 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960987+0000", 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022656493 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961043+0000", 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6416999999999999e-05 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961089+0000", 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6152000000000001e-05 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8774403329999991, 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.862 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:00.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7993097440000003, 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.868 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:00.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:00.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:85 3.36 e55/20 3:6fe880e3:::10000000002.00000052:head v 55'2, mlcod=0'0)", 2024-09-16T01:51:00.874 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7769234179999991, 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052325050999999997, 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042769+0000", 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3520000000000001e-06 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051520+0000", 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087513980000000005 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051666+0000", 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014630900000000001 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051711+0000", 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4768999999999998e-05 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052370+0000", 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065890100000000004 2024-09-16T01:51:00.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052478+0000", 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010805 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095028+0000", 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042550194 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095091+0000", 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3077999999999995e-05 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:98 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.6810825400000002, 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052298391, 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 98 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138610+0000", 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166283+0000", 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027672980999999999 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166310+0000", 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7712000000000002e-05 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166324+0000", 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3239e-05 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166414+0000", 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0062000000000006e-05 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166567+0000", 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015352700000000001 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166726+0000", 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015874500000000001 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.187851+0000", 2024-09-16T01:51:00.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021125117999999998 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190785+0000", 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029334970000000002 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190850+0000", 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5560000000000002e-05 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190906+0000", 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5451000000000002e-05 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:140 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2643346149999992, 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050007939000000001, 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555356+0000", 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1039999999999999e-06 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591189+0000", 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035832756 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591217+0000", 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8263e-05 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591230+0000", 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591354+0000", 2024-09-16T01:51:00.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012385200000000001 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591571+0000", 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217099 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591739+0000", 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016809299999999999 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.598744+0000", 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070049939999999996 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605250+0000", 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065054079999999999 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605310+0000", 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0263000000000002e-05 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605363+0000", 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3140999999999998e-05 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2270968579999995, 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.891 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:00.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.0029403240000008, 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.897 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:00.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:00.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:181 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.9229606029999999, 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046020995000000002, 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.903 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 181 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896731+0000", 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-06 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931107+0000", 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034375920999999997 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931130+0000", 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3668000000000001e-05 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931141+0000", 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0366999999999999e-05 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931281+0000", 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140622 2024-09-16T01:51:00.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931432+0000", 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150469 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931599+0000", 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167338 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.940280+0000", 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086811649999999994 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942632+0000", 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002352224 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942694+0000", 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2014999999999995e-05 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942750+0000", 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5763999999999997e-05 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7056633009999995, 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.909 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:00.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.915 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.232855448, 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:00.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:00.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.921 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.0450288620000001, 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.922 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:00.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:00.927 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:463 3.26 3:65689812:::10000000002.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.95385556599999999, 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112035999999998, 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.928 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 463 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865836+0000", 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9719999999999999e-06 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883238+0000", 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017402081999999999 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883259+0000", 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1143999999999999e-05 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883269+0000", 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-06 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883356+0000", 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7114999999999995e-05 2024-09-16T01:51:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883467+0000", 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110513 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883617+0000", 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150216 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.906097+0000", 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022480257 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914840+0000", 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087431309999999995 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914898+0000", 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7643999999999999e-05 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914946+0000", 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8161999999999997e-05 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:484 3.e 3:723b89cc:::10000000002.000000e1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.71198609700000004, 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054305622999999997, 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 484 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107706+0000", 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145546+0000", 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037839795000000002 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145584+0000", 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8556000000000002e-05 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145598+0000", 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4238e-05 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145708+0000", 2024-09-16T01:51:00.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109584 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145834+0000", 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012588200000000001 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145990+0000", 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015650199999999999 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.160245+0000", 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014254119000000001 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161887+0000", 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016424009999999999 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161956+0000", 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8996000000000001e-05 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.162009+0000", 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3381999999999998e-05 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.53621245699999998, 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:00.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:00.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:00.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:00.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:00.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:00.946 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:00.947 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:51:00.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 rewriters = 273369.72 kB/sec 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 rewriters = 273315.35 kB/sec 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 273369.72 kB/sec 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 273369.72 kB/sec 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 273369.72 kB/sec 2024-09-16T01:51:00.949 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:00.950 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:00.947+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:01.049 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:01.048+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:01.149 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:01.148+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:01.206 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:01.206 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:01.206 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:01.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.187+0000 7f5fb2600640 1 Processor -- start 2024-09-16T01:51:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.187+0000 7f5fb2600640 1 -- start start 2024-09-16T01:51:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.188+0000 7f5fb2600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5fac168200 0x7f5fac0c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:01.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.188+0000 7f5fb2600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5fac0c7430 0x7f5fac0c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.188+0000 7f5fb2600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fac16ad60 con 0x7f5fac164d70 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.188+0000 7f5fb2600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5fac16b040 con 0x7f5fac168200 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.188+0000 7f5fb2600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5fac16b320 con 0x7f5fac0c7430 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.188+0000 7f5faaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5fac168200 0x7f5fac0c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.189+0000 7f5fab400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5fac164d70 0x7f5fac167af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43354/0 (socket says 172.21.15.146:43354) 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.189+0000 7f5fab400640 1 -- 172.21.15.146:0/1264183880 learned_addr learned my addr 172.21.15.146:0/1264183880 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.189+0000 7f5faa000640 1 -- 172.21.15.146:0/1264183880 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233116502 0 0) 0x7f5fac16ad60 con 0x7f5fac164d70 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.189+0000 7f5fabe00640 1 --2- 172.21.15.146:0/1264183880 >> v2:172.21.15.161:3300/0 conn(0x7f5fac0c7430 0x7f5fac0c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.230+0000 7f5faa000640 1 -- 172.21.15.146:0/1264183880 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f90003500 con 0x7f5fac164d70 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.230+0000 7f5faaa00640 1 -- 172.21.15.146:0/1264183880 >> v2:172.21.15.161:3300/0 conn(0x7f5fac0c7430 msgr2=0x7f5fac0c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.230+0000 7f5faaa00640 1 --2- 172.21.15.146:0/1264183880 >> v2:172.21.15.161:3300/0 conn(0x7f5fac0c7430 0x7f5fac0c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.230+0000 7f5faaa00640 1 -- 172.21.15.146:0/1264183880 >> v1:172.21.15.146:6789/0 conn(0x7f5fac164d70 legacy=0x7f5fac167af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:01.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.230+0000 7f5faaa00640 1 -- 172.21.15.146:0/1264183880 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fac16b600 con 0x7f5fac168200 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.231+0000 7f5faaa00640 1 --2- 172.21.15.146:0/1264183880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac0c8ad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5fac071e00 tx=0x7f5f9802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6462f8bdaa14b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.231+0000 7f5faa000640 1 -- 172.21.15.146:0/1264183880 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98033470 con 0x7f5fac168200 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.231+0000 7f5faa000640 1 -- 172.21.15.146:0/1264183880 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9802e070 con 0x7f5fac168200 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.231+0000 7f5faa000640 1 -- 172.21.15.146:0/1264183880 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98033810 con 0x7f5fac168200 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 -- 172.21.15.146:0/1264183880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 msgr2=0x7f5fac0c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 --2- 172.21.15.146:0/1264183880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac0c8ad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5fac071e00 tx=0x7f5f9802a1c0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 -- 172.21.15.146:0/1264183880 shutdown_connections 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 --2- 172.21.15.146:0/1264183880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac0c8ad0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 --2- 172.21.15.146:0/1264183880 >> v2:172.21.15.161:3300/0 conn(0x7f5fac0c7430 0x7f5fac0c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 -- 172.21.15.146:0/1264183880 >> 172.21.15.146:0/1264183880 conn(0x7f5fac06f3c0 msgr2=0x7f5fac06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:01.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.232+0000 7f5fb2600640 1 -- 172.21.15.146:0/1264183880 shutdown_connections 2024-09-16T01:51:01.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.233+0000 7f5fb2600640 1 -- 172.21.15.146:0/1264183880 wait complete. 2024-09-16T01:51:01.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.234+0000 7f5fb2600640 1 Processor -- start 2024-09-16T01:51:01.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.234+0000 7f5fb2600640 1 -- start start 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fb2600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5fac0c7430 0x7f5fac169980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fb2600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fac164d70 0x7f5fac16a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fb2600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac1fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fb2600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5fac0c6bb0 con 0x7f5fac168200 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fb2600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5fac0c3970 con 0x7f5fac164d70 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fb2600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5fac0718c0 con 0x7f5fac0c7430 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac1fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac1fd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50932/0 (socket says 172.21.15.146:50932) 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5faaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fac164d70 0x7f5fac16a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fabe00640 1 -- 172.21.15.146:0/4015485019 learned_addr learned my addr 172.21.15.146:0/4015485019 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.235+0000 7f5fab400640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5fac0c7430 0x7f5fac169980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:01.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5faaa00640 1 -- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5fac0c7430 msgr2=0x7f5fac169980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:01.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5faaa00640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5fac0c7430 0x7f5fac169980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5faaa00640 1 -- 172.21.15.146:0/4015485019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 msgr2=0x7f5fac1fd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:01.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5faaa00640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac1fd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5faaa00640 1 -- 172.21.15.146:0/4015485019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fac0c4980 con 0x7f5fac164d70 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5faaa00640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fac164d70 0x7f5fac16a460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5f9802f000 tx=0x7f5f98004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98043070 con 0x7f5fac164d70 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9800c040 con 0x7f5fac164d70 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fac0c4c60 con 0x7f5fac164d70 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.236+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98004430 con 0x7f5fac164d70 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.237+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fac0c5220 con 0x7f5fac164d70 2024-09-16T01:51:01.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.238+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fac165150 con 0x7f5fac164d70 2024-09-16T01:51:01.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.239+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5f98046070 con 0x7f5fac164d70 2024-09-16T01:51:01.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.240+0000 7f5fa3400640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f8c06c2b0 0x7f5f8c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:01.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.240+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5741+0+0 (secure 0 0 0) 0x7f5f980ab260 con 0x7f5fac164d70 2024-09-16T01:51:01.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.240+0000 7f5fab400640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f8c06c2b0 0x7f5f8c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:01.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.241+0000 7f5fab400640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f8c06c2b0 0x7f5f8c06e770 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c001200 tx=0x7f5f9c000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:01.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.245+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5f98079360 con 0x7f5fac164d70 2024-09-16T01:51:01.250 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:01.249+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:01.351 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:01.350+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:01.384 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:01.384 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:01.384 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:01.384 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:01.384 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:01.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:01.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.445+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f5fac168690 con 0x7f5fac164d70 2024-09-16T01:51:01.452 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:01.451+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:01.554 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:01.553+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:01.599 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:01.599 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:01.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:01.599 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.341133391, 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.600 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:01.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:14 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.243813079000001, 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051400593000000001, 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 14 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:01.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352602+0000", 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393382+0000", 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040779871000000002 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393422+0000", 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9765000000000003e-05 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393436+0000", 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3633e-05 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393509+0000", 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3847e-05 2024-09-16T01:51:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393696+0000", 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186911 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393839+0000", 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014274700000000001 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.401310+0000", 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074705040000000002 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403876+0000", 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025662020000000001 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403939+0000", 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3423999999999994e-05 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.404001+0000", 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1649000000000006e-05 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.142760521, 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.03395789, 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:66 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.774728394, 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053741734999999999, 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821687+0000", 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3750000000000001e-06 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862658+0000", 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040970816 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862692+0000", 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3420000000000002e-05 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862706+0000", 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4017e-05 2024-09-16T01:51:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862879+0000", 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173015 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863040+0000", 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016103800000000001 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863192+0000", 2024-09-16T01:51:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152738 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.867722+0000", 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045298689999999997 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875297+0000", 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075744860000000001 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875372+0000", 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5049999999999995e-05 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875427+0000", 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4911000000000003e-05 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.714389257000001, 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:75 3.30 3:0d2dbd17:::10000000002.00000048:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.650307829999999, 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050908518999999999, 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946107+0000", 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7710000000000001e-06 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960806+0000", 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014698386000000001 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960821+0000", 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960832+0000", 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0698e-05 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960964+0000", 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013269700000000001 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961122+0000", 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157765 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961267+0000", 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014486100000000001 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981986+0000", 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020719158000000001 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996886+0000", 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014899654999999999 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996958+0000", 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2095999999999995e-05 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997014+0000", 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6258999999999998e-05 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:110 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.360051543000001, 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053794485000000003, 2024-09-16T01:51:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 110 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236365+0000", 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1590000000000002e-06 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278463+0000", 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042097744999999999 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278494+0000", 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1009000000000001e-05 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278508+0000", 2024-09-16T01:51:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3901e-05 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278639+0000", 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013188100000000001 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278806+0000", 2024-09-16T01:51:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016689799999999999 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278974+0000", 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016719799999999999 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.286538+0000", 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075647910000000004 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290050+0000", 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035116969999999998 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290114+0000", 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4298e-05 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290156+0000", 2024-09-16T01:51:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1907999999999999e-05 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.215726517, 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:160 3.9 3:93988fc9:::10000000002.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8467364400000008, 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056552534000000002, 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 160 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749679+0000", 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.956e-06 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794169+0000", 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04449061 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794202+0000", 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2626999999999998e-05 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794217+0000", 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794305+0000", 2024-09-16T01:51:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8158999999999994e-05 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794460+0000", 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155277 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794625+0000", 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164854 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.801888+0000", 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072630239999999999 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806094+0000", 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042063919999999998 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806178+0000", 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3287000000000004e-05 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806229+0000", 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765000000000003e-05 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8021538330000002, 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:164 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7742818160000002, 2024-09-16T01:51:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053828972000000003, 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 164 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822134+0000", 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841247+0000", 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019113208 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841279+0000", 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2265999999999999e-05 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841291+0000", 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2305e-05 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841376+0000", 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4507999999999998e-05 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841535+0000", 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159216 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841687+0000", 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151846 2024-09-16T01:51:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859703+0000", 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018016197000000001 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875876+0000", 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016172628000000001 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875929+0000", 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3108e-05 2024-09-16T01:51:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875960+0000", 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1491000000000001e-05 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:197 3.1d 3:bb6dfbbf:::10000000002.000000c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.4892747590000006, 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057555969999999998, 2024-09-16T01:51:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 197 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107141+0000", 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148704+0000", 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041563044 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148738+0000", 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4480999999999997e-05 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148754+0000", 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6021000000000001e-05 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148909+0000", 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154735 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149071+0000", 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016181600000000001 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149226+0000", 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015550000000000001 2024-09-16T01:51:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.158210+0000", 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089836529999999994 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164352+0000", 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061416810000000004 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164426+0000", 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4192999999999999e-05 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164695+0000", 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026864800000000002 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:449 3.28 3:144fb21c:::10000000002.000000be:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.9053010260000001, 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05217542, 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 449 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691115+0000", 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.056e-06 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726045+0000", 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034929984999999997 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726077+0000", 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1390000000000003e-05 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726093+0000", 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6047000000000001e-05 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726207+0000", 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011452599999999999 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726335+0000", 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012742299999999999 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726492+0000", 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015772599999999999 2024-09-16T01:51:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.740613+0000", 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014120071999999999 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743157+0000", 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025441550000000002 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743229+0000", 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2102999999999997e-05 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743288+0000", 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8937000000000002e-05 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.815780741, 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.7476761000000001, 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.605216435, 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:497 3.30 3:0c580b6a:::10000000002.000000ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.404741024, 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052947199, 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 497 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191674+0000", 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.575e-06 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220734+0000", 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029059623999999999 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220763+0000", 2024-09-16T01:51:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9099e-05 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220777+0000", 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4299e-05 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220897+0000", 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011993999999999999 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.221085+0000", 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000188457 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.221327+0000", 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024153299999999999 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.233648+0000", 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012321544 2024-09-16T01:51:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244466+0000", 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010817795 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244519+0000", 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3316999999999998e-05 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244620+0000", 2024-09-16T01:51:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100016 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:499 3.2f 3:f78980a5:::10000000002.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.373513185, 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055062123999999997, 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 499 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222902+0000", 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261223+0000", 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038321506999999998 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261257+0000", 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3526999999999999e-05 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261272+0000", 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4748e-05 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261399+0000", 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012685100000000001 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261500+0000", 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101556 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261663+0000", 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016259900000000001 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.270356+0000", 2024-09-16T01:51:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086932399999999996 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277838+0000", 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074817470000000004 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277906+0000", 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8454000000000006e-05 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277962+0000", 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6075000000000001e-05 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2577502119999999, 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:01.729 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:01.730 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:01.731 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:01.729+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:01.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.813+0000 7f5fa3400640 1 -- 172.21.15.146:0/4015485019 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v61) ==== 101+0+0 (secure 0 0 0) 0x7f5f9807d660 con 0x7f5fac164d70 2024-09-16T01:51:01.815 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f8c06c2b0 msgr2=0x7f5f8c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f8c06c2b0 0x7f5f8c06e770 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c001200 tx=0x7f5f9c000bc0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fac164d70 msgr2=0x7f5fac16a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fac164d70 0x7f5fac16a460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5f9802f000 tx=0x7f5f98004730 comp rx=0 tx=0).stop 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 shutdown_connections 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f8c06c2b0 0x7f5f8c06e770 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fac168200 0x7f5fac1fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5fac164d70 0x7f5fac16a460 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 --2- 172.21.15.146:0/4015485019 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5fac0c7430 0x7f5fac169980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:01.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 >> 172.21.15.146:0/4015485019 conn(0x7f5fac06f3c0 msgr2=0x7f5fac071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:01.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 shutdown_connections 2024-09-16T01:51:01.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:01.816+0000 7f5fb2600640 1 -- 172.21.15.146:0/4015485019 wait complete. 2024-09-16T01:51:01.831 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:01.830+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:01.839 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:01.840 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:01.849 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:01.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:01.933 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:01.932+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:01.993 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:01.993 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:01.993 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:01.993 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:01.993 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:02.002 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:02.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.024+0000 7ff7a5e00640 1 Processor -- start 2024-09-16T01:51:02.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.024+0000 7ff7a5e00640 1 -- start start 2024-09-16T01:51:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.025+0000 7ff7a5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7a00c36b0 0x7ff7a00c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.025+0000 7ff7a5e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a00c4240 con 0x7ff7a0152c30 2024-09-16T01:51:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.025+0000 7ff7a5e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a00c4500 con 0x7ff7a00c9180 2024-09-16T01:51:02.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.025+0000 7ff7a5e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff7a00c71c0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff79ea00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff7a0152c30 0x7ff7a00c6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:43360/0 (socket says 172.21.15.146:43360) 2024-09-16T01:51:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff797e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7a00c36b0 0x7ff7a00c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff79ea00640 1 -- 172.21.15.146:0/3561002976 learned_addr learned my addr 172.21.15.146:0/3561002976 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:02.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff797e00640 1 -- 172.21.15.146:0/3561002976 >> v1:172.21.15.161:6789/0 conn(0x7ff7a00c9180 legacy=0x7ff7a00c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff797e00640 1 -- 172.21.15.146:0/3561002976 >> v1:172.21.15.146:6789/0 conn(0x7ff7a0152c30 legacy=0x7ff7a00c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff797e00640 1 -- 172.21.15.146:0/3561002976 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a00c47e0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.026+0000 7ff79e000640 1 -- 172.21.15.146:0/3561002976 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1401884038 0 0) 0x7ff7a00c4240 con 0x7ff7a0152c30 2024-09-16T01:51:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.027+0000 7ff797e00640 1 --2- 172.21.15.146:0/3561002976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a00c8c40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff7880097a0 tx=0x7ff78802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec66cbf4f04d272 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.027+0000 7ff79e000640 1 -- 172.21.15.146:0/3561002976 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff788037070 con 0x7ff7a00c36b0 2024-09-16T01:51:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff79e000640 1 -- 172.21.15.146:0/3561002976 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff788032440 con 0x7ff7a00c36b0 2024-09-16T01:51:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff79e000640 1 -- 172.21.15.146:0/3561002976 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff788032810 con 0x7ff7a00c36b0 2024-09-16T01:51:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3561002976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 msgr2=0x7ff7a00c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3561002976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a00c8c40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff7880097a0 tx=0x7ff78802a230 comp rx=0 tx=0).stop 2024-09-16T01:51:02.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3561002976 shutdown_connections 2024-09-16T01:51:02.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3561002976 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a00c8c40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3561002976 >> 172.21.15.146:0/3561002976 conn(0x7ff7a006bc10 msgr2=0x7ff7a006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:02.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3561002976 shutdown_connections 2024-09-16T01:51:02.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.028+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3561002976 wait complete. 2024-09-16T01:51:02.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.030+0000 7ff7a5e00640 1 Processor -- start 2024-09-16T01:51:02.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.030+0000 7ff7a5e00640 1 -- start start 2024-09-16T01:51:02.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff7a5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a01f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff7a5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff7a00c9180 0x7ff7a00ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff7a5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0152c30 0x7ff7a00bd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff7a5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a00c4da0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff7a5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a00c6630 con 0x7ff7a0152c30 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff7a5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a00c8700 con 0x7ff7a00c9180 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a01f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a01f15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50958/0 (socket says 172.21.15.146:50958) 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 -- 172.21.15.146:0/3884721009 learned_addr learned my addr 172.21.15.146:0/3884721009 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff7a00c9180 0x7ff7a00ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0152c30 0x7ff7a00bd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff7a00c9180 msgr2=0x7ff7a00ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff7a00c9180 0x7ff7a00ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0152c30 msgr2=0x7ff7a00bd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7a0152c30 0x7ff7a00bd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.031+0000 7ff79ea00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a00733e0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.032+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a01f15f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c4d0 tx=0x7ff79000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.032+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790018070 con 0x7ff7a00c36b0 2024-09-16T01:51:02.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.032+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7900074e0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.032+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a0073680 con 0x7ff7a00c36b0 2024-09-16T01:51:02.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.032+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790004a20 con 0x7ff7a00c36b0 2024-09-16T01:51:02.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.032+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a01f0df0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.033+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff7a014d600 con 0x7ff7a00c36b0 2024-09-16T01:51:02.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.034+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff79001b070 con 0x7ff7a00c36b0 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.035+0000 7ff797400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff77c06c2b0 0x7ff77c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.035+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5741+0+0 (secure 0 0 0) 0x7ff79008a7f0 con 0x7ff7a00c36b0 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.035+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v61) ==== 54+0+16 (secure 0 0 0) 0x7ff7a014d600 con 0x7ff7a00c36b0 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.035+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff77c06c2b0 0x7ff77c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.036+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff7a0072b40 0x7ff7a006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.036+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7a01f4e90 con 0x7ff7a0072b40 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.036+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff7a0072b40 0x7ff7a006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.036+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff77c06c2b0 0x7ff77c06e770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff788034000 tx=0x7ff78802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.036+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff7a0072b40 0x7ff7a006ea70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.037+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a01f4e90 con 0x7ff7a0072b40 2024-09-16T01:51:02.043 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:02.042+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:02.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.052+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a00c8c40 con 0x7ff7a0072b40 2024-09-16T01:51:02.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.053+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a00c8c40 con 0x7ff7a0072b40 2024-09-16T01:51:02.054 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:02.055 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:02.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.054+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff7a0071a10 0x7ff7a0071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.054+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff7a00c7530 con 0x7ff7a0071a10 2024-09-16T01:51:02.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.054+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff7a0071a10 0x7ff7a0071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.054+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff7a0071a10 0x7ff7a0071df0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.055+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a00c7530 con 0x7ff7a0071a10 2024-09-16T01:51:02.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.070+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a014ede0 con 0x7ff7a0071a10 2024-09-16T01:51:02.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.071+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a014ede0 con 0x7ff7a0071a10 2024-09-16T01:51:02.072 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:02.073 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:02.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.072+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff7a0148b00 0x7ff7a0148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.072+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff7a0149420 con 0x7ff7a0148b00 2024-09-16T01:51:02.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.072+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff7a0148b00 0x7ff7a0148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.072+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff7a0148b00 0x7ff7a0148ee0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.073+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a0149420 con 0x7ff7a0148b00 2024-09-16T01:51:02.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.087+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a014eb00 con 0x7ff7a0148b00 2024-09-16T01:51:02.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.088+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a014eb00 con 0x7ff7a0148b00 2024-09-16T01:51:02.089 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:02.090 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:02.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.088+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff7a0062f00 0x7ff7a01e1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.088+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff7a014e820 con 0x7ff7a0062f00 2024-09-16T01:51:02.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.089+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff7a0062f00 0x7ff7a01e1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.089+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff7a0062f00 0x7ff7a01e1d70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.090+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a014e820 con 0x7ff7a0062f00 2024-09-16T01:51:02.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.103+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a014e820 con 0x7ff7a0062f00 2024-09-16T01:51:02.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.105+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a014e820 con 0x7ff7a0062f00 2024-09-16T01:51:02.106 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:02.107 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:02.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.105+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff7a01e27c0 0x7ff7a002c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.105+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff7a014e820 con 0x7ff7a01e27c0 2024-09-16T01:51:02.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.106+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff7a01e27c0 0x7ff7a002c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.106+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff7a01e27c0 0x7ff7a002c640 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.107+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a014e820 con 0x7ff7a01e27c0 2024-09-16T01:51:02.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.120+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a014e820 con 0x7ff7a01e27c0 2024-09-16T01:51:02.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.122+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a014e820 con 0x7ff7a01e27c0 2024-09-16T01:51:02.123 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:02.124 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:02.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.122+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff7a002d090 0x7ff7a002d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.122+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff7a01e2c40 con 0x7ff7a002d090 2024-09-16T01:51:02.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.123+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff7a002d090 0x7ff7a002d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.123+0000 7ff797e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff7a002d090 0x7ff7a002d4f0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.124+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a01e2c40 con 0x7ff7a002d090 2024-09-16T01:51:02.133 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:02.132+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:02.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.138+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a01e2c40 con 0x7ff7a002d090 2024-09-16T01:51:02.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.139+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a01e2c40 con 0x7ff7a002d090 2024-09-16T01:51:02.141 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.140+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff7a0207a20 0x7ff7a0209e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.140+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff7a002db30 con 0x7ff7a0207a20 2024-09-16T01:51:02.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.140+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff7a0207a20 0x7ff7a0209e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.141+0000 7ff79f400640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff7a0207a20 0x7ff7a0209e10 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.142+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a002db30 con 0x7ff7a0207a20 2024-09-16T01:51:02.147 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.147 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.156 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:02.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.156+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a002db00 con 0x7ff7a0207a20 2024-09-16T01:51:02.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.158+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a002db00 con 0x7ff7a0207a20 2024-09-16T01:51:02.159 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:02.159 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.158+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff7a020a860 0x7ff7a020ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.158+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff7a020fe20 con 0x7ff7a020a860 2024-09-16T01:51:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.158+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff7a020a860 0x7ff7a020ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.159+0000 7ff79ea00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff7a020a860 0x7ff7a020ac80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.159+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7a020fe20 con 0x7ff7a020a860 2024-09-16T01:51:02.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.173+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7a020b2d0 con 0x7ff7a020a860 2024-09-16T01:51:02.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff797400640 1 -- 172.21.15.146:0/3884721009 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7a020b2d0 con 0x7ff7a020a860 2024-09-16T01:51:02.176 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:02.177 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:02.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff7a0072b40 msgr2=0x7ff7a006ea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7ff7a0072b40 0x7ff7a006ea70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff7a0071a10 msgr2=0x7ff7a0071df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7ff7a0071a10 0x7ff7a0071df0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff7a0148b00 msgr2=0x7ff7a0148ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7ff7a0148b00 0x7ff7a0148ee0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff7a0062f00 msgr2=0x7ff7a01e1d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff7a0062f00 0x7ff7a01e1d70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff7a01e27c0 msgr2=0x7ff7a002c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7ff7a01e27c0 0x7ff7a002c640 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff7a002d090 msgr2=0x7ff7a002d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff7a002d090 0x7ff7a002d4f0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff7a0207a20 msgr2=0x7ff7a0209e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7ff7a0207a20 0x7ff7a0209e10 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff7a020a860 msgr2=0x7ff7a020ac80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff7a020a860 0x7ff7a020ac80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:02.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff79f400640 1 -- 172.21.15.146:0/3884721009 reap_dead start 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff79f400640 1 -- 172.21.15.146:0/3884721009 reap_dead start 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff77c06c2b0 msgr2=0x7ff77c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff77c06c2b0 0x7ff77c06e770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff788034000 tx=0x7ff78802e000 comp rx=0 tx=0).stop 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 msgr2=0x7ff7a01f15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 --2- 172.21.15.146:0/3884721009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7a00c36b0 0x7ff7a01f15f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c4d0 tx=0x7ff79000c990 comp rx=0 tx=0).stop 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff79f400640 1 -- 172.21.15.146:0/3884721009 reap_dead start 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff79f400640 1 -- 172.21.15.146:0/3884721009 reap_dead start 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff79f400640 1 -- 172.21.15.146:0/3884721009 reap_dead start 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 shutdown_connections 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.175+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 >> 172.21.15.146:0/3884721009 conn(0x7ff7a006bc10 msgr2=0x7ff7a0150970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.176+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 shutdown_connections 2024-09-16T01:51:02.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:02.176+0000 7ff7a5e00640 1 -- 172.21.15.146:0/3884721009 wait complete. 2024-09-16T01:51:02.240 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:02.239+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:02.300 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.300 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.309 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:02.335 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:02.335+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:02.437 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:02.436+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:02.452 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.452 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.461 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:02.537 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:02.536+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:02.565 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.565 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.574 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:02.638 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:02.637+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:02.719 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.719 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.728 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:02.772 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:02.771+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:02.837 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.837 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.839 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:02.838+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:02.843 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:02.940 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:02.939+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:02.987 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:02.987 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:02.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:03.042 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:03.040+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:03.140 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:03.141 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:03.142 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:03.141+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:03.149 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:03.246 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:03.245+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:03.253 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:03.253 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:03.262 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:03.346 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:03.345+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:03.365 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:03.366 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:03.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:03.447 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:03.446+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:03.543 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:03.543 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:03.547 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:03.547+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:03.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:03.648 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:03.647+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:03.749 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:03.748+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:03.759 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:03.759 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:03.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:03.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:51:03.859 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:03.858+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:03.927 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:03.927 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 readers = 3497330.75 kB/sec 2024-09-16T01:51:03.927 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 readers = 3488902.10 kB/sec 2024-09-16T01:51:03.927 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 3497330.75 kB/sec 2024-09-16T01:51:03.927 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 3497330.75 kB/sec 2024-09-16T01:51:03.927 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 3497330.75 kB/sec 2024-09-16T01:51:03.928 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:03.951 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:03.950+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:03.980 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:03.980 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:03.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:04.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.018+0000 7fa5bec00640 1 Processor -- start 2024-09-16T01:51:04.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.019+0000 7fa5bec00640 1 -- start start 2024-09-16T01:51:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bec00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5b80c6b70 0x7fa5b815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 0x7fa5b80cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b806d140 con 0x7fa5b80cbab0 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bec00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b815b880 con 0x7fa5b80c6b70 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b80cab20 con 0x7fa5b80c75a0 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bce00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 0x7fa5b80cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bd800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5b80c6b70 0x7fa5b815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bce00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 0x7fa5b80cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44678/0 (socket says 172.21.15.146:44678) 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bce00640 1 -- 172.21.15.146:0/3751349441 learned_addr learned my addr 172.21.15.146:0/3751349441 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bd800640 1 -- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 msgr2=0x7fa5b80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bd800640 1 --2- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 0x7fa5b80cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bd800640 1 -- 172.21.15.146:0/3751349441 >> v1:172.21.15.146:6789/0 conn(0x7fa5b80cbab0 legacy=0x7fa5b80cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bd800640 1 -- 172.21.15.146:0/3751349441 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b806d420 con 0x7fa5b80c6b70 2024-09-16T01:51:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.020+0000 7fa5bce00640 1 --2- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 0x7fa5b80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:04.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bd800640 1 --2- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815b340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0097a0 tx=0x7fa5ac02a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa956ff0677e4031 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5b7e00640 1 -- 172.21.15.146:0/3751349441 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac037070 con 0x7fa5b80c6b70 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5b7e00640 1 -- 172.21.15.146:0/3751349441 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ac02adb0 con 0x7fa5b80c6b70 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5b7e00640 1 -- 172.21.15.146:0/3751349441 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac032660 con 0x7fa5b80c6b70 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bec00640 1 -- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 msgr2=0x7fa5b815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bec00640 1 --2- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815b340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0097a0 tx=0x7fa5ac02a060 comp rx=0 tx=0).stop 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bec00640 1 -- 172.21.15.146:0/3751349441 shutdown_connections 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bec00640 1 --2- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80c75a0 0x7fa5b80cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bec00640 1 --2- 172.21.15.146:0/3751349441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815b340 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.022+0000 7fa5bec00640 1 -- 172.21.15.146:0/3751349441 >> 172.21.15.146:0/3751349441 conn(0x7fa5b80ccc20 msgr2=0x7fa5b80cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.023+0000 7fa5bec00640 1 -- 172.21.15.146:0/3751349441 shutdown_connections 2024-09-16T01:51:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.023+0000 7fa5bec00640 1 -- 172.21.15.146:0/3751349441 wait complete. 2024-09-16T01:51:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.024+0000 7fa5bec00640 1 Processor -- start 2024-09-16T01:51:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.024+0000 7fa5bec00640 1 -- start start 2024-09-16T01:51:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.025+0000 7fa5bec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.025+0000 7fa5bec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 0x7fa5b8157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.025+0000 7fa5bec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80cbab0 0x7fa5b8151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.025+0000 7fa5bec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b80cb050 con 0x7fa5b80c75a0 2024-09-16T01:51:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.025+0000 7fa5bec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b80cad70 con 0x7fa5b80c6b70 2024-09-16T01:51:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.025+0000 7fa5bec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b806cd20 con 0x7fa5b80cbab0 2024-09-16T01:51:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 0x7fa5b8157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 0x7fa5b8157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40880/0 (socket says 172.21.15.146:40880) 2024-09-16T01:51:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 -- 172.21.15.146:0/4269455656 learned_addr learned my addr 172.21.15.146:0/4269455656 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5be200640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80cbab0 0x7fa5b8151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bd800640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 -- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80cbab0 msgr2=0x7fa5b8151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80cbab0 0x7fa5b8151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 -- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 msgr2=0x7fa5b815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 -- 172.21.15.146:0/4269455656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b8153c50 con 0x7fa5b80c75a0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.026+0000 7fa5bce00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 0x7fa5b8157630 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400e930 tx=0x7fa5a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.027+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4017070 con 0x7fa5b80c75a0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.027+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a400ca00 con 0x7fa5b80c75a0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.027+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a400cdf0 con 0x7fa5b80c75a0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.027+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b80c35d0 con 0x7fa5b80c75a0 2024-09-16T01:51:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.027+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b8156e30 con 0x7fa5b80c75a0 2024-09-16T01:51:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.028+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b81f4720 con 0x7fa5b80c75a0 2024-09-16T01:51:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.029+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa5a401a070 con 0x7fa5b80c75a0 2024-09-16T01:51:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.030+0000 7fa5b5600640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa59806c2b0 0x7fa59806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.030+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5741+0+0 (secure 0 0 0) 0x7fa5a408b230 con 0x7fa5b80c75a0 2024-09-16T01:51:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.030+0000 7fa5bd800640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa59806c2b0 0x7fa59806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.031+0000 7fa5bd800640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa59806c2b0 0x7fa59806e770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac034000 tx=0x7fa5ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.034+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa5b81f4720 con 0x7fa5b80c75a0 2024-09-16T01:51:04.052 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:04.051+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:04.154 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:04.154 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:04.154 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:04.152+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:04.166 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:04.192 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:04.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.226+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fa5b806da90 con 0x7fa5b80c75a0 2024-09-16T01:51:04.254 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:04.253+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:04.355 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:04.354+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:04.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.362+0000 7f63e6000640 1 Processor -- start 2024-09-16T01:51:04.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.362+0000 7f63e6000640 1 -- start start 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63e6000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63e0152c30 0x7f63e00c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63e6000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63e00c6ab0 0x7f63e00c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63e6000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e00c4240 con 0x7f63e00c6e90 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63e6000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f63e00c4da0 con 0x7f63e0152c30 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63e6000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f63e00c4500 con 0x7f63e00c6ab0 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63df400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63e0152c30 0x7f63e00c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63df400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63e0152c30 0x7f63e00c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40886/0 (socket says 172.21.15.146:40886) 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63df400640 1 -- 172.21.15.146:0/2878757370 learned_addr learned my addr 172.21.15.146:0/2878757370 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:04.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63dfe00640 1 --1- 172.21.15.146:0/2878757370 >> v1:172.21.15.146:6789/0 conn(0x7f63e00c6e90 0x7f63e00c7ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45650/0 (socket says 172.21.15.146:45650) 2024-09-16T01:51:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.363+0000 7f63dea00640 1 --2- 172.21.15.146:0/2878757370 >> v2:172.21.15.161:3300/0 conn(0x7f63e00c6ab0 0x7f63e00c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.364+0000 7f63df400640 1 -- 172.21.15.146:0/2878757370 >> v2:172.21.15.161:3300/0 conn(0x7f63e00c6ab0 msgr2=0x7f63e00c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.364+0000 7f63df400640 1 --2- 172.21.15.146:0/2878757370 >> v2:172.21.15.161:3300/0 conn(0x7f63e00c6ab0 0x7f63e00c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.364+0000 7f63df400640 1 -- 172.21.15.146:0/2878757370 >> v1:172.21.15.146:6789/0 conn(0x7f63e00c6e90 legacy=0x7f63e00c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.364+0000 7f63df400640 1 -- 172.21.15.146:0/2878757370 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e00c47e0 con 0x7f63e0152c30 2024-09-16T01:51:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.364+0000 7f63de000640 1 -- 172.21.15.146:0/2878757370 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2964910106 0 0) 0x7f63e00c4240 con 0x7f63e00c6e90 2024-09-16T01:51:04.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.365+0000 7f63df400640 1 --2- 172.21.15.146:0/2878757370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0152c30 0x7f63e00c36b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f63e0153010 tx=0x7f63d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23db11a8f75d3dee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.365+0000 7f63de000640 1 -- 172.21.15.146:0/2878757370 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0037070 con 0x7f63e0152c30 2024-09-16T01:51:04.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.365+0000 7f63de000640 1 -- 172.21.15.146:0/2878757370 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d0032440 con 0x7f63e0152c30 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.365+0000 7f63de000640 1 -- 172.21.15.146:0/2878757370 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0032810 con 0x7f63e0152c30 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 -- 172.21.15.146:0/2878757370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0152c30 msgr2=0x7f63e00c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 --2- 172.21.15.146:0/2878757370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0152c30 0x7f63e00c36b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f63e0153010 tx=0x7f63d002a270 comp rx=0 tx=0).stop 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 -- 172.21.15.146:0/2878757370 shutdown_connections 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 --2- 172.21.15.146:0/2878757370 >> v2:172.21.15.161:3300/0 conn(0x7f63e00c6ab0 0x7f63e00c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 --2- 172.21.15.146:0/2878757370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0152c30 0x7f63e00c36b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 -- 172.21.15.146:0/2878757370 >> 172.21.15.146:0/2878757370 conn(0x7f63e006bc10 msgr2=0x7f63e006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 -- 172.21.15.146:0/2878757370 shutdown_connections 2024-09-16T01:51:04.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.366+0000 7f63e6000640 1 -- 172.21.15.146:0/2878757370 wait complete. 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.367+0000 7f63e6000640 1 Processor -- start 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.368+0000 7f63e6000640 1 -- start start 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.368+0000 7f63e6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 0x7f63e01e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.368+0000 7f63e6000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63e00c6e90 0x7f63e00c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63e6000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63e0152c30 0x7f63e00bea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63e6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f63e00c8700 con 0x7f63e00c6ab0 2024-09-16T01:51:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63e6000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f63e00c6630 con 0x7f63e00c6e90 2024-09-16T01:51:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63e6000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f63e00c8880 con 0x7f63e0152c30 2024-09-16T01:51:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 0x7f63e01e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 0x7f63e01e3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40888/0 (socket says 172.21.15.146:40888) 2024-09-16T01:51:04.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 -- 172.21.15.146:0/2127063934 learned_addr learned my addr 172.21.15.146:0/2127063934 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:04.371 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:04.371 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63e00c6e90 0x7f63e00c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63e0152c30 0x7f63e00bea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63e0152c30 msgr2=0x7f63e00bea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63e0152c30 0x7f63e00bea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63e00c6e90 msgr2=0x7f63e00c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63e00c6e90 0x7f63e00c2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.369+0000 7f63df400640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e00ca600 con 0x7f63e00c6ab0 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.370+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 0x7f63e01e3fa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.370+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0037040 con 0x7f63e00c6ab0 2024-09-16T01:51:04.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.370+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d0004030 con 0x7f63e00c6ab0 2024-09-16T01:51:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.370+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0004400 con 0x7f63e00c6ab0 2024-09-16T01:51:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.370+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e00c1f70 con 0x7f63e00c6ab0 2024-09-16T01:51:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.370+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e01f2220 con 0x7f63e00c6ab0 2024-09-16T01:51:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.371+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f63e00c68a0 con 0x7f63e00c6ab0 2024-09-16T01:51:04.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.372+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f63d003a070 con 0x7f63e00c6ab0 2024-09-16T01:51:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.373+0000 7f63d7400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f63c006c2b0 0x7f63c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.373+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5741+0+0 (secure 0 0 0) 0x7f63d00ab9b0 con 0x7f63e00c6ab0 2024-09-16T01:51:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.373+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v61) ==== 54+0+16 (secure 0 0 0) 0x7f63d0041020 con 0x7f63e00c6ab0 2024-09-16T01:51:04.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.373+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f63c006c2b0 0x7f63c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.373+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f63e00cb4f0 0x7f63e00cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.373+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63e00cbe10 con 0x7f63e00cb4f0 2024-09-16T01:51:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.374+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f63e00cb4f0 0x7f63e00cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.374+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f63c006c2b0 0x7f63c006e770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f63e00bff80 tx=0x7f63cc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.374+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f63e00cb4f0 0x7f63e00cb8d0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.375+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63d8001b30 con 0x7f63e00cb4f0 2024-09-16T01:51:04.384 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:04.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.390+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e014e920 con 0x7f63e00cb4f0 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.392+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e014e920 con 0x7f63e00cb4f0 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.392+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f63e01f4da0 0x7f63e01f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.392+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f63e01f56c0 con 0x7f63e01f4da0 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.392+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f63e01f4da0 0x7f63e01f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.393+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f63e01f4da0 0x7f63e01f5180 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.393+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e01f56c0 con 0x7f63e01f4da0 2024-09-16T01:51:04.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.408+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e0050090 con 0x7f63e01f4da0 2024-09-16T01:51:04.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.410+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e0050090 con 0x7f63e01f4da0 2024-09-16T01:51:04.411 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:04.412 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:04.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.410+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f63e0148a70 0x7f63e0148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.410+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f63e0149390 con 0x7f63e0148a70 2024-09-16T01:51:04.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.410+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f63e0148a70 0x7f63e0148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.411+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f63e0148a70 0x7f63e0148e50 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.412+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e0149390 con 0x7f63e0148a70 2024-09-16T01:51:04.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.426+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e00c7530 con 0x7f63e0148a70 2024-09-16T01:51:04.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.427+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e00c7530 con 0x7f63e0148a70 2024-09-16T01:51:04.428 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.427+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f63e0062f00 0x7f63e0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.428+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f63e0072550 con 0x7f63e0062f00 2024-09-16T01:51:04.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.428+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f63e0062f00 0x7f63e0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.428+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f63e0062f00 0x7f63e0150870 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.429+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e0072550 con 0x7f63e0062f00 2024-09-16T01:51:04.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.444+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e00635d0 con 0x7f63e0062f00 2024-09-16T01:51:04.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.446+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e00635d0 con 0x7f63e0062f00 2024-09-16T01:51:04.447 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:04.447 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:04.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.446+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f63e01512c0 0x7f63e002c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.446+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f63e0072550 con 0x7f63e01512c0 2024-09-16T01:51:04.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.446+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f63e01512c0 0x7f63e002c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.447+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f63e01512c0 0x7f63e002c640 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.447+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e0072550 con 0x7f63e01512c0 2024-09-16T01:51:04.456 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:04.455+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:04.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.461+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e0072550 con 0x7f63e01512c0 2024-09-16T01:51:04.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.463+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e0072550 con 0x7f63e01512c0 2024-09-16T01:51:04.464 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:04.465 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:04.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.463+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f63e002d090 0x7f63e002d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.463+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f63e0072270 con 0x7f63e002d090 2024-09-16T01:51:04.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.464+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f63e002d090 0x7f63e002d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.464+0000 7f63dea00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f63e002d090 0x7f63e002d4d0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.465+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e0072270 con 0x7f63e002d090 2024-09-16T01:51:04.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.478+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e0072270 con 0x7f63e002d090 2024-09-16T01:51:04.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.480+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e0072270 con 0x7f63e002d090 2024-09-16T01:51:04.481 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:04.482 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:04.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.481+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f63e02083b0 0x7f63e020a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.481+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f63e006eb10 con 0x7f63e02083b0 2024-09-16T01:51:04.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.481+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f63e02083b0 0x7f63e020a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.482+0000 7f63dfe00640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f63e02083b0 0x7f63e020a7a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.483+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e006eb10 con 0x7f63e02083b0 2024-09-16T01:51:04.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.496+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e006eb10 con 0x7f63e02083b0 2024-09-16T01:51:04.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.498+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e006eb10 con 0x7f63e02083b0 2024-09-16T01:51:04.499 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:04.500 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:04.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.498+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f63e020b1f0 0x7f63e020d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:04.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.498+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f63e00008d0 con 0x7f63e020b1f0 2024-09-16T01:51:04.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.499+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f63e020b1f0 0x7f63e020d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:04.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.499+0000 7f63df400640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f63e020b1f0 0x7f63e020d5e0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:04.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.500+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63e00008d0 con 0x7f63e020b1f0 2024-09-16T01:51:04.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.527+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f63e00008d0 con 0x7f63e020b1f0 2024-09-16T01:51:04.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.529+0000 7f63d7400640 1 -- 172.21.15.146:0/2127063934 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f63e00008d0 con 0x7f63e020b1f0 2024-09-16T01:51:04.530 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:04.531 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.529+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f63e00cb4f0 msgr2=0x7f63e00cb8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.529+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f63e00cb4f0 0x7f63e00cb8d0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.529+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f63e01f4da0 msgr2=0x7f63e01f5180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.529+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f63e01f4da0 0x7f63e01f5180 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f63e0148a70 msgr2=0x7f63e0148e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f63e0148a70 0x7f63e0148e50 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f63e0062f00 msgr2=0x7f63e0150870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f63e0062f00 0x7f63e0150870 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f63e01512c0 msgr2=0x7f63e002c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f63e01512c0 0x7f63e002c640 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f63e002d090 msgr2=0x7f63e002d4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f63e002d090 0x7f63e002d4d0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f63e02083b0 msgr2=0x7f63e020a7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f63e02083b0 0x7f63e020a7a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f63e020b1f0 msgr2=0x7f63e020d5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f63e020b1f0 0x7f63e020d5e0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63dfe00640 1 -- 172.21.15.146:0/2127063934 reap_dead start 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63dfe00640 1 -- 172.21.15.146:0/2127063934 reap_dead start 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63dfe00640 1 -- 172.21.15.146:0/2127063934 reap_dead start 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63dfe00640 1 -- 172.21.15.146:0/2127063934 reap_dead start 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63dfe00640 1 -- 172.21.15.146:0/2127063934 reap_dead start 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f63c006c2b0 msgr2=0x7f63c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f63c006c2b0 0x7f63c006e770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f63e00bff80 tx=0x7f63cc009190 comp rx=0 tx=0).stop 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 msgr2=0x7f63e01e3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 0x7f63e01e3fa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d00046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 shutdown_connections 2024-09-16T01:51:04.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f63c006c2b0 0x7f63c006e770 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 --2- 172.21.15.146:0/2127063934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e00c6ab0 0x7f63e01e3fa0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 >> 172.21.15.146:0/2127063934 conn(0x7f63e006bc10 msgr2=0x7f63e0072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:04.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 shutdown_connections 2024-09-16T01:51:04.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.530+0000 7f63e6000640 1 -- 172.21.15.146:0/2127063934 wait complete. 2024-09-16T01:51:04.568 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:04.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 136.79764079399999, 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.569 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:51:04.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:51:04.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:7 3.2a 3:557f550e:::10000000002.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.30730046, 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081303988999999993, 2024-09-16T01:51:04.576 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 7 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258696+0000", 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7320000000000001e-06 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305572+0000", 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046875737000000001 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305609+0000", 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7704999999999999e-05 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305623+0000", 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-09-16T01:51:04.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310196+0000", 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045734269999999997 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310344+0000", 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147181 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310523+0000", 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017908400000000001 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.327195+0000", 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016672724999999999 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339893+0000", 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012697559000000001 2024-09-16T01:51:04.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339953+0000", 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0084999999999998e-05 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339998+0000", 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5218e-05 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:20 3.21 3:85380590:::10000000002.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.154293402, 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055783419000000001, 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.583 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 20 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411703+0000", 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.84e-06 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441653+0000", 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029949640999999999 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441685+0000", 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2452e-05 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441702+0000", 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441798+0000", 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6501000000000003e-05 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441964+0000", 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165635 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.442314+0000", 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035073800000000002 2024-09-16T01:51:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.455571+0000", 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013256273000000001 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467368+0000", 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011797656 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467432+0000", 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3585999999999997e-05 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467485+0000", 2024-09-16T01:51:04.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2729999999999998e-05 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:22 3.10 3:0a7b425d:::10000000002.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.124255092, 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050802171, 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.591 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 22 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441740+0000", 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441741+0000", 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.641e-06 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459091+0000", 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017350175999999998 2024-09-16T01:51:04.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459122+0000", 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1041000000000003e-05 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459136+0000", 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.384e-05 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461076+0000", 2024-09-16T01:51:04.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001939213 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461219+0000", 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143101 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.461360+0000", 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014149900000000001 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492196+0000", 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030836319000000001 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492406+0000", 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020962800000000001 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492461+0000", 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5251999999999999e-05 2024-09-16T01:51:04.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.492542+0000", 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0461000000000001e-05 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:23 3.21 3:87fa6b7a:::10000000002.00000014:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.106814907, 2024-09-16T01:51:04.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049069672000000002, 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 23 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459180+0000", 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.459181+0000", 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.356e-06 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499609+0000", 2024-09-16T01:51:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040428104999999999 2024-09-16T01:51:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499638+0000", 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8640999999999999e-05 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499650+0000", 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1989e-05 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499729+0000", 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8869000000000007e-05 2024-09-16T01:51:04.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.499916+0000", 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018738500000000001 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.500068+0000", 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015149099999999999 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.506758+0000", 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066901900000000004 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508128+0000", 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013700330000000001 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508197+0000", 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9077000000000003e-05 2024-09-16T01:51:04.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.508249+0000", 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2536e-05 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:43 3.1a 3:5917aad8:::10000000002.00000028:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.965349896999999, 2024-09-16T01:51:04.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041950973000000003, 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 43 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:51:04.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600645+0000", 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.600647+0000", 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.136e-06 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623155+0000", 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022507547999999999 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623186+0000", 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1078000000000003e-05 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623196+0000", 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0825999999999999e-05 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624399+0000", 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012021969999999999 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624531+0000", 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013200899999999999 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624681+0000", 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015020999999999999 2024-09-16T01:51:04.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.639604+0000", 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014922962 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642466+0000", 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002862028 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642534+0000", 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8503999999999993e-05 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.642596+0000", 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1475000000000003e-05 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:44 3.19 3:9961cbc1:::10000000002.00000029:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.611 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.942300681000001, 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044190814000000002, 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 44 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:51:04.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623694+0000", 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.623696+0000", 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000002e-06 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660636+0000", 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036939840000000002 2024-09-16T01:51:04.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660662+0000", 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5891e-05 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660672+0000", 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0362e-05 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660758+0000", 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5834999999999994e-05 2024-09-16T01:51:04.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660916+0000", 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158572 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:04.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.661072+0000", 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015595600000000001 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667770+0000", 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066975170000000001 2024-09-16T01:51:04.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667785+0000", 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4882000000000001e-05 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667837+0000", 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2343999999999999e-05 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.667885+0000", 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7679000000000002e-05 2024-09-16T01:51:04.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:59 3.1c 3:394f8a26:::10000000002.00000038:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:51:04.622 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.773259093, 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051871274000000002, 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.623 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 59 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792736+0000", 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.792738+0000", 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1100000000000001e-06 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824239+0000", 2024-09-16T01:51:04.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031501159000000001 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824261+0000", 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2515e-05 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824272+0000", 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0696999999999999e-05 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824397+0000", 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012514399999999999 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824552+0000", 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015457599999999999 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.824715+0000", 2024-09-16T01:51:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163657 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.832638+0000", 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079224959999999994 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844470+0000", 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011831734999999999 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844542+0000", 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2194999999999996e-05 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.844607+0000", 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4989999999999999e-05 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:91 3.2a 3:5747858d:::10000000002.00000058:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.505405804, 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045690485000000003, 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 91 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060589+0000", 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060591+0000", 2024-09-16T01:51:04.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1009999999999999e-06 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094735+0000", 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034143896999999999 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094758+0000", 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3485999999999998e-05 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094769+0000", 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0844e-05 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094852+0000", 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2627999999999996e-05 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095011+0000", 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015938000000000001 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095160+0000", 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014846799999999999 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.105436+0000", 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010275986000000001 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106140+0000", 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000703905 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106219+0000", 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9672999999999994e-05 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.106279+0000", 2024-09-16T01:51:04.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0117000000000001e-05 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:92 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.471188911, 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040038826999999999, 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.637 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 92 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:51:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094806+0000", 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094807+0000", 2024-09-16T01:51:04.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1319999999999999e-06 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126953+0000", 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032146474000000001 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126978+0000", 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4737e-05 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.126992+0000", 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3543000000000001e-05 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127074+0000", 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1889000000000002e-05 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127230+0000", 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156238 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127387+0000", 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157207 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.131783+0000", 2024-09-16T01:51:04.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004395805 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134755+0000", 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029720520000000002 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134795+0000", 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9909e-05 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.134845+0000", 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9840999999999999e-05 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:97 3.1a 3:5b190c62:::10000000002.0000005e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.438681055, 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040400441000000002, 2024-09-16T01:51:04.645 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 97 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127314+0000", 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.127315+0000", 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.626e-06 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156590+0000", 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029275042000000001 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156619+0000", 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8200000000000001e-05 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156631+0000", 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.252e-05 2024-09-16T01:51:04.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156742+0000", 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011125599999999999 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.156905+0000", 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016292499999999999 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:04.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.157055+0000", 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014968699999999999 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.162589+0000", 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0055339530000000003 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167534+0000", 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049449699999999999 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167659+0000", 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000125575 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.167714+0000", 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4687e-05 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:124 3.21 3:844ac05a:::10000000002.00000079:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.166307895999999, 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052276376999999999, 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 124 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.652 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399687+0000", 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.399689+0000", 2024-09-16T01:51:04.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5100000000000001e-06 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443104+0000", 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043414547999999997 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443122+0000", 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8071e-05 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443132+0000", 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0492999999999999e-05 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443200+0000", 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7248000000000004e-05 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443353+0000", 2024-09-16T01:51:04.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015338799999999999 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.443504+0000", 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150681 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447533+0000", 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040287889999999996 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451892+0000", 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043593520000000004 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451932+0000", 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0104000000000001e-05 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.451963+0000", 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1192999999999998e-05 2024-09-16T01:51:04.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:139 3.1c 3:384ad1d9:::10000000002.00000088:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.025292778000001, 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061931110999999997, 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540704+0000", 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.221e-06 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586535+0000", 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045830901 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586563+0000", 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8388e-05 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586591+0000", 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7424999999999999e-05 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586694+0000", 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103269 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586849+0000", 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154857 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.587008+0000", 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015895699999999999 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591909+0000", 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049009609999999997 2024-09-16T01:51:04.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602502+0000", 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010592773 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602564+0000", 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2154000000000002e-05 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602633+0000", 2024-09-16T01:51:04.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9204999999999999e-05 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:148 3.1c 3:3afba7d1:::10000000002.00000091:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.941720614999999, 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047360674999999998, 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.664 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 148 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624274+0000", 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.624276+0000", 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1639999999999999e-06 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653877+0000", 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029600346 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653900+0000", 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3399000000000001e-05 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.653912+0000", 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2232000000000001e-05 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654079+0000", 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016647999999999999 2024-09-16T01:51:04.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654232+0000", 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015308100000000001 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.654384+0000", 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152573 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.662170+0000", 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077856990000000001 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671503+0000", 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093331279999999996 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671572+0000", 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8417999999999994e-05 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.671635+0000", 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3155000000000001e-05 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:191 3.1a 3:5ab92411:::10000000002.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.529329452000001, 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045215946999999999, 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 191 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036665+0000", 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.036668+0000", 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4880000000000001e-06 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066884+0000", 2024-09-16T01:51:04.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030216601999999999 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066908+0000", 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3555999999999999e-05 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.066918+0000", 2024-09-16T01:51:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0443e-05 2024-09-16T01:51:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069411+0000", 2024-09-16T01:51:04.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024928149999999998 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069698+0000", 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287025 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.069886+0000", 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018816899999999999 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.075677+0000", 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0057910540000000003 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081769+0000", 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060919659999999999 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081834+0000", 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4797999999999999e-05 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.081881+0000", 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7030999999999997e-05 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:441 3.1c 3:3913eadd:::10000000002.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9573127259999996, 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072104032999999998, 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 441 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608684+0000", 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4959999999999999e-06 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650015+0000", 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041330193000000001 2024-09-16T01:51:04.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650045+0000", 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0335e-05 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650057+0000", 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650158+0000", 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100837 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650266+0000", 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107975 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650426+0000", 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016051800000000001 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.663669+0000", 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013242594 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680672+0000", 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017003041999999999 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680737+0000", 2024-09-16T01:51:04.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4750000000000002e-05 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680786+0000", 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9217e-05 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:447 3.1a 3:5ab92411:::10000000002.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.664227+0000", 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.901768143, 2024-09-16T01:51:04.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043974170999999999, 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 447 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.664227+0000", 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.664227+0000", 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.664229+0000", 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9230000000000001e-06 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680449+0000", 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016220535000000001 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680479+0000", 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9935000000000001e-05 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680493+0000", 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3539e-05 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680615+0000", 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000122423 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680713+0000", 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7816e-05 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680856+0000", 2024-09-16T01:51:04.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142917 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.701876+0000", 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021019880000000001 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.708073+0000", 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061970219999999999 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.708147+0000", 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4072000000000005e-05 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.708201+0000", 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4109e-05 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:454 3.37 3:ec162ecd:::10000000002.000000c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.732176+0000", 2024-09-16T01:51:04.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8338190690000005, 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050739933000000001, 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 454 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.732176+0000", 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.732176+0000", 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.732178+0000", 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4219999999999999e-06 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.761814+0000", 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029635946 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.761961+0000", 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147083 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.762014+0000", 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2902999999999998e-05 2024-09-16T01:51:04.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.765825+0000", 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038106080000000001 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.766230+0000", 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040490300000000002 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.766806+0000", 2024-09-16T01:51:04.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00057623199999999998 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.776801+0000", 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099953569999999999 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.782797+0000", 2024-09-16T01:51:04.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0059956180000000003 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.782865+0000", 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8300000000000007e-05 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.782916+0000", 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0560999999999999e-05 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:461 3.19 3:9a784cd8:::10000000002.000000ca:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.835939+0000", 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.7300561459999999, 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046963036, 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 461 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.835939+0000", 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.835939+0000", 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.835941+0000", 2024-09-16T01:51:04.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.526e-06 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.868859+0000", 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032917517 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.868890+0000", 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0944000000000001e-05 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.868903+0000", 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3004e-05 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.869041+0000", 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013840400000000001 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.869137+0000", 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6471000000000005e-05 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.869278+0000", 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014032000000000001 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.874951+0000", 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056731580000000002 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.882790+0000", 2024-09-16T01:51:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078386399999999992 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.882853+0000", 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2949999999999999e-05 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.882902+0000", 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9101999999999998e-05 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.1801884539999996, 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:51:04.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:04.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:51:04.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:04.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:51:04.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:04.712 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:04.713 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:04.569+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:04.714 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:04.766 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:04.754+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:04.816 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:04.815+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.818+0000 7fa5b5600640 1 -- 172.21.15.146:0/4269455656 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v62) ==== 87+0+0 (secure 0 0 0) 0x7fa5a4059330 con 0x7fa5b80c75a0 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa59806c2b0 msgr2=0x7fa59806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa59806c2b0 0x7fa59806e770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac034000 tx=0x7fa5ac02e000 comp rx=0 tx=0).stop 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 msgr2=0x7fa5b8157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 0x7fa5b8157630 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400e930 tx=0x7fa5a400edf0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 shutdown_connections 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa59806c2b0 0x7fa59806e770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5b80cbab0 0x7fa5b8151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b80c75a0 0x7fa5b8157630 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.821+0000 7fa5bec00640 1 --2- 172.21.15.146:0/4269455656 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5b80c6b70 0x7fa5b815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.822+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 >> 172.21.15.146:0/4269455656 conn(0x7fa5b80ccc20 msgr2=0x7fa5b80cf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.822+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 shutdown_connections 2024-09-16T01:51:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:04.822+0000 7fa5bec00640 1 -- 172.21.15.146:0/4269455656 wait complete. 2024-09-16T01:51:04.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:04.872 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:04.872 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T01:51:04.872 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:51:04.872 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:51:04.872 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T01:51:04.929 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:04.930 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:04.930 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:04.931 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:04.929+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:04.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:05.018 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:05.017+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:05.045 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:05.046 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T01:51:05.046 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:05.118 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:05.118+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:05.145 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:05.145 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:05.145 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:05.145 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:05.145 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:05.159 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:05.220 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:05.219+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:05.321 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:05.320+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:05.359 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.101636473999999, 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:51:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:51:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.788690699, 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:42 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.747608230999999, 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051426843, 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 42 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608928+0000", 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5689999999999999e-06 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648858+0000", 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039929485000000001 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648879+0000", 2024-09-16T01:51:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1172e-05 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648889+0000", 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0512e-05 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648975+0000", 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5473e-05 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649103+0000", 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012862999999999999 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649230+0000", 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126561 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.655882+0000", 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066518819999999996 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660228+0000", 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043460699999999996 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660293+0000", 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5050999999999998e-05 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660353+0000", 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0438e-05 2024-09-16T01:51:05.377 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.635145016999999, 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.378 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:05.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:05.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:67 3.16 3:6a386151:::10000000002.00000040:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.520627185, 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046477360000000002, 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 67 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835910+0000", 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.872997+0000", 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03708761 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873029+0000", 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1911e-05 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873046+0000", 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6702999999999999e-05 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873132+0000", 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6034999999999999e-05 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873288+0000", 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015656000000000001 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873419+0000", 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130381 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.880727+0000", 2024-09-16T01:51:05.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073081919999999998 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882276+0000", 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015495330000000001 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882336+0000", 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9964999999999999e-05 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882385+0000", 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8588999999999998e-05 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:77 3.3c 3:3db0268f:::10000000002.0000004a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.446836031, 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051390551, 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 77 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:05.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909702+0000", 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9699999999999999e-06 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927863+0000", 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018160824999999998 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927887+0000", 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4802000000000001e-05 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927900+0000", 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928016+0000", 2024-09-16T01:51:05.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116123 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928158+0000", 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141977 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928294+0000", 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135733 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.938330+0000", 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010036092999999999 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960987+0000", 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022656493 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961043+0000", 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6416999999999999e-05 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961089+0000", 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6152000000000001e-05 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.414285337000001, 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.396 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:05.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:05.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.336154748, 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:05.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:85 3.36 e55/20 3:6fe880e3:::10000000002.00000052:head v 55'2, mlcod=0'0)", 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.313768422000001, 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052325050999999997, 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.408 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042769+0000", 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3520000000000001e-06 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051520+0000", 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087513980000000005 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051666+0000", 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014630900000000001 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051711+0000", 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4768999999999998e-05 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052370+0000", 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065890100000000004 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052478+0000", 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010805 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095028+0000", 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042550194 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095091+0000", 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3077999999999995e-05 2024-09-16T01:51:05.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:98 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.217927544, 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052298391, 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.414 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 98 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138610+0000", 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166283+0000", 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027672980999999999 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166310+0000", 2024-09-16T01:51:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7712000000000002e-05 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166324+0000", 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3239e-05 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166414+0000", 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0062000000000006e-05 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166567+0000", 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015352700000000001 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166726+0000", 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015874500000000001 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.187851+0000", 2024-09-16T01:51:05.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021125117999999998 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190785+0000", 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029334970000000002 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190850+0000", 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5560000000000002e-05 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190906+0000", 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5451000000000002e-05 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:140 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.801179618999999, 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050007939000000001, 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.420 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555356+0000", 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1039999999999999e-06 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591189+0000", 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035832756 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591217+0000", 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8263e-05 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591230+0000", 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591354+0000", 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012385200000000001 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591571+0000", 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217099 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591739+0000", 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016809299999999999 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.598744+0000", 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070049939999999996 2024-09-16T01:51:05.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605250+0000", 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065054079999999999 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605310+0000", 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0263000000000002e-05 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605363+0000", 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3140999999999998e-05 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.763941861999999, 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:05.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:05.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:05.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.539785328000001, 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:05.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:05.437 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:181 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.459805607, 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046020995000000002, 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.438 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 181 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896731+0000", 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-06 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931107+0000", 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034375920999999997 2024-09-16T01:51:05.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931130+0000", 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3668000000000001e-05 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931141+0000", 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0366999999999999e-05 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931281+0000", 2024-09-16T01:51:05.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140622 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931432+0000", 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150469 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931599+0000", 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167338 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.940280+0000", 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086811649999999994 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942632+0000", 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002352224 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942694+0000", 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2014999999999995e-05 2024-09-16T01:51:05.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942750+0000", 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5763999999999997e-05 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.242508304999999, 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:05.444 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7697004520000004, 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.450 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:05.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5818738660000005, 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.456 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:05.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:463 3.26 3:65689812:::10000000002.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4907005699999996, 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112035999999998, 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.462 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 463 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865836+0000", 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9719999999999999e-06 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883238+0000", 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017402081999999999 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883259+0000", 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1143999999999999e-05 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883269+0000", 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-06 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883356+0000", 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7114999999999995e-05 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883467+0000", 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110513 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883617+0000", 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150216 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.906097+0000", 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022480257 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914840+0000", 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087431309999999995 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914898+0000", 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7643999999999999e-05 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914946+0000", 2024-09-16T01:51:05.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8161999999999997e-05 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:484 3.e 3:723b89cc:::10000000002.000000e1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2488311010000004, 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054305622999999997, 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 484 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107706+0000", 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145546+0000", 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037839795000000002 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145584+0000", 2024-09-16T01:51:05.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8556000000000002e-05 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145598+0000", 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4238e-05 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145708+0000", 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109584 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145834+0000", 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012588200000000001 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145990+0000", 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015650199999999999 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.160245+0000", 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014254119000000001 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161887+0000", 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016424009999999999 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161956+0000", 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8996000000000001e-05 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.162009+0000", 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3381999999999998e-05 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.473 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0730574610000003, 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:05.474 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:05.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:05.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:05.479 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:05.480 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:05.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:05.482 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:05.480+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:05.582 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:05.581+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:05.639 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:05.639 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:05.639 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:05.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:05.682 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:05.681+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:05.783 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:05.782+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:05.853 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:05.853 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:05.853 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:05.853 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:05.853 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:05.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:05.884 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:05.883+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:05.985 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:05.984+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:06.067 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:06.067 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:06.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:06.067 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:06.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.067 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.809055955, 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:06.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:06.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:06.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:14 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.073 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.711735643000001, 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051400593000000001, 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 14 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:06.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352602+0000", 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393382+0000", 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040779871000000002 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393422+0000", 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9765000000000003e-05 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393436+0000", 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3633e-05 2024-09-16T01:51:06.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393509+0000", 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3847e-05 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393696+0000", 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186911 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393839+0000", 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014274700000000001 2024-09-16T01:51:06.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.401310+0000", 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074705040000000002 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403876+0000", 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025662020000000001 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403939+0000", 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3423999999999994e-05 2024-09-16T01:51:06.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.404001+0000", 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1649000000000006e-05 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.610683085, 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:06.079 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:06.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.501880454, 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.085 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:06.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:06.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:66 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.242650958, 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053741734999999999, 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.091 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:06.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821687+0000", 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3750000000000001e-06 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862658+0000", 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040970816 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862692+0000", 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3420000000000002e-05 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862706+0000", 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4017e-05 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862879+0000", 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173015 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863040+0000", 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016103800000000001 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863192+0000", 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152738 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.867722+0000", 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045298689999999997 2024-09-16T01:51:06.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875297+0000", 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075744860000000001 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875372+0000", 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5049999999999995e-05 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875427+0000", 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4911000000000003e-05 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.182311821000001, 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:06.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:06.102 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:75 3.30 3:0d2dbd17:::10000000002.00000048:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.118230393999999, 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050908518999999999, 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.103 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946107+0000", 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7710000000000001e-06 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960806+0000", 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014698386000000001 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960821+0000", 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960832+0000", 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0698e-05 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960964+0000", 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013269700000000001 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961122+0000", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157765 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961267+0000", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014486100000000001 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981986+0000", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020719158000000001 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996886+0000", 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014899654999999999 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996958+0000", 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2095999999999995e-05 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997014+0000", 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6258999999999998e-05 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:110 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.827974106999999, 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053794485000000003, 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 110 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.109 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236365+0000", 2024-09-16T01:51:06.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1590000000000002e-06 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278463+0000", 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042097744999999999 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278494+0000", 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1009000000000001e-05 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278508+0000", 2024-09-16T01:51:06.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3901e-05 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278639+0000", 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013188100000000001 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278806+0000", 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016689799999999999 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278974+0000", 2024-09-16T01:51:06.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016719799999999999 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.286538+0000", 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075647910000000004 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290050+0000", 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035116969999999998 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290114+0000", 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4298e-05 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290156+0000", 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1907999999999999e-05 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.683649081, 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:06.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:160 3.9 3:93988fc9:::10000000002.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.314659003999999, 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056552534000000002, 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.120 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 160 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749679+0000", 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.956e-06 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794169+0000", 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04449061 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794202+0000", 2024-09-16T01:51:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2626999999999998e-05 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794217+0000", 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794305+0000", 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8158999999999994e-05 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794460+0000", 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155277 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794625+0000", 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164854 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.801888+0000", 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072630239999999999 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806094+0000", 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042063919999999998 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806178+0000", 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3287000000000004e-05 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806229+0000", 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765000000000003e-05 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.125 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.270076397, 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.126 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:06.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:06.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.131 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:164 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.24220438, 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053828972000000003, 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.132 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 164 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822134+0000", 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841247+0000", 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019113208 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841279+0000", 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2265999999999999e-05 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841291+0000", 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2305e-05 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841376+0000", 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4507999999999998e-05 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841535+0000", 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159216 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841687+0000", 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151846 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859703+0000", 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018016197000000001 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875876+0000", 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016172628000000001 2024-09-16T01:51:06.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875929+0000", 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3108e-05 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875960+0000", 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1491000000000001e-05 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.137 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:197 3.1d 3:bb6dfbbf:::10000000002.000000c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.957197323000001, 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057555969999999998, 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 197 2024-09-16T01:51:06.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107141+0000", 2024-09-16T01:51:06.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148704+0000", 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041563044 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148738+0000", 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4480999999999997e-05 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148754+0000", 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6021000000000001e-05 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148909+0000", 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154735 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149071+0000", 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016181600000000001 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149226+0000", 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015550000000000001 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.158210+0000", 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089836529999999994 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164352+0000", 2024-09-16T01:51:06.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061416810000000004 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164426+0000", 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4192999999999999e-05 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164695+0000", 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026864800000000002 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:449 3.28 3:144fb21c:::10000000002.000000be:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.3732235900000003, 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05217542, 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 449 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691115+0000", 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.056e-06 2024-09-16T01:51:06.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726045+0000", 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034929984999999997 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726077+0000", 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1390000000000003e-05 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726093+0000", 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6047000000000001e-05 2024-09-16T01:51:06.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726207+0000", 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011452599999999999 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726335+0000", 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012742299999999999 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726492+0000", 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015772599999999999 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.740613+0000", 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014120071999999999 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743157+0000", 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025441550000000002 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743229+0000", 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2102999999999997e-05 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743288+0000", 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8937000000000002e-05 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.149 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.2837033050000004, 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.150 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:06.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:06.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.2155986639999998, 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.156 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:06.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:06.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.0731389990000002, 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:06.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:06.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:497 3.30 3:0c580b6a:::10000000002.000000ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.170 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.872663588, 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052947199, 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 497 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191674+0000", 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.575e-06 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220734+0000", 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029059623999999999 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220763+0000", 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9099e-05 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220777+0000", 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4299e-05 2024-09-16T01:51:06.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220897+0000", 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011993999999999999 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.221085+0000", 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000188457 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.221327+0000", 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024153299999999999 2024-09-16T01:51:06.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.233648+0000", 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012321544 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244466+0000", 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010817795 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244519+0000", 2024-09-16T01:51:06.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3316999999999998e-05 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244620+0000", 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100016 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:499 3.2f 3:f78980a5:::10000000002.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:06.176 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.8414357490000004, 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055062123999999997, 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 499 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222902+0000", 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261223+0000", 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038321506999999998 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261257+0000", 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3526999999999999e-05 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261272+0000", 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4748e-05 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261399+0000", 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012685100000000001 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261500+0000", 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101556 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261663+0000", 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016259900000000001 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.270356+0000", 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086932399999999996 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277838+0000", 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074817470000000004 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277906+0000", 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8454000000000006e-05 2024-09-16T01:51:06.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277962+0000", 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6075000000000001e-05 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7256727759999997, 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:06.182 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:06.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:06.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:06.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:06.189 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:06.191 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:06.191 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:06.190+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:06.291 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:06.290+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:06.309 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:06.309 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:06.309 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:06.309 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:06.318 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:06.392 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:06.391+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 re-readers = 2713355.00 kB/sec 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 re-readers = 2708827.58 kB/sec 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 2713355.00 kB/sec 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 2713355.00 kB/sec 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 2713355.00 kB/sec 2024-09-16T01:51:06.437 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:06.464 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:06.464 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:06.464 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:06.464 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:06.464 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:06.473 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:06.517 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:06.516+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:06.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:06.580 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:06.580 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5 3.7 3:e001da36:::10000000002.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.317566043999999, 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044893833000000001, 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.581 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260358+0000", 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.672e-06 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297299+0000", 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036941017999999999 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297321+0000", 2024-09-16T01:51:06.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2223000000000001e-05 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297332+0000", 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.058e-05 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297462+0000", 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013043399999999999 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.584 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297624+0000", 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000162263 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297768+0000", 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014397800000000001 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.303867+0000", 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060983820000000003 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305178+0000", 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013110330000000001 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305214+0000", 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6380000000000001e-05 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305250+0000", 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.587e-05 2024-09-16T01:51:06.586 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:13 3.2 3:438a76aa:::10000000002.0000000a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.223713714999999, 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045857943999999998, 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.587 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 13 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354210+0000", 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.203e-06 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388636+0000", 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034425859000000003 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388673+0000", 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7004e-05 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388693+0000", 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0194999999999999e-05 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388771+0000", 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8737000000000002e-05 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.590 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388904+0000", 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013215899999999999 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.389016+0000", 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112642 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.392245+0000", 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032287370000000002 2024-09-16T01:51:06.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.399956+0000", 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0077109190000000001 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400015+0000", 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9119000000000001e-05 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400066+0000", 2024-09-16T01:51:06.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1369999999999998e-05 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:30 3.7 3:e33cd625:::10000000002.0000001b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.093704789, 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035807259000000001, 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.593 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 30 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:51:06.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484219+0000", 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512447+0000", 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028228652 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512476+0000", 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8283e-05 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512489+0000", 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3049e-05 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512566+0000", 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7420000000000001e-05 2024-09-16T01:51:06.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512720+0000", 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015377000000000001 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512891+0000", 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017147199999999999 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.516819+0000", 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003927458 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519938+0000", 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031196489999999999 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519989+0000", 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0380000000000002e-05 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.520025+0000", 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5839e-05 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.028301251999999, 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.599 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:06.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:06.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.781267671, 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:06.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:06.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.722108585000001, 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.612 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:06.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:82 3.2b 3:d50470f2:::10000000002.0000004f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.615156747, 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047169015000000002, 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 82 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962768+0000", 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3269999999999999e-06 2024-09-16T01:51:06.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002536+0000", 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039768464000000003 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002551+0000", 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002558+0000", 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8730000000000001e-06 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002630+0000", 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2025000000000002e-05 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002751+0000", 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121197 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002871+0000", 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011995 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.006288+0000", 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003417032 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009828+0000", 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00353996 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009879+0000", 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0473999999999998e-05 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009934+0000", 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5872999999999999e-05 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.623 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:114 3.8 3:10128d86:::10000000002.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.309268396, 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038796303999999997, 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 114 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268656+0000", 2024-09-16T01:51:06.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.858e-06 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303790+0000", 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035134060000000002 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303806+0000", 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6611000000000001e-05 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303814+0000", 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4329999999999998e-06 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303869+0000", 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5639e-05 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303979+0000", 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110043 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.304078+0000", 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9006999999999996e-05 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.306722+0000", 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002643074 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307380+0000", 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00065801399999999995 2024-09-16T01:51:06.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307414+0000", 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4082000000000003e-05 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307450+0000", 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6483000000000003e-05 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:132 3.2b 3:d52309a9:::10000000002.00000081:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.108231271999999, 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034693794, 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 132 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469692+0000", 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.516e-06 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496852+0000", 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027159651999999999 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496868+0000", 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5869999999999999e-05 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496875+0000", 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.391e-06 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496952+0000", 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6320000000000001e-05 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497109+0000", 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015718700000000001 2024-09-16T01:51:06.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497263+0000", 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015406799999999999 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.500555+0000", 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032925720000000001 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504317+0000", 2024-09-16T01:51:06.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037611150000000002 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504351+0000", 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4214000000000001e-05 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504385+0000", 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3889e-05 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.635 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.810530272999999, 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:06.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:06.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:06.641 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.766270043, 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.642 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:06.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:06.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.735770222999999, 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.648 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:06.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:06.653 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.694214701, 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.654 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:06.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:06.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:183 3.1b 3:db82751e:::10000000002.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.606828085, 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040945861, 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 183 2024-09-16T01:51:06.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971104+0000", 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8719999999999997e-06 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999810+0000", 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028705819 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999832+0000", 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2560000000000001e-05 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999847+0000", 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4956e-05 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999905+0000", 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8084000000000002e-05 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000017+0000", 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111222 2024-09-16T01:51:06.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000113+0000", 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6131999999999994e-05 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.002657+0000", 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025443089999999998 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011949+0000", 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0092920350000000006 2024-09-16T01:51:06.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011989+0000", 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9987999999999997e-05 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.012040+0000", 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0884000000000003e-05 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.556168007, 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:06.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:06.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:06.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.4372082800000001, 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.672 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:06.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.4024119810000002, 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:06.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:06.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:06.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:06.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:489 3.7 3:e3fe2874:::10000000002.000000e6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.686 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.220274+0000", 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.3576480850000001, 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045008859999999998, 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 489 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220274+0000", 2024-09-16T01:51:06.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220274+0000", 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220276+0000", 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4640000000000001e-06 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258890+0000", 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038614895000000003 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258911+0000", 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0965e-05 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258924+0000", 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2282e-05 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259005+0000", 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1768999999999996e-05 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259151+0000", 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014523699999999999 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259306+0000", 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015550500000000001 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265082+0000", 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057754939999999999 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265201+0000", 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119397 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265249+0000", 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7951000000000002e-05 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265283+0000", 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3900999999999999e-05 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.3189630279999998, 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.692 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:06.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:06.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:507 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.307551+0000", 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.2703714359999996, 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040029878999999997, 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:06.698 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 507 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.307551+0000", 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.307551+0000", 2024-09-16T01:51:06.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.307552+0000", 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.505e-06 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335233+0000", 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027680284999999999 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335251+0000", 2024-09-16T01:51:06.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8958e-05 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335289+0000", 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7333000000000003e-05 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335358+0000", 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9493999999999996e-05 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335427+0000", 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8984000000000002e-05 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335525+0000", 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7819000000000007e-05 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.338819+0000", 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032943130000000001 2024-09-16T01:51:06.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.347505+0000", 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086852309999999999 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.347541+0000", 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6266000000000001e-05 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.347581+0000", 2024-09-16T01:51:06.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9691000000000003e-05 2024-09-16T01:51:06.704 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.704 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.704 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.704 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:06.704 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:06.704 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:06.706 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:06.706 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:06.704+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:06.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.698+0000 7f9aa0800640 1 Processor -- start 2024-09-16T01:51:06.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.698+0000 7f9aa0800640 1 -- start start 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.699+0000 7f9aa0800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a98151240 0x7f9a980c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.699+0000 7f9aa0800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a98157850 con 0x7f9a98072520 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.699+0000 7f9aa0800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a98157b30 con 0x7f9a98150e60 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.699+0000 7f9aa0800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9a98072a70 con 0x7f9a98151240 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9de00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9a98150e60 0x7f9a9806ea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50700/0 (socket says 172.21.15.146:50700) 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9de00640 1 -- 172.21.15.146:0/3783585182 learned_addr learned my addr 172.21.15.146:0/3783585182 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9e800640 1 --1- 172.21.15.146:0/3783585182 >> v1:172.21.15.146:6789/0 conn(0x7f9a98072520 0x7f9a981616b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45660/0 (socket says 172.21.15.146:45660) 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3783585182 >> v2:172.21.15.146:3300/0 conn(0x7f9a98151240 0x7f9a980c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9d400640 1 -- 172.21.15.146:0/3783585182 >> v1:172.21.15.161:6789/0 conn(0x7f9a98150e60 legacy=0x7f9a9806ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9d400640 1 -- 172.21.15.146:0/3783585182 >> v1:172.21.15.146:6789/0 conn(0x7f9a98072520 legacy=0x7f9a981616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.700+0000 7f9a9d400640 1 -- 172.21.15.146:0/3783585182 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a98072cc0 con 0x7f9a98151240 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.701+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3783585182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98151240 0x7f9a980c6ab0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097a0 tx=0x7f9a8802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1f09da0f051c1d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.701+0000 7f9a9ca00640 1 -- 172.21.15.146:0/3783585182 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88037070 con 0x7f9a98151240 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.701+0000 7f9a9ca00640 1 -- 172.21.15.146:0/3783585182 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88032440 con 0x7f9a98151240 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.701+0000 7f9a9ca00640 1 -- 172.21.15.146:0/3783585182 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88032810 con 0x7f9a98151240 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 -- 172.21.15.146:0/3783585182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98151240 msgr2=0x7f9a980c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3783585182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98151240 0x7f9a980c6ab0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097a0 tx=0x7f9a8802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 -- 172.21.15.146:0/3783585182 shutdown_connections 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3783585182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98151240 0x7f9a980c6ab0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 -- 172.21.15.146:0/3783585182 >> 172.21.15.146:0/3783585182 conn(0x7f9a9806bc10 msgr2=0x7f9a9806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 -- 172.21.15.146:0/3783585182 shutdown_connections 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.702+0000 7f9aa0800640 1 -- 172.21.15.146:0/3783585182 wait complete. 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.704+0000 7f9aa0800640 1 Processor -- start 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.704+0000 7f9aa0800640 1 -- start start 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9aa0800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98072520 0x7f9a98143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9aa0800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 0x7f9a980c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9aa0800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a98151240 0x7f9a98140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9aa0800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9806f180 con 0x7f9a98072520 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9aa0800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a98073310 con 0x7f9a98151240 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9aa0800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9806f300 con 0x7f9a98150e60 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98072520 0x7f9a98143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a98151240 0x7f9a98140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 0x7f9a980c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 0x7f9a980c2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44714/0 (socket says 172.21.15.146:44714) 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 -- 172.21.15.146:0/3071330191 learned_addr learned my addr 172.21.15.146:0/3071330191 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a98151240 msgr2=0x7f9a98140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a98151240 0x7f9a98140bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98072520 msgr2=0x7f9a98143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a98072520 0x7f9a98143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.705+0000 7f9a9d400640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a981435c0 con 0x7f9a98150e60 2024-09-16T01:51:06.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.706+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 0x7f9a980c2540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9a88000c00 tx=0x7f9a880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.706+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88037040 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.706+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a88042070 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.706+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88030bd0 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.706+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a980c2a80 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.706+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a980ca200 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.707+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9a980c59b0 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.708+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9a88047460 con 0x7f9a98150e60 2024-09-16T01:51:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.709+0000 7f9a92000640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7c06c2b0 0x7f9a7c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.709+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 5741+0+0 (secure 0 0 0) 0x7f9a880abf90 con 0x7f9a98150e60 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.709+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7c06c2b0 0x7f9a7c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.709+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v62) ==== 54+0+16 (secure 0 0 0) 0x7f9a980c59b0 con 0x7f9a98150e60 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.709+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a9806f480 0x7f9a980ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.709+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a98071eb0 con 0x7f9a9806f480 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.710+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7c06c2b0 0x7f9a7c06e770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c005880 tx=0x7f9a8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.710+0000 7f9a9e800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a9806f480 0x7f9a980ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.710+0000 7f9a9e800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a9806f480 0x7f9a980ca400 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.711+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a98071eb0 con 0x7f9a9806f480 2024-09-16T01:51:06.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.725+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a98150660 con 0x7f9a9806f480 2024-09-16T01:51:06.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.727+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a98150660 con 0x7f9a9806f480 2024-09-16T01:51:06.729 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:06.729 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:06.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.727+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a9806f860 0x7f9a981f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.727+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9a9814e590 con 0x7f9a9806f860 2024-09-16T01:51:06.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.727+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a9806f860 0x7f9a981f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.728+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a9806f860 0x7f9a981f1540 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.729+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a9814e590 con 0x7f9a9806f860 2024-09-16T01:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.743+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a9814f860 con 0x7f9a9806f860 2024-09-16T01:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.745+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a9814f860 con 0x7f9a9806f860 2024-09-16T01:51:06.746 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.745+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a981f4760 0x7f9a981f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.745+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9a9806ed50 con 0x7f9a981f4760 2024-09-16T01:51:06.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.745+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a981f4760 0x7f9a981f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.745+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a981f4760 0x7f9a981f4b80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.746+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a9806ed50 con 0x7f9a981f4760 2024-09-16T01:51:06.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.759+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a98062630 con 0x7f9a981f4760 2024-09-16T01:51:06.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.761+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a98062630 con 0x7f9a981f4760 2024-09-16T01:51:06.762 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:06.763 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:06.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.761+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a98063110 0x7f9a980c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.761+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9a98062630 con 0x7f9a98063110 2024-09-16T01:51:06.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.762+0000 7f9a9e800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a98063110 0x7f9a980c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.762+0000 7f9a9e800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a98063110 0x7f9a980c4830 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.763+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a98062630 con 0x7f9a98063110 2024-09-16T01:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.776+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a98062630 con 0x7f9a98063110 2024-09-16T01:51:06.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.778+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a98062630 con 0x7f9a98063110 2024-09-16T01:51:06.779 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:06.779 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:06.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.778+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a9802c640 0x7f9a9802cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.778+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9a98062630 con 0x7f9a9802c640 2024-09-16T01:51:06.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.778+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a9802c640 0x7f9a9802cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.779+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a9802c640 0x7f9a9802cab0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.780+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a98062630 con 0x7f9a9802c640 2024-09-16T01:51:06.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.793+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a98062630 con 0x7f9a9802c640 2024-09-16T01:51:06.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.795+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a98062630 con 0x7f9a9802c640 2024-09-16T01:51:06.796 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:06.797 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:06.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.795+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a9802d500 0x7f9a982075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.795+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9a98063570 con 0x7f9a9802d500 2024-09-16T01:51:06.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.795+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a9802d500 0x7f9a982075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.796+0000 7f9a9d400640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a9802d500 0x7f9a982075e0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.796+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a98063570 con 0x7f9a9802d500 2024-09-16T01:51:06.805 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:06.805+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.809+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a98063570 con 0x7f9a9802d500 2024-09-16T01:51:06.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.811+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a98063570 con 0x7f9a9802d500 2024-09-16T01:51:06.812 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.811+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a98208030 0x7f9a9820a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.811+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9a980c5780 con 0x7f9a98208030 2024-09-16T01:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.812+0000 7f9a9e800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a98208030 0x7f9a9820a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.812+0000 7f9a9e800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a98208030 0x7f9a9820a420 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.813+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a980c5780 con 0x7f9a98208030 2024-09-16T01:51:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.827+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a980c4dd0 con 0x7f9a98208030 2024-09-16T01:51:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.829+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a980c4dd0 con 0x7f9a98208030 2024-09-16T01:51:06.830 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.829+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a9820ae70 0x7f9a9820d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.829+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9a98210030 con 0x7f9a9820ae70 2024-09-16T01:51:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.830+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a9820ae70 0x7f9a9820d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.830+0000 7f9a9de00640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a9820ae70 0x7f9a9820d260 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:06.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.831+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a98210030 con 0x7f9a9820ae70 2024-09-16T01:51:06.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:51:06.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.846+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a9820d7a0 con 0x7f9a9820ae70 2024-09-16T01:51:06.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9a92000640 1 -- 172.21.15.146:0/3071330191 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a9820d7a0 con 0x7f9a9820ae70 2024-09-16T01:51:06.849 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a9806f480 msgr2=0x7f9a980ca400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a9806f480 0x7f9a980ca400 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a9806f860 msgr2=0x7f9a981f1540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a9806f860 0x7f9a981f1540 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a981f4760 msgr2=0x7f9a981f4b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.848+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a981f4760 0x7f9a981f4b80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a98063110 msgr2=0x7f9a980c4830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a98063110 0x7f9a980c4830 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a9802c640 msgr2=0x7f9a9802cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a9802c640 0x7f9a9802cab0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a9802d500 msgr2=0x7f9a982075e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a9802d500 0x7f9a982075e0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a98208030 msgr2=0x7f9a9820a420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a98208030 0x7f9a9820a420 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a9820ae70 msgr2=0x7f9a9820d260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a9820ae70 0x7f9a9820d260 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9a9e800640 1 -- 172.21.15.146:0/3071330191 reap_dead start 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9a9e800640 1 -- 172.21.15.146:0/3071330191 reap_dead start 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9a9e800640 1 -- 172.21.15.146:0/3071330191 reap_dead start 2024-09-16T01:51:06.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9a9e800640 1 -- 172.21.15.146:0/3071330191 reap_dead start 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9a9e800640 1 -- 172.21.15.146:0/3071330191 reap_dead start 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7c06c2b0 msgr2=0x7f9a7c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7c06c2b0 0x7f9a7c06e770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c005880 tx=0x7f9a8c009040 comp rx=0 tx=0).stop 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 msgr2=0x7f9a980c2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 0x7f9a980c2540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9a88000c00 tx=0x7f9a880046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 shutdown_connections 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9a7c06c2b0 0x7f9a7c06e770 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 --2- 172.21.15.146:0/3071330191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a98150e60 0x7f9a980c2540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 >> 172.21.15.146:0/3071330191 conn(0x7f9a9806bc10 msgr2=0x7f9a9814fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 shutdown_connections 2024-09-16T01:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.849+0000 7f9aa0800640 1 -- 172.21.15.146:0/3071330191 wait complete. 2024-09-16T01:51:06.855 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:06.855 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:06.864 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:06.908 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:06.907+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.995+0000 7f1d3d600640 1 Processor -- start 2024-09-16T01:51:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.996+0000 7f1d3d600640 1 -- start start 2024-09-16T01:51:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d3d600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1d38152850 0x7f1d38158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d3d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d3d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d38072f50 con 0x7f1d38158bb0 2024-09-16T01:51:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d3d600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1d38073210 con 0x7f1d38152850 2024-09-16T01:51:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d3d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3815cb50 con 0x7f1d380c6bb0 2024-09-16T01:51:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d2e000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1d38158bb0 0x7f1d380c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45664/0 (socket says 172.21.15.146:45664) 2024-09-16T01:51:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d36a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1d38152850 0x7f1d38158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d2e000640 1 -- 172.21.15.146:0/473075546 learned_addr learned my addr 172.21.15.146:0/473075546 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:06.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d37400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.998+0000 7f1d37400640 1 -- 172.21.15.146:0/473075546 >> v2:172.21.15.146:3300/0 conn(0x7f1d38152850 msgr2=0x7f1d38158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.998+0000 7f1d37400640 1 --2- 172.21.15.146:0/473075546 >> v2:172.21.15.146:3300/0 conn(0x7f1d38152850 0x7f1d38158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.997+0000 7f1d36000640 1 -- 172.21.15.146:0/473075546 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2656767746 0 0) 0x7f1d38072f50 con 0x7f1d38158bb0 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.998+0000 7f1d37400640 1 -- 172.21.15.146:0/473075546 >> v1:172.21.15.146:6789/0 conn(0x7f1d38158bb0 legacy=0x7f1d380c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.998+0000 7f1d37400640 1 -- 172.21.15.146:0/473075546 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3815bd90 con 0x7f1d380c6bb0 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.998+0000 7f1d36a00640 1 --2- 172.21.15.146:0/473075546 >> v2:172.21.15.146:3300/0 conn(0x7f1d38152850 0x7f1d38158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d37400640 1 --2- 172.21.15.146:0/473075546 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c9700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009d90 tx=0x7f1d3002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6900809a1bf5ae77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d36000640 1 -- 172.21.15.146:0/473075546 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30007740 con 0x7f1d380c6bb0 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d36000640 1 -- 172.21.15.146:0/473075546 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d300078e0 con 0x7f1d380c6bb0 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d36000640 1 -- 172.21.15.146:0/473075546 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30007cb0 con 0x7f1d380c6bb0 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 -- 172.21.15.146:0/473075546 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 msgr2=0x7f1d380c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 --2- 172.21.15.146:0/473075546 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c9700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009d90 tx=0x7f1d3002a660 comp rx=0 tx=0).stop 2024-09-16T01:51:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 -- 172.21.15.146:0/473075546 shutdown_connections 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 --2- 172.21.15.146:0/473075546 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c9700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 --2- 172.21.15.146:0/473075546 >> v2:172.21.15.146:3300/0 conn(0x7f1d38152850 0x7f1d38158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 -- 172.21.15.146:0/473075546 >> 172.21.15.146:0/473075546 conn(0x7f1d3806f3c0 msgr2=0x7f1d3806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:06.999+0000 7f1d3d600640 1 -- 172.21.15.146:0/473075546 shutdown_connections 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.000+0000 7f1d3d600640 1 -- 172.21.15.146:0/473075546 wait complete. 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.001+0000 7f1d3d600640 1 Processor -- start 2024-09-16T01:51:07.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.001+0000 7f1d3d600640 1 -- start start 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d3d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d3d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d38152850 0x7f1d38159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d3d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d38158bb0 0x7f1d3815a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d3d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3815c630 con 0x7f1d38158bb0 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d3d600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3815c350 con 0x7f1d38152850 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d3d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d38070200 con 0x7f1d380c6bb0 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d36a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d2e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d38152850 0x7f1d38159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d36a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44728/0 (socket says 172.21.15.146:44728) 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d36a00640 1 -- 172.21.15.146:0/3869482163 learned_addr learned my addr 172.21.15.146:0/3869482163 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.002+0000 7f1d37400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d38158bb0 0x7f1d3815a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d36a00640 1 -- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d38152850 msgr2=0x7f1d38159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d36a00640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d38152850 0x7f1d38159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d36a00640 1 -- 172.21.15.146:0/3869482163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d38158bb0 msgr2=0x7f1d3815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d36a00640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d38158bb0 0x7f1d3815a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d36a00640 1 -- 172.21.15.146:0/3869482163 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d381f0d70 con 0x7f1d380c6bb0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d36a00640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c4aa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800ec10 tx=0x7f1d2800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28017070 con 0x7f1d380c6bb0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2800cc30 con 0x7f1d380c6bb0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d381f1050 con 0x7f1d380c6bb0 2024-09-16T01:51:07.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28013470 con 0x7f1d380c6bb0 2024-09-16T01:51:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.003+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d381f1610 con 0x7f1d380c6bb0 2024-09-16T01:51:07.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.004+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d380718a0 con 0x7f1d380c6bb0 2024-09-16T01:51:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.005+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1d2801a070 con 0x7f1d380c6bb0 2024-09-16T01:51:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.006+0000 7f1d2f400640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1d1406c2b0 0x7f1d1406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.006+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 5741+0+0 (secure 0 0 0) 0x7f1d2808b6b0 con 0x7f1d380c6bb0 2024-09-16T01:51:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.006+0000 7f1d2e000640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1d1406c2b0 0x7f1d1406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:07.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.007+0000 7f1d2e000640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1d1406c2b0 0x7f1d1406e770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d3813b5d0 tx=0x7f1d20007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:07.009 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:07.007+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:07.009 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.010 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.010+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1d280597a0 con 0x7f1d380c6bb0 2024-09-16T01:51:07.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:07.109 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:07.108+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:07.124 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.125 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:07.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.193+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f1d381f1850 con 0x7f1d380c6bb0 2024-09-16T01:51:07.209 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:07.208+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:07.238 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.238 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.246 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:07.310 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:07.309+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:07.351 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.351 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.359 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:07.411 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:07.410+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:07.504 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.505 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.512 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:07.511+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:07.513 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:07.613 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:07.612+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:07.616 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.617 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.625 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:07.714 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:07.713+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:07.769 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.769 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.778 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:07.821 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:07.820+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:07.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.829+0000 7f1d2f400640 1 -- 172.21.15.146:0/3869482163 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v63) ==== 97+0+0 (secure 0 0 0) 0x7f1d2805daa0 con 0x7f1d380c6bb0 2024-09-16T01:51:07.830 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:51:07.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1d1406c2b0 msgr2=0x7f1d1406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1d1406c2b0 0x7f1d1406e770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d3813b5d0 tx=0x7f1d20007de0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 msgr2=0x7f1d380c4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c4aa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800ec10 tx=0x7f1d2800c520 comp rx=0 tx=0).stop 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 shutdown_connections 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1d1406c2b0 0x7f1d1406e770 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d38158bb0 0x7f1d3815a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d38152850 0x7f1d38159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 --2- 172.21.15.146:0/3869482163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d380c6bb0 0x7f1d380c4aa0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 >> 172.21.15.146:0/3869482163 conn(0x7f1d3806f3c0 msgr2=0x7f1d3806da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 shutdown_connections 2024-09-16T01:51:07.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:07.831+0000 7f1d3d600640 1 -- 172.21.15.146:0/3869482163 wait complete. 2024-09-16T01:51:07.884 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:07.885 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:07.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:07.940 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:07.939+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:08.018 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:08.017+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:08.052 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:08.052 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:08.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:08.151 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:08.150+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:08.220 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:08.219+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:08.266 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:08.267 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:08.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:08.320 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:08.319+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:08.421 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:08.420+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:08.477 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:08.477 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:08.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:08.522 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:08.521+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:08.623 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:08.622+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:08.648 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:08.648 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:08.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:08.724 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:08.723+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:08.818 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:08.818 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:08.825 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:08.824+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:08.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:08.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:08.926 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:08.925+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:09.027 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:09.026+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.029+0000 7f757a800640 1 Processor -- start 2024-09-16T01:51:09.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.029+0000 7f757a800640 1 -- start start 2024-09-16T01:51:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.030+0000 7f757a800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75740d0490 0x7f75740c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.030+0000 7f757a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7574154da0 0x7f75740c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f757a800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f757415e220 con 0x7f75740d0870 2024-09-16T01:51:09.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f757a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f75740c6160 con 0x7f7574154da0 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f757a800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f757415e500 con 0x7f75740d0490 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7574154da0 0x7f75740c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7574154da0 0x7f75740c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40930/0 (socket says 172.21.15.146:40930) 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 -- 172.21.15.146:0/665333246 learned_addr learned my addr 172.21.15.146:0/665333246 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579400640 1 --2- 172.21.15.146:0/665333246 >> v2:172.21.15.161:3300/0 conn(0x7f75740d0490 0x7f75740c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 -- 172.21.15.146:0/665333246 >> v2:172.21.15.161:3300/0 conn(0x7f75740d0490 msgr2=0x7f75740c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 --2- 172.21.15.146:0/665333246 >> v2:172.21.15.161:3300/0 conn(0x7f75740d0490 0x7f75740c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 -- 172.21.15.146:0/665333246 >> v1:172.21.15.161:6789/0 conn(0x7f75740d0870 legacy=0x7f7574154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7579e00640 1 -- 172.21.15.146:0/665333246 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757415e7e0 con 0x7f7574154da0 2024-09-16T01:51:09.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.031+0000 7f7573e00640 1 -- 172.21.15.146:0/665333246 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052823818 0 0) 0x7f757415e220 con 0x7f75740d0870 2024-09-16T01:51:09.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.032+0000 7f7579e00640 1 --2- 172.21.15.146:0/665333246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7574154da0 0x7f75740c6b30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f75740cb260 tx=0x7f756c02a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463e70e7c1d46811 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.032+0000 7f7573e00640 1 -- 172.21.15.146:0/665333246 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c036070 con 0x7f7574154da0 2024-09-16T01:51:09.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f7573e00640 1 -- 172.21.15.146:0/665333246 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f756c031600 con 0x7f7574154da0 2024-09-16T01:51:09.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f7573e00640 1 -- 172.21.15.146:0/665333246 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c0319f0 con 0x7f7574154da0 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 -- 172.21.15.146:0/665333246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7574154da0 msgr2=0x7f75740c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 --2- 172.21.15.146:0/665333246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7574154da0 0x7f75740c6b30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f75740cb260 tx=0x7f756c02a7c0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 -- 172.21.15.146:0/665333246 shutdown_connections 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 --2- 172.21.15.146:0/665333246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7574154da0 0x7f75740c6b30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 --2- 172.21.15.146:0/665333246 >> v2:172.21.15.161:3300/0 conn(0x7f75740d0490 0x7f75740c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 -- 172.21.15.146:0/665333246 >> 172.21.15.146:0/665333246 conn(0x7f757406d6f0 msgr2=0x7f757406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 -- 172.21.15.146:0/665333246 shutdown_connections 2024-09-16T01:51:09.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.033+0000 7f757a800640 1 -- 172.21.15.146:0/665333246 wait complete. 2024-09-16T01:51:09.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.035+0000 7f757a800640 1 Processor -- start 2024-09-16T01:51:09.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.035+0000 7f757a800640 1 -- start start 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f757a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 0x7f7574155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f757a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75740d0870 0x7f75741f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f757a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7574154da0 0x7f757415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f757a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f757415dec0 con 0x7f75740d0870 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f757a800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f757415eda0 con 0x7f75740d0490 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f757a800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75740c51f0 con 0x7f7574154da0 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 0x7f7574155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7578a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75740d0870 0x7f75741f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 0x7f7574155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56966/0 (socket says 172.21.15.146:56966) 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7574154da0 0x7f757415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 -- 172.21.15.146:0/2431859432 learned_addr learned my addr 172.21.15.146:0/2431859432 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7574154da0 msgr2=0x7f757415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7574154da0 0x7f757415a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75740d0870 msgr2=0x7f75741f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75740d0870 0x7f75741f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7579400640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7574157dd0 con 0x7f75740d0490 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.036+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75740d0870 0x7f75741f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.037+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 0x7f7574155610 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f756800e9e0 tx=0x7f756800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.037+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7568017070 con 0x7f75740d0490 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.037+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f756800c9d0 con 0x7f75740d0490 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.037+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75741580b0 con 0x7f75740d0490 2024-09-16T01:51:09.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.037+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756800cdc0 con 0x7f75740d0490 2024-09-16T01:51:09.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.037+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75741f14c0 con 0x7f75740d0490 2024-09-16T01:51:09.040 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:09.040 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:09.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.038+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f757406e6b0 con 0x7f75740d0490 2024-09-16T01:51:09.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.039+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f756801a070 con 0x7f75740d0490 2024-09-16T01:51:09.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.040+0000 7f7571600640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f755406c2b0 0x7f755406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.040+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5741+0+0 (secure 0 0 0) 0x7f756808b550 con 0x7f75740d0490 2024-09-16T01:51:09.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.041+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v63) ==== 54+0+16 (secure 0 0 0) 0x7f757406e6b0 con 0x7f75740d0490 2024-09-16T01:51:09.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.041+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f755406c2b0 0x7f755406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.041+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7574159860 0x7f7574159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.041+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75740cdb10 con 0x7f7574159860 2024-09-16T01:51:09.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.041+0000 7f7579e00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7574159860 0x7f7574159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.041+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f755406c2b0 0x7f755406e770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f75740b2ab0 tx=0x7f7560009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.042+0000 7f7579e00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7574159860 0x7f7574159c40 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.042+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75740cdb10 con 0x7f7574159860 2024-09-16T01:51:09.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:09.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.057+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7574154690 con 0x7f7574159860 2024-09-16T01:51:09.062 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.059+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7574154690 con 0x7f7574159860 2024-09-16T01:51:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.059+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75741f4060 0x7f75741f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.059+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f75741f4980 con 0x7f75741f4060 2024-09-16T01:51:09.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.060+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75741f4060 0x7f75741f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.060+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75741f4060 0x7f75741f4440 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.061+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75741f4980 con 0x7f75741f4060 2024-09-16T01:51:09.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.076+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75740cc3f0 con 0x7f75741f4060 2024-09-16T01:51:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.078+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75740cc3f0 con 0x7f75741f4060 2024-09-16T01:51:09.079 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.078+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f757414e010 0x7f757414e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.078+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f75740cd830 con 0x7f757414e010 2024-09-16T01:51:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.078+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f757414e010 0x7f757414e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.079+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f757414e010 0x7f757414e430 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.080+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75740cd830 con 0x7f757414e010 2024-09-16T01:51:09.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.093+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75740cb470 con 0x7f757414e010 2024-09-16T01:51:09.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.095+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75740cb470 con 0x7f757414e010 2024-09-16T01:51:09.096 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:09.097 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:09.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.095+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7574062f00 0x7f75740ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.095+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7574063320 con 0x7f7574062f00 2024-09-16T01:51:09.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.095+0000 7f7579e00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7574062f00 0x7f75740ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.096+0000 7f7579e00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7574062f00 0x7f75740ce090 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.097+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7574063320 con 0x7f7574062f00 2024-09-16T01:51:09.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.111+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75740ce5d0 con 0x7f7574062f00 2024-09-16T01:51:09.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.113+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75740ce5d0 con 0x7f7574062f00 2024-09-16T01:51:09.114 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:09.114 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:09.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.113+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75740ceae0 0x7f757402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.113+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f75740cef20 con 0x7f75740ceae0 2024-09-16T01:51:09.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.114+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75740ceae0 0x7f757402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.114+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75740ceae0 0x7f757402c640 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.115+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75740cef20 con 0x7f75740ceae0 2024-09-16T01:51:09.128 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:09.127+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:09.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.128+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f75740cef20 con 0x7f75740ceae0 2024-09-16T01:51:09.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.131+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75740cef20 con 0x7f75740ceae0 2024-09-16T01:51:09.132 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.131+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f757402d090 0x7f757402d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.131+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f757402da10 con 0x7f757402d090 2024-09-16T01:51:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.131+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f757402d090 0x7f757402d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.132+0000 7f7578a00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f757402d090 0x7f757402d4d0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.133+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f757402da10 con 0x7f757402d090 2024-09-16T01:51:09.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.146+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f757402da10 con 0x7f757402d090 2024-09-16T01:51:09.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.149+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f757402da10 con 0x7f757402d090 2024-09-16T01:51:09.150 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.149+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7574207600 0x7f75742099f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.149+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f757420c7c0 con 0x7f7574207600 2024-09-16T01:51:09.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.149+0000 7f7579e00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7574207600 0x7f75742099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.150+0000 7f7579e00640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7574207600 0x7f75742099f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.151+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f757420c7c0 con 0x7f7574207600 2024-09-16T01:51:09.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.164+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7574209f30 con 0x7f7574207600 2024-09-16T01:51:09.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.166+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7574209f30 con 0x7f7574207600 2024-09-16T01:51:09.167 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.167+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f757420a650 0x7f757420aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.167+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f757420fb90 con 0x7f757420a650 2024-09-16T01:51:09.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.167+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f757420a650 0x7f757420aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.167+0000 7f7579400640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f757420a650 0x7f757420aab0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.168+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f757420fb90 con 0x7f757420a650 2024-09-16T01:51:09.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.182+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f757420b100 con 0x7f757420a650 2024-09-16T01:51:09.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f7571600640 1 -- 172.21.15.146:0/2431859432 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f757420b100 con 0x7f757420a650 2024-09-16T01:51:09.185 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7574159860 msgr2=0x7f7574159c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7574159860 0x7f7574159c40 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75741f4060 msgr2=0x7f75741f4440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75741f4060 0x7f75741f4440 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f757414e010 msgr2=0x7f757414e430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f757414e010 0x7f757414e430 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7574062f00 msgr2=0x7f75740ce090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7574062f00 0x7f75740ce090 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75740ceae0 msgr2=0x7f757402c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75740ceae0 0x7f757402c640 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f757402d090 msgr2=0x7f757402d4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f757402d090 0x7f757402d4d0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7574207600 msgr2=0x7f75742099f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7574207600 0x7f75742099f0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f757420a650 msgr2=0x7f757420aab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f757420a650 0x7f757420aab0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f7579e00640 1 -- 172.21.15.146:0/2431859432 reap_dead start 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f7579e00640 1 -- 172.21.15.146:0/2431859432 reap_dead start 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f7579e00640 1 -- 172.21.15.146:0/2431859432 reap_dead start 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f7579e00640 1 -- 172.21.15.146:0/2431859432 reap_dead start 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f7579e00640 1 -- 172.21.15.146:0/2431859432 reap_dead start 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f755406c2b0 msgr2=0x7f755406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f755406c2b0 0x7f755406e770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f75740b2ab0 tx=0x7f7560009040 comp rx=0 tx=0).stop 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 msgr2=0x7f7574155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.184+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 0x7f7574155610 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f756800e9e0 tx=0x7f756800eea0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.185+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 shutdown_connections 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.185+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f755406c2b0 0x7f755406e770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.185+0000 7f757a800640 1 --2- 172.21.15.146:0/2431859432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75740d0490 0x7f7574155610 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:09.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.185+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 >> 172.21.15.146:0/2431859432 conn(0x7f757406d6f0 msgr2=0x7f75740c8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.185+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 shutdown_connections 2024-09-16T01:51:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:09.185+0000 7f757a800640 1 -- 172.21.15.146:0/2431859432 wait complete. 2024-09-16T01:51:09.221 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:09.221 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:09.221 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:09.229 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:09.228+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:09.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:09.330 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:09.329+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:09.431 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:09.431 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:09.431 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:09.431 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:09.431 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:09.431 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:09.430+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:09.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:09.532 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:09.531+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:09.633 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:09.632+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:09.644 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:09.645 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:09.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:09.645 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:09.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.645 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:09.646 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:09.646 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.386497469999998, 2024-09-16T01:51:09.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:51:09.646 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.646 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.646 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.647 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:51:09.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.647 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:09.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:09.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:51:09.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:51:09.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:51:09.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:51:09.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:51:09.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:51:09.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:51:09.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:51:09.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:51:09.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:09.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:51:09.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:51:09.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:51:09.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.073551694999999, 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:09.658 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:09.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:09.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:09.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:42 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.032469227, 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051426843, 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 42 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608928+0000", 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5689999999999999e-06 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648858+0000", 2024-09-16T01:51:09.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039929485000000001 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648879+0000", 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1172e-05 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648889+0000", 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0512e-05 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648975+0000", 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5473e-05 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649103+0000", 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012862999999999999 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649230+0000", 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126561 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.655882+0000", 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066518819999999996 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660228+0000", 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043460699999999996 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660293+0000", 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5050999999999998e-05 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660353+0000", 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0438e-05 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.920006012999998, 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:09.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:09.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:09.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:67 3.16 3:6a386151:::10000000002.00000040:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.805488181000001, 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046477360000000002, 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 67 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835910+0000", 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.872997+0000", 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03708761 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873029+0000", 2024-09-16T01:51:09.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1911e-05 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873046+0000", 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6702999999999999e-05 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873132+0000", 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6034999999999999e-05 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873288+0000", 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015656000000000001 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873419+0000", 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130381 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.880727+0000", 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073081919999999998 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882276+0000", 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015495330000000001 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882336+0000", 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9964999999999999e-05 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882385+0000", 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8588999999999998e-05 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:77 3.3c 3:3db0268f:::10000000002.0000004a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.731697026999999, 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051390551, 2024-09-16T01:51:09.684 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 77 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909702+0000", 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9699999999999999e-06 2024-09-16T01:51:09.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927863+0000", 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018160824999999998 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927887+0000", 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4802000000000001e-05 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927900+0000", 2024-09-16T01:51:09.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928016+0000", 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116123 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928158+0000", 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141977 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928294+0000", 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135733 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.938330+0000", 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010036092999999999 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960987+0000", 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022656493 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961043+0000", 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6416999999999999e-05 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961089+0000", 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6152000000000001e-05 2024-09-16T01:51:09.690 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.699146333000002, 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.691 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:09.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:09.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.621015744000001, 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.697 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:09.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:09.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:85 3.36 e55/20 3:6fe880e3:::10000000002.00000052:head v 55'2, mlcod=0'0)", 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:09.703 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.598629418000002, 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052325050999999997, 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042769+0000", 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3520000000000001e-06 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051520+0000", 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087513980000000005 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051666+0000", 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014630900000000001 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051711+0000", 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4768999999999998e-05 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052370+0000", 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065890100000000004 2024-09-16T01:51:09.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052478+0000", 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010805 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095028+0000", 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042550194 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095091+0000", 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3077999999999995e-05 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:98 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.502788540000001, 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052298391, 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.708 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 98 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138610+0000", 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166283+0000", 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027672980999999999 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166310+0000", 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7712000000000002e-05 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166324+0000", 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3239e-05 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166414+0000", 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0062000000000006e-05 2024-09-16T01:51:09.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166567+0000", 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015352700000000001 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166726+0000", 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015874500000000001 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.187851+0000", 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021125117999999998 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190785+0000", 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029334970000000002 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190850+0000", 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5560000000000002e-05 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190906+0000", 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5451000000000002e-05 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:140 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.086040615000002, 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050007939000000001, 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555356+0000", 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1039999999999999e-06 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591189+0000", 2024-09-16T01:51:09.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035832756 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591217+0000", 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8263e-05 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591230+0000", 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591354+0000", 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012385200000000001 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591571+0000", 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217099 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591739+0000", 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016809299999999999 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.598744+0000", 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070049939999999996 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605250+0000", 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065054079999999999 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605310+0000", 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0263000000000002e-05 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605363+0000", 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3140999999999998e-05 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.048802857999998, 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:09.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:09.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:09.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.824646324, 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:09.726 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:09.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:09.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:181 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.744666602999999, 2024-09-16T01:51:09.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046020995000000002, 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 181 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896731+0000", 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-06 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931107+0000", 2024-09-16T01:51:09.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034375920999999997 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931130+0000", 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3668000000000001e-05 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931141+0000", 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0366999999999999e-05 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931281+0000", 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140622 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931432+0000", 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150469 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931599+0000", 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167338 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.940280+0000", 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086811649999999994 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942632+0000", 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002352224 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942694+0000", 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2014999999999995e-05 2024-09-16T01:51:09.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942750+0000", 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5763999999999997e-05 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.527369301, 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:09.738 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:09.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:09.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:09.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:09.744 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.054561447999999, 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:09.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:09.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:09.750 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8667348619999995, 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:09.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:463 3.26 3:65689812:::10000000002.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7755615660000004, 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112035999999998, 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 463 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.757 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865836+0000", 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9719999999999999e-06 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883238+0000", 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017402081999999999 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883259+0000", 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1143999999999999e-05 2024-09-16T01:51:09.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883269+0000", 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-06 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883356+0000", 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7114999999999995e-05 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883467+0000", 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110513 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883617+0000", 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150216 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.906097+0000", 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022480257 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914840+0000", 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087431309999999995 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914898+0000", 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7643999999999999e-05 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914946+0000", 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8161999999999997e-05 2024-09-16T01:51:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:484 3.e 3:723b89cc:::10000000002.000000e1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5336920969999994, 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054305622999999997, 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 484 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107706+0000", 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145546+0000", 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037839795000000002 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145584+0000", 2024-09-16T01:51:09.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8556000000000002e-05 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145598+0000", 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4238e-05 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145708+0000", 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109584 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145834+0000", 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012588200000000001 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145990+0000", 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015650199999999999 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.160245+0000", 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014254119000000001 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161887+0000", 2024-09-16T01:51:09.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016424009999999999 2024-09-16T01:51:09.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161956+0000", 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8996000000000001e-05 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.162009+0000", 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3381999999999998e-05 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3579184570000002, 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:09.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:09.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:09.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:09.775 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:09.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:09.777 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:09.776+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:09.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:51:09.877 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:09.876+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:09.939 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:09.939 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:09.939 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:09.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:09.978 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:09.977+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:10.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.011+0000 7f65cec00640 1 Processor -- start 2024-09-16T01:51:10.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.011+0000 7f65cec00640 1 -- start start 2024-09-16T01:51:10.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.012+0000 7f65cec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.012+0000 7f65cec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c815b1d0 con 0x7f65c80cc4a0 2024-09-16T01:51:10.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.012+0000 7f65cec00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c815b4b0 con 0x7f65c80cb2f0 2024-09-16T01:51:10.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.012+0000 7f65cec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f65c80cbd00 con 0x7f65c80cc0c0 2024-09-16T01:51:10.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.012+0000 7f65ccc00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f65c80cc4a0 0x7f65c8165430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45678/0 (socket says 172.21.15.146:45678) 2024-09-16T01:51:10.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.012+0000 7f65ccc00640 1 -- 172.21.15.146:0/3564754301 learned_addr learned my addr 172.21.15.146:0/3564754301 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:10.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c7e00640 1 --2- 172.21.15.146:0/3564754301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048305851 0 0) 0x7f65c815b1d0 con 0x7f65c80cc4a0 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65ac003520 con 0x7f65c80cc4a0 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212000342 0 0) 0x7f65c815b4b0 con 0x7f65c80cb2f0 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65c815b1d0 con 0x7f65c80cb2f0 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c7e00640 1 -- 172.21.15.146:0/3564754301 >> v1:172.21.15.161:6789/0 conn(0x7f65c80cb2f0 legacy=0x7f65c815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c7e00640 1 -- 172.21.15.146:0/3564754301 >> v1:172.21.15.146:6789/0 conn(0x7f65c80cc4a0 legacy=0x7f65c8165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.013+0000 7f65c7e00640 1 -- 172.21.15.146:0/3564754301 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c815b790 con 0x7f65c80cc0c0 2024-09-16T01:51:10.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.014+0000 7f65c7e00640 1 --2- 172.21.15.146:0/3564754301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c815aab0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f65b80097d0 tx=0x7f65b802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7790c08ba2610f5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.014+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8037070 con 0x7f65c80cc0c0 2024-09-16T01:51:10.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.014+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b8032440 con 0x7f65c80cc0c0 2024-09-16T01:51:10.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.014+0000 7f65c6a00640 1 -- 172.21.15.146:0/3564754301 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8032810 con 0x7f65c80cc0c0 2024-09-16T01:51:10.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 -- 172.21.15.146:0/3564754301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 msgr2=0x7f65c815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 --2- 172.21.15.146:0/3564754301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c815aab0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f65b80097d0 tx=0x7f65b802a210 comp rx=0 tx=0).stop 2024-09-16T01:51:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 -- 172.21.15.146:0/3564754301 shutdown_connections 2024-09-16T01:51:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 --2- 172.21.15.146:0/3564754301 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c815aab0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 -- 172.21.15.146:0/3564754301 >> 172.21.15.146:0/3564754301 conn(0x7f65c806d560 msgr2=0x7f65c806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 -- 172.21.15.146:0/3564754301 shutdown_connections 2024-09-16T01:51:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.015+0000 7f65cec00640 1 -- 172.21.15.146:0/3564754301 wait complete. 2024-09-16T01:51:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.017+0000 7f65cec00640 1 Processor -- start 2024-09-16T01:51:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.017+0000 7f65cec00640 1 -- start start 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65cec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65c80cb2f0 0x7f65c814ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65cec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c80c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65cec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 0x7f65c81f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65cec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f65c815e540 con 0x7f65c80cc4a0 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65cec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f65c815ba70 con 0x7f65c80cb2f0 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65cec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f65c80cc8b0 con 0x7f65c80cc0c0 2024-09-16T01:51:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65ccc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 0x7f65c81f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65c7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c80c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65ccc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 0x7f65c81f0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40952/0 (socket says 172.21.15.146:40952) 2024-09-16T01:51:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65ccc00640 1 -- 172.21.15.146:0/232771943 learned_addr learned my addr 172.21.15.146:0/232771943 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65c7e00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65c80cb2f0 0x7f65c814ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65ccc00640 1 -- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 msgr2=0x7f65c80c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.018+0000 7f65ccc00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c80c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65ccc00640 1 -- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65c80cb2f0 msgr2=0x7f65c814ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65ccc00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65c80cb2f0 0x7f65c814ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65ccc00640 1 -- 172.21.15.146:0/232771943 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c80c8120 con 0x7f65c80cc4a0 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65c7400640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c80c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65c7e00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65c80cb2f0 0x7f65c814ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65ccc00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 0x7f65c81f0ce0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f65c000cad0 tx=0x7f65c000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0017070 con 0x7f65c80cc4a0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c000ac50 con 0x7f65c80cc4a0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0012400 con 0x7f65c80cc4a0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c80c9860 con 0x7f65c80cc4a0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.019+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c80c7e40 con 0x7f65c80cc4a0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.020+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c80c5560 con 0x7f65c80cc4a0 2024-09-16T01:51:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.021+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f65c001a070 con 0x7f65c80cc4a0 2024-09-16T01:51:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.022+0000 7f65bfe00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f65a806c2e0 0x7f65a806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.022+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5741+0+0 (secure 0 0 0) 0x7f65c008b1b0 con 0x7f65c80cc4a0 2024-09-16T01:51:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.022+0000 7f65c7e00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f65a806c2e0 0x7f65a806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.023+0000 7f65c7e00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f65a806c2e0 0x7f65a806e7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f65b8034000 tx=0x7f65b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.027+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f65c0059210 con 0x7f65c80cc4a0 2024-09-16T01:51:10.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:10.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:10.059 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T01:51:10.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-16T01:51:10.106 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:10.105+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:10.120 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:10.120 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:10.120 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:10.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:10.219 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:10.218+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:10.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.219+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f65c81f0990 con 0x7f65c80cc4a0 2024-09-16T01:51:10.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.219+0000 7f92b1c00640 1 Processor -- start 2024-09-16T01:51:10.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.220+0000 7f92b1c00640 1 -- start start 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92b1c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92ac0c6bb0 0x7f92ac0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92b1c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92ac0c58c0 0x7f92ac0c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92b1c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92ac072f50 con 0x7f92ac152850 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92b1c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f92ac073210 con 0x7f92ac0c58c0 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92b1c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f92ac15cb50 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92ab400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92ac0c58c0 0x7f92ac0c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92ab400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92ac0c58c0 0x7f92ac0c5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40958/0 (socket says 172.21.15.146:40958) 2024-09-16T01:51:10.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92ab400640 1 -- 172.21.15.146:0/3102412324 learned_addr learned my addr 172.21.15.146:0/3102412324 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:10.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.221+0000 7f92aa000640 1 --2- 172.21.15.146:0/3102412324 >> v2:172.21.15.161:3300/0 conn(0x7f92ac0c6bb0 0x7f92ac0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.222+0000 7f92ab400640 1 -- 172.21.15.146:0/3102412324 >> v2:172.21.15.161:3300/0 conn(0x7f92ac0c6bb0 msgr2=0x7f92ac0c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:10.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.222+0000 7f92ab400640 1 --2- 172.21.15.146:0/3102412324 >> v2:172.21.15.161:3300/0 conn(0x7f92ac0c6bb0 0x7f92ac0c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.222+0000 7f92ab400640 1 -- 172.21.15.146:0/3102412324 >> v1:172.21.15.161:6789/0 conn(0x7f92ac152850 legacy=0x7f92ac158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.222+0000 7f92ab400640 1 -- 172.21.15.146:0/3102412324 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac15bd90 con 0x7f92ac0c58c0 2024-09-16T01:51:10.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.223+0000 7f92ab400640 1 --2- 172.21.15.146:0/3102412324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c58c0 0x7f92ac0c5d10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0c67f0 tx=0x7f92a402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e5a29af49a233f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.223+0000 7f92a9600640 1 -- 172.21.15.146:0/3102412324 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4036070 con 0x7f92ac0c58c0 2024-09-16T01:51:10.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.223+0000 7f92a9600640 1 -- 172.21.15.146:0/3102412324 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a4031600 con 0x7f92ac0c58c0 2024-09-16T01:51:10.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.223+0000 7f92a9600640 1 -- 172.21.15.146:0/3102412324 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a40319d0 con 0x7f92ac0c58c0 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 -- 172.21.15.146:0/3102412324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c58c0 msgr2=0x7f92ac0c5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 --2- 172.21.15.146:0/3102412324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c58c0 0x7f92ac0c5d10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0c67f0 tx=0x7f92a402a730 comp rx=0 tx=0).stop 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 -- 172.21.15.146:0/3102412324 shutdown_connections 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 --2- 172.21.15.146:0/3102412324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c58c0 0x7f92ac0c5d10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 --2- 172.21.15.146:0/3102412324 >> v2:172.21.15.161:3300/0 conn(0x7f92ac0c6bb0 0x7f92ac0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 -- 172.21.15.146:0/3102412324 >> 172.21.15.146:0/3102412324 conn(0x7f92ac06f3c0 msgr2=0x7f92ac06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 -- 172.21.15.146:0/3102412324 shutdown_connections 2024-09-16T01:51:10.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.224+0000 7f92b1c00640 1 -- 172.21.15.146:0/3102412324 wait complete. 2024-09-16T01:51:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.226+0000 7f92b1c00640 1 Processor -- start 2024-09-16T01:51:10.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.226+0000 7f92b1c00640 1 -- start start 2024-09-16T01:51:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92b1c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92ac0c58c0 0x7f92ac1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92b1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c6bb0 0x7f92ac0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92b1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92ac152850 0x7f92ac0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92b1c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac15c630 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92b1c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac15c350 con 0x7f92ac152850 2024-09-16T01:51:10.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92b1c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f92ac070200 con 0x7f92ac0c58c0 2024-09-16T01:51:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92aa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c6bb0 0x7f92ac0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92aa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c6bb0 0x7f92ac0ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40972/0 (socket says 172.21.15.146:40972) 2024-09-16T01:51:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92aa000640 1 -- 172.21.15.146:0/2373856740 learned_addr learned my addr 172.21.15.146:0/2373856740 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92ab400640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92ac152850 0x7f92ac0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.227+0000 7f92aaa00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92ac0c58c0 0x7f92ac1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92aa000640 1 -- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92ac0c58c0 msgr2=0x7f92ac1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92aa000640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92ac0c58c0 0x7f92ac1420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92aa000640 1 -- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92ac152850 msgr2=0x7f92ac0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92aa000640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92ac152850 0x7f92ac0cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92aa000640 1 -- 172.21.15.146:0/2373856740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac142910 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92aa000640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c6bb0 0x7f92ac0ca950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f929800e9b0 tx=0x7f929800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298017070 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929800c9a0 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ac142c10 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929800cd90 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.228+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ac141140 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.229+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f92ac0718c0 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.231+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f929801a070 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.232+0000 7f92a2a00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f928c06c2e0 0x7f928c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.232+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5741+0+0 (secure 0 0 0) 0x7f929808b300 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.232+0000 7f92a2a00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f928c0744e0 0x7f928c076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.232+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f928c076fd0 con 0x7f928c0744e0 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.232+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f9298092050 con 0x7f92ac0c6bb0 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.232+0000 7f92aaa00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f928c06c2e0 0x7f928c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.233+0000 7f92ab400640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f928c0744e0 0x7f928c076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.233+0000 7f92aaa00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f928c06c2e0 0x7f928c06e7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0cd0d0 tx=0x7f929c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.233+0000 7f92ab400640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f928c0744e0 0x7f928c076900 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.234+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f928c076fd0 con 0x7f928c0744e0 2024-09-16T01:51:10.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.251+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f92ac152000 con 0x7f928c0744e0 2024-09-16T01:51:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.251+0000 7f92a2a00640 1 -- 172.21.15.146:0/2373856740 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f92ac152000 con 0x7f928c0744e0 2024-09-16T01:51:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.251+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f928c0744e0 msgr2=0x7f928c076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f928c0744e0 0x7f928c076900 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f928c06c2e0 msgr2=0x7f928c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f928c06c2e0 0x7f928c06e7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0cd0d0 tx=0x7f929c009040 comp rx=0 tx=0).stop 2024-09-16T01:51:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c6bb0 msgr2=0x7f92ac0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 --2- 172.21.15.146:0/2373856740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac0c6bb0 0x7f92ac0ca950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f929800e9b0 tx=0x7f929800ee70 comp rx=0 tx=0).stop 2024-09-16T01:51:10.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92ab400640 1 -- 172.21.15.146:0/2373856740 reap_dead start 2024-09-16T01:51:10.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 shutdown_connections 2024-09-16T01:51:10.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 >> 172.21.15.146:0/2373856740 conn(0x7f92ac06f3c0 msgr2=0x7f92ac0722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 shutdown_connections 2024-09-16T01:51:10.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.252+0000 7f92b1c00640 1 -- 172.21.15.146:0/2373856740 wait complete. 2024-09-16T01:51:10.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-16T01:51:10.310 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:10.309+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:10.344 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.085978879999999, 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:10.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:10.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:14 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.350 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.988658567999998, 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051400593000000001, 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 14 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352600+0000", 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.352602+0000", 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393382+0000", 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040779871000000002 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393422+0000", 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9765000000000003e-05 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393436+0000", 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3633e-05 2024-09-16T01:51:10.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393509+0000", 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3847e-05 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393696+0000", 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186911 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.393839+0000", 2024-09-16T01:51:10.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014274700000000001 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.401310+0000", 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074705040000000002 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403876+0000", 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025662020000000001 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.403939+0000", 2024-09-16T01:51:10.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3423999999999994e-05 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.404001+0000", 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1649000000000006e-05 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:10.356 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.887606009999999, 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:10.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.778803378999999, 2024-09-16T01:51:10.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:10.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:66 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.368 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.519573883, 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053741734999999999, 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 66 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821685+0000", 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.821687+0000", 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3750000000000001e-06 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862658+0000", 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040970816 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862692+0000", 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3420000000000002e-05 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862706+0000", 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4017e-05 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.862879+0000", 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173015 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863040+0000", 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016103800000000001 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.863192+0000", 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152738 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.867722+0000", 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045298689999999997 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875297+0000", 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075744860000000001 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875372+0000", 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5049999999999995e-05 2024-09-16T01:51:10.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.875427+0000", 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4911000000000003e-05 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.459234746, 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:10.374 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:10.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:10.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:10.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:75 3.30 3:0d2dbd17:::10000000002.00000048:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.395153318999999, 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050908518999999999, 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 75 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946105+0000", 2024-09-16T01:51:10.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.946107+0000", 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7710000000000001e-06 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960806+0000", 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014698386000000001 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960821+0000", 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5173e-05 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960832+0000", 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0698e-05 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960964+0000", 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013269700000000001 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961122+0000", 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157765 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961267+0000", 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014486100000000001 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981986+0000", 2024-09-16T01:51:10.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020719158000000001 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996886+0000", 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014899654999999999 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996958+0000", 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2095999999999995e-05 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997014+0000", 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6258999999999998e-05 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:110 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.104897032, 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053794485000000003, 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.386 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 110 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236362+0000", 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.236365+0000", 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1590000000000002e-06 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278463+0000", 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042097744999999999 2024-09-16T01:51:10.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278494+0000", 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1009000000000001e-05 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278508+0000", 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3901e-05 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278639+0000", 2024-09-16T01:51:10.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013188100000000001 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278806+0000", 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016689799999999999 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.278974+0000", 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016719799999999999 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.286538+0000", 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075647910000000004 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290050+0000", 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035116969999999998 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290114+0000", 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4298e-05 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.290156+0000", 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1907999999999999e-05 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.960572006, 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:10.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:10.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:160 3.9 3:93988fc9:::10000000002.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.591581929, 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056552534000000002, 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 160 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749679+0000", 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.956e-06 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794169+0000", 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04449061 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794202+0000", 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2626999999999998e-05 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794217+0000", 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794305+0000", 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8158999999999994e-05 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794460+0000", 2024-09-16T01:51:10.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155277 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794625+0000", 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164854 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.801888+0000", 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072630239999999999 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806094+0000", 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042063919999999998 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806178+0000", 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3287000000000004e-05 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806229+0000", 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765000000000003e-05 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.546999322000001, 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.404 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:10.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:10.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:10.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:164 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.409 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.519127305000001, 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053828972000000003, 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 164 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:10.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822134+0000", 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841247+0000", 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019113208 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841279+0000", 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2265999999999999e-05 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841291+0000", 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2305e-05 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841376+0000", 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4507999999999998e-05 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841535+0000", 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159216 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841687+0000", 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151846 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859703+0000", 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018016197000000001 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875876+0000", 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016172628000000001 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875929+0000", 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3108e-05 2024-09-16T01:51:10.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875960+0000", 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1491000000000001e-05 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:197 3.1d 3:bb6dfbbf:::10000000002.000000c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.234120248, 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057555969999999998, 2024-09-16T01:51:10.415 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 197 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107141+0000", 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148704+0000", 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041563044 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148738+0000", 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4480999999999997e-05 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148754+0000", 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6021000000000001e-05 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148909+0000", 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154735 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149071+0000", 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016181600000000001 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149226+0000", 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015550000000000001 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.158210+0000", 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089836529999999994 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164352+0000", 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061416810000000004 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164426+0000", 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4192999999999999e-05 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164695+0000", 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026864800000000002 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:449 3.28 3:144fb21c:::10000000002.000000be:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.650146514999999, 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05217542, 2024-09-16T01:51:10.421 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 449 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691112+0000", 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.691115+0000", 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.056e-06 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726045+0000", 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034929984999999997 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726077+0000", 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1390000000000003e-05 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726093+0000", 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6047000000000001e-05 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726207+0000", 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011452599999999999 2024-09-16T01:51:10.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726335+0000", 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012742299999999999 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.726492+0000", 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015772599999999999 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.740613+0000", 2024-09-16T01:51:10.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014120071999999999 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743157+0000", 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025441550000000002 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743229+0000", 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2102999999999997e-05 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.743288+0000", 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8937000000000002e-05 2024-09-16T01:51:10.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.56062623, 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:10.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:10.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:10.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.492521589000001, 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.433 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:10.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:10.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:10.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:10.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.350061924, 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:10.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:10.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:497 3.30 3:0c580b6a:::10000000002.000000ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.149586512999999, 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052947199, 2024-09-16T01:51:10.448 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 497 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191672+0000", 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.191674+0000", 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.575e-06 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220734+0000", 2024-09-16T01:51:10.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029059623999999999 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220763+0000", 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9099e-05 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220777+0000", 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4299e-05 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.220897+0000", 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011993999999999999 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.221085+0000", 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000188457 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.221327+0000", 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024153299999999999 2024-09-16T01:51:10.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.233648+0000", 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012321544 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244466+0000", 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010817795 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244519+0000", 2024-09-16T01:51:10.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3316999999999998e-05 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.244620+0000", 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100016 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:499 3.2f 3:f78980a5:::10000000002.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:10.454 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.118358674, 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055062123999999997, 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 499 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:10.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222902+0000", 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261223+0000", 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038321506999999998 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261257+0000", 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3526999999999999e-05 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261272+0000", 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4748e-05 2024-09-16T01:51:10.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261399+0000", 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012685100000000001 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261500+0000", 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101556 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261663+0000", 2024-09-16T01:51:10.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016259900000000001 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.270356+0000", 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086932399999999996 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277838+0000", 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074817470000000004 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277906+0000", 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8454000000000006e-05 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277962+0000", 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6075000000000001e-05 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:10.460 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.002595701000001, 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:10.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:10.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:10.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:10.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:10.468 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:10.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:10.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.466+0000 7fb666000640 1 Processor -- start 2024-09-16T01:51:10.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.466+0000 7fb666000640 1 -- start start 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb666000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6600c6410 0x7fb660158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb666000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6600c5280 0x7fb6600c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb666000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb660151e70 con 0x7fb660151a90 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb666000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb6600c8130 con 0x7fb6600c5280 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb666000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb6600c8410 con 0x7fb6600c6410 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65fe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6600c5280 0x7fb6600c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb656a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6600c6410 0x7fb660158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65f400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb660151a90 0x7fb6600c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50744/0 (socket says 172.21.15.146:50744) 2024-09-16T01:51:10.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65f400640 1 -- 172.21.15.146:0/1570382348 learned_addr learned my addr 172.21.15.146:0/1570382348 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:10.472 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65fe00640 1 -- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6600c6410 msgr2=0x7fb660158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65fe00640 1 --2- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6600c6410 0x7fb660158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65fe00640 1 -- 172.21.15.146:0/1570382348 >> v1:172.21.15.161:6789/0 conn(0x7fb660151a90 legacy=0x7fb6600c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65fe00640 1 -- 172.21.15.146:0/1570382348 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6600c86f0 con 0x7fb6600c5280 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb656a00640 1 --2- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6600c6410 0x7fb660158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.468+0000 7fb65ea00640 1 -- 172.21.15.146:0/1570382348 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1111990634 0 0) 0x7fb660151e70 con 0x7fb660151a90 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.469+0000 7fb65fe00640 1 --2- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c5280 0x7fb6600c5730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb658009d90 tx=0x7fb65802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c0990de401e227 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb65ea00640 1 -- 172.21.15.146:0/1570382348 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658007740 con 0x7fb6600c5280 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb65ea00640 1 -- 172.21.15.146:0/1570382348 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6580078e0 con 0x7fb6600c5280 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb65ea00640 1 -- 172.21.15.146:0/1570382348 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658007cb0 con 0x7fb6600c5280 2024-09-16T01:51:10.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 -- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c5280 msgr2=0x7fb6600c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 --2- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c5280 0x7fb6600c5730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb658009d90 tx=0x7fb65802a660 comp rx=0 tx=0).stop 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 -- 172.21.15.146:0/1570382348 shutdown_connections 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 --2- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c5280 0x7fb6600c5730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 --2- 172.21.15.146:0/1570382348 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6600c6410 0x7fb660158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 -- 172.21.15.146:0/1570382348 >> 172.21.15.146:0/1570382348 conn(0x7fb66006f3c0 msgr2=0x7fb66006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 -- 172.21.15.146:0/1570382348 shutdown_connections 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.470+0000 7fb666000640 1 -- 172.21.15.146:0/1570382348 wait complete. 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.472+0000 7fb666000640 1 Processor -- start 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.472+0000 7fb666000640 1 -- start start 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb666000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6600c5280 0x7fb6601480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb666000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c6410 0x7fb660149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb666000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb660151a90 0x7fb660150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb666000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb660157b20 con 0x7fb6600c5280 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb666000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb660157840 con 0x7fb6600c6410 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb666000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb660158580 con 0x7fb660151a90 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb65f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6600c5280 0x7fb6601480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb65fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb660151a90 0x7fb660150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb656a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c6410 0x7fb660149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb65f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6600c5280 0x7fb6601480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40984/0 (socket says 172.21.15.146:40984) 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb65fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb660151a90 0x7fb660150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44774/0 (socket says 172.21.15.146:44774) 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.473+0000 7fb65f400640 1 -- 172.21.15.146:0/1638780916 learned_addr learned my addr 172.21.15.146:0/1638780916 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb65f400640 1 -- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb660151a90 msgr2=0x7fb660150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb65f400640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb660151a90 0x7fb660150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb65f400640 1 -- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c6410 msgr2=0x7fb660149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb65f400640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6600c6410 0x7fb660149f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb65f400640 1 -- 172.21.15.146:0/1638780916 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6600cbf50 con 0x7fb6600c5280 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb65f400640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6600c5280 0x7fb6601480d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb65000c420 tx=0x7fb65000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6500074e0 con 0x7fb6600c5280 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6500047e0 con 0x7fb6600c5280 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650004c10 con 0x7fb6600c5280 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6600cc710 con 0x7fb6600c5280 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.474+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb660148b40 con 0x7fb6600c5280 2024-09-16T01:51:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.475+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb66014a460 con 0x7fb6600c5280 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.476+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb650011070 con 0x7fb6600c5280 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb657e00640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb63c06c2e0 0x7fb63c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5741+0+0 (secure 0 0 0) 0x7fb65008aff0 con 0x7fb6600c5280 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb657e00640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fb63c0744e0 0x7fb63c076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb63c076fd0 con 0x7fb63c0744e0 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fb65008d610 con 0x7fb6600c5280 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb656a00640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb63c06c2e0 0x7fb63c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.477+0000 7fb65fe00640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fb63c0744e0 0x7fb63c076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.478+0000 7fb656a00640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb63c06c2e0 0x7fb63c06e7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb648009560 tx=0x7fb648009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.478+0000 7fb65fe00640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fb63c0744e0 0x7fb63c076900 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.479+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb63c076fd0 con 0x7fb63c0744e0 2024-09-16T01:51:10.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.495+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fb660157230 con 0x7fb63c0744e0 2024-09-16T01:51:10.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.495+0000 7fb657e00640 1 -- 172.21.15.146:0/1638780916 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fb660157230 con 0x7fb63c0744e0 2024-09-16T01:51:10.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fb63c0744e0 msgr2=0x7fb63c076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fb63c0744e0 0x7fb63c076900 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb63c06c2e0 msgr2=0x7fb63c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb63c06c2e0 0x7fb63c06e7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb648009560 tx=0x7fb648009190 comp rx=0 tx=0).stop 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6600c5280 msgr2=0x7fb6601480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 --2- 172.21.15.146:0/1638780916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6600c5280 0x7fb6601480d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb65000c420 tx=0x7fb65000c8e0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb65fe00640 1 -- 172.21.15.146:0/1638780916 reap_dead start 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 shutdown_connections 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 >> 172.21.15.146:0/1638780916 conn(0x7fb66006f3c0 msgr2=0x7fb6600c9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 shutdown_connections 2024-09-16T01:51:10.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.496+0000 7fb666000640 1 -- 172.21.15.146:0/1638780916 wait complete. 2024-09-16T01:51:10.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-16T01:51:10.570 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:10.569+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:10.595 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:10.595 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:10.595 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:10.595 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:10.604 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:10.688 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:10.687+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:10.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.708+0000 7fab89600640 1 Processor -- start 2024-09-16T01:51:10.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.708+0000 7fab89600640 1 -- start start 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.709+0000 7fab89600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab8415b340 0x7fab84166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.709+0000 7fab89600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab8406d140 con 0x7fab840c6b70 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.709+0000 7fab89600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab8406d420 con 0x7fab840c75a0 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.709+0000 7fab89600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fab840cab20 con 0x7fab8415b340 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab88c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab8415b340 0x7fab84166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab88c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab8415b340 0x7fab84166170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40996/0 (socket says 172.21.15.146:40996) 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab83400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fab840c75a0 0x7fab840cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50748/0 (socket says 172.21.15.146:50748) 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab88c00640 1 -- 172.21.15.146:0/2776017150 learned_addr learned my addr 172.21.15.146:0/2776017150 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab83e00640 1 --1- 172.21.15.146:0/2776017150 >> v1:172.21.15.146:6789/0 conn(0x7fab840c6b70 0x7fab840cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45690/0 (socket says 172.21.15.146:45690) 2024-09-16T01:51:10.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab88c00640 1 -- 172.21.15.146:0/2776017150 >> v1:172.21.15.161:6789/0 conn(0x7fab840c75a0 legacy=0x7fab840cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab82a00640 1 -- 172.21.15.146:0/2776017150 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014585656 0 0) 0x7fab8406d420 con 0x7fab840c75a0 2024-09-16T01:51:10.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab88c00640 1 -- 172.21.15.146:0/2776017150 >> v1:172.21.15.146:6789/0 conn(0x7fab840c6b70 legacy=0x7fab840cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.710+0000 7fab88c00640 1 -- 172.21.15.146:0/2776017150 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8406d700 con 0x7fab8415b340 2024-09-16T01:51:10.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.711+0000 7fab88c00640 1 --2- 172.21.15.146:0/2776017150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab84166170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009e50 tx=0x7fab7c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17ac8a6a22df6de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.711+0000 7fab82a00640 1 -- 172.21.15.146:0/2776017150 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c036070 con 0x7fab8415b340 2024-09-16T01:51:10.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.711+0000 7fab82a00640 1 -- 172.21.15.146:0/2776017150 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab7c031600 con 0x7fab8415b340 2024-09-16T01:51:10.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.711+0000 7fab82a00640 1 -- 172.21.15.146:0/2776017150 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c0319d0 con 0x7fab8415b340 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 -- 172.21.15.146:0/2776017150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 msgr2=0x7fab84166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 --2- 172.21.15.146:0/2776017150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab84166170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009e50 tx=0x7fab7c02a730 comp rx=0 tx=0).stop 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 -- 172.21.15.146:0/2776017150 shutdown_connections 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 --2- 172.21.15.146:0/2776017150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab84166170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 -- 172.21.15.146:0/2776017150 >> 172.21.15.146:0/2776017150 conn(0x7fab840ccc20 msgr2=0x7fab840cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 -- 172.21.15.146:0/2776017150 shutdown_connections 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.712+0000 7fab89600640 1 -- 172.21.15.146:0/2776017150 wait complete. 2024-09-16T01:51:10.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.713+0000 7fab89600640 1 Processor -- start 2024-09-16T01:51:10.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.714+0000 7fab89600640 1 -- start start 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.714+0000 7fab89600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab840c6b70 0x7fab841f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.714+0000 7fab89600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab840c75a0 0x7fab8415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab89600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab8415dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab89600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fab8415b7c0 con 0x7fab8415b340 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab89600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab840cb050 con 0x7fab840c75a0 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab89600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fab840cbc80 con 0x7fab840c6b70 2024-09-16T01:51:10.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab8415dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab8415dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41006/0 (socket says 172.21.15.146:41006) 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 -- 172.21.15.146:0/1145633650 learned_addr learned my addr 172.21.15.146:0/1145633650 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab83400640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab840c75a0 0x7fab8415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab83e00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab840c6b70 0x7fab841f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 -- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab840c6b70 msgr2=0x7fab841f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab840c6b70 0x7fab841f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 -- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab840c75a0 msgr2=0x7fab8415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab840c75a0 0x7fab8415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.715+0000 7fab88c00640 1 -- 172.21.15.146:0/1145633650 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab84151920 con 0x7fab8415b340 2024-09-16T01:51:10.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.716+0000 7fab88c00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab8415dc90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fab7c034040 tx=0x7fab7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.716+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c036050 con 0x7fab8415b340 2024-09-16T01:51:10.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.716+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab7c042070 con 0x7fab8415b340 2024-09-16T01:51:10.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.716+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8415baa0 con 0x7fab8415b340 2024-09-16T01:51:10.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.716+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c009510 con 0x7fab8415b340 2024-09-16T01:51:10.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.716+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab841f4750 con 0x7fab8415b340 2024-09-16T01:51:10.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.717+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab8406cd20 con 0x7fab8415b340 2024-09-16T01:51:10.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.718+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fab7c007bf0 con 0x7fab8415b340 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab7be00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab6406c2e0 0x7fab6406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5741+0+0 (secure 0 0 0) 0x7fab7c0ac1a0 con 0x7fab8415b340 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab83e00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab6406c2e0 0x7fab6406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab7be00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fab640744e0 0x7fab64076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab7c0020b0 con 0x7fab640744e0 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7fab7c0b3050 con 0x7fab8415b340 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.719+0000 7fab83400640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fab640744e0 0x7fab64076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.720+0000 7fab83e00640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab6406c2e0 0x7fab6406e7a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fab74000f60 tx=0x7fab74000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.720+0000 7fab83400640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fab640744e0 0x7fab64076900 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:10.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.721+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab7c0020b0 con 0x7fab640744e0 2024-09-16T01:51:10.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.738+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fab840ca280 con 0x7fab640744e0 2024-09-16T01:51:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab7be00640 1 -- 172.21.15.146:0/1145633650 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fab840ca280 con 0x7fab640744e0 2024-09-16T01:51:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fab640744e0 msgr2=0x7fab64076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fab640744e0 0x7fab64076900 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab6406c2e0 msgr2=0x7fab6406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab6406c2e0 0x7fab6406e7a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fab74000f60 tx=0x7fab74000ed0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 msgr2=0x7fab8415dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 --2- 172.21.15.146:0/1145633650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab8415b340 0x7fab8415dc90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fab7c034040 tx=0x7fab7c004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab88c00640 1 -- 172.21.15.146:0/1145633650 reap_dead start 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 shutdown_connections 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 >> 172.21.15.146:0/1145633650 conn(0x7fab840ccc20 msgr2=0x7fab840ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 shutdown_connections 2024-09-16T01:51:10.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.739+0000 7fab89600640 1 -- 172.21.15.146:0/1145633650 wait complete. 2024-09-16T01:51:10.751 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:10.751 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:10.751 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:10.752 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:10.752 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:10.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-16T01:51:10.760 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:10.802 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:10.801+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:10.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.835+0000 7f65bfe00640 1 -- 172.21.15.146:0/232771943 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v64) ==== 91+0+0 (secure 0 0 0) 0x7f65c005d510 con 0x7f65c80cc4a0 2024-09-16T01:51:10.836 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f65a806c2e0 msgr2=0x7f65a806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f65a806c2e0 0x7f65a806e7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f65b8034000 tx=0x7f65b802e000 comp rx=0 tx=0).stop 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 msgr2=0x7f65c81f0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 0x7f65c81f0ce0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f65c000cad0 tx=0x7f65c000cf90 comp rx=0 tx=0).stop 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 shutdown_connections 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f65a806c2e0 0x7f65a806e7a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.839+0000 7f65cec00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65c80cc4a0 0x7f65c81f0ce0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.840+0000 7f65cec00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65c80cc0c0 0x7f65c80c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.840+0000 7f65cec00640 1 --2- 172.21.15.146:0/232771943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f65c80cb2f0 0x7f65c814ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:10.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.840+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 >> 172.21.15.146:0/232771943 conn(0x7f65c806d560 msgr2=0x7f65c80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:10.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.840+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 shutdown_connections 2024-09-16T01:51:10.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.840+0000 7f65cec00640 1 -- 172.21.15.146:0/232771943 wait complete. 2024-09-16T01:51:10.901 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:10.900+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:10.908 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:10.908 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5 3.7 3:e001da36:::10000000002.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.645860313, 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044893833000000001, 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.909 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260356+0000", 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.260358+0000", 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.672e-06 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297299+0000", 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036941017999999999 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297321+0000", 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2223000000000001e-05 2024-09-16T01:51:10.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297332+0000", 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.058e-05 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297462+0000", 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013043399999999999 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297624+0000", 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000162263 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.297768+0000", 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014397800000000001 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.303867+0000", 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060983820000000003 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305178+0000", 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013110330000000001 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305214+0000", 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6380000000000001e-05 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.305250+0000", 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.587e-05 2024-09-16T01:51:10.914 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:13 3.2 3:438a76aa:::10000000002.0000000a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.552007983999999, 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045857943999999998, 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.915 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 13 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354208+0000", 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.354210+0000", 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.203e-06 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388636+0000", 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034425859000000003 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388673+0000", 2024-09-16T01:51:10.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7004e-05 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388693+0000", 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0194999999999999e-05 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388771+0000", 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8737000000000002e-05 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388904+0000", 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013215899999999999 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.389016+0000", 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112642 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.392245+0000", 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032287370000000002 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.399956+0000", 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0077109190000000001 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400015+0000", 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9119000000000001e-05 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.400066+0000", 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1369999999999998e-05 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:30 3.7 3:e33cd625:::10000000002.0000001b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.421999058000001, 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035807259000000001, 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 30 2024-09-16T01:51:10.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484217+0000", 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.484219+0000", 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512447+0000", 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028228652 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512476+0000", 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8283e-05 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512489+0000", 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3049e-05 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512566+0000", 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7420000000000001e-05 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512720+0000", 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015377000000000001 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.512891+0000", 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017147199999999999 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.516819+0000", 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003927458 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519938+0000", 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031196489999999999 2024-09-16T01:51:10.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.519989+0000", 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0380000000000002e-05 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.520025+0000", 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5839e-05 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.356595520999999, 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:10.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:10.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.932 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.109561939999999, 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:10.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:10.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.938 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.050402854000001, 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:10.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:82 3.2b 3:d50470f2:::10000000002.0000004f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.943451016000001, 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047169015000000002, 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 82 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962765+0000", 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.962768+0000", 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3269999999999999e-06 2024-09-16T01:51:10.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002536+0000", 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039768464000000003 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002551+0000", 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002558+0000", 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8730000000000001e-06 2024-09-16T01:51:10.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002630+0000", 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2025000000000002e-05 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002751+0000", 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121197 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.002871+0000", 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011995 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.006288+0000", 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003417032 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009828+0000", 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00353996 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009879+0000", 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0473999999999998e-05 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.009934+0000", 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5872999999999999e-05 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.950 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:114 3.8 3:10128d86:::10000000002.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.637562665000001, 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038796303999999997, 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 114 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268654+0000", 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.268656+0000", 2024-09-16T01:51:10.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.858e-06 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303790+0000", 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035134060000000002 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303806+0000", 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6611000000000001e-05 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303814+0000", 2024-09-16T01:51:10.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4329999999999998e-06 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303869+0000", 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5639e-05 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303979+0000", 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110043 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.304078+0000", 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9006999999999996e-05 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.306722+0000", 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002643074 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307380+0000", 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00065801399999999995 2024-09-16T01:51:10.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307414+0000", 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4082000000000003e-05 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.307450+0000", 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6483000000000003e-05 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:132 3.2b 3:d52309a9:::10000000002.00000081:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.436525541000002, 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034693794, 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 132 2024-09-16T01:51:10.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469691+0000", 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.469692+0000", 2024-09-16T01:51:10.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.516e-06 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496852+0000", 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027159651999999999 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496868+0000", 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5869999999999999e-05 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496875+0000", 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.391e-06 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.496952+0000", 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6320000000000001e-05 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497109+0000", 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015718700000000001 2024-09-16T01:51:10.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.497263+0000", 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015406799999999999 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.500555+0000", 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032925720000000001 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504317+0000", 2024-09-16T01:51:10.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037611150000000002 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504351+0000", 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4214000000000001e-05 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.504385+0000", 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3889e-05 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.962 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.138824541999998, 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:10.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:10.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.968 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.094564311999999, 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:10.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:10.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.064064492, 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:10.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.022508970000001, 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:10.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:10.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:10.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:10.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:183 3.1b 3:db82751e:::10000000002.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.986 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 18.935122354000001, 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040945861, 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 183 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:51:10.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971094+0000", 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.971104+0000", 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8719999999999997e-06 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999810+0000", 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028705819 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999832+0000", 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2560000000000001e-05 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999847+0000", 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4956e-05 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.999905+0000", 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8084000000000002e-05 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000017+0000", 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111222 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.000113+0000", 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6131999999999994e-05 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.002657+0000", 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025443089999999998 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011949+0000", 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0092920350000000006 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.011989+0000", 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9987999999999997e-05 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.012040+0000", 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0884000000000003e-05 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 18.884462276000001, 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.992 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:10.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:10.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:10.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.765502549000001, 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:10.998 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:10.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:11.001 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.730706250000001, 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.005 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:11.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:11.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:489 3.7 3:e3fe2874:::10000000002.000000e6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.220274+0000", 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.685942354, 2024-09-16T01:51:11.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045008859999999998, 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 489 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220274+0000", 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220274+0000", 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220276+0000", 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4640000000000001e-06 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258890+0000", 2024-09-16T01:51:11.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038614895000000003 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258911+0000", 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0965e-05 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258924+0000", 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2282e-05 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259005+0000", 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1768999999999996e-05 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259151+0000", 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014523699999999999 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259306+0000", 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015550500000000001 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265082+0000", 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057754939999999999 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265201+0000", 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119397 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265249+0000", 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7951000000000002e-05 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.265283+0000", 2024-09-16T01:51:11.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3900999999999999e-05 2024-09-16T01:51:11.018 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.018 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.020 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.020 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.020 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.647257296999999, 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:11.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:507 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.307551+0000", 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.598665705, 2024-09-16T01:51:11.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040029878999999997, 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 507 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.307551+0000", 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.307551+0000", 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.307552+0000", 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.505e-06 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335233+0000", 2024-09-16T01:51:11.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027680284999999999 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335251+0000", 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8958e-05 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335289+0000", 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7333000000000003e-05 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335358+0000", 2024-09-16T01:51:11.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9493999999999996e-05 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335427+0000", 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8984000000000002e-05 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.335525+0000", 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7819000000000007e-05 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.338819+0000", 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032943130000000001 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.347505+0000", 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086852309999999999 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.347541+0000", 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6266000000000001e-05 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.347581+0000", 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9691000000000003e-05 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.032 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:11.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.916+0000 7f739ae00640 1 Processor -- start 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.916+0000 7f739ae00640 1 -- start start 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739ae00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f73940c6b70 0x7f739415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 0x7f73940cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739ae00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940cbab0 0x7f73940cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f739415b880 con 0x7f73940c75a0 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739ae00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f73940cab20 con 0x7f73940c6b70 2024-09-16T01:51:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739ae00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f739406d140 con 0x7f73940cbab0 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f7399000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 0x7f73940cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f7399a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f73940c6b70 0x7f739415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f7399000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 0x7f73940cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41012/0 (socket says 172.21.15.146:41012) 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f7399000640 1 -- 172.21.15.146:0/422273487 learned_addr learned my addr 172.21.15.146:0/422273487 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.917+0000 7f739a400640 1 --2- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940cbab0 0x7f73940cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.918+0000 7f7399a00640 1 -- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940cbab0 msgr2=0x7f73940cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.918+0000 7f7399a00640 1 --2- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940cbab0 0x7f73940cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.918+0000 7f7399a00640 1 -- 172.21.15.146:0/422273487 >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 msgr2=0x7f73940cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.918+0000 7f7399a00640 1 --2- 172.21.15.146:0/422273487 >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 0x7f73940cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.918+0000 7f7399a00640 1 -- 172.21.15.146:0/422273487 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739406d420 con 0x7f73940c6b70 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.918+0000 7f7399000640 1 --2- 172.21.15.146:0/422273487 >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 0x7f73940cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f7399a00640 1 --2- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940c6b70 0x7f739415b340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7394075ec0 tx=0x7f7388029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a64ce42f6cea87 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f7393e00640 1 -- 172.21.15.146:0/422273487 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388037070 con 0x7f73940c6b70 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f7393e00640 1 -- 172.21.15.146:0/422273487 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738802ad30 con 0x7f73940c6b70 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f7393e00640 1 -- 172.21.15.146:0/422273487 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388032660 con 0x7f73940c6b70 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 -- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940c6b70 msgr2=0x7f739415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 --2- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940c6b70 0x7f739415b340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7394075ec0 tx=0x7f7388029fe0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 -- 172.21.15.146:0/422273487 shutdown_connections 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 --2- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940cbab0 0x7f73940cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 --2- 172.21.15.146:0/422273487 >> v2:172.21.15.146:3300/0 conn(0x7f73940c75a0 0x7f73940cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 --2- 172.21.15.146:0/422273487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940c6b70 0x7f739415b340 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 -- 172.21.15.146:0/422273487 >> 172.21.15.146:0/422273487 conn(0x7f73940ccc20 msgr2=0x7f73940cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 -- 172.21.15.146:0/422273487 shutdown_connections 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.919+0000 7f739ae00640 1 -- 172.21.15.146:0/422273487 wait complete. 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.921+0000 7f739ae00640 1 Processor -- start 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.921+0000 7f739ae00640 1 -- start start 2024-09-16T01:51:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739ae00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940c6b70 0x7f739415bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940c75a0 0x7f739415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739ae00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940cbab0 0x7f739415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f73940cb050 con 0x7f73940c75a0 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739ae00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f73940cad70 con 0x7f73940cbab0 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739ae00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f739406cd20 con 0x7f73940c6b70 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f7399a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940c6b70 0x7f739415bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f7399000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940c75a0 0x7f739415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940cbab0 0x7f739415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f7399000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940c75a0 0x7f739415d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41018/0 (socket says 172.21.15.146:41018) 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399000640 1 -- 172.21.15.146:0/904151966 learned_addr learned my addr 172.21.15.146:0/904151966 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f7399a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940c6b70 0x7f739415bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44794/0 (socket says 172.21.15.146:44794) 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.922+0000 7f739a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940cbab0 0x7f739415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:57052/0 (socket says 172.21.15.146:57052) 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399a00640 1 -- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940cbab0 msgr2=0x7f739415dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399a00640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940cbab0 0x7f739415dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399a00640 1 -- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940c75a0 msgr2=0x7f739415d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399a00640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940c75a0 0x7f739415d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399a00640 1 -- 172.21.15.146:0/904151966 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73940c3140 con 0x7f73940c6b70 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399000640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940c75a0 0x7f739415d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f739a400640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f73940cbab0 0x7f739415dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.923+0000 7f7399a00640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940c6b70 0x7f739415bc90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f73940caca0 tx=0x7f73880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.924+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388037040 con 0x7f73940c6b70 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.924+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738802ab30 con 0x7f73940c6b70 2024-09-16T01:51:11.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.924+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738803d660 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.924+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73941f0b90 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.924+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73941f1150 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.924+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73940ca280 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.926+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7388044020 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.926+0000 7f7391600640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f737406c2b0 0x7f737406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7f73880ab240 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f7399000640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f737406c2b0 0x7f737406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f7391600640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f73740744b0 0x7f73740768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7374076fa0 con 0x7f73740744b0 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f73880ab5f0 con 0x7f73940c6b70 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f739a400640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f73740744b0 0x7f73740768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.927+0000 7f7399000640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f737406c2b0 0x7f737406e770 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f739415d790 tx=0x7f7380009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.928+0000 7f739a400640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f73740744b0 0x7f73740768d0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.928+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7374076fa0 con 0x7f73740744b0 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.945+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f739406da90 con 0x7f73740744b0 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.945+0000 7f7391600640 1 -- 172.21.15.146:0/904151966 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f739406da90 con 0x7f73740744b0 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f73740744b0 msgr2=0x7f73740768d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f73740744b0 0x7f73740768d0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f737406c2b0 msgr2=0x7f737406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f737406c2b0 0x7f737406e770 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f739415d790 tx=0x7f7380009240 comp rx=0 tx=0).stop 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940c6b70 msgr2=0x7f739415bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 --2- 172.21.15.146:0/904151966 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f73940c6b70 0x7f739415bc90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f73940caca0 tx=0x7f73880046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739a400640 1 -- 172.21.15.146:0/904151966 reap_dead start 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 shutdown_connections 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 >> 172.21.15.146:0/904151966 conn(0x7f73940ccc20 msgr2=0x7f73940c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 shutdown_connections 2024-09-16T01:51:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:10.946+0000 7f739ae00640 1 -- 172.21.15.146:0/904151966 wait complete. 2024-09-16T01:51:11.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-16T01:51:11.047 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:11.032+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:11.134 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:11.133+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:11.142 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:11.142 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:11.142 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:11.142 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:11.151 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:11.202 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.235+0000 7f2a73a00640 1 Processor -- start 2024-09-16T01:51:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.235+0000 7f2a73a00640 1 -- start start 2024-09-16T01:51:11.237 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:11.236+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.237+0000 7f2a73a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2a6c0cb2f0 0x7f2a6c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.237+0000 7f2a73a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a6c0cc4a0 0x7f2a6c165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.237+0000 7f2a73a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6c15b1d0 con 0x7f2a6c0cc0c0 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.237+0000 7f2a73a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6c0cbd00 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a73a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6c15ba70 con 0x7f2a6c0cc4a0 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a6be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2a6c0cb2f0 0x7f2a6c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a6c0cc4a0 0x7f2a6c165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2a6c0cc0c0 0x7f2a6c15e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45700/0 (socket says 172.21.15.146:45700) 2024-09-16T01:51:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71000640 1 -- 172.21.15.146:0/2717611021 learned_addr learned my addr 172.21.15.146:0/2717611021 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71a00640 1 -- 172.21.15.146:0/2717611021 >> v2:172.21.15.146:3300/0 conn(0x7f2a6c0cb2f0 msgr2=0x7f2a6c15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71a00640 1 --2- 172.21.15.146:0/2717611021 >> v2:172.21.15.146:3300/0 conn(0x7f2a6c0cb2f0 0x7f2a6c15aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71a00640 1 -- 172.21.15.146:0/2717611021 >> v1:172.21.15.146:6789/0 conn(0x7f2a6c0cc0c0 legacy=0x7f2a6c15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.238+0000 7f2a71a00640 1 -- 172.21.15.146:0/2717611021 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c15b4b0 con 0x7f2a6c0cc4a0 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.239+0000 7f2a71a00640 1 --2- 172.21.15.146:0/2717611021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cc4a0 0x7f2a6c165430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c06e100 tx=0x7f2a6402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8664828447d99b8d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.239+0000 7f2a6b400640 1 -- 172.21.15.146:0/2717611021 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64007740 con 0x7f2a6c0cc4a0 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.239+0000 7f2a6b400640 1 -- 172.21.15.146:0/2717611021 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a640078e0 con 0x7f2a6c0cc4a0 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.239+0000 7f2a6b400640 1 -- 172.21.15.146:0/2717611021 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64007cb0 con 0x7f2a6c0cc4a0 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 -- 172.21.15.146:0/2717611021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cc4a0 msgr2=0x7f2a6c165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 --2- 172.21.15.146:0/2717611021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cc4a0 0x7f2a6c165430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c06e100 tx=0x7f2a6402a680 comp rx=0 tx=0).stop 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 -- 172.21.15.146:0/2717611021 shutdown_connections 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 --2- 172.21.15.146:0/2717611021 >> v2:172.21.15.146:3300/0 conn(0x7f2a6c0cb2f0 0x7f2a6c15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 --2- 172.21.15.146:0/2717611021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cc4a0 0x7f2a6c165430 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 -- 172.21.15.146:0/2717611021 >> 172.21.15.146:0/2717611021 conn(0x7f2a6c06d560 msgr2=0x7f2a6c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 -- 172.21.15.146:0/2717611021 shutdown_connections 2024-09-16T01:51:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.240+0000 7f2a73a00640 1 -- 172.21.15.146:0/2717611021 wait complete. 2024-09-16T01:51:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.242+0000 7f2a73a00640 1 Processor -- start 2024-09-16T01:51:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.242+0000 7f2a73a00640 1 -- start start 2024-09-16T01:51:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a73a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cb2f0 0x7f2a6c14a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a73a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2a6c0cc0c0 0x7f2a6c0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a73a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a6c0cc4a0 0x7f2a6c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a73a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c15e1d0 con 0x7f2a6c0cc4a0 2024-09-16T01:51:11.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a73a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c15e540 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a73a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c0cc8b0 con 0x7f2a6c0cc0c0 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a71000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cb2f0 0x7f2a6c14a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2a6c0cc0c0 0x7f2a6c0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a71a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a6c0cc4a0 0x7f2a6c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2a6c0cc0c0 0x7f2a6c0ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44810/0 (socket says 172.21.15.146:44810) 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a6be00640 1 -- 172.21.15.146:0/1839668160 learned_addr learned my addr 172.21.15.146:0/1839668160 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:11.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.243+0000 7f2a71000640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cb2f0 0x7f2a6c14a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:57074/0 (socket says 172.21.15.146:57074) 2024-09-16T01:51:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a71000640 1 -- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2a6c0cc0c0 msgr2=0x7f2a6c0ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a71000640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2a6c0cc0c0 0x7f2a6c0ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a71000640 1 -- 172.21.15.146:0/1839668160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a6c0cc4a0 msgr2=0x7f2a6c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a71000640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a6c0cc4a0 0x7f2a6c0cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a71000640 1 -- 172.21.15.146:0/1839668160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c1f38e0 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a71000640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cb2f0 0x7f2a6c14a8b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c370 tx=0x7f2a6000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60017070 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a60005ce0 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6c0b3050 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60007310 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.244+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6c1f3e80 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.245+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a6c0c5560 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.246+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2a6001a070 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.247+0000 7f2a68c00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2a4c06c2b0 0x7f2a4c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.247+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7f2a6008a7f0 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.247+0000 7f2a6be00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2a4c06c2b0 0x7f2a4c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.247+0000 7f2a68c00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f2a4c074510 0x7f2a4c076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.247+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a4c077000 con 0x7f2a4c074510 2024-09-16T01:51:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.247+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f2a60091050 con 0x7f2a6c0cb2f0 2024-09-16T01:51:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.248+0000 7f2a71a00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f2a4c074510 0x7f2a4c076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.248+0000 7f2a6be00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2a4c06c2b0 0x7f2a4c06e770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c154460 tx=0x7f2a58009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.248+0000 7f2a71a00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f2a4c074510 0x7f2a4c076930 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.249+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a4c077000 con 0x7f2a4c074510 2024-09-16T01:51:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.269+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f2a6c0cbea0 con 0x7f2a4c074510 2024-09-16T01:51:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a68c00640 1 -- 172.21.15.146:0/1839668160 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f2a6c0cbea0 con 0x7f2a4c074510 2024-09-16T01:51:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f2a4c074510 msgr2=0x7f2a4c076930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a73a00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f2a4c074510 0x7f2a4c076930 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2a4c06c2b0 msgr2=0x7f2a4c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a73a00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2a4c06c2b0 0x7f2a4c06e770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c154460 tx=0x7f2a58009190 comp rx=0 tx=0).stop 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cb2f0 msgr2=0x7f2a6c14a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.270+0000 7f2a73a00640 1 --2- 172.21.15.146:0/1839668160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a6c0cb2f0 0x7f2a6c14a8b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c370 tx=0x7f2a6000c830 comp rx=0 tx=0).stop 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.271+0000 7f2a71a00640 1 -- 172.21.15.146:0/1839668160 reap_dead start 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.271+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 shutdown_connections 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.271+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 >> 172.21.15.146:0/1839668160 conn(0x7f2a6c06d560 msgr2=0x7f2a6c0c6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.271+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 shutdown_connections 2024-09-16T01:51:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.271+0000 7f2a73a00640 1 -- 172.21.15.146:0/1839668160 wait complete. 2024-09-16T01:51:11.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-16T01:51:11.295 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:11.295 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:11.295 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:11.295 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:11.296 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:11.304 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:11.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.363+0000 7f08abe00640 1 Processor -- start 2024-09-16T01:51:11.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.363+0000 7f08abe00640 1 -- start start 2024-09-16T01:51:11.365 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:11.364+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08abe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a40c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08abe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f08a40c9180 0x7f08a40c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08abe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08a40c4240 con 0x7f08a4152c30 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08abe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f08a40c71c0 con 0x7f08a40c9180 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08abe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f08a40c3a90 con 0x7f08a40c36b0 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a9400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f08a4152c30 0x7f08a40c6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45706/0 (socket says 172.21.15.146:45706) 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a9400640 1 -- 172.21.15.146:0/3136701466 learned_addr learned my addr 172.21.15.146:0/3136701466 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a8a00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a40c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a9e00640 1 --2- 172.21.15.146:0/3136701466 >> v2:172.21.15.161:3300/0 conn(0x7f08a40c9180 0x7f08a40c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a9e00640 1 -- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 msgr2=0x7f08a40c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:11.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a9e00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a40c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a9e00640 1 -- 172.21.15.146:0/3136701466 >> v1:172.21.15.146:6789/0 conn(0x7f08a4152c30 legacy=0x7f08a40c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.366+0000 7f08a9e00640 1 -- 172.21.15.146:0/3136701466 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a40c4500 con 0x7f08a40c9180 2024-09-16T01:51:11.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.365+0000 7f08a3e00640 1 -- 172.21.15.146:0/3136701466 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982287049 0 0) 0x7f08a40c4240 con 0x7f08a4152c30 2024-09-16T01:51:11.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.366+0000 7f08a8a00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a40c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:11.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.366+0000 7f08a9e00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a40c9180 0x7f08a40c7ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f089c009d90 tx=0x7f089c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e161add9a23646b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.367+0000 7f08a3e00640 1 -- 172.21.15.146:0/3136701466 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c007740 con 0x7f08a40c9180 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.367+0000 7f08a3e00640 1 -- 172.21.15.146:0/3136701466 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089c0078e0 con 0x7f08a40c9180 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.367+0000 7f08a3e00640 1 -- 172.21.15.146:0/3136701466 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c007cb0 con 0x7f08a40c9180 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 -- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a40c9180 msgr2=0x7f08a40c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a40c9180 0x7f08a40c7ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f089c009d90 tx=0x7f089c02a660 comp rx=0 tx=0).stop 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 -- 172.21.15.146:0/3136701466 shutdown_connections 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a40c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 --2- 172.21.15.146:0/3136701466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a40c9180 0x7f08a40c7ff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 -- 172.21.15.146:0/3136701466 >> 172.21.15.146:0/3136701466 conn(0x7f08a406bc10 msgr2=0x7f08a406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 -- 172.21.15.146:0/3136701466 shutdown_connections 2024-09-16T01:51:11.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.368+0000 7f08abe00640 1 -- 172.21.15.146:0/3136701466 wait complete. 2024-09-16T01:51:11.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.370+0000 7f08abe00640 1 Processor -- start 2024-09-16T01:51:11.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.370+0000 7f08abe00640 1 -- start start 2024-09-16T01:51:11.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08abe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a414c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08abe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a40c9180 0x7f08a414e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08abe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 0x7f08a41eff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08abe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f08a40c4ac0 con 0x7f08a40c9180 2024-09-16T01:51:11.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08abe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f08a40c4da0 con 0x7f08a4152c30 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08abe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f08a40c6630 con 0x7f08a40c36b0 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08a9400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a414c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08a9e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 0x7f08a41eff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08a8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a40c9180 0x7f08a414e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08a9e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 0x7f08a41eff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:57098/0 (socket says 172.21.15.146:57098) 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 learned_addr learned my addr 172.21.15.146:0/672014487 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.372+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a414c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44830/0 (socket says 172.21.15.146:44830) 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 msgr2=0x7f08a414c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a414c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a40c9180 msgr2=0x7f08a414e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a40c9180 0x7f08a414e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a414c6d0 con 0x7f08a4152c30 2024-09-16T01:51:11.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08a40c36b0 0x7f08a414c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 0x7f08a41eff80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f089c009eb0 tx=0x7f089c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c040070 con 0x7f08a4152c30 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f089c031c40 con 0x7f08a4152c30 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c009450 con 0x7f08a4152c30 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a414b550 con 0x7f08a4152c30 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.373+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a414bb10 con 0x7f08a4152c30 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.374+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f08a40c6160 con 0x7f08a4152c30 2024-09-16T01:51:11.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.375+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f089c039040 con 0x7f08a4152c30 2024-09-16T01:51:11.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.376+0000 7f08a1600640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f088406c2e0 0x7f088406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7f089c0ac4d0 con 0x7f08a4152c30 2024-09-16T01:51:11.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f088406c2e0 0x7f088406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v64) ==== 54+0+16 (secure 0 0 0) 0x7f08a40c6160 con 0x7f08a4152c30 2024-09-16T01:51:11.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f08a41599f0 0x7f08a41f04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08a4073a20 con 0x7f08a41599f0 2024-09-16T01:51:11.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f088406c2e0 0x7f088406e7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0898000f60 tx=0x7f0898007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.377+0000 7f08a8a00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f08a41599f0 0x7f08a41f04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.378+0000 7f08a8a00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f08a41599f0 0x7f08a41f04c0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.379+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a4073a20 con 0x7f08a41599f0 2024-09-16T01:51:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.394+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a41e4110 con 0x7f08a41599f0 2024-09-16T01:51:11.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.396+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a41e4110 con 0x7f08a41599f0 2024-09-16T01:51:11.397 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:11.398 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:11.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.397+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f08a414bce0 0x7f08a4148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.397+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f08a41e2260 con 0x7f08a414bce0 2024-09-16T01:51:11.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.397+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f08a414bce0 0x7f08a4148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.397+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f08a414bce0 0x7f08a4148990 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.398+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a41e2260 con 0x7f08a414bce0 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:19 3.1f 3:f8cab2eb:::10000000002.00000010:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.010659064999999, 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075358271000000004, 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.411 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 19 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398086+0000", 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2149999999999998e-06 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438214+0000", 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040127599999999999 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.413 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438231+0000", 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7028e-05 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438238+0000", 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438293+0000", 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5007999999999999e-05 2024-09-16T01:51:11.414 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438402+0000", 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000109379 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438491+0000", 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8774000000000001e-05 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.442442+0000", 2024-09-16T01:51:11.415 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039511320000000004 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473344+0000", 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030902466999999999 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473394+0000", 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9264000000000001e-05 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.416 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473438+0000", 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4579000000000001e-05 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:24 3.0 3:028eacff:::10000000002.00000015:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.966777670999999, 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.073041954000000006, 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.417 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 24 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441963+0000", 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7379999999999999e-06 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508113+0000", 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066149523000000002 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508127+0000", 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4678000000000001e-05 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508135+0000", 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4959999999999999e-06 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508200+0000", 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5062999999999997e-05 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508374+0000", 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000174322 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508512+0000", 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013740099999999999 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514816+0000", 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063044199999999998 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514914+0000", 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7590000000000006e-05 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514951+0000", 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7348999999999998e-05 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.515003+0000", 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2373999999999997e-05 2024-09-16T01:51:11.422 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.885538910000001, 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.423 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:51:11.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:51:11.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:51:11.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:50 3.17 3:eaf1bb0b:::10000000002.0000002f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.692083517, 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058893003999999999, 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 50 2024-09-16T01:51:11.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716657+0000", 2024-09-16T01:51:11.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9290000000000001e-06 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747413+0000", 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030755714 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747428+0000", 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4452e-05 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747438+0000", 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0436e-05 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747490+0000", 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1632000000000003e-05 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747608+0000", 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000117934 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.436 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747716+0000", 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107953 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.751490+0000", 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037745750000000001 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775454+0000", 2024-09-16T01:51:11.437 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023963661000000001 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775504+0000", 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0003999999999998e-05 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775549+0000", 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4713999999999999e-05 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.438 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:63 3.17 3:ead2f1a2:::10000000002.0000003c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.789728+0000", 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.619011144000002, 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046076147999999997, 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 63 2024-09-16T01:51:11.439 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.789728+0000", 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.789728+0000", 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.789730+0000", 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9999999999999999e-06 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.821221+0000", 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031491096000000003 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.821245+0000", 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3969999999999999e-05 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.441 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.821289+0000", 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4456999999999998e-05 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.821385+0000", 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5043000000000005e-05 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.821493+0000", 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108718 2024-09-16T01:51:11.442 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.821589+0000", 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6150000000000006e-05 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.825051+0000", 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034616650000000001 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.443 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.835724+0000", 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010672922 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.835762+0000", 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7567000000000001e-05 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.835804+0000", 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2559999999999999e-05 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.444 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:80 3.3d 3:be92a730:::10000000002.0000004d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.949123+0000", 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.459615639999999, 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045371450000000001, 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.445 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 80 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.949123+0000", 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.949123+0000", 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.949126+0000", 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.43e-06 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.987748+0000", 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038622525999999997 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.987773+0000", 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4148999999999999e-05 2024-09-16T01:51:11.447 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.987784+0000", 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1103e-05 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.987858+0000", 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3863999999999996e-05 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.987970+0000", 2024-09-16T01:51:11.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112446 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.988060+0000", 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0180999999999996e-05 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.991602+0000", 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003541981 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.994423+0000", 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028206310000000001 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.994460+0000", 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7168000000000001e-05 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.994495+0000", 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4971000000000001e-05 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.450 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:99 3.29 3:94fce3f9:::10000000002.00000060:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.143942+0000", 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.264796666999999, 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047097868000000001, 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.451 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 99 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.143942+0000", 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.143942+0000", 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.143944+0000", 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.699e-06 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.165963+0000", 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022019064000000001 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.165976+0000", 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2616000000000001e-05 2024-09-16T01:51:11.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.165983+0000", 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0500000000000003e-06 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.166047+0000", 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4539e-05 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.166165+0000", 2024-09-16T01:51:11.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000117458 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.166264+0000", 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9622000000000003e-05 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.169143+0000", 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028782479999999999 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.190946+0000", 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.021802788 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.190995+0000", 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9363000000000002e-05 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.191040+0000", 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5420999999999998e-05 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.456 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:113 3.0 3:03abc375:::10000000002.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.256493+0000", 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.152246385000002, 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046521774000000002, 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.457 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 113 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.256493+0000", 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.256493+0000", 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.256495+0000", 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8500000000000001e-06 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.296265+0000", 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039770207000000002 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.296278+0000", 2024-09-16T01:51:11.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3413e-05 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.296285+0000", 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7870000000000004e-06 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.296337+0000", 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2092000000000002e-05 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.460 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.296465+0000", 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012790499999999999 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.296576+0000", 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110709 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.299464+0000", 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002888439 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.302948+0000", 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003483646 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.302981+0000", 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3719e-05 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.303014+0000", 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3006999999999998e-05 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.462 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:122 3.0 3:00eacacc:::10000000002.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.038825664000001, 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068671870999999995, 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.463 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 122 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369915+0000", 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5990000000000001e-06 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434232+0000", 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064317038000000007 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434250+0000", 2024-09-16T01:51:11.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8028000000000001e-05 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434258+0000", 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6720000000000004e-06 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434311+0000", 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2914999999999997e-05 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434411+0000", 2024-09-16T01:51:11.466 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100045 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434501+0000", 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494000000000005e-05 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.437130+0000", 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026284059999999998 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.467 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438521+0000", 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013915290000000001 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438553+0000", 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1992999999999997e-05 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438585+0000", 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2152e-05 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.468 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:134 3.17 3:e85118c5:::10000000002.00000083:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.944121795000001, 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061709079999999999, 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.469 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 134 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464620+0000", 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8499999999999998e-06 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518730+0000", 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054109808000000002 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518748+0000", 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8289000000000001e-05 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518756+0000", 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8539999999999997e-06 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518809+0000", 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2879e-05 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518914+0000", 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104719 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.519010+0000", 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6248000000000004e-05 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.522952+0000", 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039422440000000001 2024-09-16T01:51:11.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526248+0000", 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003296003 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526290+0000", 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1708000000000001e-05 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526326+0000", 2024-09-16T01:51:11.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6477999999999999e-05 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:136 3.1e 3:7b5aaf6d:::10000000002.00000085:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.522050+0000", 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.886688642999999, 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045368964999999997, 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.475 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 136 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.522050+0000", 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.522050+0000", 2024-09-16T01:51:11.476 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.522053+0000", 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.097e-06 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556320+0000", 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034266985999999999 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556345+0000", 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5171999999999999e-05 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556353+0000", 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4400000000000005e-06 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556423+0000", 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0117999999999995e-05 2024-09-16T01:51:11.478 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556605+0000", 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00018205599999999999 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556815+0000", 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00020950499999999999 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.562573+0000", 2024-09-16T01:51:11.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057577990000000001 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.567293+0000", 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004720223 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.567367+0000", 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4196000000000006e-05 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.567419+0000", 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2373000000000002e-05 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:142 3.31 3:8d07500b:::10000000002.0000008b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.556392+0000", 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.852347347999999, 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050701733999999998, 2024-09-16T01:51:11.481 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 142 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556392+0000", 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556392+0000", 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.556393+0000", 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3400000000000001e-06 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.588795+0000", 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032401606999999999 2024-09-16T01:51:11.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.588812+0000", 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7547e-05 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.588820+0000", 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9820000000000008e-06 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.588880+0000", 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0146999999999999e-05 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.588990+0000", 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000109499 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.589089+0000", 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9145e-05 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.591498+0000", 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024092779999999999 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.607005+0000", 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015506984 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.607044+0000", 2024-09-16T01:51:11.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8745000000000003e-05 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.607093+0000", 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9459999999999997e-05 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:173 3.38 3:1ee175b9:::10000000002.000000aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.846328+0000", 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.562411212000001, 2024-09-16T01:51:11.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052069183999999998, 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 173 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.846328+0000", 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.488 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.846328+0000", 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.846329+0000", 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4750000000000001e-06 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.878246+0000", 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031916739 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.878265+0000", 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9040000000000001e-05 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.878274+0000", 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8920000000000006e-06 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.490 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.878338+0000", 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4307999999999995e-05 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.878481+0000", 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000142535 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.878582+0000", 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010088400000000001 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.881847+0000", 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032649739999999999 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.898230+0000", 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016383393999999999 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.898318+0000", 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7447999999999994e-05 2024-09-16T01:51:11.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.898397+0000", 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9494999999999997e-05 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:182 3.23 3:c56934e7:::10000000002.000000b3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.974016+0000", 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.434723341000002, 2024-09-16T01:51:11.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050008195999999998, 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 182 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.974016+0000", 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.974016+0000", 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.974018+0000", 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.204e-06 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.018343+0000", 2024-09-16T01:51:11.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044325126999999999 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.018364+0000", 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0866e-05 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.018372+0000", 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9799999999999998e-06 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.018432+0000", 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0297999999999998e-05 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.018544+0000", 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011204099999999999 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.018642+0000", 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7410000000000004e-05 2024-09-16T01:51:11.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.022267+0000", 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036248489999999999 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.023917+0000", 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001650863 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.023976+0000", 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8730999999999997e-05 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.024024+0000", 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7827000000000001e-05 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:452 3.3e 3:7c415ecd:::10000000002.000000c1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:59.705900+0000", 2024-09-16T01:51:11.499 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.702838634000001, 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049869273999999998, 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 452 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.500 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.705900+0000", 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.705900+0000", 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.705902+0000", 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.412e-06 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.751419+0000", 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045517047999999997 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.751432+0000", 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2781e-05 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.751439+0000", 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.99e-06 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.751502+0000", 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3132999999999993e-05 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.751596+0000", 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4252999999999994e-05 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.751702+0000", 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000105455 2024-09-16T01:51:11.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.754863+0000", 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031613919999999998 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.755700+0000", 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00083697199999999995 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.755735+0000", 2024-09-16T01:51:11.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4879000000000003e-05 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.755770+0000", 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4959000000000002e-05 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:472 3.31 3:8c3036ad:::10000000002.000000d5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.465062573000001, 2024-09-16T01:51:11.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060972124000000003, 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 472 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943678+0000", 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5549999999999999e-06 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.507 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974973+0000", 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031295264000000003 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974987+0000", 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3349e-05 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974994+0000", 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3520000000000001e-06 2024-09-16T01:51:11.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975074+0000", 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9864000000000006e-05 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975140+0000", 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6377999999999994e-05 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975233+0000", 2024-09-16T01:51:11.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2359000000000001e-05 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.978088+0000", 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028552489999999998 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004578+0000", 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026490519000000001 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004615+0000", 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6455999999999998e-05 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004649+0000", 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3779000000000003e-05 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:481 3.1e 3:7aa08efe:::10000000002.000000de:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.511 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.054241+0000", 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.354497802999999, 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054298632999999999, 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 481 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.054241+0000", 2024-09-16T01:51:11.512 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.054241+0000", 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.054243+0000", 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.624e-06 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.093593+0000", 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039349793000000001 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.093608+0000", 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5271e-05 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.093617+0000", 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9570000000000008e-06 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.093677+0000", 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0300999999999997e-05 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.093745+0000", 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7823000000000004e-05 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.515 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.093845+0000", 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9963999999999994e-05 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096922+0000", 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030767020000000002 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.108467+0000", 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011545339 2024-09-16T01:51:11.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.108507+0000", 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9662000000000001e-05 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.108540+0000", 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3197000000000001e-05 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:483 3.27 3:e6494e75:::10000000002.000000e0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.311958476999999, 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059372405000000003, 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 483 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096782+0000", 2024-09-16T01:51:11.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137952+0000", 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041169532000000002 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137964+0000", 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1725999999999999e-05 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137988+0000", 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4020999999999999e-05 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138046+0000", 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8585000000000002e-05 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138124+0000", 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7312999999999997e-05 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138239+0000", 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000115278 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.143288+0000", 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050496630000000002 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156064+0000", 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012775269000000001 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156103+0000", 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8957e-05 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156153+0000", 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0324000000000001e-05 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:500 3.29 3:96fc3913:::10000000002.000000f1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.225014+0000", 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.183724765000001, 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048112883000000002, 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 500 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.225014+0000", 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.225014+0000", 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.225016+0000", 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9099999999999999e-06 2024-09-16T01:51:11.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255055+0000", 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030038624999999999 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255075+0000", 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9936999999999999e-05 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255086+0000", 2024-09-16T01:51:11.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.118e-05 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255182+0000", 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5775999999999999e-05 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255283+0000", 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101197 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:11.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255430+0000", 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014669999999999999 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.259648+0000", 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042183089999999999 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.273050+0000", 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013401621000000001 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.273089+0000", 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9410999999999999e-05 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.273127+0000", 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8216999999999997e-05 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.529 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:501 3.1f 3:fa3756cc:::10000000002.000000f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.255127+0000", 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.153612154999999, 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046665076, 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 501 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255127+0000", 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255127+0000", 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.255128+0000", 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0899999999999999e-06 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.278981+0000", 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023852885000000001 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.278994+0000", 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3298999999999999e-05 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.279000+0000", 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:51:11.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.279045+0000", 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4671e-05 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.279106+0000", 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1562000000000005e-05 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.279197+0000", 2024-09-16T01:51:11.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1195999999999999e-05 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.282139+0000", 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002941745 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.301643+0000", 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.019503669000000001 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.301723+0000", 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0241000000000006e-05 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.301792+0000", 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8977e-05 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:11.535 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.427+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a40c8c40 con 0x7f08a414bce0 2024-09-16T01:51:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.428+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a40c8c40 con 0x7f08a414bce0 2024-09-16T01:51:11.537 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:11.538 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:11.539 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:11.535+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.428+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f08a4149290 0x7f08a41f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.429+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f08a41f25b0 con 0x7f08a4149290 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.429+0000 7f08a8a00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f08a4149290 0x7f08a41f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.429+0000 7f08a8a00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f08a4149290 0x7f08a41f2070 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.430+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a41f25b0 con 0x7f08a4149290 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.444+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a4062630 con 0x7f08a4149290 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.446+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a4062630 con 0x7f08a4149290 2024-09-16T01:51:11.540 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.446+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f08a4063110 0x7f08a414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.446+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f08a4062630 con 0x7f08a4063110 2024-09-16T01:51:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.446+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f08a4063110 0x7f08a414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.447+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f08a4063110 0x7f08a414caf0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.447+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a4062630 con 0x7f08a4063110 2024-09-16T01:51:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.461+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a4062630 con 0x7f08a4063110 2024-09-16T01:51:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.463+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a4062630 con 0x7f08a4063110 2024-09-16T01:51:11.542 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.463+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f08a414d540 0x7f08a402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.463+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f08a414d960 con 0x7f08a414d540 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.463+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f08a414d540 0x7f08a402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.464+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f08a414d540 0x7f08a402c640 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.465+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a414d960 con 0x7f08a414d540 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.479+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a414d960 con 0x7f08a414d540 2024-09-16T01:51:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.481+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a414d960 con 0x7f08a414d540 2024-09-16T01:51:11.544 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.481+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f08a402d090 0x7f08a402d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.481+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f08a402da50 con 0x7f08a402d090 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.482+0000 7f08a8a00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f08a402d090 0x7f08a402d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.482+0000 7f08a8a00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f08a402d090 0x7f08a402d510 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.483+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a402da50 con 0x7f08a402d090 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.497+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a402da50 con 0x7f08a402d090 2024-09-16T01:51:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.498+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a402da50 con 0x7f08a402d090 2024-09-16T01:51:11.546 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:11.546 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.499+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f08a42033f0 0x7f08a42057e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.499+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f08a42085b0 con 0x7f08a42033f0 2024-09-16T01:51:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.499+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f08a42033f0 0x7f08a42057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.499+0000 7f08a9e00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f08a42033f0 0x7f08a42057e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.500+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a42085b0 con 0x7f08a42033f0 2024-09-16T01:51:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.500+0000 7ff6dbe00640 1 Processor -- start 2024-09-16T01:51:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.501+0000 7ff6dbe00640 1 -- start start 2024-09-16T01:51:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6dbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc157d40 0x7ff6dc0c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6dbe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc0c1150 con 0x7ff6dc1515f0 2024-09-16T01:51:11.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6dbe00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc073350 con 0x7ff6dc1519d0 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6dbe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc073630 con 0x7ff6dc157d40 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6d2000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff6dc1519d0 0x7ff6dc157630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50754/0 (socket says 172.21.15.146:50754) 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6d2000640 1 -- 172.21.15.146:0/3854699246 learned_addr learned my addr 172.21.15.146:0/3854699246 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6daa00640 1 --1- 172.21.15.146:0/3854699246 >> v1:172.21.15.146:6789/0 conn(0x7ff6dc1515f0 0x7ff6dc0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:45714/0 (socket says 172.21.15.146:45714) 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6db400640 1 --2- 172.21.15.146:0/3854699246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc157d40 0x7ff6dc0c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6da000640 1 -- 172.21.15.146:0/3854699246 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922584177 0 0) 0x7ff6dc073350 con 0x7ff6dc1519d0 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6da000640 1 -- 172.21.15.146:0/3854699246 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c0003520 con 0x7ff6dc1519d0 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6db400640 1 -- 172.21.15.146:0/3854699246 >> v1:172.21.15.161:6789/0 conn(0x7ff6dc1519d0 legacy=0x7ff6dc157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.502+0000 7ff6da000640 1 -- 172.21.15.146:0/3854699246 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3492432494 0 0) 0x7ff6dc0c1150 con 0x7ff6dc1515f0 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.503+0000 7ff6db400640 1 -- 172.21.15.146:0/3854699246 >> v1:172.21.15.146:6789/0 conn(0x7ff6dc1515f0 legacy=0x7ff6dc0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.503+0000 7ff6db400640 1 -- 172.21.15.146:0/3854699246 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc0c13f0 con 0x7ff6dc157d40 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.503+0000 7ff6db400640 1 --2- 172.21.15.146:0/3854699246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc157d40 0x7ff6dc0c8060 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009dc0 tx=0x7ff6d402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f8dd78ef450476b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6da000640 1 -- 172.21.15.146:0/3854699246 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4007740 con 0x7ff6dc157d40 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6da000640 1 -- 172.21.15.146:0/3854699246 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d40078e0 con 0x7ff6dc157d40 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6da000640 1 -- 172.21.15.146:0/3854699246 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4007cb0 con 0x7ff6dc157d40 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3854699246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc157d40 msgr2=0x7ff6dc0c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 --2- 172.21.15.146:0/3854699246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc157d40 0x7ff6dc0c8060 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009dc0 tx=0x7ff6d402a760 comp rx=0 tx=0).stop 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3854699246 shutdown_connections 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 --2- 172.21.15.146:0/3854699246 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc157d40 0x7ff6dc0c8060 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3854699246 >> 172.21.15.146:0/3854699246 conn(0x7ff6dc067950 msgr2=0x7ff6dc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3854699246 shutdown_connections 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.504+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3854699246 wait complete. 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.506+0000 7ff6dbe00640 1 Processor -- start 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.506+0000 7ff6dbe00640 1 -- start start 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6dbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc1515f0 0x7ff6dc154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6dbe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6dc1519d0 0x7ff6dc1e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6dbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6dc157d40 0x7ff6dc1ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6dbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc074060 con 0x7ff6dc157d40 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6dbe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc0c19b0 con 0x7ff6dc1519d0 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6dbe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc0741e0 con 0x7ff6dc1515f0 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6daa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc1515f0 0x7ff6dc154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6d2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6dc1519d0 0x7ff6dc1e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6daa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc1515f0 0x7ff6dc154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44852/0 (socket says 172.21.15.146:44852) 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6daa00640 1 -- 172.21.15.146:0/3161022228 learned_addr learned my addr 172.21.15.146:0/3161022228 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.507+0000 7ff6db400640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6dc157d40 0x7ff6dc1ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6daa00640 1 -- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6dc1519d0 msgr2=0x7ff6dc1e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6daa00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6dc1519d0 0x7ff6dc1e1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6daa00640 1 -- 172.21.15.146:0/3161022228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6dc157d40 msgr2=0x7ff6dc1ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6daa00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6dc157d40 0x7ff6dc1ebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6daa00640 1 -- 172.21.15.146:0/3161022228 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc0cb500 con 0x7ff6dc1515f0 2024-09-16T01:51:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6daa00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc1515f0 0x7ff6dc154370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00c4d0 tx=0x7ff6cc00c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc018070 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc007500 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc004a20 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1f0310 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.508+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1ef550 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.509+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6dc154de0 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.510+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff6cc004bc0 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.511+0000 7ff6d3e00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff6b806c2b0 0x7ff6b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.511+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7ff6cc08a800 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.511+0000 7ff6d3e00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff6b80744b0 0x7ff6b80768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.511+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6cc010e50 con 0x7ff6b80744b0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.511+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7ff6cc08abb0 con 0x7ff6dc1515f0 2024-09-16T01:51:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.512+0000 7ff6d2000640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff6b806c2b0 0x7ff6b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.512+0000 7ff6db400640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff6b80744b0 0x7ff6b80768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.512+0000 7ff6d2000640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff6b806c2b0 0x7ff6b806e770 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc156980 tx=0x7ff6c4007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.512+0000 7ff6db400640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff6b80744b0 0x7ff6b80768d0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.513+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6cc010e50 con 0x7ff6b80744b0 2024-09-16T01:51:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.516+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a4205d20 con 0x7f08a42033f0 2024-09-16T01:51:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.518+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a4205d20 con 0x7f08a42033f0 2024-09-16T01:51:11.557 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.518+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f08a4207c90 0x7f08a42080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.518+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f08a420b920 con 0x7f08a4207c90 2024-09-16T01:51:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.518+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f08a4207c90 0x7f08a42080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.519+0000 7f08a9400640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f08a4207c90 0x7f08a42080d0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.519+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08a420b920 con 0x7f08a4207c90 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.529+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7ff6dc072d90 con 0x7ff6b80744b0 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6d3e00640 1 -- 172.21.15.146:0/3161022228 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7ff6dc072d90 con 0x7ff6b80744b0 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff6b80744b0 msgr2=0x7ff6b80768d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6dbe00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7ff6b80744b0 0x7ff6b80768d0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff6b806c2b0 msgr2=0x7ff6b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6dbe00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff6b806c2b0 0x7ff6b806e770 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc156980 tx=0x7ff6c4007d30 comp rx=0 tx=0).stop 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc1515f0 msgr2=0x7ff6dc154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6dbe00640 1 --2- 172.21.15.146:0/3161022228 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff6dc1515f0 0x7ff6dc154370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00c4d0 tx=0x7ff6cc00c990 comp rx=0 tx=0).stop 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.530+0000 7ff6db400640 1 -- 172.21.15.146:0/3161022228 reap_dead start 2024-09-16T01:51:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.531+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 shutdown_connections 2024-09-16T01:51:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.531+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 >> 172.21.15.146:0/3161022228 conn(0x7ff6dc067950 msgr2=0x7ff6dc0cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.531+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 shutdown_connections 2024-09-16T01:51:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.531+0000 7ff6dbe00640 1 -- 172.21.15.146:0/3161022228 wait complete. 2024-09-16T01:51:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.533+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08a4205d20 con 0x7f08a4207c90 2024-09-16T01:51:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.535+0000 7f08a1600640 1 -- 172.21.15.146:0/672014487 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08a4205d20 con 0x7f08a4207c90 2024-09-16T01:51:11.560 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f08a41599f0 msgr2=0x7f08a41f04c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f08a41599f0 0x7f08a41f04c0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f08a414bce0 msgr2=0x7f08a4148990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f08a414bce0 0x7f08a4148990 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f08a4149290 msgr2=0x7f08a41f2070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f08a4149290 0x7f08a41f2070 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f08a4063110 msgr2=0x7f08a414caf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f08a4063110 0x7f08a414caf0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f08a414d540 msgr2=0x7f08a402c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f08a414d540 0x7f08a402c640 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f08a402d090 msgr2=0x7f08a402d510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f08a402d090 0x7f08a402d510 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 reap_dead start 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f08a42033f0 msgr2=0x7f08a42057e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f08a42033f0 0x7f08a42057e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 reap_dead start 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f08a4207c90 msgr2=0x7f08a42080d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f08a4207c90 0x7f08a42080d0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 reap_dead start 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08a9e00640 1 -- 172.21.15.146:0/672014487 reap_dead start 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f088406c2e0 msgr2=0x7f088406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f088406c2e0 0x7f088406e7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0898000f60 tx=0x7f0898007de0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 msgr2=0x7f08a41eff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 0x7f08a41eff80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f089c009eb0 tx=0x7f089c004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 shutdown_connections 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f088406c2e0 0x7f088406e7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 --2- 172.21.15.146:0/672014487 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08a4152c30 0x7f08a41eff80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 >> 172.21.15.146:0/672014487 conn(0x7f08a406bc10 msgr2=0x7f08a4150450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 shutdown_connections 2024-09-16T01:51:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.536+0000 7f08abe00640 1 -- 172.21.15.146:0/672014487 wait complete. 2024-09-16T01:51:11.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-16T01:51:11.637 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:11.636+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:11.682 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:11.683 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:11.691 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:11.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.759+0000 7fc65b000640 1 Processor -- start 2024-09-16T01:51:11.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.759+0000 7fc65b000640 1 -- start start 2024-09-16T01:51:11.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc65b000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6540c6410 0x7fc654158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc65b000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6540c5280 0x7fc6540c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc65b000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc654151e70 con 0x7fc654151a90 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc65b000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc6540c8130 con 0x7fc6540c6410 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc65b000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc6540c8410 con 0x7fc6540c5280 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc659000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6540c5280 0x7fc6540c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc653e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc654151a90 0x7fc6540c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50768/0 (socket says 172.21.15.146:50768) 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc653e00640 1 -- 172.21.15.146:0/2118000935 learned_addr learned my addr 172.21.15.146:0/2118000935 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc653400640 1 --2- 172.21.15.146:0/2118000935 >> v2:172.21.15.146:3300/0 conn(0x7fc6540c6410 0x7fc654158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc659000640 1 -- 172.21.15.146:0/2118000935 >> v2:172.21.15.146:3300/0 conn(0x7fc6540c6410 msgr2=0x7fc654158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc659000640 1 --2- 172.21.15.146:0/2118000935 >> v2:172.21.15.146:3300/0 conn(0x7fc6540c6410 0x7fc654158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc659000640 1 -- 172.21.15.146:0/2118000935 >> v1:172.21.15.161:6789/0 conn(0x7fc654151a90 legacy=0x7fc6540c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc659000640 1 -- 172.21.15.146:0/2118000935 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6540c86f0 con 0x7fc6540c5280 2024-09-16T01:51:11.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.761+0000 7fc652a00640 1 -- 172.21.15.146:0/2118000935 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322729112 0 0) 0x7fc654151e70 con 0x7fc654151a90 2024-09-16T01:51:11.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.762+0000 7fc659000640 1 --2- 172.21.15.146:0/2118000935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 0x7fc6540c56f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc65406f350 tx=0x7fc64c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=782b2317cf6f3072 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc652a00640 1 -- 172.21.15.146:0/2118000935 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c007740 con 0x7fc6540c5280 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc652a00640 1 -- 172.21.15.146:0/2118000935 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc64c0078e0 con 0x7fc6540c5280 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc652a00640 1 -- 172.21.15.146:0/2118000935 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c007cb0 con 0x7fc6540c5280 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 -- 172.21.15.146:0/2118000935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 msgr2=0x7fc6540c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 --2- 172.21.15.146:0/2118000935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 0x7fc6540c56f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc65406f350 tx=0x7fc64c02a680 comp rx=0 tx=0).stop 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 -- 172.21.15.146:0/2118000935 shutdown_connections 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 --2- 172.21.15.146:0/2118000935 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 0x7fc6540c56f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 --2- 172.21.15.146:0/2118000935 >> v2:172.21.15.146:3300/0 conn(0x7fc6540c6410 0x7fc654158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 -- 172.21.15.146:0/2118000935 >> 172.21.15.146:0/2118000935 conn(0x7fc65406f3c0 msgr2=0x7fc65406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 -- 172.21.15.146:0/2118000935 shutdown_connections 2024-09-16T01:51:11.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.763+0000 7fc65b000640 1 -- 172.21.15.146:0/2118000935 wait complete. 2024-09-16T01:51:11.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.765+0000 7fc65b000640 1 Processor -- start 2024-09-16T01:51:11.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.765+0000 7fc65b000640 1 -- start start 2024-09-16T01:51:11.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc65b000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 0x7fc6540cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc65b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6540c6410 0x7fc6540cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc65b000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc654151a90 0x7fc654153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc65b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc654157b20 con 0x7fc6540c6410 2024-09-16T01:51:11.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc65b000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc654157840 con 0x7fc6540c5280 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc65b000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc654158580 con 0x7fc654151a90 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc654151a90 0x7fc654153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc653400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6540c6410 0x7fc6540cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc653e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 0x7fc6540cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc654151a90 0x7fc654153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44856/0 (socket says 172.21.15.146:44856) 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 -- 172.21.15.146:0/3362909569 learned_addr learned my addr 172.21.15.146:0/3362909569 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 -- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 msgr2=0x7fc6540cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6540c5280 0x7fc6540cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 -- 172.21.15.146:0/3362909569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6540c6410 msgr2=0x7fc6540cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6540c6410 0x7fc6540cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc659000640 1 -- 172.21.15.146:0/3362909569 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc654146eb0 con 0x7fc654151a90 2024-09-16T01:51:11.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.766+0000 7fc653400640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6540c6410 0x7fc6540cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:11.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.767+0000 7fc659000640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc654151a90 0x7fc654153470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009e80 tx=0x7fc64c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.767+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c040070 con 0x7fc654151a90 2024-09-16T01:51:11.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.767+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc64c031c40 con 0x7fc654151a90 2024-09-16T01:51:11.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.767+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c009450 con 0x7fc654151a90 2024-09-16T01:51:11.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.767+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6541f2020 con 0x7fc654151a90 2024-09-16T01:51:11.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.767+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc654142a90 con 0x7fc654151a90 2024-09-16T01:51:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.768+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc654152340 con 0x7fc654151a90 2024-09-16T01:51:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.769+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc64c039040 con 0x7fc654151a90 2024-09-16T01:51:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc64be00640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc63406c2b0 0x7fc63406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7fc64c0ac5c0 con 0x7fc654151a90 2024-09-16T01:51:11.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc653e00640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc63406c2b0 0x7fc63406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc64be00640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc634074510 0x7fc634076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc64c04be50 con 0x7fc634074510 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fc64c0b3050 con 0x7fc654151a90 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.770+0000 7fc653400640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc634074510 0x7fc634076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.771+0000 7fc653e00640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc63406c2b0 0x7fc63406e770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc644007fd0 tx=0x7fc644007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.771+0000 7fc653400640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc634074510 0x7fc634076930 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:11.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.771+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c04be50 con 0x7fc634074510 2024-09-16T01:51:11.775 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:11.774+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.788+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fc654075810 con 0x7fc634074510 2024-09-16T01:51:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.788+0000 7fc64be00640 1 -- 172.21.15.146:0/3362909569 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fc654075810 con 0x7fc634074510 2024-09-16T01:51:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc634074510 msgr2=0x7fc634076930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fc634074510 0x7fc634076930 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc63406c2b0 msgr2=0x7fc63406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc63406c2b0 0x7fc63406e770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc644007fd0 tx=0x7fc644007de0 comp rx=0 tx=0).stop 2024-09-16T01:51:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc654151a90 msgr2=0x7fc654153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 --2- 172.21.15.146:0/3362909569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc654151a90 0x7fc654153470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009e80 tx=0x7fc64c004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc659000640 1 -- 172.21.15.146:0/3362909569 reap_dead start 2024-09-16T01:51:11.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 shutdown_connections 2024-09-16T01:51:11.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 >> 172.21.15.146:0/3362909569 conn(0x7fc65406f3c0 msgr2=0x7fc654155390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:11.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 shutdown_connections 2024-09-16T01:51:11.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:11.789+0000 7fc65b000640 1 -- 172.21.15.146:0/3362909569 wait complete. 2024-09-16T01:51:11.836 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:11.836 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:11.839 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:11.838+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:11.845 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:11.940 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:11.940+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:11.989 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:11.990 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:11.998 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:12.041 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:12.040+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:12.102 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:12.102 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:12.111 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:12.154 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:12.153+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:12.214 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:12.214 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:12.223 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:12.264 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:12.263+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:12.346 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:12.345+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:12.367 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:12.367 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:12.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:12.449 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:12.448+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:12.536 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:12.536 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:12.549 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:12.590 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:12.589+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:12.653 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:12.652+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:12.748 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:12.748 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:12.754 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:12.754+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:12.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:12.855 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:12.854+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:12.861 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:51:12.957 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:12.956+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:12.964 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:12.964 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:12.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.019+0000 7fe5d4600640 1 Processor -- start 2024-09-16T01:51:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.019+0000 7fe5d4600640 1 -- start start 2024-09-16T01:51:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.020+0000 7fe5d4600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe5cc0cb2f0 0x7fe5cc15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.020+0000 7fe5d4600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe5cc0cc4a0 0x7fe5cc165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.020+0000 7fe5d4600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5cc15b1d0 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.020+0000 7fe5d4600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe5cc0cbd00 con 0x7fe5cc0cb2f0 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.020+0000 7fe5d4600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe5cc15ba70 con 0x7fe5cc0cc4a0 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe5cc0cb2f0 0x7fe5cc15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1c00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe5cc0cc0c0 0x7fe5cc15e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60866/0 (socket says 172.21.15.146:60866) 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1c00640 1 -- 172.21.15.146:0/1657069794 learned_addr learned my addr 172.21.15.146:0/1657069794 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d2600640 1 --2- 172.21.15.146:0/1657069794 >> v2:172.21.15.161:3300/0 conn(0x7fe5cc0cc4a0 0x7fe5cc165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1200640 1 -- 172.21.15.146:0/1657069794 >> v2:172.21.15.161:3300/0 conn(0x7fe5cc0cc4a0 msgr2=0x7fe5cc165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1200640 1 --2- 172.21.15.146:0/1657069794 >> v2:172.21.15.161:3300/0 conn(0x7fe5cc0cc4a0 0x7fe5cc165430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1200640 1 -- 172.21.15.146:0/1657069794 >> v1:172.21.15.146:6789/0 conn(0x7fe5cc0cc0c0 legacy=0x7fe5cc15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5d1200640 1 -- 172.21.15.146:0/1657069794 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc15b4b0 con 0x7fe5cc0cb2f0 2024-09-16T01:51:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.021+0000 7fe5cbe00640 1 -- 172.21.15.146:0/1657069794 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582132594 0 0) 0x7fe5cc15b1d0 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.022+0000 7fe5d1200640 1 --2- 172.21.15.146:0/1657069794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc15aab0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc06e100 tx=0x7fe5b802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85a886b7dfd2b74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5cbe00640 1 -- 172.21.15.146:0/1657069794 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8037070 con 0x7fe5cc0cb2f0 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5cbe00640 1 -- 172.21.15.146:0/1657069794 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b8032440 con 0x7fe5cc0cb2f0 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5cbe00640 1 -- 172.21.15.146:0/1657069794 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8032810 con 0x7fe5cc0cb2f0 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 -- 172.21.15.146:0/1657069794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 msgr2=0x7fe5cc15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 --2- 172.21.15.146:0/1657069794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc15aab0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc06e100 tx=0x7fe5b802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 -- 172.21.15.146:0/1657069794 shutdown_connections 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 --2- 172.21.15.146:0/1657069794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc15aab0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 --2- 172.21.15.146:0/1657069794 >> v2:172.21.15.161:3300/0 conn(0x7fe5cc0cc4a0 0x7fe5cc165430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 -- 172.21.15.146:0/1657069794 >> 172.21.15.146:0/1657069794 conn(0x7fe5cc06d560 msgr2=0x7fe5cc06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 -- 172.21.15.146:0/1657069794 shutdown_connections 2024-09-16T01:51:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.023+0000 7fe5d4600640 1 -- 172.21.15.146:0/1657069794 wait complete. 2024-09-16T01:51:13.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.025+0000 7fe5d4600640 1 Processor -- start 2024-09-16T01:51:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.026+0000 7fe5d4600640 1 -- start start 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d4600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc14f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d4600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5cc0cc0c0 0x7fe5cc159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d4600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5cc0cc4a0 0x7fe5cc1f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d4600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc15e1d0 con 0x7fe5cc0cb2f0 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d4600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc15e540 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d4600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc0cc8b0 con 0x7fe5cc0cc4a0 2024-09-16T01:51:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc14f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc14f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60258/0 (socket says 172.21.15.146:60258) 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d1c00640 1 -- 172.21.15.146:0/2279329288 learned_addr learned my addr 172.21.15.146:0/2279329288 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.027+0000 7fe5d1200640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5cc0cc0c0 0x7fe5cc159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d2600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5cc0cc4a0 0x7fe5cc1f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d1200640 1 -- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5cc0cc4a0 msgr2=0x7fe5cc1f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d1200640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5cc0cc4a0 0x7fe5cc1f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d1200640 1 -- 172.21.15.146:0/2279329288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 msgr2=0x7fe5cc14f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d1200640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc14f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d1200640 1 -- 172.21.15.146:0/2279329288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc147580 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5d1200640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5cc0cc0c0 0x7fe5cc159d10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8031000 tx=0x7fe5b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.028+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8037050 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.029+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b8038040 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.029+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc147820 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.029+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8004430 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.029+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc15db90 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.030+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5cc1586f0 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.031+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe5b803c6d0 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.032+0000 7fe5c9600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe5ac06c2b0 0x7fe5ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.032+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7fe5b80abd20 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.033+0000 7fe5d1c00640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe5ac06c2b0 0x7fe5ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.033+0000 7fe5d1c00640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe5ac06c2b0 0x7fe5ac06e770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0000f60 tx=0x7fe5c0007e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.036+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe5cc1586f0 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.060 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:13.057+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:13.140 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:13.141 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:13.153 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:13.159 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:13.157+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:13.194 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:13.194 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:13.194 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:13.194 INFO:tasks.workunit.client.0.smithi146.stdout:iozone test complete. 2024-09-16T01:51:13.197 INFO:tasks.workunit.client.0.smithi146.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-09-16T01:51:13.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.227+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fe5cc0c7350 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.259 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:13.258+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Iozone: Performance Test of File I/O 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Version $Revision: 3.494 $ 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Compiled for 64 bit mode. 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Build: linux-AMD64 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-16T01:51:13.276 INFO:tasks.workunit.client.0.smithi146.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Run began: Mon Sep 16 01:51:13 2024 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Include close in write timing 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Include fsync in write timing 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: File size set to 1048576 kB 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Record Size 1024 kB 2024-09-16T01:51:13.277 INFO:tasks.workunit.client.0.smithi146.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: Output is in kBytes/sec 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: Time Resolution = 0.000001 seconds. 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: Processor cache size set to 1024 kBytes. 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: Processor cache line size set to 32 bytes. 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: File stride size set to 17 * record size. 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: Throughput test with 1 process 2024-09-16T01:51:13.278 INFO:tasks.workunit.client.0.smithi146.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-09-16T01:51:13.358 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:13.358 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:13.360 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:13.360+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:13.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:13.461 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:13.461+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:13.562 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:13.561+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:13.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:13.578 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:13.578 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:13.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:13.664 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:13.663+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:13.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.727+0000 7f5f65400640 1 Processor -- start 2024-09-16T01:51:13.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.728+0000 7f5f65400640 1 -- start start 2024-09-16T01:51:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f65400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f60152a10 0x7f5f600c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f65400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f60070470 con 0x7f5f600c95d0 2024-09-16T01:51:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f65400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f60070ae0 con 0x7f5f600c5140 2024-09-16T01:51:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f65400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5f600ca4f0 con 0x7f5f60152a10 2024-09-16T01:51:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f5e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f60152a10 0x7f5f600c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f5f200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5f600c95d0 0x7f5f601630b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60870/0 (socket says 172.21.15.146:60870) 2024-09-16T01:51:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.729+0000 7f5f5f200640 1 -- 172.21.15.146:0/683453959 learned_addr learned my addr 172.21.15.146:0/683453959 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.730+0000 7f5f5d400640 1 -- 172.21.15.146:0/683453959 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1396695729 0 0) 0x7f5f60070470 con 0x7f5f600c95d0 2024-09-16T01:51:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.730+0000 7f5f5d400640 1 -- 172.21.15.146:0/683453959 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f44003520 con 0x7f5f600c95d0 2024-09-16T01:51:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.730+0000 7f5f5d400640 1 -- 172.21.15.146:0/683453959 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2022270068 0 0) 0x7f5f60070ae0 con 0x7f5f600c5140 2024-09-16T01:51:13.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.730+0000 7f5f5e800640 1 -- 172.21.15.146:0/683453959 >> v1:172.21.15.161:6789/0 conn(0x7f5f600c5140 legacy=0x7f5f600ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.730+0000 7f5f5e800640 1 -- 172.21.15.146:0/683453959 >> v1:172.21.15.146:6789/0 conn(0x7f5f600c95d0 legacy=0x7f5f601630b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.730+0000 7f5f5e800640 1 -- 172.21.15.146:0/683453959 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f60070dc0 con 0x7f5f60152a10 2024-09-16T01:51:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.731+0000 7f5f5e800640 1 --2- 172.21.15.146:0/683453959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f600c8cf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5f500097a0 tx=0x7f5f5002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2d15e4553e0c3f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.731+0000 7f5f5d400640 1 -- 172.21.15.146:0/683453959 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50037070 con 0x7f5f60152a10 2024-09-16T01:51:13.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.731+0000 7f5f5d400640 1 -- 172.21.15.146:0/683453959 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f50032640 con 0x7f5f60152a10 2024-09-16T01:51:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.731+0000 7f5f5d400640 1 -- 172.21.15.146:0/683453959 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f500329f0 con 0x7f5f60152a10 2024-09-16T01:51:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 -- 172.21.15.146:0/683453959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 msgr2=0x7f5f600c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 --2- 172.21.15.146:0/683453959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f600c8cf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5f500097a0 tx=0x7f5f5002a2c0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 -- 172.21.15.146:0/683453959 shutdown_connections 2024-09-16T01:51:13.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 --2- 172.21.15.146:0/683453959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f600c8cf0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 -- 172.21.15.146:0/683453959 >> 172.21.15.146:0/683453959 conn(0x7f5f6006c2a0 msgr2=0x7f5f60067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:13.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 -- 172.21.15.146:0/683453959 shutdown_connections 2024-09-16T01:51:13.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.732+0000 7f5f65400640 1 -- 172.21.15.146:0/683453959 wait complete. 2024-09-16T01:51:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.734+0000 7f5f65400640 1 Processor -- start 2024-09-16T01:51:13.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.734+0000 7f5f65400640 1 -- start start 2024-09-16T01:51:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f65400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f600c5140 0x7f5f601f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f65400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5f600c95d0 0x7f5f601f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f65400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f6014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f65400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5f600c86f0 con 0x7f5f60152a10 2024-09-16T01:51:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f65400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5f60070130 con 0x7f5f600c5140 2024-09-16T01:51:13.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f65400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5f60152df0 con 0x7f5f600c95d0 2024-09-16T01:51:13.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f5f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f6014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f5de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5f600c95d0 0x7f5f601f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f5de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5f600c95d0 0x7f5f601f46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37410/0 (socket says 172.21.15.146:37410) 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f5f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f6014dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60268/0 (socket says 172.21.15.146:60268) 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f5de00640 1 -- 172.21.15.146:0/3449779529 learned_addr learned my addr 172.21.15.146:0/3449779529 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.735+0000 7f5f5e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f600c5140 0x7f5f601f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f5de00640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f600c5140 msgr2=0x7f5f601f11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f600c5140 0x7f5f601f11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f5de00640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 msgr2=0x7f5f6014dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f60152a10 0x7f5f6014dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f5de00640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f601f1730 con 0x7f5f600c95d0 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5f600c95d0 0x7f5f601f46d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00c410 tx=0x7f5f4c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c017070 con 0x7f5f600c95d0 2024-09-16T01:51:13.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f4c0035f0 con 0x7f5f600c95d0 2024-09-16T01:51:13.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0039e0 con 0x7f5f600c95d0 2024-09-16T01:51:13.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.736+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f601536a0 con 0x7f5f600c95d0 2024-09-16T01:51:13.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.737+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f60153c60 con 0x7f5f600c95d0 2024-09-16T01:51:13.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.737+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5f60153ea0 con 0x7f5f600c95d0 2024-09-16T01:51:13.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.740+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5f4c01a070 con 0x7f5f600c95d0 2024-09-16T01:51:13.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.740+0000 7f5f56a00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f4006c2b0 0x7f5f4006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.741+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 5741+0+0 (secure 0 0 0) 0x7f5f4c08aae0 con 0x7f5f600c95d0 2024-09-16T01:51:13.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.741+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v64) ==== 54+0+16 (secure 0 0 0) 0x7f5f60153ea0 con 0x7f5f600c95d0 2024-09-16T01:51:13.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.741+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f4006c2b0 0x7f5f4006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.741+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5f601f3d50 0x7f5f6014e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.741+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f600c4680 con 0x7f5f601f3d50 2024-09-16T01:51:13.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.742+0000 7f5f5f200640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5f601f3d50 0x7f5f6014e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.742+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f4006c2b0 0x7f5f4006e770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5f50034000 tx=0x7f5f5002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.742+0000 7f5f5f200640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5f601f3d50 0x7f5f6014e0f0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.743+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f600c4680 con 0x7f5f601f3d50 2024-09-16T01:51:13.765 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:13.764+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:13.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.773+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f600c8cf0 con 0x7f5f601f3d50 2024-09-16T01:51:13.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.774+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f600c8cf0 con 0x7f5f601f3d50 2024-09-16T01:51:13.776 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:13.777 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:13.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.775+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f5f6014eb60 0x7f5f6014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.775+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5f6014f1b0 con 0x7f5f6014eb60 2024-09-16T01:51:13.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.775+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f5f6014eb60 0x7f5f6014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.776+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f5f6014eb60 0x7f5f6014caf0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.776+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6014f1b0 con 0x7f5f6014eb60 2024-09-16T01:51:13.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.792+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f60156200 con 0x7f5f6014eb60 2024-09-16T01:51:13.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.793+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f60156200 con 0x7f5f6014eb60 2024-09-16T01:51:13.794 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:13.795 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:13.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.794+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5f60062630 0x7f5f60062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.794+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5f600ceb70 con 0x7f5f60062630 2024-09-16T01:51:13.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.794+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5f60062630 0x7f5f60062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.795+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5f60062630 0x7f5f60062a50 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.795+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f600ceb70 con 0x7f5f60062630 2024-09-16T01:51:13.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.810+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f6014bab0 con 0x7f5f60062630 2024-09-16T01:51:13.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.812+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f6014bab0 con 0x7f5f60062630 2024-09-16T01:51:13.813 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:13.813 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:13.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.812+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f5f60063350 0x7f5f6002cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.812+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5f6006e430 con 0x7f5f60063350 2024-09-16T01:51:13.814 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:13.814 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:13.814 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:13.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.812+0000 7f5f5f200640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f5f60063350 0x7f5f6002cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.813+0000 7f5f5f200640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f5f60063350 0x7f5f6002cb50 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.814+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6006e430 con 0x7f5f60063350 2024-09-16T01:51:13.827 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:13.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.830+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f6002d360 con 0x7f5f60063350 2024-09-16T01:51:13.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.832+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f6002d360 con 0x7f5f60063350 2024-09-16T01:51:13.833 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:13.834 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:13.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.832+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f5f6006cd20 0x7f5f6006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.832+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5f6002d2e0 con 0x7f5f6006cd20 2024-09-16T01:51:13.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.832+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f5f6006cd20 0x7f5f6006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.833+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f5f6006cd20 0x7f5f6006d140 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.834+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6002d2e0 con 0x7f5f6006cd20 2024-09-16T01:51:13.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.843+0000 7fe5c9600640 1 -- 172.21.15.146:0/2279329288 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v65) ==== 101+0+0 (secure 0 0 0) 0x7fe5cc0c7350 con 0x7fe5cc0cc0c0 2024-09-16T01:51:13.845 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:51:13.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe5ac06c2b0 msgr2=0x7fe5ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe5ac06c2b0 0x7fe5ac06e770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0000f60 tx=0x7fe5c0007e10 comp rx=0 tx=0).stop 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5cc0cc0c0 msgr2=0x7fe5cc159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5cc0cc0c0 0x7fe5cc159d10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8031000 tx=0x7fe5b80046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 shutdown_connections 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe5ac06c2b0 0x7fe5ac06e770 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5cc0cc4a0 0x7fe5cc1f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5cc0cc0c0 0x7fe5cc159d10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 --2- 172.21.15.146:0/2279329288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5cc0cb2f0 0x7fe5cc14f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 >> 172.21.15.146:0/2279329288 conn(0x7fe5cc06d560 msgr2=0x7fe5cc1527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 shutdown_connections 2024-09-16T01:51:13.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.846+0000 7fe5d4600640 1 -- 172.21.15.146:0/2279329288 wait complete. 2024-09-16T01:51:13.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.851+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f6014b7d0 con 0x7f5f6006cd20 2024-09-16T01:51:13.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.853+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f6014b7d0 con 0x7f5f6006cd20 2024-09-16T01:51:13.855 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:13.855 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:13.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.853+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5f6002d090 0x7f5f6006db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.854+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f5f6006e0d0 con 0x7f5f6002d090 2024-09-16T01:51:13.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.854+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5f6002d090 0x7f5f6006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.854+0000 7f5f5de00640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5f6002d090 0x7f5f6006db90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.855+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6006e0d0 con 0x7f5f6002d090 2024-09-16T01:51:13.866 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:13.866+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:13.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.873+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f6006e0d0 con 0x7f5f6002d090 2024-09-16T01:51:13.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.875+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f6006e0d0 con 0x7f5f6002d090 2024-09-16T01:51:13.877 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:13.878 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:13.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.875+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5f60207650 0x7f5f60209a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.875+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5f6006e440 con 0x7f5f60207650 2024-09-16T01:51:13.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.876+0000 7f5f5f200640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5f60207650 0x7f5f60209a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.876+0000 7f5f5f200640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5f60207650 0x7f5f60209a40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.877+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6006e440 con 0x7f5f60207650 2024-09-16T01:51:13.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.897+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f6006e440 con 0x7f5f60207650 2024-09-16T01:51:13.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.899+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f6006e440 con 0x7f5f60207650 2024-09-16T01:51:13.901 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.900+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f5f6020bc50 0x7f5f6020c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.900+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5f6020f9d0 con 0x7f5f6020bc50 2024-09-16T01:51:13.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.900+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f5f6020bc50 0x7f5f6020c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.901+0000 7f5f5e800640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f5f6020bc50 0x7f5f6020c0b0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:13.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.902+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f6020f9d0 con 0x7f5f6020bc50 2024-09-16T01:51:13.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.916+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5f60209fe0 con 0x7f5f6020bc50 2024-09-16T01:51:13.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f56a00640 1 -- 172.21.15.146:0/3449779529 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5f60209fe0 con 0x7f5f6020bc50 2024-09-16T01:51:13.919 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5f601f3d50 msgr2=0x7f5f6014e0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f5f601f3d50 0x7f5f6014e0f0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f5f6014eb60 msgr2=0x7f5f6014caf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f5f6014eb60 0x7f5f6014caf0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5f60062630 msgr2=0x7f5f60062a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5f60062630 0x7f5f60062a50 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f5f60063350 msgr2=0x7f5f6002cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f5f60063350 0x7f5f6002cb50 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f5f6006cd20 msgr2=0x7f5f6006d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f5f6006cd20 0x7f5f6006d140 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5f6002d090 msgr2=0x7f5f6006db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5f6002d090 0x7f5f6006db90 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5f60207650 msgr2=0x7f5f60209a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5f60207650 0x7f5f60209a40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f5f6020bc50 msgr2=0x7f5f6020c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f5f6020bc50 0x7f5f6020c0b0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f5f200640 1 -- 172.21.15.146:0/3449779529 reap_dead start 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f5f200640 1 -- 172.21.15.146:0/3449779529 reap_dead start 2024-09-16T01:51:13.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f4006c2b0 msgr2=0x7f5f4006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5f4006c2b0 0x7f5f4006e770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5f50034000 tx=0x7f5f5002e000 comp rx=0 tx=0).stop 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5f600c95d0 msgr2=0x7f5f601f46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.918+0000 7f5f65400640 1 --2- 172.21.15.146:0/3449779529 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5f600c95d0 0x7f5f601f46d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00c410 tx=0x7f5f4c00c8d0 comp rx=0 tx=0).stop 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f5f200640 1 -- 172.21.15.146:0/3449779529 reap_dead start 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f5f200640 1 -- 172.21.15.146:0/3449779529 reap_dead start 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f5f200640 1 -- 172.21.15.146:0/3449779529 reap_dead start 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 shutdown_connections 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 >> 172.21.15.146:0/3449779529 conn(0x7f5f6006c2a0 msgr2=0x7f5f60150230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 shutdown_connections 2024-09-16T01:51:13.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:13.919+0000 7f5f65400640 1 -- 172.21.15.146:0/3449779529 wait complete. 2024-09-16T01:51:13.984 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:13.979+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:14.052 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:14.053 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:14.053 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:14.053 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:14.053 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:14.072 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:14.067+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:14.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:14.169 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:14.168+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:14.269 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:14.268+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.083255575999999, 2024-09-16T01:51:14.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:51:14.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:51:14.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:51:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:51:14.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.348 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.770309801, 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:14.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.354 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:42 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.729227333000001, 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051426843, 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 42 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608927+0000", 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608928+0000", 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5689999999999999e-06 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648858+0000", 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039929485000000001 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648879+0000", 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1172e-05 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648889+0000", 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0512e-05 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.648975+0000", 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5473e-05 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649103+0000", 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012862999999999999 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.649230+0000", 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126561 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.655882+0000", 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066518819999999996 2024-09-16T01:51:14.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660228+0000", 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0043460699999999996 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660293+0000", 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5050999999999998e-05 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.660353+0000", 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0438e-05 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.616764118999999, 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:14.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:14.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:67 3.16 3:6a386151:::10000000002.00000040:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.502246286999998, 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046477360000000002, 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.368 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 67 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835908+0000", 2024-09-16T01:51:14.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.835910+0000", 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.872997+0000", 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.03708761 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873029+0000", 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1911e-05 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873046+0000", 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6702999999999999e-05 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873132+0000", 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6034999999999999e-05 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873288+0000", 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015656000000000001 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.873419+0000", 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000130381 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.880727+0000", 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0073081919999999998 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882276+0000", 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015495330000000001 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882336+0000", 2024-09-16T01:51:14.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9964999999999999e-05 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882385+0000", 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8588999999999998e-05 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:77 3.3c 3:3db0268f:::10000000002.0000004a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:14.374 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.428455133, 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051390551, 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 77 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:14.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909699+0000", 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.909702+0000", 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9699999999999999e-06 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927863+0000", 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018160824999999998 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927887+0000", 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4802000000000001e-05 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.927900+0000", 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928016+0000", 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116123 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928158+0000", 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141977 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.928294+0000", 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000135733 2024-09-16T01:51:14.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.938330+0000", 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010036092999999999 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.960987+0000", 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022656493 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961043+0000", 2024-09-16T01:51:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6416999999999999e-05 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.961089+0000", 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6152000000000001e-05 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:14.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.395904438999999, 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:14.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:14.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:14.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.317773850000002, 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:14.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:14.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.392 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:85 3.36 e55/20 3:6fe880e3:::10000000002.00000052:head v 55'2, mlcod=0'0)", 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.295387523999999, 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052325050999999997, 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042769+0000", 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3520000000000001e-06 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051520+0000", 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087513980000000005 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051666+0000", 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014630900000000001 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051711+0000", 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4768999999999998e-05 2024-09-16T01:51:14.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052370+0000", 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065890100000000004 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052478+0000", 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010805 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095028+0000", 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042550194 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095091+0000", 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3077999999999995e-05 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:98 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:14.397 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.199546646000002, 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052298391, 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 98 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:14.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138607+0000", 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.138610+0000", 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4990000000000001e-06 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166283+0000", 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027672980999999999 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166310+0000", 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7712000000000002e-05 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166324+0000", 2024-09-16T01:51:14.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3239e-05 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166414+0000", 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0062000000000006e-05 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166567+0000", 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015352700000000001 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.166726+0000", 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015874500000000001 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.187851+0000", 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021125117999999998 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190785+0000", 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029334970000000002 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190850+0000", 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5560000000000002e-05 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.190906+0000", 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5451000000000002e-05 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.403 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:140 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.782798720999999, 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050007939000000001, 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 140 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555355+0000", 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.555356+0000", 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1039999999999999e-06 2024-09-16T01:51:14.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591189+0000", 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035832756 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591217+0000", 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8263e-05 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591230+0000", 2024-09-16T01:51:14.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2965999999999999e-05 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591354+0000", 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012385200000000001 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591571+0000", 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000217099 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:14.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591739+0000", 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016809299999999999 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.598744+0000", 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0070049939999999996 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605250+0000", 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0065054079999999999 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605310+0000", 2024-09-16T01:51:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0263000000000002e-05 2024-09-16T01:51:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.605363+0000", 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3140999999999998e-05 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.745560963999999, 2024-09-16T01:51:14.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:14.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:14.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:14.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:14.416 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.52140443, 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:14.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:14.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:14.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:181 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.441424709, 2024-09-16T01:51:14.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046020995000000002, 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 181 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896729+0000", 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.896731+0000", 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4419999999999999e-06 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931107+0000", 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034375920999999997 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931130+0000", 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3668000000000001e-05 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931141+0000", 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0366999999999999e-05 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931281+0000", 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140622 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931432+0000", 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150469 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.931599+0000", 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167338 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.940280+0000", 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086811649999999994 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942632+0000", 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002352224 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942694+0000", 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2014999999999995e-05 2024-09-16T01:51:14.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.942750+0000", 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5763999999999997e-05 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.224127407000001, 2024-09-16T01:51:14.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:14.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:14.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.751319554, 2024-09-16T01:51:14.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:14.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.563492968, 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:14.440 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:14.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:14.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:463 3.26 3:65689812:::10000000002.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:14.446 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.472319671999999, 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112035999999998, 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 463 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865834+0000", 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.865836+0000", 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9719999999999999e-06 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883238+0000", 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017402081999999999 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883259+0000", 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1143999999999999e-05 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883269+0000", 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7999999999999993e-06 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883356+0000", 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7114999999999995e-05 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883467+0000", 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110513 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.883617+0000", 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150216 2024-09-16T01:51:14.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.906097+0000", 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022480257 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914840+0000", 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087431309999999995 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914898+0000", 2024-09-16T01:51:14.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7643999999999999e-05 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.914946+0000", 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8161999999999997e-05 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:484 3.e 3:723b89cc:::10000000002.000000e1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.230450203, 2024-09-16T01:51:14.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054305622999999997, 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 484 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107706+0000", 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145546+0000", 2024-09-16T01:51:14.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037839795000000002 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145584+0000", 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8556000000000002e-05 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145598+0000", 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4238e-05 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145708+0000", 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109584 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145834+0000", 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012588200000000001 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145990+0000", 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015650199999999999 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.160245+0000", 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014254119000000001 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161887+0000", 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016424009999999999 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161956+0000", 2024-09-16T01:51:14.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8996000000000001e-05 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.162009+0000", 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3381999999999998e-05 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.054676562999999, 2024-09-16T01:51:14.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:14.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:14.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:14.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:14.464 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:14.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:14.466 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:14.465+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:14.567 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:14.566+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:14.667 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:14.666+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:14.768 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:14.768+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:14.813 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:14.814 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:14.814 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:14.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:14.869 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:14.868+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:14.971 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:14.969+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:15.072 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:15.071+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:15.147 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:15.147 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:15.147 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:15.147 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:15.147 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:15.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:15.172 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:15.171+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:15.273 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:15.272+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:15.375 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:15.373+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:15.479 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:15.477+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:15.510 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.250675173000001, 2024-09-16T01:51:15.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.512 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:15.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:15.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.052302303000001, 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:15.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:15.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.527 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.943499672000002, 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:15.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:15.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.623931038999999, 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:15.534 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:15.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:15.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:15.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.542 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.125268298999998, 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.543 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:15.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:15.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:160 3.9 3:93988fc9:::10000000002.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:15.552 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.756278221999999, 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056552534000000002, 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 160 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.553 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749679+0000", 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.956e-06 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794169+0000", 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04449061 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794202+0000", 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2626999999999998e-05 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794217+0000", 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794305+0000", 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8158999999999994e-05 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794460+0000", 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155277 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794625+0000", 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164854 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.801888+0000", 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072630239999999999 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806094+0000", 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042063919999999998 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806178+0000", 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3287000000000004e-05 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806229+0000", 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765000000000003e-05 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.711695615, 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:15.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:15.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:164 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.683823598, 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053828972000000003, 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 164 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822132+0000", 2024-09-16T01:51:15.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822134+0000", 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841247+0000", 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019113208 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841279+0000", 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2265999999999999e-05 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841291+0000", 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2305e-05 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841376+0000", 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4507999999999998e-05 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841535+0000", 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159216 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.841687+0000", 2024-09-16T01:51:15.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151846 2024-09-16T01:51:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.859703+0000", 2024-09-16T01:51:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018016197000000001 2024-09-16T01:51:15.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875876+0000", 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016172628000000001 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875929+0000", 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3108e-05 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.875960+0000", 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1491000000000001e-05 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.575 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:197 3.1d 3:bb6dfbbf:::10000000002.000000c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.398816540999999, 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057555969999999998, 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.576 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 197 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107141+0000", 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148704+0000", 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041563044 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148738+0000", 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4480999999999997e-05 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148754+0000", 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6021000000000001e-05 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148909+0000", 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154735 2024-09-16T01:51:15.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149071+0000", 2024-09-16T01:51:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016181600000000001 2024-09-16T01:51:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149226+0000", 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015550000000000001 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.158210+0000", 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089836529999999994 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164352+0000", 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061416810000000004 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164426+0000", 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4192999999999999e-05 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164695+0000", 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026864800000000002 2024-09-16T01:51:15.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.725322522999999, 2024-09-16T01:51:15.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:15.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:15.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:15.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.657217881999999, 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.593 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:15.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:15.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.514758217000001, 2024-09-16T01:51:15.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:15.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:15.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:15.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:15.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:15.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:15.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:499 3.2f 3:f78980a5:::10000000002.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.283054967, 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055062123999999997, 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 499 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222900+0000", 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.222902+0000", 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8199999999999999e-06 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261223+0000", 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038321506999999998 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261257+0000", 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3526999999999999e-05 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261272+0000", 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4748e-05 2024-09-16T01:51:15.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261399+0000", 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012685100000000001 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261500+0000", 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101556 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.261663+0000", 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016259900000000001 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.270356+0000", 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086932399999999996 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277838+0000", 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074817470000000004 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277906+0000", 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8454000000000006e-05 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.277962+0000", 2024-09-16T01:51:15.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6075000000000001e-05 2024-09-16T01:51:15.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.620 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.167291993999999, 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:15.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:15.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:529 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.537651+0000", 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.96830384300000005, 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056028116000000003, 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 529 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537651+0000", 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537651+0000", 2024-09-16T01:51:15.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537655+0000", 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4599999999999999e-06 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561331+0000", 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023676256 2024-09-16T01:51:15.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561354+0000", 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3404000000000002e-05 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561368+0000", 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:15.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561461+0000", 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3003000000000004e-05 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.562117+0000", 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065668599999999997 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.562219+0000", 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101128 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.579034+0000", 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016815192999999999 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.593564+0000", 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014530112 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.593631+0000", 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7574e-05 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.593679+0000", 2024-09-16T01:51:15.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7981e-05 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:544 3.14 3:28ae9428:::100000001f7.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.86191809200000002, 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064751230000000007, 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 544 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644039+0000", 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6619999999999999e-06 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665383+0000", 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021343964 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665408+0000", 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.561e-05 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665422+0000", 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3947999999999999e-05 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665607+0000", 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018433499999999999 2024-09-16T01:51:15.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665831+0000", 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022424399999999999 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665986+0000", 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155654 2024-09-16T01:51:15.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.682208+0000", 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016221920000000001 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708669+0000", 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026460513000000001 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708734+0000", 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4888000000000006e-05 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708788+0000", 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4491999999999999e-05 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:549 3.11 3:8ae93890:::100000001f7.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.81971807100000005, 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081050590000000006, 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 549 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.646 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686239+0000", 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754451+0000", 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068212138000000005 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754474+0000", 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2733000000000001e-05 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754483+0000", 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1349999999999998e-06 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754562+0000", 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8319999999999996e-05 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754725+0000", 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016298699999999999 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754875+0000", 2024-09-16T01:51:15.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149942 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.762160+0000", 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072859630000000003 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767169+0000", 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050088609999999999 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767231+0000", 2024-09-16T01:51:15.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1158e-05 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767288+0000", 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7108000000000003e-05 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:597 3.11 3:89a80ce8:::100000001f7.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.203535+0000", 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.30241991200000001, 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055367708000000002, 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.655 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.203535+0000", 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.203535+0000", 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.203537+0000", 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236338+0000", 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032800274999999997 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236359+0000", 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1372000000000001e-05 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236371+0000", 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2359e-05 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236534+0000", 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016273200000000001 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.237346+0000", 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081165000000000002 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.237822+0000", 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047652099999999999 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.251989+0000", 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014166972999999999 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.258782+0000", 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067926820000000004 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.258851+0000", 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9148999999999998e-05 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.258903+0000", 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1774999999999998e-05 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.661 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:606 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.231304024, 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058221427999999999, 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.662 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274653+0000", 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322327+0000", 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047674080000000001 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322357+0000", 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9813000000000001e-05 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322369+0000", 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2074e-05 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322494+0000", 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012469 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322653+0000", 2024-09-16T01:51:15.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158987 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322803+0000", 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150433 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331674+0000", 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088701219999999994 2024-09-16T01:51:15.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332772+0000", 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001098552 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332829+0000", 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6941000000000003e-05 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332873+0000", 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3516000000000001e-05 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:615 3.30 3:0f21c3d5:::100000001f7.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.382520+0000", 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.123435056, 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054458120999999998, 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:15.670 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 615 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.382520+0000", 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.382520+0000", 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.382522+0000", 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6050000000000001e-06 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.404430+0000", 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021908229000000001 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.404480+0000", 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9985000000000003e-05 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.404492+0000", 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2544999999999999e-05 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.404889+0000", 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039679899999999998 2024-09-16T01:51:15.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.405043+0000", 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015331500000000001 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.405191+0000", 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148813 2024-09-16T01:51:15.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.417977+0000", 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012785342 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.436314+0000", 2024-09-16T01:51:15.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018337636000000001 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.436925+0000", 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061089 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:15.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.436978+0000", 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2961999999999999e-05 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:15.678 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:15.680 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:15.687 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:15.681+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:15.781 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:15.780+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:15.796 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:15.796 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:15.796 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:15.797 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:15.806 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:15.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:51:15.882 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:15.881+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:15.937 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:15.939 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:15.939 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:15.939 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:15.939 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:15.939 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:15.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:15.987 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:15.986+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:16.085 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:16.084+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:16.109 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:16.109 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:16.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.556800365000001, 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.110 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:16.117 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.309766784000001, 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:16.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.250607698, 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.124 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:16.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.339029386, 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.130 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:16.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:16.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.294769156000001, 2024-09-16T01:51:16.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:16.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:16.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:16.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:16.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.264269336000002, 2024-09-16T01:51:16.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:16.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.222713813999999, 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:16.153 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:16.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:16.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:16.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:16.159 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.084667119999999, 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:16.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:16.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:16.165 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.965707393000001, 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.166 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:16.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:16.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.174 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.930911094000001, 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.175 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:16.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:16.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.184 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.847462140999999, 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:16.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:16.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.6481593540000001, 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.191 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:16.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:51:16.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:51:16.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:51:16.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.503282416, 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.198 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:51:16.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:51:16.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.204 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.4626379919999999, 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:51:16.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:51:16.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:51:16.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:51:16.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.212 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.3046951579999999, 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:51:16.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:51:16.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:51:16.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.1795678970000001, 2024-09-16T01:51:16.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:51:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:576 3.1b 3:d81af1ca:::100000001f7.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.0707652109999999, 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052702763999999999, 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 576 2024-09-16T01:51:16.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035658+0000", 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.03e-06 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075189+0000", 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039530957999999998 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075204+0000", 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5071e-05 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075214+0000", 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075299+0000", 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4807000000000003e-05 2024-09-16T01:51:16.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075401+0000", 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010220700000000001 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075495+0000", 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4006999999999996e-05 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.078964+0000", 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003468741 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088273+0000", 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0093088089999999995 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088311+0000", 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8402000000000003e-05 2024-09-16T01:51:16.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088359+0000", 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7920999999999997e-05 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:598 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.233 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.89068534499999996, 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054052254000000001, 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 598 2024-09-16T01:51:16.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215738+0000", 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.526e-06 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250373+0000", 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034634160999999997 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250391+0000", 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7986e-05 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250400+0000", 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1439999999999992e-06 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250455+0000", 2024-09-16T01:51:16.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5578999999999998e-05 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250562+0000", 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106717 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250665+0000", 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010307200000000001 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.253888+0000", 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032231019999999998 2024-09-16T01:51:16.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269702+0000", 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015813473000000002 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269742+0000", 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0154000000000002e-05 2024-09-16T01:51:16.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269788+0000", 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.634e-05 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:614 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.356692+0000", 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.74972966299999999, 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048508810999999999, 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 614 2024-09-16T01:51:16.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.356692+0000", 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.356692+0000", 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.356694+0000", 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2409999999999998e-06 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383766+0000", 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027072213000000001 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383784+0000", 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8198000000000001e-05 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383792+0000", 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6969999999999993e-06 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383871+0000", 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9102000000000003e-05 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383976+0000", 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104579 2024-09-16T01:51:16.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.384069+0000", 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3017999999999996e-05 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.387522+0000", 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003453065 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.405058+0000", 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017536169000000001 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.405136+0000", 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7762999999999994e-05 2024-09-16T01:51:16.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.405200+0000", 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4765999999999996e-05 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:16.249 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.142767966, 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:51:16.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:16.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:51:16.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:16.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:16.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:16.257 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:16.259 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.189+0000 7f5a0b000640 1 Processor -- start 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.189+0000 7f5a0b000640 1 -- start start 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.191+0000 7f5a0b000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a040c7310 0x7f5a040c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.191+0000 7f5a0b000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a04156830 con 0x7f5a0406c260 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.191+0000 7f5a0b000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a04071bc0 con 0x7f5a0406be80 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.191+0000 7f5a0b000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5a04156af0 con 0x7f5a040c7310 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.191+0000 7f5a03e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5a0406be80 0x7f5a04159de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57942/0 (socket says 172.21.15.146:57942) 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.191+0000 7f5a03e00640 1 -- 172.21.15.146:0/3214425560 learned_addr learned my addr 172.21.15.146:0/3214425560 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3103021328 0 0) 0x7f5a04071bc0 con 0x7f5a0406be80 2024-09-16T01:51:16.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59e8003520 con 0x7f5a0406be80 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 748704474 0 0) 0x7f59e8003520 con 0x7f5a0406be80 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a040c7310 msgr2=0x7f5a040c7da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a040c7310 0x7f5a040c7da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 >> v1:172.21.15.146:6789/0 conn(0x7f5a0406c260 legacy=0x7f5a0414d230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a04156dd0 con 0x7f5a0406be80 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f59f40019d0 con 0x7f5a0406be80 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 >> v1:172.21.15.161:6789/0 conn(0x7f5a0406be80 legacy=0x7f5a04159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e80039b0 0x7f59e8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.192+0000 7f5a02a00640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8004800 0x7f59e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.193+0000 7f5a02a00640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e8005230 0x7f59e80079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.193+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04156af0 con 0x7f59e8004800 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.193+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59e80081b0 con 0x7f59e8005230 2024-09-16T01:51:16.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.193+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f59e80083b0 con 0x7f59e80039b0 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.194+0000 7f5a03e00640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e80039b0 0x7f59e8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.196+0000 7f5a03400640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8004800 0x7f59e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.197+0000 7f5a09000640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e8005230 0x7f59e80079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.197+0000 7f5a09000640 1 -- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e80039b0 msgr2=0x7f59e8003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.197+0000 7f5a09000640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e80039b0 0x7f59e8003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.197+0000 7f5a09000640 1 -- 172.21.15.146:0/3214425560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8004800 msgr2=0x7f59e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.197+0000 7f5a09000640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8004800 0x7f59e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.197+0000 7f5a09000640 1 -- 172.21.15.146:0/3214425560 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a04156dd0 con 0x7f59e8005230 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.199+0000 7f5a09000640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e8005230 0x7f59e80079a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5a040b1710 tx=0x7f59e8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38825d860b8b57ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.200+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc037070 con 0x7f59e8005230 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.200+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc02a870 con 0x7f59e8005230 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.200+0000 7f5a02a00640 1 -- 172.21.15.146:0/3214425560 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc009350 con 0x7f59e8005230 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.201+0000 7f5a0b000640 1 -- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e8005230 msgr2=0x7f59e80079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.201+0000 7f5a0b000640 1 --2- 172.21.15.146:0/3214425560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e8005230 0x7f59e80079a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5a040b1710 tx=0x7f59e8005830 comp rx=0 tx=0).stop 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.201+0000 7f5a09000640 1 -- 172.21.15.146:0/3214425560 reap_dead start 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.202+0000 7f5a0b000640 1 -- 172.21.15.146:0/3214425560 shutdown_connections 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.202+0000 7f5a0b000640 1 -- 172.21.15.146:0/3214425560 >> 172.21.15.146:0/3214425560 conn(0x7f5a040cbf90 msgr2=0x7f5a040cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.202+0000 7f5a0b000640 1 -- 172.21.15.146:0/3214425560 shutdown_connections 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.202+0000 7f5a0b000640 1 -- 172.21.15.146:0/3214425560 wait complete. 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.206+0000 7f5a0b000640 1 Processor -- start 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.206+0000 7f5a0b000640 1 -- start start 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.208+0000 7f5a0b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 0x7f5a041ec070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.209+0000 7f5a03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 0x7f5a041ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.209+0000 7f5a03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 0x7f5a041ec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60284/0 (socket says 172.21.15.146:60284) 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.209+0000 7f5a03e00640 1 -- 172.21.15.146:0/347054878 learned_addr learned my addr 172.21.15.146:0/347054878 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.209+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a040ae340 0x7f5a041ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.209+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a0406cde0 0x7f5a041eec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.209+0000 7f5a03400640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a040ae340 0x7f5a041ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.210+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04071780 con 0x7f59e8005230 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.210+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04157390 con 0x7f5a040ae340 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.210+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5a04071900 con 0x7f5a0406cde0 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.211+0000 7f5a09000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a0406cde0 0x7f5a041eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.212+0000 7f5a03e00640 1 -- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a0406cde0 msgr2=0x7f5a041eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.212+0000 7f5a03e00640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a0406cde0 0x7f5a041eec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.212+0000 7f5a03e00640 1 -- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a040ae340 msgr2=0x7f5a041ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.212+0000 7f5a03e00640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a040ae340 0x7f5a041ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.212+0000 7f5a03e00640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a041edcd0 con 0x7f59e8005230 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.213+0000 7f5a03e00640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 0x7f5a041ec070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59f4007730 tx=0x7f59f4001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.214+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4017020 con 0x7f59e8005230 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.214+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f400ab80 con 0x7f59e8005230 2024-09-16T01:51:16.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.214+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f40133f0 con 0x7f59e8005230 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.214+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a041ec9a0 con 0x7f59e8005230 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.214+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041edf40 con 0x7f59e8005230 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.218+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc005290 con 0x7f59e8005230 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.218+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f59f401a020 con 0x7f59e8005230 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.220+0000 7f59fbe00640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59e406c330 0x7f59e406e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.220+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5741+0+0 (secure 0 0 0) 0x7f59f408b980 con 0x7f59e8005230 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.221+0000 7f5a03400640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59e406c330 0x7f59e406e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.222+0000 7f5a03400640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59e406c330 0x7f59e406e7f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5a040cbe80 tx=0x7f59f0006f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.229+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f59f4021020 con 0x7f59e8005230 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.230+0000 7f6138e00640 1 Processor -- start 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.230+0000 7f6138e00640 1 -- start start 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f6138e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c7ff0 0x7f61340c8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f6138e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61340c89c0 con 0x7f6134152c30 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f6138e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61340c4240 con 0x7f61340c36b0 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f6138e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f61340c71c0 con 0x7f61340c7ff0 2024-09-16T01:51:16.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f612be00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f61340c36b0 0x7f61340c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57946/0 (socket says 172.21.15.146:57946) 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f612be00640 1 -- 172.21.15.146:0/2635901492 learned_addr learned my addr 172.21.15.146:0/2635901492 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.232+0000 7f6132800640 1 --2- 172.21.15.146:0/2635901492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c7ff0 0x7f61340c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.233+0000 7f6131400640 1 -- 172.21.15.146:0/2635901492 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2600125713 0 0) 0x7f61340c4240 con 0x7f61340c36b0 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.233+0000 7f6132800640 1 -- 172.21.15.146:0/2635901492 >> v1:172.21.15.161:6789/0 conn(0x7f61340c36b0 legacy=0x7f61340c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.233+0000 7f6132800640 1 -- 172.21.15.146:0/2635901492 >> v1:172.21.15.146:6789/0 conn(0x7f6134152c30 legacy=0x7f61340c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.233+0000 7f6132800640 1 -- 172.21.15.146:0/2635901492 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61340c4500 con 0x7f61340c7ff0 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.234+0000 7f6132800640 1 --2- 172.21.15.146:0/2635901492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c7ff0 0x7f61340c8480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f612c009dc0 tx=0x7f612c02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=327e02c013d5c297 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.234+0000 7f6131400640 1 -- 172.21.15.146:0/2635901492 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c007740 con 0x7f61340c7ff0 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6131400640 1 -- 172.21.15.146:0/2635901492 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f612c0078e0 con 0x7f61340c7ff0 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6131400640 1 -- 172.21.15.146:0/2635901492 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c007cb0 con 0x7f61340c7ff0 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6138e00640 1 -- 172.21.15.146:0/2635901492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c7ff0 msgr2=0x7f61340c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6138e00640 1 --2- 172.21.15.146:0/2635901492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c7ff0 0x7f61340c8480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f612c009dc0 tx=0x7f612c02a6d0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6138e00640 1 -- 172.21.15.146:0/2635901492 shutdown_connections 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6138e00640 1 --2- 172.21.15.146:0/2635901492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c7ff0 0x7f61340c8480 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.235+0000 7f6138e00640 1 -- 172.21.15.146:0/2635901492 >> 172.21.15.146:0/2635901492 conn(0x7f613406bc10 msgr2=0x7f613406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.236+0000 7f6138e00640 1 -- 172.21.15.146:0/2635901492 shutdown_connections 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.236+0000 7f6138e00640 1 -- 172.21.15.146:0/2635901492 wait complete. 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.237+0000 7f6138e00640 1 Processor -- start 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.237+0000 7f6138e00640 1 -- start start 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.238+0000 7f6138e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c36b0 0x7f6134153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.238+0000 7f6138e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61340c7ff0 0x7f613414e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.238+0000 7f6138e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6134152c30 0x7f6134155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.238+0000 7f6138e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61340c4da0 con 0x7f61340c7ff0 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.238+0000 7f6138e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61340c4ac0 con 0x7f6134152c30 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.238+0000 7f6138e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f61340c6630 con 0x7f61340c36b0 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61340c7ff0 0x7f613414e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61340c7ff0 0x7f613414e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60296/0 (socket says 172.21.15.146:60296) 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f6131e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c36b0 0x7f6134153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 -- 172.21.15.146:0/1995436492 learned_addr learned my addr 172.21.15.146:0/1995436492 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6134152c30 0x7f6134155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c36b0 msgr2=0x7f6134153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61340c36b0 0x7f6134153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6134152c30 msgr2=0x7f6134155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6134152c30 0x7f6134155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.239+0000 7f612be00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61341f2d50 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.240+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61340c7ff0 0x7f613414e2c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f611c00c3f0 tx=0x7f611c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.240+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c0074e0 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.240+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f611c0047e0 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.240+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c004bf0 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.241+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61341f2ff0 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.241+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61341ef2d0 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.242+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f61340696d0 con 0x7f61340c7ff0 2024-09-16T01:51:16.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.244+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f611c011680 con 0x7f61340c7ff0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.246+0000 7f612a000640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f611006c2b0 0x7f611006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.246+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5741+0+0 (secure 0 0 0) 0x7f611c08f120 con 0x7f61340c7ff0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.246+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v65) ==== 54+0+16 (secure 0 0 0) 0x7f61340696d0 con 0x7f61340c7ff0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.247+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f611006c2b0 0x7f611006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.247+0000 7f6123400640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6104001650 0x7f6104003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.248+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6104006bb0 con 0x7f6104001650 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.248+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6104001650 0x7f6104003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.249+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6104001650 0x7f6104003b10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.250+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f611006c2b0 0x7f611006e770 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6124007fd0 tx=0x7f6124009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.251+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6104006bb0 con 0x7f6104001650 2024-09-16T01:51:16.281 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:16.258+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:16.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.299+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6104005c00 con 0x7f6104001650 2024-09-16T01:51:16.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.311+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6104005c00 con 0x7f6104001650 2024-09-16T01:51:16.315 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:16.316 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:16.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.315+0000 7f6123400640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6104004190 0x7f6104004640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.315+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6104004b80 con 0x7f6104004190 2024-09-16T01:51:16.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.316+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6104004190 0x7f6104004640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.317+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6104004190 0x7f6104004640 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.318+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6104004b80 con 0x7f6104004190 2024-09-16T01:51:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.349+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61040050f0 con 0x7f6104004190 2024-09-16T01:51:16.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.352+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61040050f0 con 0x7f6104004190 2024-09-16T01:51:16.353 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:16.354 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:16.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.352+0000 7f6123400640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6104009340 0x7f610400b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.352+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f610400e660 con 0x7f6104009340 2024-09-16T01:51:16.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.353+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6104009340 0x7f610400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.353+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6104009340 0x7f610400b730 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.355+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f610400e660 con 0x7f6104009340 2024-09-16T01:51:16.358 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:16.358+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:16.382 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:16.382 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:16.383 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:16.383 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:16.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.383+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f610400bfa0 con 0x7f6104009340 2024-09-16T01:51:16.392 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:16.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.390+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f610400bfa0 con 0x7f6104009340 2024-09-16T01:51:16.394 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:16.394 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:16.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.391+0000 7f6123400640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f610400c150 0x7f610400c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.391+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f610400cb20 con 0x7f610400c150 2024-09-16T01:51:16.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.391+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f610400c150 0x7f610400c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.392+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f610400c150 0x7f610400c5e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.395+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f610400cb20 con 0x7f610400c150 2024-09-16T01:51:16.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.424+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f610400d030 con 0x7f610400c150 2024-09-16T01:51:16.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.428+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f610400d030 con 0x7f610400c150 2024-09-16T01:51:16.429 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:16.429 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:16.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.428+0000 7f6123400640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6104010820 0x7f6104012c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.428+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6104015a60 con 0x7f6104010820 2024-09-16T01:51:16.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.429+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6104010820 0x7f6104012c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.433+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6104010820 0x7f6104012c10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.434+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6104015a60 con 0x7f6104010820 2024-09-16T01:51:16.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.468+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6104013420 con 0x7f6104010820 2024-09-16T01:51:16.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.470+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6104013420 con 0x7f6104010820 2024-09-16T01:51:16.475 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:16.475 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:16.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.472+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6100001650 0x7f6100003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.472+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6100006bb0 con 0x7f6100001650 2024-09-16T01:51:16.477 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:16.476+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:16.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.480+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6100001650 0x7f6100003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.482+0000 7f612be00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6100001650 0x7f6100003b10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.484+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6100006bb0 con 0x7f6100001650 2024-09-16T01:51:16.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.520+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61000058d0 con 0x7f6100001650 2024-09-16T01:51:16.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.522+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61000058d0 con 0x7f6100001650 2024-09-16T01:51:16.523 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:16.523 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:16.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.523+0000 7f6123400640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6104013b40 0x7f6104013f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.523+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6104018e30 con 0x7f6104013b40 2024-09-16T01:51:16.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.523+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6104013b40 0x7f6104013f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.532+0000 7f6132800640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6104013b40 0x7f6104013f80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.540+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6104018e30 con 0x7f6104013b40 2024-09-16T01:51:16.560 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:16.559+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:16.562 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:16.562 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:16.573 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:16.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.573+0000 7f6123400640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61040144c0 con 0x7f6104013b40 2024-09-16T01:51:16.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.575+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61040144c0 con 0x7f6104013b40 2024-09-16T01:51:16.576 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:16.577 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:16.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.576+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6100006020 0x7f6100006440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:16.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.576+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f610000a000 con 0x7f6100006020 2024-09-16T01:51:16.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.579+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6100006020 0x7f6100006440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:16.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.580+0000 7f6131e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6100006020 0x7f6100006440 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:16.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.581+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f610000a000 con 0x7f6100006020 2024-09-16T01:51:16.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.610+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6100008e80 con 0x7f6100006020 2024-09-16T01:51:16.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.612+0000 7f612a000640 1 -- 172.21.15.146:0/1995436492 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6100008e80 con 0x7f6100006020 2024-09-16T01:51:16.614 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:16.615 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:16.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6104001650 msgr2=0x7f6104003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6104001650 0x7f6104003b10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6104004190 msgr2=0x7f6104004640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6104004190 0x7f6104004640 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6104009340 msgr2=0x7f610400b730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6104009340 0x7f610400b730 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f610400c150 msgr2=0x7f610400c5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f610400c150 0x7f610400c5e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6104010820 msgr2=0x7f6104012c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6104010820 0x7f6104012c10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6100001650 msgr2=0x7f6100003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6100001650 0x7f6100003b10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6104013b40 msgr2=0x7f6104013f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6104013b40 0x7f6104013f80 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6100006020 msgr2=0x7f6100006440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.614+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6100006020 0x7f6100006440 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.615+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f611006c2b0 msgr2=0x7f611006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.615+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f611006c2b0 0x7f611006e770 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6124007fd0 tx=0x7f6124009040 comp rx=0 tx=0).stop 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.615+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61340c7ff0 msgr2=0x7f613414e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.615+0000 7f6138e00640 1 --2- 172.21.15.146:0/1995436492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61340c7ff0 0x7f613414e2c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f611c00c3f0 tx=0x7f611c00c8b0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.615+0000 7f6132800640 1 -- 172.21.15.146:0/1995436492 reap_dead start 2024-09-16T01:51:16.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.615+0000 7f6132800640 1 -- 172.21.15.146:0/1995436492 reap_dead start 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6132800640 1 -- 172.21.15.146:0/1995436492 reap_dead start 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6132800640 1 -- 172.21.15.146:0/1995436492 reap_dead start 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6132800640 1 -- 172.21.15.146:0/1995436492 reap_dead start 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 shutdown_connections 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 >> 172.21.15.146:0/1995436492 conn(0x7f613406bc10 msgr2=0x7f613414f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 shutdown_connections 2024-09-16T01:51:16.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.616+0000 7f6138e00640 1 -- 172.21.15.146:0/1995436492 wait complete. 2024-09-16T01:51:16.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.625+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f59dc005580 con 0x7f59e8005230 2024-09-16T01:51:16.679 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:16.678+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:16.730 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:16.731 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:16.742 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:16.786 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:16.785+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:16.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:16.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:16.806 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-16T01:51:16.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-09-16T01:51:16.862 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:16.861+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:16.863 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:16.863 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:16.873 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:16.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.904+0000 7f59fbe00640 1 -- 172.21.15.146:0/347054878 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v66) ==== 87+0+0 (secure 0 0 0) 0x7f59f4059af0 con 0x7f59e8005230 2024-09-16T01:51:16.905 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:51:16.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.906+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59e406c330 msgr2=0x7f59e406e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.906+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59e406c330 0x7f59e406e7f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5a040cbe80 tx=0x7f59f0006f00 comp rx=0 tx=0).stop 2024-09-16T01:51:16.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.906+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 msgr2=0x7f5a041ec070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:16.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.906+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 0x7f5a041ec070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59f4007730 tx=0x7f59f4001aa0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 shutdown_connections 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59e406c330 0x7f59e406e7f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a0406cde0 0x7f5a041eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a040ae340 0x7f5a041ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 --2- 172.21.15.146:0/347054878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e8005230 0x7f5a041ec070 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 >> 172.21.15.146:0/347054878 conn(0x7f5a040cbf90 msgr2=0x7f5a040cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 shutdown_connections 2024-09-16T01:51:16.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:16.907+0000 7f5a0b000640 1 -- 172.21.15.146:0/347054878 wait complete. 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 initial writers = 288188.44 kB/sec 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 initial writers = 288129.30 kB/sec 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 288188.44 kB/sec 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 288188.44 kB/sec 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 288188.44 kB/sec 2024-09-16T01:51:16.949 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:16.963 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:16.963+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:17.019 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.019 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:17.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.022+0000 7f26f5e00640 1 Processor -- start 2024-09-16T01:51:17.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.023+0000 7f26f5e00640 1 -- start start 2024-09-16T01:51:17.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c75a0 0x7f26f015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:17.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f5e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f006d140 con 0x7f26f00cc040 2024-09-16T01:51:17.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f5e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f006d420 con 0x7f26f00c6b70 2024-09-16T01:51:17.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f26f015b880 con 0x7f26f00c75a0 2024-09-16T01:51:17.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f4a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f26f00c6b70 0x7f26f00cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57948/0 (socket says 172.21.15.146:57948) 2024-09-16T01:51:17.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26efe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c75a0 0x7f26f015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f5400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f26f00cc040 0x7f26f0165160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60880/0 (socket says 172.21.15.146:60880) 2024-09-16T01:51:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26f4a00640 1 -- 172.21.15.146:0/111814644 learned_addr learned my addr 172.21.15.146:0/111814644 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26efe00640 1 -- 172.21.15.146:0/111814644 >> v1:172.21.15.161:6789/0 conn(0x7f26f00c6b70 legacy=0x7f26f00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26efe00640 1 -- 172.21.15.146:0/111814644 >> v1:172.21.15.146:6789/0 conn(0x7f26f00cc040 legacy=0x7f26f0165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26ef400640 1 -- 172.21.15.146:0/111814644 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367019509 0 0) 0x7f26f006d420 con 0x7f26f00c6b70 2024-09-16T01:51:17.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.024+0000 7f26efe00640 1 -- 172.21.15.146:0/111814644 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f006d700 con 0x7f26f00c75a0 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.025+0000 7f26efe00640 1 --2- 172.21.15.146:0/111814644 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c75a0 0x7f26f015b340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097d0 tx=0x7f26dc02a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=974db2c1fef158d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26ef400640 1 -- 172.21.15.146:0/111814644 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc037070 con 0x7f26f00c75a0 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26ef400640 1 -- 172.21.15.146:0/111814644 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc02ad10 con 0x7f26f00c75a0 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26ef400640 1 -- 172.21.15.146:0/111814644 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc032660 con 0x7f26f00c75a0 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 -- 172.21.15.146:0/111814644 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c75a0 msgr2=0x7f26f015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 --2- 172.21.15.146:0/111814644 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c75a0 0x7f26f015b340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097d0 tx=0x7f26dc02a080 comp rx=0 tx=0).stop 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 -- 172.21.15.146:0/111814644 shutdown_connections 2024-09-16T01:51:17.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 --2- 172.21.15.146:0/111814644 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c75a0 0x7f26f015b340 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 -- 172.21.15.146:0/111814644 >> 172.21.15.146:0/111814644 conn(0x7f26f00ccc20 msgr2=0x7f26f00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:17.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 -- 172.21.15.146:0/111814644 shutdown_connections 2024-09-16T01:51:17.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.026+0000 7f26f5e00640 1 -- 172.21.15.146:0/111814644 wait complete. 2024-09-16T01:51:17.029 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.028+0000 7f26f5e00640 1 Processor -- start 2024-09-16T01:51:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.028+0000 7f26f5e00640 1 -- start start 2024-09-16T01:51:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 0x7f26f01f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:17.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26f00c75a0 0x7f26f01f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26f00cc040 0x7f26f0156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f26f00cb050 con 0x7f26f00cc040 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26f00cab20 con 0x7f26f00c75a0 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f26f00cbc80 con 0x7f26f00c6b70 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 0x7f26f01f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26efe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26f00c75a0 0x7f26f01f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 0x7f26f01f0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37468/0 (socket says 172.21.15.146:37468) 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 -- 172.21.15.146:0/751914812 learned_addr learned my addr 172.21.15.146:0/751914812 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f5400640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26f00cc040 0x7f26f0156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 -- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26f00c75a0 msgr2=0x7f26f01f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26f00c75a0 0x7f26f01f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 -- 172.21.15.146:0/751914812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26f00cc040 msgr2=0x7f26f0156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.029+0000 7f26f4a00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26f00cc040 0x7f26f0156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26f4a00640 1 -- 172.21.15.146:0/751914812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f015bba0 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26f4a00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 0x7f26f01f0540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f26e400c3f0 tx=0x7f26e400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e40074e0 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e40047e0 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4004c10 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f015be80 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.030+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f00ce650 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.031+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f006cd20 con 0x7f26f00c6b70 2024-09-16T01:51:17.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.032+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f26e4011070 con 0x7f26f00c6b70 2024-09-16T01:51:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.033+0000 7f26ecc00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f26d006c2e0 0x7f26d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.033+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(66..66 src has 1..66) ==== 5741+0+0 (secure 0 0 0) 0x7f26e408aef0 con 0x7f26f00c6b70 2024-09-16T01:51:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.033+0000 7f26efe00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f26d006c2e0 0x7f26d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:17.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.034+0000 7f26efe00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f26d006c2e0 0x7f26d006e7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f26dc034000 tx=0x7f26dc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:17.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.037+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f26e4058fe0 con 0x7f26f00c6b70 2024-09-16T01:51:17.076 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:17.075+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:17.135 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.135 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:17.144 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:17.185 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:17.184+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:17.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.226+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd in", "ids": ["7"]} v 0) -- 0x7f26f00cda30 con 0x7f26f00c6b70 2024-09-16T01:51:17.228 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:51:17.227+0000 7f49bee00640 -1 mon.a@0(leader).osd e66 definitely_dead 0 2024-09-16T01:51:17.266 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:17.265+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:17.289 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.290 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:17.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:17.381 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:17.380+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:17.441 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.442 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:17.450 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:17.468 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:17.467+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:17.570 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:17.568+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:17.595 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.595 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:17.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:17.670 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:17.669+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:17.764 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.765 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:17.771 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:17.770+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:17.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:17.872 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:17.871+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.914+0000 7f26ecc00640 1 -- 172.21.15.146:0/751914812 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["7"]}]=0 marked in osd.7. v67) ==== 85+0+0 (secure 0 0 0) 0x7f26e405d2e0 con 0x7f26f00c6b70 2024-09-16T01:51:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:marked in osd.7. 2024-09-16T01:51:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f26d006c2e0 msgr2=0x7f26d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f26d006c2e0 0x7f26d006e7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f26dc034000 tx=0x7f26dc02e000 comp rx=0 tx=0).stop 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 msgr2=0x7f26f01f0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 0x7f26f01f0540 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f26e400c3f0 tx=0x7f26e400c8b0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 shutdown_connections 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f26d006c2e0 0x7f26d006e7a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26f00cc040 0x7f26f0156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26f00c75a0 0x7f26f01f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 --2- 172.21.15.146:0/751914812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26f00c6b70 0x7f26f01f0540 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.918+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 >> 172.21.15.146:0/751914812 conn(0x7f26f00ccc20 msgr2=0x7f26f00ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.923+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 shutdown_connections 2024-09-16T01:51:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:17.923+0000 7f26f5e00640 1 -- 172.21.15.146:0/751914812 wait complete. 2024-09-16T01:51:17.947 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-16T01:51:17.973 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:17.972+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:17.991 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:17.991 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:18.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:18.091 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:18.090+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:18.175 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:18.174+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:18.203 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:18.203 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:18.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:18.302 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:18.301+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:18.378 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:18.377+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:18.413 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:18.413 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:18.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:18.512 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:18.511+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:18.581 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:18.581+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:18.623 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:18.624 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:18.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:18.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:18.682 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:18.682+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:18.786 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:18.785+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:18.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.831+0000 7f9ccd600640 1 Processor -- start 2024-09-16T01:51:18.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.831+0000 7f9ccd600640 1 -- start start 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.832+0000 7f9ccd600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9cc8154f40 0x7f9cc80c7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.832+0000 7f9ccd600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9cc815ab70 0x7f9cc815afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9ccd600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc80cb600 con 0x7f9cc815eac0 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9ccd600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc80cbbc0 con 0x7f9cc815ab70 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9ccd600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc8155320 con 0x7f9cc8154f40 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9cc815ab70 0x7f9cc815afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9cc815ab70 0x7f9cc815afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60318/0 (socket says 172.21.15.146:60318) 2024-09-16T01:51:18.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 -- 172.21.15.146:0/2478014548 learned_addr learned my addr 172.21.15.146:0/2478014548 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:18.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cbfe00640 1 --1- 172.21.15.146:0/2478014548 >> v1:172.21.15.161:6789/0 conn(0x7f9cc815eac0 0x7f9cc80d02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57962/0 (socket says 172.21.15.146:57962) 2024-09-16T01:51:18.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/2478014548 >> v2:172.21.15.161:3300/0 conn(0x7f9cc8154f40 0x7f9cc80c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 -- 172.21.15.146:0/2478014548 >> v2:172.21.15.161:3300/0 conn(0x7f9cc8154f40 msgr2=0x7f9cc80c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:18.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc6000640 1 -- 172.21.15.146:0/2478014548 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1841483008 0 0) 0x7f9cc80cb600 con 0x7f9cc815eac0 2024-09-16T01:51:18.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 --2- 172.21.15.146:0/2478014548 >> v2:172.21.15.161:3300/0 conn(0x7f9cc8154f40 0x7f9cc80c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:18.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 -- 172.21.15.146:0/2478014548 >> v1:172.21.15.161:6789/0 conn(0x7f9cc815eac0 legacy=0x7f9cc80d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:18.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc7400640 1 -- 172.21.15.146:0/2478014548 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc80cb8e0 con 0x7f9cc815ab70 2024-09-16T01:51:18.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.833+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/2478014548 >> v2:172.21.15.161:3300/0 conn(0x7f9cc8154f40 0x7f9cc80c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:18.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.835+0000 7f9cc7400640 1 --2- 172.21.15.146:0/2478014548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815ab70 0x7f9cc815afa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80cbd40 tx=0x7f9cc002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366a9ccdc90f78aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.835+0000 7f9cc6000640 1 -- 172.21.15.146:0/2478014548 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0036070 con 0x7f9cc815ab70 2024-09-16T01:51:18.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.835+0000 7f9cc6000640 1 -- 172.21.15.146:0/2478014548 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc0031600 con 0x7f9cc815ab70 2024-09-16T01:51:18.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.835+0000 7f9cc6000640 1 -- 172.21.15.146:0/2478014548 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc00319d0 con 0x7f9cc815ab70 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 -- 172.21.15.146:0/2478014548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815ab70 msgr2=0x7f9cc815afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 --2- 172.21.15.146:0/2478014548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815ab70 0x7f9cc815afa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80cbd40 tx=0x7f9cc002a730 comp rx=0 tx=0).stop 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 -- 172.21.15.146:0/2478014548 shutdown_connections 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 --2- 172.21.15.146:0/2478014548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815ab70 0x7f9cc815afa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 --2- 172.21.15.146:0/2478014548 >> v2:172.21.15.161:3300/0 conn(0x7f9cc8154f40 0x7f9cc80c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 -- 172.21.15.146:0/2478014548 >> 172.21.15.146:0/2478014548 conn(0x7f9cc806cb50 msgr2=0x7f9cc806e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 -- 172.21.15.146:0/2478014548 shutdown_connections 2024-09-16T01:51:18.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.836+0000 7f9ccd600640 1 -- 172.21.15.146:0/2478014548 wait complete. 2024-09-16T01:51:18.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.838+0000 7f9ccd600640 1 Processor -- start 2024-09-16T01:51:18.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.838+0000 7f9ccd600640 1 -- start start 2024-09-16T01:51:18.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.838+0000 7f9ccd600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 0x7f9cc81e7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9ccd600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9cc815ab70 0x7f9cc8155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9ccd600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815eac0 0x7f9cc81569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9ccd600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc815b4e0 con 0x7f9cc815eac0 2024-09-16T01:51:18.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9ccd600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc80b6210 con 0x7f9cc8154f40 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9ccd600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc815eee0 con 0x7f9cc815ab70 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815eac0 0x7f9cc81569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cbfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9cc815ab70 0x7f9cc8155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 0x7f9cc81e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 0x7f9cc81e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56380/0 (socket says 172.21.15.146:56380) 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 -- 172.21.15.146:0/836372753 learned_addr learned my addr 172.21.15.146:0/836372753 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9cc815ab70 msgr2=0x7f9cc8155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9cc815ab70 0x7f9cc8155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815eac0 msgr2=0x7f9cc81569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815eac0 0x7f9cc81569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc6a00640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc81538e0 con 0x7f9cc8154f40 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.839+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc815eac0 0x7f9cc81569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.840+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 0x7f9cc81e7200 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800e960 tx=0x7f9cb800ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.840+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8017070 con 0x7f9cc8154f40 2024-09-16T01:51:18.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.840+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb800ca00 con 0x7f9cc8154f40 2024-09-16T01:51:18.843 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:18.843 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.840+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc80ce480 con 0x7f9cc8154f40 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.840+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb800cdd0 con 0x7f9cc8154f40 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.840+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc80cea40 con 0x7f9cc8154f40 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.841+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9cc815afa0 con 0x7f9cc8154f40 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.842+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9cb801a070 con 0x7f9cc8154f40 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.843+0000 7f9cbea00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9ca406c2b0 0x7f9ca406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.843+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5741+0+0 (secure 0 0 0) 0x7f9cb808b360 con 0x7f9cc8154f40 2024-09-16T01:51:18.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.843+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v67) ==== 54+0+16 (secure 0 0 0) 0x7f9cb808b710 con 0x7f9cc8154f40 2024-09-16T01:51:18.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.843+0000 7f9cbfe00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9ca406c2b0 0x7f9ca406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.844+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9cc80cec40 0x7f9cc80d0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.844+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cc80cbd70 con 0x7f9cc80cec40 2024-09-16T01:51:18.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.844+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9cc80cec40 0x7f9cc80d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.844+0000 7f9cbfe00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9ca406c2b0 0x7f9ca406e770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8150910 tx=0x7f9cb0007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.845+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9cc80cec40 0x7f9cc80d0320 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.845+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc80cbd70 con 0x7f9cc80cec40 2024-09-16T01:51:18.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:18.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.861+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc80cc8b0 con 0x7f9cc80cec40 2024-09-16T01:51:18.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.862+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc80cc8b0 con 0x7f9cc80cec40 2024-09-16T01:51:18.864 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:18.865 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:18.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.863+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9cc81561d0 0x7f9cc8150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.863+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9cc8150560 con 0x7f9cc81561d0 2024-09-16T01:51:18.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.863+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9cc81561d0 0x7f9cc8150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.864+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9cc81561d0 0x7f9cc8150020 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.864+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc8150560 con 0x7f9cc81561d0 2024-09-16T01:51:18.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.879+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc80cd5a0 con 0x7f9cc81561d0 2024-09-16T01:51:18.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.881+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc80cd5a0 con 0x7f9cc81561d0 2024-09-16T01:51:18.886 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:18.887 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:18.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.881+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9cc81f3d20 0x7f9cc81f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.881+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9cc815bbe0 con 0x7f9cc81f3d20 2024-09-16T01:51:18.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.882+0000 7f9cbfe00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9cc81f3d20 0x7f9cc81f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.882+0000 7f9cbfe00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9cc81f3d20 0x7f9cc81f4180 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.883+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc815bbe0 con 0x7f9cc81f3d20 2024-09-16T01:51:18.888 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:18.887+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:18.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.897+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc8062880 con 0x7f9cc81f3d20 2024-09-16T01:51:18.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.898+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc8062880 con 0x7f9cc81f3d20 2024-09-16T01:51:18.899 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:18.900 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:18.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.899+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9cc8063360 0x7f9cc8152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.899+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9cc8152ca0 con 0x7f9cc8063360 2024-09-16T01:51:18.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.899+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9cc8063360 0x7f9cc8152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.900+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9cc8063360 0x7f9cc8152760 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.900+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc8152ca0 con 0x7f9cc8063360 2024-09-16T01:51:18.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.922+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc8153180 con 0x7f9cc8063360 2024-09-16T01:51:18.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:51:18.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.927+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc8153180 con 0x7f9cc8063360 2024-09-16T01:51:18.933 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.928+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9cc802c640 0x7f9cc802caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.928+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9cc802cfe0 con 0x7f9cc802c640 2024-09-16T01:51:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.928+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9cc802c640 0x7f9cc802caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.929+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9cc802c640 0x7f9cc802caa0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.931+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc802cfe0 con 0x7f9cc802c640 2024-09-16T01:51:18.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.958+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc802cfe0 con 0x7f9cc802c640 2024-09-16T01:51:18.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.960+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc802cfe0 con 0x7f9cc802c640 2024-09-16T01:51:18.961 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:18.962 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:18.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.960+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9cc802d660 0x7f9cc8206ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.960+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9cc802dae0 con 0x7f9cc802d660 2024-09-16T01:51:18.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.961+0000 7f9cbfe00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9cc802d660 0x7f9cc8206ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.961+0000 7f9cbfe00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9cc802d660 0x7f9cc8206ba0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.962+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc802dae0 con 0x7f9cc802d660 2024-09-16T01:51:18.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.976+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc802dae0 con 0x7f9cc802d660 2024-09-16T01:51:18.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.978+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc802dae0 con 0x7f9cc802d660 2024-09-16T01:51:18.979 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:18.979 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:18.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.978+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9cc82075f0 0x7f9cc82099e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.978+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9cc820c880 con 0x7f9cc82075f0 2024-09-16T01:51:18.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.979+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9cc82075f0 0x7f9cc82099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.979+0000 7f9cc7400640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9cc82075f0 0x7f9cc82099e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.980+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc820c880 con 0x7f9cc82075f0 2024-09-16T01:51:18.990 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:18.989+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:18.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.994+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc820c880 con 0x7f9cc82075f0 2024-09-16T01:51:18.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.995+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc820c880 con 0x7f9cc82075f0 2024-09-16T01:51:18.996 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:18.997 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:18.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.996+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9cc820a430 0x7f9cc820a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:18.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.996+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9cc820fc20 con 0x7f9cc820a430 2024-09-16T01:51:18.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.996+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9cc820a430 0x7f9cc820a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:18.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.997+0000 7f9cc6a00640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9cc820a430 0x7f9cc820a870 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:18.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:18.998+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc820fc20 con 0x7f9cc820a430 2024-09-16T01:51:19.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.026+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9cc820ae70 con 0x7f9cc820a430 2024-09-16T01:51:19.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.028+0000 7f9cbea00640 1 -- 172.21.15.146:0/836372753 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9cc820ae70 con 0x7f9cc820a430 2024-09-16T01:51:19.029 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9cc80cec40 msgr2=0x7f9cc80d0320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9cc80cec40 0x7f9cc80d0320 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9cc81561d0 msgr2=0x7f9cc8150020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9cc81561d0 0x7f9cc8150020 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9cc81f3d20 msgr2=0x7f9cc81f4180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9cc81f3d20 0x7f9cc81f4180 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9cc8063360 msgr2=0x7f9cc8152760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9cc8063360 0x7f9cc8152760 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9cc802c640 msgr2=0x7f9cc802caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9cc802c640 0x7f9cc802caa0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9cc802d660 msgr2=0x7f9cc8206ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9cc802d660 0x7f9cc8206ba0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9cc82075f0 msgr2=0x7f9cc82099e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9cc82075f0 0x7f9cc82099e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9cc820a430 msgr2=0x7f9cc820a870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9cc820a430 0x7f9cc820a870 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9cc7400640 1 -- 172.21.15.146:0/836372753 reap_dead start 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9cc7400640 1 -- 172.21.15.146:0/836372753 reap_dead start 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9cc7400640 1 -- 172.21.15.146:0/836372753 reap_dead start 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9cc7400640 1 -- 172.21.15.146:0/836372753 reap_dead start 2024-09-16T01:51:19.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9cc7400640 1 -- 172.21.15.146:0/836372753 reap_dead start 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9ca406c2b0 msgr2=0x7f9ca406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9ca406c2b0 0x7f9ca406e770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8150910 tx=0x7f9cb0007d50 comp rx=0 tx=0).stop 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 msgr2=0x7f9cc81e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.029+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 0x7f9cc81e7200 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800e960 tx=0x7f9cb800ee20 comp rx=0 tx=0).stop 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.030+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 shutdown_connections 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.030+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9ca406c2b0 0x7f9ca406e770 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.030+0000 7f9ccd600640 1 --2- 172.21.15.146:0/836372753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cc8154f40 0x7f9cc81e7200 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.030+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 >> 172.21.15.146:0/836372753 conn(0x7f9cc806cb50 msgr2=0x7f9cc81590e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:19.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.030+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 shutdown_connections 2024-09-16T01:51:19.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.030+0000 7f9ccd600640 1 -- 172.21.15.146:0/836372753 wait complete. 2024-09-16T01:51:19.043 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:19.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:19.043 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:19.043 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:19.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:19.091 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:19.090+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:19.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.112+0000 7f220d200640 1 Processor -- start 2024-09-16T01:51:19.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.113+0000 7f220d200640 1 -- start start 2024-09-16T01:51:19.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.114+0000 7f220d200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22080cc0c0 0x7f220815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:19.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.114+0000 7f220d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22080cc4a0 0x7f2208165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:19.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.114+0000 7f220d200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220815b1d0 con 0x7f22080cb2f0 2024-09-16T01:51:19.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.114+0000 7f220d200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f22080cbd00 con 0x7f22080cc4a0 2024-09-16T01:51:19.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.114+0000 7f220d200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f220815ba70 con 0x7f22080cc0c0 2024-09-16T01:51:19.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22080cc0c0 0x7f220815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:19.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22080cc0c0 0x7f220815aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56386/0 (socket says 172.21.15.146:56386) 2024-09-16T01:51:19.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 -- 172.21.15.146:0/2884894861 learned_addr learned my addr 172.21.15.146:0/2884894861 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:19.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2206800640 1 --2- 172.21.15.146:0/2884894861 >> v2:172.21.15.146:3300/0 conn(0x7f22080cc4a0 0x7f2208165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:19.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 -- 172.21.15.146:0/2884894861 >> v2:172.21.15.146:3300/0 conn(0x7f22080cc4a0 msgr2=0x7f2208165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:19.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 --2- 172.21.15.146:0/2884894861 >> v2:172.21.15.146:3300/0 conn(0x7f22080cc4a0 0x7f2208165430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 -- 172.21.15.146:0/2884894861 >> v1:172.21.15.146:6789/0 conn(0x7f22080cb2f0 legacy=0x7f220815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.115+0000 7f2205e00640 1 -- 172.21.15.146:0/2884894861 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220815b4b0 con 0x7f22080cc0c0 2024-09-16T01:51:19.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.116+0000 7f2205e00640 1 --2- 172.21.15.146:0/2884894861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f220815aab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f22080ce150 tx=0x7f21f802a740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75fca517e384c235 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:19.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.117+0000 7f2204a00640 1 -- 172.21.15.146:0/2884894861 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8036070 con 0x7f22080cc0c0 2024-09-16T01:51:19.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.117+0000 7f2204a00640 1 -- 172.21.15.146:0/2884894861 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f8031600 con 0x7f22080cc0c0 2024-09-16T01:51:19.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.117+0000 7f2204a00640 1 -- 172.21.15.146:0/2884894861 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f80319d0 con 0x7f22080cc0c0 2024-09-16T01:51:19.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.117+0000 7f220d200640 1 -- 172.21.15.146:0/2884894861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 msgr2=0x7f220815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.117+0000 7f220d200640 1 --2- 172.21.15.146:0/2884894861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f220815aab0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f22080ce150 tx=0x7f21f802a740 comp rx=0 tx=0).stop 2024-09-16T01:51:19.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.118+0000 7f220d200640 1 -- 172.21.15.146:0/2884894861 shutdown_connections 2024-09-16T01:51:19.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.118+0000 7f220d200640 1 --2- 172.21.15.146:0/2884894861 >> v2:172.21.15.146:3300/0 conn(0x7f22080cc4a0 0x7f2208165430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.118+0000 7f220d200640 1 --2- 172.21.15.146:0/2884894861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f220815aab0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.118+0000 7f220d200640 1 -- 172.21.15.146:0/2884894861 >> 172.21.15.146:0/2884894861 conn(0x7f220806d560 msgr2=0x7f220806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:19.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.118+0000 7f220d200640 1 -- 172.21.15.146:0/2884894861 shutdown_connections 2024-09-16T01:51:19.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.118+0000 7f220d200640 1 -- 172.21.15.146:0/2884894861 wait complete. 2024-09-16T01:51:19.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.119+0000 7f220d200640 1 Processor -- start 2024-09-16T01:51:19.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.119+0000 7f220d200640 1 -- start start 2024-09-16T01:51:19.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.120+0000 7f220d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 0x7f22081f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:19.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.120+0000 7f220d200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f2208154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:19.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.120+0000 7f220d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22080cc4a0 0x7f22081f1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:19.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.120+0000 7f220d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f220815e1d0 con 0x7f22080cc4a0 2024-09-16T01:51:19.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.120+0000 7f220d200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f220815e540 con 0x7f22080cc0c0 2024-09-16T01:51:19.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.120+0000 7f220d200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f22080cc8b0 con 0x7f22080cb2f0 2024-09-16T01:51:19.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 0x7f22081f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:19.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 0x7f22081f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37488/0 (socket says 172.21.15.146:37488) 2024-09-16T01:51:19.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 -- 172.21.15.146:0/3555080469 learned_addr learned my addr 172.21.15.146:0/3555080469 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:19.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205400640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f2208154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:19.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2206800640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22080cc4a0 0x7f22081f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:19.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 -- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 msgr2=0x7f2208154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f2208154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 -- 172.21.15.146:0/3555080469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22080cc4a0 msgr2=0x7f22081f1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22080cc4a0 0x7f22081f1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 -- 172.21.15.146:0/3555080469 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22080b3660 con 0x7f22080cb2f0 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f2205e00640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 0x7f22081f3650 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f21f8034040 tx=0x7f21f8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.121+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8036050 con 0x7f22080cb2f0 2024-09-16T01:51:19.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.122+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f8042070 con 0x7f22080cb2f0 2024-09-16T01:51:19.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.122+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8009510 con 0x7f22080cb2f0 2024-09-16T01:51:19.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.122+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22080c7d00 con 0x7f22080cb2f0 2024-09-16T01:51:19.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.122+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22080c82a0 con 0x7f22080cb2f0 2024-09-16T01:51:19.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.123+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22080c5560 con 0x7f22080cb2f0 2024-09-16T01:51:19.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.124+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f21f8007c30 con 0x7f22080cb2f0 2024-09-16T01:51:19.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.125+0000 7f21fe000640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f21e406c2b0 0x7f21e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:19.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.125+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(68..68 src has 1..68) ==== 5741+0+0 (secure 0 0 0) 0x7f21f80ac2d0 con 0x7f22080cb2f0 2024-09-16T01:51:19.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.125+0000 7f2205400640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f21e406c2b0 0x7f21e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:19.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.126+0000 7f2205400640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f21e406c2b0 0x7f21e406e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f220815e070 tx=0x7f21f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:19.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.129+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f21f807a330 con 0x7f22080cb2f0 2024-09-16T01:51:19.195 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:19.194+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:19.269 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:19.270 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:19.270 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:19.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:19.323 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:19.322+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:19.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.335+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f22080b65e0 con 0x7f22080cb2f0 2024-09-16T01:51:19.396 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:19.395+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:19.486 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:19.487 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:19.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:19.487 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:19.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:6 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:19.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.228082283999999, 2024-09-16T01:51:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053803974999999997, 2024-09-16T01:51:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.488 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 6 2024-09-16T01:51:19.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.490 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:19.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254898+0000", 2024-09-16T01:51:19.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.254900+0000", 2024-09-16T01:51:19.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.863e-06 2024-09-16T01:51:19.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284843+0000", 2024-09-16T01:51:19.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029942949999999999 2024-09-16T01:51:19.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284880+0000", 2024-09-16T01:51:19.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.697e-05 2024-09-16T01:51:19.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.284894+0000", 2024-09-16T01:51:19.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3382e-05 2024-09-16T01:51:19.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285037+0000", 2024-09-16T01:51:19.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000142996 2024-09-16T01:51:19.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285182+0000", 2024-09-16T01:51:19.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014590600000000001 2024-09-16T01:51:19.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:19.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.285317+0000", 2024-09-16T01:51:19.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013416200000000001 2024-09-16T01:51:19.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.306224+0000", 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020907194 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308590+0000", 2024-09-16T01:51:19.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023660220000000002 2024-09-16T01:51:19.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308653+0000", 2024-09-16T01:51:19.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3008000000000004e-05 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.308702+0000", 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9521999999999998e-05 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.501 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.502 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.502 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:19.502 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.915136509, 2024-09-16T01:51:19.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:19.503 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.503 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.503 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:19.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:19.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:19.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:19.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.761590826999999, 2024-09-16T01:51:19.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:19.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:19.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:19.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.540731146999999, 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:19.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:19.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:19.525 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.462600557999998, 2024-09-16T01:51:19.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:19.526 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.526 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:19.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:19.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:19.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:19.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:19.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:19.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:85 3.36 e55/20 3:6fe880e3:::10000000002.00000052:head v 55'2, mlcod=0'0)", 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.440214231999999, 2024-09-16T01:51:19.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052325050999999997, 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042766+0000", 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042769+0000", 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3520000000000001e-06 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051520+0000", 2024-09-16T01:51:19.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087513980000000005 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051666+0000", 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014630900000000001 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.051711+0000", 2024-09-16T01:51:19.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4768999999999998e-05 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052370+0000", 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065890100000000004 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.052478+0000", 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010805 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095028+0000", 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042550194 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095091+0000", 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3077999999999995e-05 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.890387671999999, 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.540 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:19.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:19.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:19.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:19.546 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.666231138000001, 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:19.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:19.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.368954115000001, 2024-09-16T01:51:19.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:19.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:19.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.896146261999998, 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:19.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:19.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:19.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.708319675999999, 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:19.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:484 3.e 3:723b89cc:::10000000002.000000e1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.375276911, 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054305622999999997, 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 484 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107704+0000", 2024-09-16T01:51:19.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.107706+0000", 2024-09-16T01:51:19.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:19.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145546+0000", 2024-09-16T01:51:19.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037839795000000002 2024-09-16T01:51:19.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145584+0000", 2024-09-16T01:51:19.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8556000000000002e-05 2024-09-16T01:51:19.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145598+0000", 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4238e-05 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145708+0000", 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109584 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145834+0000", 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012588200000000001 2024-09-16T01:51:19.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.145990+0000", 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015650199999999999 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.160245+0000", 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014254119000000001 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161887+0000", 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016424009999999999 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.161956+0000", 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8996000000000001e-05 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.162009+0000", 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3381999999999998e-05 2024-09-16T01:51:19.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.199503271000001, 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.581 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:19.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8206573720000003, 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.587 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:51:19.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:562 3.16 3:6824bda2:::100000001f7.0000002f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.593 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.839661+0000", 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.6433198860000005, 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055000097999999997, 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 562 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.594 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.839661+0000", 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.839661+0000", 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.839663+0000", 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5380000000000001e-06 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.869188+0000", 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029524679000000002 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.870761+0000", 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001573128 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.870774+0000", 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3029e-05 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.870865+0000", 2024-09-16T01:51:19.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0597e-05 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.871015+0000", 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015075099999999999 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.871160+0000", 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014403899999999999 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.883539+0000", 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012379415 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.894575+0000", 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011035967000000001 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.894633+0000", 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8396000000000001e-05 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.894661+0000", 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:607 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.1922705809999998, 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057026931000000003, 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.601 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 607 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290713+0000", 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6900000000000001e-06 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330961+0000", 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040248199999999998 2024-09-16T01:51:19.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330992+0000", 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1251000000000003e-05 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331005+0000", 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3006999999999999e-05 2024-09-16T01:51:19.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332660+0000", 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016549589999999999 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332819+0000", 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159137 2024-09-16T01:51:19.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332968+0000", 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148942 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.342537+0000", 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095683910000000007 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347616+0000", 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005079793 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347682+0000", 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5622000000000003e-05 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347737+0000", 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4938999999999997e-05 2024-09-16T01:51:19.607 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:634 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9680821669999999, 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068283101999999998, 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 634 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514901+0000", 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5299999999999999e-06 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549207+0000", 2024-09-16T01:51:19.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034306171000000003 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549224+0000", 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6976000000000001e-05 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549235+0000", 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0682999999999999e-05 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549568+0000", 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033294599999999998 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549714+0000", 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014617499999999999 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549858+0000", 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144356 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.566205+0000", 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01634683 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583078+0000", 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016872402000000002 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583135+0000", 2024-09-16T01:51:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7286e-05 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583182+0000", 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6746999999999998e-05 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:688 3.26 3:642b2ad4:::100000001f7.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.4511146950000002, 2024-09-16T01:51:19.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062867175999999997, 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 688 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:19.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031869+0000", 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5509999999999998e-06 2024-09-16T01:51:19.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084368+0000", 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052499410000000003 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084392+0000", 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084405+0000", 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2561000000000001e-05 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084491+0000", 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5945000000000005e-05 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084662+0000", 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017081299999999999 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084812+0000", 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993999999999999 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.091507+0000", 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066950079999999997 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094630+0000", 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031236570000000002 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094700+0000", 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9575999999999999e-05 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094733+0000", 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3260999999999999e-05 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:693 3.16 3:68ba535b:::100000001f7.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.105928+0000", 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.3770523859999999, 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052746521999999997, 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.621 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 693 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.105928+0000", 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.105928+0000", 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.105931+0000", 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6819999999999999e-06 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.124440+0000", 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018508862000000001 2024-09-16T01:51:19.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.124479+0000", 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8844999999999998e-05 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.124504+0000", 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5303000000000002e-05 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.124635+0000", 2024-09-16T01:51:19.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013112900000000001 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.124795+0000", 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016026300000000001 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.124948+0000", 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015311099999999999 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.145472+0000", 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020523350999999999 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.158544+0000", 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01307265 2024-09-16T01:51:19.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.158623+0000", 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8627999999999993e-05 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.158675+0000", 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1697999999999999e-05 2024-09-16T01:51:19.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:696 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.3558568310000001, 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071884475000000003, 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:19.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 696 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:19.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127126+0000", 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9369999999999998e-06 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159227+0000", 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032101044000000002 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:19.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159250+0000", 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3332999999999998e-05 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159260+0000", 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159343+0000", 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3373000000000003e-05 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:19.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159484+0000", 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014100399999999999 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159646+0000", 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162044 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.186104+0000", 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026457794999999999 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198896+0000", 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012791749 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198959+0000", 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3387999999999997e-05 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:19.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.199008+0000", 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9118e-05 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:19.634 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:19.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:19.636 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:19.635+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:19.737 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:19.736+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:19.837 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:19.837+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:19.842 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:19.842 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:19.843 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:19.843 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:19.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:19.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.932+0000 7f21fe000640 1 -- 172.21.15.146:0/3555080469 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v69) ==== 97+0+0 (secure 0 0 0) 0x7f21f807e630 con 0x7f22080cb2f0 2024-09-16T01:51:19.934 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f21e406c2b0 msgr2=0x7f21e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f21e406c2b0 0x7f21e406e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f220815e070 tx=0x7f21f0009040 comp rx=0 tx=0).stop 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 msgr2=0x7f22081f3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 0x7f22081f3650 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f21f8034040 tx=0x7f21f8004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 shutdown_connections 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f21e406c2b0 0x7f21e406e770 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22080cc4a0 0x7f22081f1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22080cc0c0 0x7f2208154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 --2- 172.21.15.146:0/3555080469 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22080cb2f0 0x7f22081f3650 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:19.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.934+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 >> 172.21.15.146:0/3555080469 conn(0x7f220806d560 msgr2=0x7f22080ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:19.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.935+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 shutdown_connections 2024-09-16T01:51:19.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:19.935+0000 7f220d200640 1 -- 172.21.15.146:0/3555080469 wait complete. 2024-09-16T01:51:19.938 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:19.937+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:20.039 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:20.038+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:20.066 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:20.066 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:20.066 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:20.066 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:20.066 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:20.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:20.140 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:20.139+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:20.249 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:20.248+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.051626002999999, 2024-09-16T01:51:20.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:20.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.853253132999999, 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:20.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.325 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.744450501999999, 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:20.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:20.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:20.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.424881869, 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:20.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.926219129, 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:20.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:20.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:20.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:160 3.9 3:93988fc9:::10000000002.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.557229052, 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056552534000000002, 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 160 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749677+0000", 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.749679+0000", 2024-09-16T01:51:20.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.956e-06 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794169+0000", 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04449061 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794202+0000", 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2626999999999998e-05 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794217+0000", 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4583000000000001e-05 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794305+0000", 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8158999999999994e-05 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794460+0000", 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155277 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794625+0000", 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164854 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.801888+0000", 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072630239999999999 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806094+0000", 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042063919999999998 2024-09-16T01:51:20.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806178+0000", 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3287000000000004e-05 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.806229+0000", 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765000000000003e-05 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.512646445000001, 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:20.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:20.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:197 3.1d 3:bb6dfbbf:::10000000002.000000c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.199767371, 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057555969999999998, 2024-09-16T01:51:20.361 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 197 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.362 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107139+0000", 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.107141+0000", 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:20.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148704+0000", 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041563044 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148738+0000", 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4480999999999997e-05 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148754+0000", 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6021000000000001e-05 2024-09-16T01:51:20.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.148909+0000", 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154735 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149071+0000", 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016181600000000001 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.149226+0000", 2024-09-16T01:51:20.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015550000000000001 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.158210+0000", 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089836529999999994 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164352+0000", 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061416810000000004 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164426+0000", 2024-09-16T01:51:20.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4192999999999999e-05 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.164695+0000", 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026864800000000002 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.526273353000001, 2024-09-16T01:51:20.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.373 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:20.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:20.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:20.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.383 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.458168711999999, 2024-09-16T01:51:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:20.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:20.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:20.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:20.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:20.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.315709046999999, 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:20.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:20.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:20.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:20.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:20.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.968242824000001, 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:20.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:20.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:529 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.537651+0000", 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7692546729999998, 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056028116000000003, 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 529 2024-09-16T01:51:20.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537651+0000", 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537651+0000", 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537655+0000", 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4599999999999999e-06 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561331+0000", 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023676256 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561354+0000", 2024-09-16T01:51:20.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3404000000000002e-05 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561368+0000", 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.561461+0000", 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3003000000000004e-05 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.562117+0000", 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00065668599999999997 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.562219+0000", 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101128 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.579034+0000", 2024-09-16T01:51:20.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016815192999999999 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.593564+0000", 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014530112 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.593631+0000", 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7574e-05 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.593679+0000", 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7981e-05 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:544 3.14 3:28ae9428:::100000001f7.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6628689220000004, 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064751230000000007, 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.428 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 544 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644039+0000", 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6619999999999999e-06 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665383+0000", 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021343964 2024-09-16T01:51:20.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665408+0000", 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.561e-05 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665422+0000", 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3947999999999999e-05 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665607+0000", 2024-09-16T01:51:20.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018433499999999999 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665831+0000", 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022424399999999999 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665986+0000", 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155654 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.682208+0000", 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016221920000000001 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708669+0000", 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026460513000000001 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708734+0000", 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4888000000000006e-05 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708788+0000", 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4491999999999999e-05 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:549 3.11 3:8ae93890:::100000001f7.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6206689010000002, 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081050590000000006, 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.434 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 549 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686239+0000", 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754451+0000", 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068212138000000005 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754474+0000", 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2733000000000001e-05 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754483+0000", 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1349999999999998e-06 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754562+0000", 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8319999999999996e-05 2024-09-16T01:51:20.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754725+0000", 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016298699999999999 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754875+0000", 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149942 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.762160+0000", 2024-09-16T01:51:20.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072859630000000003 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767169+0000", 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050088609999999999 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767231+0000", 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1158e-05 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767288+0000", 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7108000000000003e-05 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:597 3.11 3:89a80ce8:::100000001f7.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.203535+0000", 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1033707420000001, 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055367708000000002, 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.440 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 597 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.203535+0000", 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.203535+0000", 2024-09-16T01:51:20.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.203537+0000", 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236338+0000", 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032800274999999997 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236359+0000", 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1372000000000001e-05 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236371+0000", 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2359e-05 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.236534+0000", 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016273200000000001 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.237346+0000", 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081165000000000002 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.237822+0000", 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047652099999999999 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.251989+0000", 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014166972999999999 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.258782+0000", 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067926820000000004 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.258851+0000", 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9148999999999998e-05 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.258903+0000", 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1774999999999998e-05 2024-09-16T01:51:20.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:606 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0322548539999996, 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058221427999999999, 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274653+0000", 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322327+0000", 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047674080000000001 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322357+0000", 2024-09-16T01:51:20.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9813000000000001e-05 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322369+0000", 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2074e-05 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322494+0000", 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012469 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322653+0000", 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158987 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322803+0000", 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150433 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331674+0000", 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088701219999999994 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332772+0000", 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001098552 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332829+0000", 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6941000000000003e-05 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332873+0000", 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3516000000000001e-05 2024-09-16T01:51:20.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:626 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.842262979, 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057812853999999997, 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 626 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464645+0000", 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9759999999999998e-06 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505192+0000", 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040546806999999997 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505211+0000", 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9386e-05 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505220+0000", 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4810000000000006e-06 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505316+0000", 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6748000000000003e-05 2024-09-16T01:51:20.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505479+0000", 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016267300000000001 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505668+0000", 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018906700000000001 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.520824+0000", 2024-09-16T01:51:20.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015155686999999999 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522336+0000", 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015126359999999999 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522404+0000", 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7174000000000004e-05 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522456+0000", 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2219000000000003e-05 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:629 3.13 3:ca345efb:::100000001f7.00000072:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.7981142759999997, 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073672847999999999, 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.458 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 629 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:20.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508793+0000", 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5650000000000001e-06 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536849+0000", 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028055429999999999 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536888+0000", 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9128999999999998e-05 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536904+0000", 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6223e-05 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537082+0000", 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017825600000000001 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537242+0000", 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159569 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537394+0000", 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015228 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.548370+0000", 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010976324000000001 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582365+0000", 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994745999999999 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582423+0000", 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7357e-05 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582465+0000", 2024-09-16T01:51:20.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1968999999999997e-05 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:700 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.1732638440000001, 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074697384000000006, 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:20.464 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 700 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:20.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133644+0000", 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159468+0000", 2024-09-16T01:51:20.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025824323 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159551+0000", 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2286999999999993e-05 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159561+0000", 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0307e-05 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:20.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159851+0000", 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029012100000000001 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160192+0000", 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034104600000000001 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160444+0000", 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025126400000000001 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.197238+0000", 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036794265999999999 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208261+0000", 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0110233 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208304+0000", 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2833999999999997e-05 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:20.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208339+0000", 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5556000000000003e-05 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:20.470 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:20.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:20.471 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:20.390+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:20.492 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:20.491+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:20.592 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:20.592 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:20.592 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:20.592 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:20.593 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:20.591+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:20.601 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:20.693 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:20.691+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:20.724 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:20.724 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:20.725 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:20.725 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:20.725 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:20.735 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:20.822 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:20.821+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:20.895 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:20.894+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:20.900 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:20.900 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.348023934, 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.901 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:20.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.906 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.100990353, 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.907 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:20.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:20.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.913 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.041831266999999, 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.914 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:20.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 29.130252955, 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:20.920 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:20.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:20.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:20.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.926 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 29.085992725000001, 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:20.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:20.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 29.055492905000001, 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:20.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:20.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:20.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 29.013937382999998, 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:20.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:20.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:20.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:20.945 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.875890688999998, 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:20.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:20.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.951 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.756930961999998, 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.952 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:20.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.722134662999999, 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:20.960 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:20.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:20.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:20.968 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.638685710000001, 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.969 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:20.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.4393829230000001, 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.975 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:51:20.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:51:20.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.2945059849999998, 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:51:20.981 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:51:20.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.987 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.2538615609999999, 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:51:20.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:51:20.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.0959187269999999, 2024-09-16T01:51:20.994 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.995 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.996 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:51:20.997 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.998 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:51:20.999 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.000 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.9707914659999997, 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:21.001 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:21.002 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.003 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:51:21.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:51:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:576 3.1b 3:d81af1ca:::100000001f7.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.8619887799999999, 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052702763999999999, 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:21.008 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 576 2024-09-16T01:51:21.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:21.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035658+0000", 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.03e-06 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075189+0000", 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039530957999999998 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075204+0000", 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5071e-05 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075214+0000", 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075299+0000", 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4807000000000003e-05 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075401+0000", 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010220700000000001 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075495+0000", 2024-09-16T01:51:21.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4006999999999996e-05 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.078964+0000", 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003468741 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:21.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088273+0000", 2024-09-16T01:51:21.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0093088089999999995 2024-09-16T01:51:21.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:21.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088311+0000", 2024-09-16T01:51:21.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8402000000000003e-05 2024-09-16T01:51:21.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:21.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088359+0000", 2024-09-16T01:51:21.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7920999999999997e-05 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:598 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:21.017 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.6819089140000001, 2024-09-16T01:51:21.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054052254000000001, 2024-09-16T01:51:21.018 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:21.018 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:21.018 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:21.018 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 598 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215738+0000", 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.526e-06 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250373+0000", 2024-09-16T01:51:21.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034634160999999997 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250391+0000", 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7986e-05 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250400+0000", 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1439999999999992e-06 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:21.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250455+0000", 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5578999999999998e-05 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250562+0000", 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106717 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250665+0000", 2024-09-16T01:51:21.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010307200000000001 2024-09-16T01:51:21.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:21.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.253888+0000", 2024-09-16T01:51:21.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032231019999999998 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269702+0000", 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015813473000000002 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269742+0000", 2024-09-16T01:51:21.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0154000000000002e-05 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269788+0000", 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.634e-05 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.026 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.029 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:614 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:21.029 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.356692+0000", 2024-09-16T01:51:21.030 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.5409532319999997, 2024-09-16T01:51:21.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048508810999999999, 2024-09-16T01:51:21.030 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:21.030 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:21.030 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 614 2024-09-16T01:51:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.031 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:21.032 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.032 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:21.032 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.356692+0000", 2024-09-16T01:51:21.032 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.032 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.356692+0000", 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.356694+0000", 2024-09-16T01:51:21.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2409999999999998e-06 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383766+0000", 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027072213000000001 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383784+0000", 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8198000000000001e-05 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383792+0000", 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6969999999999993e-06 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383871+0000", 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9102000000000003e-05 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.383976+0000", 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104579 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.384069+0000", 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3017999999999996e-05 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.387522+0000", 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003453065 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.405058+0000", 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017536169000000001 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.405136+0000", 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7762999999999994e-05 2024-09-16T01:51:21.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.405200+0000", 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4765999999999996e-05 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.9339915349999997, 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:51:21.038 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:21.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:51:21.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:21.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:21.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:51:21.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:51:21.045 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.045 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.045 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.045 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.045 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.045 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:21.046 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:21.048 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:21.045+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:21.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:21.147 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:21.146+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:21.214 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:21.214 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:21.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.215 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:872 3.18 3:19643448:::100000001f7.00000065:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:21.215 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:21.215 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.063487423000000001, 2024-09-16T01:51:21.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.063523841999999997, 2024-09-16T01:51:21.215 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:21.216 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:21.216 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:51:21.216 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:21.216 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:21.216 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:21.216 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 872 2024-09-16T01:51:21.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.217 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:21.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:21.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:21.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:21.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:21.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:21.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:21.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149650+0000", 2024-09-16T01:51:21.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8899999999999998e-06 2024-09-16T01:51:21.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:21.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209584+0000", 2024-09-16T01:51:21.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059934556 2024-09-16T01:51:21.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:21.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209621+0000", 2024-09-16T01:51:21.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6378999999999998e-05 2024-09-16T01:51:21.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:21.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209629+0000", 2024-09-16T01:51:21.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6410000000000008e-06 2024-09-16T01:51:21.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:21.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209715+0000", 2024-09-16T01:51:21.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6013999999999993e-05 2024-09-16T01:51:21.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:21.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209807+0000", 2024-09-16T01:51:21.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2020000000000003e-05 2024-09-16T01:51:21.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:21.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:21.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:21.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209966+0000", 2024-09-16T01:51:21.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158889 2024-09-16T01:51:21.225 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.225 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:21.225 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.225 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:21.225 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-09-16T01:51:21.225 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 1 2024-09-16T01:51:21.226 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:21.227 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:21.251 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:21.249+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:21.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.286+0000 7f1c7cc00640 1 Processor -- start 2024-09-16T01:51:21.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.286+0000 7f1c7cc00640 1 -- start start 2024-09-16T01:51:21.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.289+0000 7f1c7cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c7815e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.289+0000 7f1c7cc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c780cb3a0 0x7f1c780cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c780cb3a0 0x7f1c780cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c780cb3a0 0x7f1c780cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:56410/0 (socket says 172.21.15.146:56410) 2024-09-16T01:51:21.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.289+0000 7f1c7cc00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c7815eee0 con 0x7f1c780ca270 2024-09-16T01:51:21.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 -- 172.21.15.146:0/213244330 learned_addr learned my addr 172.21.15.146:0/213244330 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:21.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1c7815e640 con 0x7f1c780cb3a0 2024-09-16T01:51:21.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c7815e920 con 0x7f1c780c68f0 2024-09-16T01:51:21.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c75e00640 1 --2- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c7815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 -- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 msgr2=0x7f1c7815e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:21.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 --2- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c7815e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 -- 172.21.15.146:0/213244330 >> v1:172.21.15.161:6789/0 conn(0x7f1c780ca270 legacy=0x7f1c7815b430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:51:21.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.290+0000 7f1c76800640 1 -- 172.21.15.146:0/213244330 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7815ec00 con 0x7f1c780cb3a0 2024-09-16T01:51:21.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.297+0000 7f1c76800640 1 --2- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 0x7f1c780cc000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009e50 tx=0x7f1c70032e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d0536f20ea15f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.299+0000 7f1c74a00640 1 -- 172.21.15.146:0/213244330 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70033940 con 0x7f1c780cb3a0 2024-09-16T01:51:21.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.300+0000 7f1c74a00640 1 -- 172.21.15.146:0/213244330 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c70033ae0 con 0x7f1c780cb3a0 2024-09-16T01:51:21.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 msgr2=0x7f1c780cc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c74a00640 1 -- 172.21.15.146:0/213244330 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70033940 con 0x7f1c780cb3a0 2024-09-16T01:51:21.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 0x7f1c780cc000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009e50 tx=0x7f1c70032e40 comp rx=0 tx=0).stop 2024-09-16T01:51:21.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 shutdown_connections 2024-09-16T01:51:21.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 0x7f1c780cc000 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/213244330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c7815e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.301+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 >> 172.21.15.146:0/213244330 conn(0x7f1c7806cb70 msgr2=0x7f1c7806e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:21.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.303+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 shutdown_connections 2024-09-16T01:51:21.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.303+0000 7f1c7cc00640 1 -- 172.21.15.146:0/213244330 wait complete. 2024-09-16T01:51:21.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.307+0000 7f1c7cc00640 1 Processor -- start 2024-09-16T01:51:21.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.307+0000 7f1c7cc00640 1 -- start start 2024-09-16T01:51:21.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.309+0000 7f1c7cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c78072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.309+0000 7f1c7cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780ca270 0x7f1c781571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.309+0000 7f1c75e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c78072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c7cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 0x7f1c78071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c7cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c780c7770 con 0x7f1c780ca270 2024-09-16T01:51:21.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c7cc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c780cbad0 con 0x7f1c780cb3a0 2024-09-16T01:51:21.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c7cc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c780c78f0 con 0x7f1c780c68f0 2024-09-16T01:51:21.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c75e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c78072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:37506/0 (socket says 172.21.15.146:37506) 2024-09-16T01:51:21.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c75e00640 1 -- 172.21.15.146:0/2404882688 learned_addr learned my addr 172.21.15.146:0/2404882688 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:21.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.310+0000 7f1c75400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780ca270 0x7f1c781571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.311+0000 7f1c75e00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 msgr2=0x7f1c78071400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:51:21.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.311+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c780cb3a0 0x7f1c78071400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.311+0000 7f1c75e00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780ca270 msgr2=0x7f1c781571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.311+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780ca270 0x7f1c781571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.311+0000 7f1c75e00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c78070590 con 0x7f1c780c68f0 2024-09-16T01:51:21.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.311+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c780ca270 0x7f1c781571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:21.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.312+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c78072a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800e9e0 tx=0x7f1c6800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.312+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68017070 con 0x7f1c780c68f0 2024-09-16T01:51:21.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.312+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c78070870 con 0x7f1c780c68f0 2024-09-16T01:51:21.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.312+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c78071940 con 0x7f1c780c68f0 2024-09-16T01:51:21.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.313+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c6800c9d0 con 0x7f1c780c68f0 2024-09-16T01:51:21.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.313+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6800cdc0 con 0x7f1c780c68f0 2024-09-16T01:51:21.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.315+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1c50005270 con 0x7f1c780c68f0 2024-09-16T01:51:21.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.320+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1c6801a070 con 0x7f1c780c68f0 2024-09-16T01:51:21.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.322+0000 7f1c6e000640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c5806c2b0 0x7f1c5806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.322+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(69..69 src has 1..69) ==== 5741+0+0 (secure 0 0 0) 0x7f1c6808b390 con 0x7f1c780c68f0 2024-09-16T01:51:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.322+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v69) ==== 54+0+16 (secure 0 0 0) 0x7f1c6808da10 con 0x7f1c780c68f0 2024-09-16T01:51:21.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.323+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c50000fa0 0x7f1c500045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.323+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c5806c2b0 0x7f1c5806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.323+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c50005550 con 0x7f1c50000fa0 2024-09-16T01:51:21.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.323+0000 7f1c76800640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c50000fa0 0x7f1c500045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.324+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c5806c2b0 0x7f1c5806e770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1c78073280 tx=0x7f1c640090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.324+0000 7f1c76800640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c50000fa0 0x7f1c500045e0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.325+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c50005550 con 0x7f1c50000fa0 2024-09-16T01:51:21.355 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:21.349+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:21.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.357+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c50004f80 con 0x7f1c50000fa0 2024-09-16T01:51:21.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.359+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c50004f80 con 0x7f1c50000fa0 2024-09-16T01:51:21.360 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:21.361 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:21.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.359+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c50001cb0 0x7f1c50002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.359+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1c50002320 con 0x7f1c50001cb0 2024-09-16T01:51:21.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.360+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c50001cb0 0x7f1c50002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.362+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c50001cb0 0x7f1c50002990 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.365+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c50002320 con 0x7f1c50001cb0 2024-09-16T01:51:21.395 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:21.395 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:21.395 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:21.395 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:21.395 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:21.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.402+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c50003200 con 0x7f1c50001cb0 2024-09-16T01:51:21.405 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:21.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.404+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c50003200 con 0x7f1c50001cb0 2024-09-16T01:51:21.406 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:21.406 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:21.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.404+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c50009c80 0x7f1c5000c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.404+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1c5000ee40 con 0x7f1c50009c80 2024-09-16T01:51:21.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.405+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c50009c80 0x7f1c5000c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.405+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c50009c80 0x7f1c5000c070 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.406+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c5000ee40 con 0x7f1c50009c80 2024-09-16T01:51:21.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.421+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c5000c800 con 0x7f1c50009c80 2024-09-16T01:51:21.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.424+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c5000c800 con 0x7f1c50009c80 2024-09-16T01:51:21.425 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:21.426 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:21.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.425+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c5000c9b0 0x7f1c5000ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.425+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1c5000d3a0 con 0x7f1c5000c9b0 2024-09-16T01:51:21.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.425+0000 7f1c76800640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c5000c9b0 0x7f1c5000ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.426+0000 7f1c76800640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c5000c9b0 0x7f1c5000ce60 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.427+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c5000d3a0 con 0x7f1c5000c9b0 2024-09-16T01:51:21.451 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:21.450+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:21.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.472+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c5000d8e0 con 0x7f1c5000c9b0 2024-09-16T01:51:21.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.478+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c5000d8e0 con 0x7f1c5000c9b0 2024-09-16T01:51:21.479 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:21.479 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:21.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.478+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c50011000 0x7f1c500133f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.478+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1c5000df80 con 0x7f1c50011000 2024-09-16T01:51:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.478+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c50011000 0x7f1c500133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.481+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c50011000 0x7f1c500133f0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.483+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c5000df80 con 0x7f1c50011000 2024-09-16T01:51:21.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.508+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c50013930 con 0x7f1c50011000 2024-09-16T01:51:21.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.509+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c50013930 con 0x7f1c50011000 2024-09-16T01:51:21.510 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:21.512 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:21.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.510+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c50014080 0x7f1c500144e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.510+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1c500193e0 con 0x7f1c50014080 2024-09-16T01:51:21.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.510+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c50014080 0x7f1c500144e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.510+0000 7f1c75400640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c50014080 0x7f1c500144e0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.511+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c500193e0 con 0x7f1c50014080 2024-09-16T01:51:21.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.534+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c500182e0 con 0x7f1c50014080 2024-09-16T01:51:21.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.537+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c500182e0 con 0x7f1c50014080 2024-09-16T01:51:21.538 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:21.540 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:21.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.537+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c50018960 0x7f1c50018de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.537+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1c5001c780 con 0x7f1c50018960 2024-09-16T01:51:21.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.537+0000 7f1c76800640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c50018960 0x7f1c50018de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.539+0000 7f1c76800640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c50018960 0x7f1c50018de0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.542 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:21.542 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:21.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.540+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c5001c780 con 0x7f1c50018960 2024-09-16T01:51:21.552 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:21.555 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:21.554+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:21.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.564+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c50014a20 con 0x7f1c50018960 2024-09-16T01:51:21.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.566+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c50014a20 con 0x7f1c50018960 2024-09-16T01:51:21.567 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:21.568 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:21.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.566+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c50015140 0x7f1c500155a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:21.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.567+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1c50015b60 con 0x7f1c50015140 2024-09-16T01:51:21.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.567+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c50015140 0x7f1c500155a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:21.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.567+0000 7f1c75e00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c50015140 0x7f1c500155a0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:21.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.568+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c50015b60 con 0x7f1c50015140 2024-09-16T01:51:21.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.591+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c50000950 con 0x7f1c50015140 2024-09-16T01:51:21.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.595+0000 7f1c6e000640 1 -- 172.21.15.146:0/2404882688 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c50000950 con 0x7f1c50015140 2024-09-16T01:51:21.597 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:21.597 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c50000fa0 msgr2=0x7f1c500045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c50000fa0 0x7f1c500045e0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c50001cb0 msgr2=0x7f1c50002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c50001cb0 0x7f1c50002990 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c50009c80 msgr2=0x7f1c5000c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c50009c80 0x7f1c5000c070 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c5000c9b0 msgr2=0x7f1c5000ce60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c5000c9b0 0x7f1c5000ce60 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c50011000 msgr2=0x7f1c500133f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c50011000 0x7f1c500133f0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c50014080 msgr2=0x7f1c500144e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c50014080 0x7f1c500144e0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c50018960 msgr2=0x7f1c50018de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c50018960 0x7f1c50018de0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.596+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c50015140 msgr2=0x7f1c500155a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c50015140 0x7f1c500155a0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c76800640 1 -- 172.21.15.146:0/2404882688 reap_dead start 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c76800640 1 -- 172.21.15.146:0/2404882688 reap_dead start 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c76800640 1 -- 172.21.15.146:0/2404882688 reap_dead start 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c76800640 1 -- 172.21.15.146:0/2404882688 reap_dead start 2024-09-16T01:51:21.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c76800640 1 -- 172.21.15.146:0/2404882688 reap_dead start 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c5806c2b0 msgr2=0x7f1c5806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c5806c2b0 0x7f1c5806e770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1c78073280 tx=0x7f1c640090e0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 msgr2=0x7f1c78072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c78072a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800e9e0 tx=0x7f1c6800eea0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 shutdown_connections 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c5806c2b0 0x7f1c5806e770 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 --2- 172.21.15.146:0/2404882688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c780c68f0 0x7f1c78072a80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:21.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.597+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 >> 172.21.15.146:0/2404882688 conn(0x7f1c7806cb70 msgr2=0x7f1c7806e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:21.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.599+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 shutdown_connections 2024-09-16T01:51:21.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:21.599+0000 7f1c7cc00640 1 -- 172.21.15.146:0/2404882688 wait complete. 2024-09-16T01:51:21.652 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:21.651+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:21.708 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:21.708 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:21.718 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:21.753 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:21.752+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:21.854 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:21.853+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:21.886 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:21.886 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:21.896 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:21.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:51:21.955 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:21.954+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:22.020 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:22.020 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:22.031 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:22.075 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:22.074+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:22.162 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:22.157+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:22.163 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:22.163 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:22.173 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:22.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.256+0000 7f705a000640 1 Processor -- start 2024-09-16T01:51:22.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.256+0000 7f705a000640 1 -- start start 2024-09-16T01:51:22.258 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:22.256+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:22.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.258+0000 7f705a000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70540c6410 0x7f7054158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:22.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.258+0000 7f705a000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70540c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:22.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.258+0000 7f705a000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7054151e70 con 0x7f7054151a90 2024-09-16T01:51:22.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.258+0000 7f705a000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f70540c8130 con 0x7f70540c6410 2024-09-16T01:51:22.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.258+0000 7f705a000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70540c8410 con 0x7f70540c5280 2024-09-16T01:51:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.258+0000 7f7053400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7054151a90 0x7f70540c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57974/0 (socket says 172.21.15.146:57974) 2024-09-16T01:51:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7053400640 1 -- 172.21.15.146:0/3077072870 learned_addr learned my addr 172.21.15.146:0/3077072870 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70540c6410 0x7f7054158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7053e00640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70540c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052000640 1 -- 172.21.15.146:0/3077072870 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502258078 0 0) 0x7f7054151e70 con 0x7f7054151a90 2024-09-16T01:51:22.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052000640 1 -- 172.21.15.146:0/3077072870 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7038003520 con 0x7f7054151a90 2024-09-16T01:51:22.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052a00640 1 -- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 msgr2=0x7f70540c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:22.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052a00640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70540c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052a00640 1 -- 172.21.15.146:0/3077072870 >> v1:172.21.15.161:6789/0 conn(0x7f7054151a90 legacy=0x7f70540c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:22.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7052a00640 1 -- 172.21.15.146:0/3077072870 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70540c86f0 con 0x7f70540c6410 2024-09-16T01:51:22.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.259+0000 7f7053e00640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70540c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:22.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.261+0000 7f7052a00640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054158040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f70400097a0 tx=0x7f704002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=375cd65e0288cd42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:22.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.261+0000 7f7052000640 1 -- 172.21.15.146:0/3077072870 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040037070 con 0x7f70540c6410 2024-09-16T01:51:22.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.261+0000 7f7052000640 1 -- 172.21.15.146:0/3077072870 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7040032440 con 0x7f70540c6410 2024-09-16T01:51:22.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.261+0000 7f7052000640 1 -- 172.21.15.146:0/3077072870 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040032810 con 0x7f70540c6410 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.262+0000 7f705a000640 1 -- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 msgr2=0x7f7054158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.262+0000 7f705a000640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054158040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f70400097a0 tx=0x7f704002a180 comp rx=0 tx=0).stop 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.262+0000 7f705a000640 1 -- 172.21.15.146:0/3077072870 shutdown_connections 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.262+0000 7f705a000640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70540c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.262+0000 7f705a000640 1 --2- 172.21.15.146:0/3077072870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054158040 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.262+0000 7f705a000640 1 -- 172.21.15.146:0/3077072870 >> 172.21.15.146:0/3077072870 conn(0x7f705406f3c0 msgr2=0x7f705406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:22.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.263+0000 7f705a000640 1 -- 172.21.15.146:0/3077072870 shutdown_connections 2024-09-16T01:51:22.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.264+0000 7f705a000640 1 -- 172.21.15.146:0/3077072870 wait complete. 2024-09-16T01:51:22.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.268+0000 7f705a000640 1 Processor -- start 2024-09-16T01:51:22.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.268+0000 7f705a000640 1 -- start start 2024-09-16T01:51:22.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f705a000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70541494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:22.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f705a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:22.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f705a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 0x7f7054147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f705a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7054157b20 con 0x7f7054151a90 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f705a000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7054157840 con 0x7f70540c6410 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f705a000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7054158580 con 0x7f70540c5280 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f7053e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 0x7f7054147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f7053e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 0x7f7054147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60370/0 (socket says 172.21.15.146:60370) 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f7053e00640 1 -- 172.21.15.146:0/567494978 learned_addr learned my addr 172.21.15.146:0/567494978 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.270+0000 7f7053400640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70541494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:22.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7052a00640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:22.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7053e00640 1 -- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 msgr2=0x7f70541494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:22.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7053e00640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70541494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7053e00640 1 -- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 msgr2=0x7f7054147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:22.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7053e00640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7053e00640 1 -- 172.21.15.146:0/567494978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7054146c30 con 0x7f7054151a90 2024-09-16T01:51:22.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7052a00640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:22.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.271+0000 7f7053e00640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 0x7f7054147650 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f704c009f00 tx=0x7f704c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:22.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.272+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c018030 con 0x7f7054151a90 2024-09-16T01:51:22.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.272+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f704c00ce50 con 0x7f7054151a90 2024-09-16T01:51:22.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.272+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c003bb0 con 0x7f7054151a90 2024-09-16T01:51:22.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.272+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541544f0 con 0x7f7054151a90 2024-09-16T01:51:22.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.273+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541f0fd0 con 0x7f7054151a90 2024-09-16T01:51:22.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.275+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7054146e00 con 0x7f7054151a90 2024-09-16T01:51:22.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.275+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f704c01b070 con 0x7f7054151a90 2024-09-16T01:51:22.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.277+0000 7f704b400640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703406c330 0x7f703406e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:22.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.277+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5741+0+0 (secure 0 0 0) 0x7f704c08be00 con 0x7f7054151a90 2024-09-16T01:51:22.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.279+0000 7f7053400640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703406c330 0x7f703406e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:22.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.280+0000 7f7053400640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703406c330 0x7f703406e7f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f70440011e0 tx=0x7f7044009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:22.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.287+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f704c010040 con 0x7f7054151a90 2024-09-16T01:51:22.335 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:22.335 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:22.345 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:22.359 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:22.358+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:22.459 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:22.458+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:22.503 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:22.503 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:22.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:22.565 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:22.562+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:22.662 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:22.661+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:22.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.689+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f70540c71f0 con 0x7f7054151a90 2024-09-16T01:51:22.763 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:22.762+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:22.799 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:22.799 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 rewriters = 341604.56 kB/sec 2024-09-16T01:51:22.799 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 rewriters = 341513.54 kB/sec 2024-09-16T01:51:22.799 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 341604.56 kB/sec 2024-09-16T01:51:22.799 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 341604.56 kB/sec 2024-09-16T01:51:22.799 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 341604.56 kB/sec 2024-09-16T01:51:22.800 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:22.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:22.864 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:22.863+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.943+0000 7f704b400640 1 -- 172.21.15.146:0/567494978 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v70) ==== 91+0+0 (secure 0 0 0) 0x7f704c05a630 con 0x7f7054151a90 2024-09-16T01:51:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:51:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.946+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703406c330 msgr2=0x7f703406e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.946+0000 7f705a000640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703406c330 0x7f703406e7f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f70440011e0 tx=0x7f7044009040 comp rx=0 tx=0).stop 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.946+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 msgr2=0x7f7054147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.946+0000 7f705a000640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 0x7f7054147650 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f704c009f00 tx=0x7f704c00c490 comp rx=0 tx=0).stop 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 shutdown_connections 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f703406c330 0x7f703406e7f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7054151a90 0x7f7054147650 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70540c6410 0x7f7054147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 --2- 172.21.15.146:0/567494978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70540c5280 0x7f70541494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 >> 172.21.15.146:0/567494978 conn(0x7f705406f3c0 msgr2=0x7f7054155390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:22.949 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:22.949 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:22.949 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T01:51:22.949 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:51:22.949 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:51:22.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-16T01:51:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 shutdown_connections 2024-09-16T01:51:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:22.947+0000 7f705a000640 1 -- 172.21.15.146:0/567494978 wait complete. 2024-09-16T01:51:22.964 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:22.963+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:23.021 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:23.021 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:23.033 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:23.054 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:23.055 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "" 2024-09-16T01:51:23.055 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:23.078 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:23.077+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:23.166 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:23.165+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:23.191 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:23.191 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:23.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:23.291 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:23.290+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:23.369 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:23.368+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:23.405 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:23.405 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:23.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:23.504 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:23.503+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:23.571 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:23.570+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:23.616 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:23.616 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:23.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:23.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:23.673 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:23.672+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:23.772 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:23.771+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:23.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.775+0000 7f86f6200640 1 Processor -- start 2024-09-16T01:51:23.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.775+0000 7f86f6200640 1 -- start start 2024-09-16T01:51:23.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.776+0000 7f86f6200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f00c7770 0x7f86f00d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.776+0000 7f86f6200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f86f00d0810 0x7f86f015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.776+0000 7f86f6200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86f00cb600 con 0x7f86f0154f40 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.776+0000 7f86f6200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f86f00cbbc0 con 0x7f86f00d0810 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.776+0000 7f86f6200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f86f0155320 con 0x7f86f00c7770 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86ef400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f86f0154f40 0x7f86f015eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44792/0 (socket says 172.21.15.146:44792) 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86ef400640 1 -- 172.21.15.146:0/337664957 learned_addr learned my addr 172.21.15.146:0/337664957 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86eea00640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f00c7770 0x7f86f00d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86efe00640 1 --2- 172.21.15.146:0/337664957 >> v2:172.21.15.161:3300/0 conn(0x7f86f00d0810 0x7f86f015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86ee000640 1 -- 172.21.15.146:0/337664957 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690749138 0 0) 0x7f86f00cb600 con 0x7f86f0154f40 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86ee000640 1 -- 172.21.15.146:0/337664957 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86d4003520 con 0x7f86f0154f40 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86efe00640 1 -- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f00c7770 msgr2=0x7f86f00d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86efe00640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f00c7770 0x7f86f00d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86efe00640 1 -- 172.21.15.146:0/337664957 >> v1:172.21.15.146:6789/0 conn(0x7f86f0154f40 legacy=0x7f86f015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86efe00640 1 -- 172.21.15.146:0/337664957 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f00cb8e0 con 0x7f86f00d0810 2024-09-16T01:51:23.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.777+0000 7f86eea00640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f00c7770 0x7f86f00d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:23.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.779+0000 7f86efe00640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00d0810 0x7f86f015ab70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009d90 tx=0x7f86e802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26de3f37f698995b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.779+0000 7f86ee000640 1 -- 172.21.15.146:0/337664957 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8007740 con 0x7f86f00d0810 2024-09-16T01:51:23.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.779+0000 7f86ee000640 1 -- 172.21.15.146:0/337664957 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86e80078e0 con 0x7f86f00d0810 2024-09-16T01:51:23.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.779+0000 7f86ee000640 1 -- 172.21.15.146:0/337664957 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8007c90 con 0x7f86f00d0810 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 -- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00d0810 msgr2=0x7f86f015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00d0810 0x7f86f015ab70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009d90 tx=0x7f86e802a660 comp rx=0 tx=0).stop 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 -- 172.21.15.146:0/337664957 shutdown_connections 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f00c7770 0x7f86f00d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 --2- 172.21.15.146:0/337664957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00d0810 0x7f86f015ab70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 -- 172.21.15.146:0/337664957 >> 172.21.15.146:0/337664957 conn(0x7f86f006cb50 msgr2=0x7f86f006e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:23.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 -- 172.21.15.146:0/337664957 shutdown_connections 2024-09-16T01:51:23.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.780+0000 7f86f6200640 1 -- 172.21.15.146:0/337664957 wait complete. 2024-09-16T01:51:23.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.781+0000 7f86f6200640 1 Processor -- start 2024-09-16T01:51:23.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.781+0000 7f86f6200640 1 -- start start 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.782+0000 7f86f6200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00c7770 0x7f86f015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.782+0000 7f86f6200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 0x7f86f015cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.782+0000 7f86f6200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f0154f40 0x7f86f015d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.782+0000 7f86f6200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f86f00b6210 con 0x7f86f00d0810 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.782+0000 7f86f6200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f86f00c7b90 con 0x7f86f00c7770 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.782+0000 7f86f6200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f86f015b0b0 con 0x7f86f0154f40 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 0x7f86f015cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86ef400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00c7770 0x7f86f015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86efe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f0154f40 0x7f86f015d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 0x7f86f015cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39932/0 (socket says 172.21.15.146:39932) 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86efe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f0154f40 0x7f86f015d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52004/0 (socket says 172.21.15.146:52004) 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86ef400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00c7770 0x7f86f015a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43542/0 (socket says 172.21.15.146:43542) 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 -- 172.21.15.146:0/2976014684 learned_addr learned my addr 172.21.15.146:0/2976014684 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f0154f40 msgr2=0x7f86f015d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f0154f40 0x7f86f015d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00c7770 msgr2=0x7f86f015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00c7770 0x7f86f015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f015a4d0 con 0x7f86f00d0810 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86f00c7770 0x7f86f015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86f0154f40 0x7f86f015d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:23.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.783+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 0x7f86f015cb90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00ebe0 tx=0x7f86dc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.784+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc017070 con 0x7f86f00d0810 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.784+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86dc00cc30 con 0x7f86f00d0810 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.784+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc013470 con 0x7f86f00d0810 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.784+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f01f49d0 con 0x7f86f00d0810 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.784+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f0158bc0 con 0x7f86f00d0810 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.785+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f86f015b4e0 con 0x7f86f00d0810 2024-09-16T01:51:23.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.787+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f86dc01a070 con 0x7f86f00d0810 2024-09-16T01:51:23.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.787+0000 7f86e7400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f86d006c2b0 0x7f86d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.787+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5741+0+0 (secure 0 0 0) 0x7f86dc08b320 con 0x7f86f00d0810 2024-09-16T01:51:23.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.788+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v70) ==== 54+0+16 (secure 0 0 0) 0x7f86dc08d9a0 con 0x7f86f00d0810 2024-09-16T01:51:23.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.788+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f86d006c2b0 0x7f86d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.788+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f86f01590e0 0x7f86f01594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.788+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86f00cbd70 con 0x7f86f01590e0 2024-09-16T01:51:23.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.788+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f86f01590e0 0x7f86f01594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.788+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f86d006c2b0 0x7f86d006e770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86e0005880 tx=0x7f86e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.789+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f86f01590e0 0x7f86f01594c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.790+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f00cbd70 con 0x7f86f01590e0 2024-09-16T01:51:23.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.804+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f00cc8b0 con 0x7f86f01590e0 2024-09-16T01:51:23.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.806+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f00cc8b0 con 0x7f86f01590e0 2024-09-16T01:51:23.808 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:23.809 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:23.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.806+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f86f00c5950 0x7f86f01f05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.806+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f86f01513f0 con 0x7f86f00c5950 2024-09-16T01:51:23.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.807+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f86f00c5950 0x7f86f01f05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.807+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f86f00c5950 0x7f86f01f05b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.808+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f01513f0 con 0x7f86f00c5950 2024-09-16T01:51:23.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.822+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f0151110 con 0x7f86f00c5950 2024-09-16T01:51:23.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.824+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f0151110 con 0x7f86f00c5950 2024-09-16T01:51:23.826 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:23.826 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:23.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.824+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f86f00c5d30 0x7f86f01f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.824+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f86f0159e90 con 0x7f86f00c5d30 2024-09-16T01:51:23.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.825+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f86f00c5d30 0x7f86f01f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.825+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f86f00c5d30 0x7f86f01f1000 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.826+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f0159e90 con 0x7f86f00c5d30 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 156.06398198700001, 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:51:23.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:51:23.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:51:23.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.842 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:7 3.2a 3:557f550e:::10000000002.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.573641653000003, 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081303988999999993, 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 7 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258696+0000", 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7320000000000001e-06 2024-09-16T01:51:23.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305572+0000", 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046875737000000001 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305609+0000", 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7704999999999999e-05 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305623+0000", 2024-09-16T01:51:23.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310196+0000", 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045734269999999997 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310344+0000", 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147181 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:23.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310523+0000", 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017908400000000001 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.327195+0000", 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016672724999999999 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339893+0000", 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012697559000000001 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339953+0000", 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0084999999999998e-05 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339998+0000", 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5218e-05 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.848 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:20 3.21 3:85380590:::10000000002.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.420634595000003, 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055783419000000001, 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 20 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411701+0000", 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.411703+0000", 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.84e-06 2024-09-16T01:51:23.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441653+0000", 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029949640999999999 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441685+0000", 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2452e-05 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441702+0000", 2024-09-16T01:51:23.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441798+0000", 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6501000000000003e-05 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.441964+0000", 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165635 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.442314+0000", 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00035073800000000002 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.455571+0000", 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013256273000000001 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467368+0000", 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011797656 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467432+0000", 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3585999999999997e-05 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.467485+0000", 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2729999999999998e-05 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.854 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:139 3.1c 3:384ad1d9:::10000000002.00000088:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.291633971000003, 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061931110999999997, 2024-09-16T01:51:23.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 139 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540702+0000", 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.540704+0000", 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.221e-06 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586535+0000", 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045830901 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586563+0000", 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8388e-05 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586591+0000", 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7424999999999999e-05 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586694+0000", 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000103269 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.586849+0000", 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154857 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.587008+0000", 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015895699999999999 2024-09-16T01:51:23.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.591909+0000", 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0049009609999999997 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602502+0000", 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010592773 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602564+0000", 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2154000000000002e-05 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.602633+0000", 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9204999999999999e-05 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:441 3.1c 3:3913eadd:::10000000002.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:23.861 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.223653919, 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072104032999999998, 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 441 2024-09-16T01:51:23.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608684+0000", 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4959999999999999e-06 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650015+0000", 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041330193000000001 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650045+0000", 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0335e-05 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650057+0000", 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650158+0000", 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100837 2024-09-16T01:51:23.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650266+0000", 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107975 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650426+0000", 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016051800000000001 2024-09-16T01:51:23.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.663669+0000", 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013242594 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680672+0000", 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017003041999999999 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680737+0000", 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4750000000000002e-05 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680786+0000", 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9217e-05 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:23.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.446529646999998, 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:51:23.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:51:23.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:524 3.19 3:9adb8721:::100000001f7.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.3534670349999995, 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092555282000000003, 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 524 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478871+0000", 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0700000000000001e-06 2024-09-16T01:51:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538563+0000", 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059691821999999999 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538594+0000", 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1285000000000003e-05 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538606+0000", 2024-09-16T01:51:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1688999999999999e-05 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538706+0000", 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100752 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538872+0000", 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016575699999999999 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.539058+0000", 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018612500000000001 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.547102+0000", 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080436529999999996 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571308+0000", 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024205946999999998 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571375+0000", 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6623000000000002e-05 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571424+0000", 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9558999999999998e-05 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:541 3.1c 3:387f027e:::100000001f7.0000001a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2115158130000001, 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084290084000000001, 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 541 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620823+0000", 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8169999999999999e-06 2024-09-16T01:51:23.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670392+0000", 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049568677999999998 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670432+0000", 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0312999999999998e-05 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670448+0000", 2024-09-16T01:51:23.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5781e-05 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670540+0000", 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1920000000000001e-05 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670705+0000", 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016490299999999999 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:23.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670881+0000", 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017675899999999999 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.683357+0000", 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012476179 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.704990+0000", 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021632195 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.705053+0000", 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3291000000000001e-05 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.705110+0000", 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7247999999999998e-05 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:557 3.10 3:0b5e6b00:::100000001f7.0000002a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.792602+0000", 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.0397336020000001, 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.069233480999999999, 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 557 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.792602+0000", 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.792602+0000", 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.792605+0000", 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0750000000000002e-06 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.816871+0000", 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024265641000000001 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.816911+0000", 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0240000000000001e-05 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.816930+0000", 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8689e-05 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.817116+0000", 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185918 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.817302+0000", 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018637300000000001 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.817444+0000", 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141865 2024-09-16T01:51:23.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.830032+0000", 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012587891 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.861681+0000", 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031649259999999999 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.861755+0000", 2024-09-16T01:51:23.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4040000000000003e-05 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.861836+0000", 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0488999999999995e-05 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:560 3.19 3:9968f8e1:::100000001f7.0000002d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.818861+0000", 2024-09-16T01:51:23.894 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.0134752749999993, 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065905354999999999, 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 560 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.818861+0000", 2024-09-16T01:51:23.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.818861+0000", 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.818863+0000", 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3439999999999999e-06 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.852356+0000", 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033492688999999999 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.852376+0000", 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0801999999999998e-05 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.852385+0000", 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1240000000000005e-06 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.852447+0000", 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2545999999999994e-05 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.852610+0000", 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016315500000000001 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.852755+0000", 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014499099999999999 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.871275+0000", 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018519357 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.884220+0000", 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012944941 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.884401+0000", 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018097400000000001 2024-09-16T01:51:23.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.884766+0000", 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036543200000000001 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:624 3.10 3:08d8b931:::100000001f7.0000006d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.428718+0000", 2024-09-16T01:51:23.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.4036175899999996, 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061237395999999999, 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 624 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.428718+0000", 2024-09-16T01:51:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.428718+0000", 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.428721+0000", 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.482470+0000", 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053749882999999998 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.482502+0000", 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1207000000000001e-05 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.482514+0000", 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2288e-05 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.482626+0000", 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112254 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.482779+0000", 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015334600000000001 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.482939+0000", 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159476 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.489348+0000", 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064094900000000003 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.489886+0000", 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000537128 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.489923+0000", 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7923000000000003e-05 2024-09-16T01:51:23.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.489956+0000", 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2221999999999997e-05 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:635 3.37 3:ee2638a1:::100000001f7.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.525652+0000", 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.3066843890000008, 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054587529000000003, 2024-09-16T01:51:23.906 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 635 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.525652+0000", 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.525652+0000", 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.525654+0000", 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0549999999999998e-06 2024-09-16T01:51:23.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.561252+0000", 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035598525999999998 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.561285+0000", 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2870000000000002e-05 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.561301+0000", 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6280999999999999e-05 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.561532+0000", 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023106299999999999 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.561703+0000", 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017070400000000001 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.561851+0000", 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014772599999999999 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.576866+0000", 2024-09-16T01:51:23.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015015662000000001 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.580134+0000", 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032673300000000001 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.580193+0000", 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9138999999999997e-05 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.580239+0000", 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6173e-05 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:681 3.10 3:0ad5cbe1:::100000001f7.000000a6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.924809+0000", 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.9075273240000001, 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070284214999999997, 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 681 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.924809+0000", 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.924809+0000", 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.924811+0000", 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981398+0000", 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056587232000000001 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981424+0000", 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5729e-05 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981436+0000", 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799e-05 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981563+0000", 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012704200000000001 2024-09-16T01:51:23.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981726+0000", 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016336599999999999 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.982383+0000", 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000656709 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.992308+0000", 2024-09-16T01:51:23.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099254040000000005 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.994970+0000", 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00266161 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.995039+0000", 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9146999999999994e-05 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.995093+0000", 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3820999999999999e-05 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.654512698, 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:51:23.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:51:23.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6189800750000001, 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:51:23.925 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:51:23.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:51:23.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:51:23.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:781 3.37 3:ef3f1fb0:::100000001f7.0000000a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.265612+0000", 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5667237900000002, 2024-09-16T01:51:23.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062977223999999998, 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 781 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.265612+0000", 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.265612+0000", 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.265614+0000", 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.291053+0000", 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025439357999999999 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.291084+0000", 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0712e-05 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.291104+0000", 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0023000000000001e-05 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.294984+0000", 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003880283 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295083+0000", 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8939999999999998e-05 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295228+0000", 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145241 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.320920+0000", 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025691453 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.328475+0000", 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0075553529999999999 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.328537+0000", 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1700999999999998e-05 2024-09-16T01:51:23.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.328589+0000", 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2630000000000003e-05 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:819 3.37 3:ef985689:::100000001f7.00000030:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.622420+0000", 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.209916008, 2024-09-16T01:51:23.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066382658999999997, 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 819 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.622420+0000", 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.622420+0000", 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.622422+0000", 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673256+0000", 2024-09-16T01:51:23.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050833761999999998 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673314+0000", 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7805e-05 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673327+0000", 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3526000000000001e-05 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673407+0000", 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9488999999999997e-05 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673508+0000", 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101817 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673673+0000", 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016442200000000001 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.682588+0000", 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089146429999999999 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.688676+0000", 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060883780000000002 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.688749+0000", 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3073999999999999e-05 2024-09-16T01:51:23.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.688803+0000", 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3575000000000001e-05 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:834 3.2a 3:57cc18b9:::100000001f7.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.742928+0000", 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.089408175, 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075641033999999996, 2024-09-16T01:51:23.943 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 834 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.742928+0000", 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.742928+0000", 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.742930+0000", 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6280000000000001e-06 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781555+0000", 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038624533000000003 2024-09-16T01:51:23.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781568+0000", 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2988e-05 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781610+0000", 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2323000000000001e-05 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781709+0000", 2024-09-16T01:51:23.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8333999999999997e-05 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781826+0000", 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117321 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.782003+0000", 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017726800000000001 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.817329+0000", 2024-09-16T01:51:23.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035325803000000003 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.818457+0000", 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011279490000000001 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.818516+0000", 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8743999999999999e-05 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.818569+0000", 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3143000000000003e-05 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:838 3.10 3:0832d2de:::100000001f7.00000043:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.806550+0000", 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.025785451, 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064653931999999997, 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.949 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 838 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.806550+0000", 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.806550+0000", 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.806552+0000", 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0470000000000001e-06 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.859652+0000", 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053100986000000003 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.859679+0000", 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6896000000000001e-05 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.859691+0000", 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1956e-05 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.859778+0000", 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7002000000000005e-05 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.859848+0000", 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0060000000000003e-05 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.859946+0000", 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7963999999999999e-05 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.868698+0000", 2024-09-16T01:51:23.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087512179999999998 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.871078+0000", 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023800179999999998 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.871143+0000", 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5556999999999996e-05 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.871204+0000", 2024-09-16T01:51:23.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1228000000000004e-05 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:873 3.10 3:0b31e270:::100000001f7.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.176797+0000", 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.6555393889999999, 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062146954999999997, 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:23.955 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 873 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.176797+0000", 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.176797+0000", 2024-09-16T01:51:23.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.176799+0000", 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6589999999999999e-06 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.224646+0000", 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047847052000000001 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.224676+0000", 2024-09-16T01:51:23.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9731e-05 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.224690+0000", 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3631999999999999e-05 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.224804+0000", 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114399 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:23.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.224922+0000", 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117834 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.225074+0000", 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152304 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.235611+0000", 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010537313 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.238811+0000", 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003199599 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.238888+0000", 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7131000000000004e-05 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.238943+0000", 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5300999999999998e-05 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.960 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:23.961 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:23.961 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:23.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.840+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f0062630 con 0x7f86f00c5d30 2024-09-16T01:51:23.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.842+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f0062630 con 0x7f86f00c5d30 2024-09-16T01:51:23.962 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.842+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f86f0063110 0x7f86f00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.842+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f86f0063570 con 0x7f86f0063110 2024-09-16T01:51:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.842+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f86f0063110 0x7f86f00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.843+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f86f0063110 0x7f86f00ce270 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.844+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f0063570 con 0x7f86f0063110 2024-09-16T01:51:23.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:23.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.858+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f00cea30 con 0x7f86f0063110 2024-09-16T01:51:23.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.859+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f00cea30 con 0x7f86f0063110 2024-09-16T01:51:23.965 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.859+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f86f002cb50 0x7f86f002cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.859+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f86f00cea30 con 0x7f86f002cb50 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.860+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f86f002cb50 0x7f86f002cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.860+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f86f002cb50 0x7f86f002cf90 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.861+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f00cea30 con 0x7f86f002cb50 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.875+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f00cea30 con 0x7f86f002cb50 2024-09-16T01:51:23.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.877+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f00cea30 con 0x7f86f002cb50 2024-09-16T01:51:23.967 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.877+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f86f00cea00 0x7f86f0206f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.877+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f86f0063570 con 0x7f86f00cea00 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.878+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f86f00cea00 0x7f86f0206f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.878+0000 7f86eea00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f86f00cea00 0x7f86f0206f70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.879+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f0063570 con 0x7f86f00cea00 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.892+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f002d9e0 con 0x7f86f00cea00 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.894+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f002d9e0 con 0x7f86f00cea00 2024-09-16T01:51:23.968 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:23.969 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:23.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.894+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f86f02079c0 0x7f86f0209db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.894+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f86f020cb80 con 0x7f86f02079c0 2024-09-16T01:51:23.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.895+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f86f02079c0 0x7f86f0209db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.895+0000 7f86efe00640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f86f02079c0 0x7f86f0209db0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.896+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f020cb80 con 0x7f86f02079c0 2024-09-16T01:51:23.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.909+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f020a2f0 con 0x7f86f02079c0 2024-09-16T01:51:23.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.912+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f020a2f0 con 0x7f86f02079c0 2024-09-16T01:51:23.970 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.912+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f86f020a800 0x7f86f020cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.912+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f86f020fcf0 con 0x7f86f020a800 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.912+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f86f020a800 0x7f86f020cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.912+0000 7f86ef400640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f86f020a800 0x7f86f020cc50 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.913+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86f020fcf0 con 0x7f86f020a800 2024-09-16T01:51:23.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.926+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f86f020ea10 con 0x7f86f020a800 2024-09-16T01:51:23.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86e7400640 1 -- 172.21.15.146:0/2976014684 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f86f020ea10 con 0x7f86f020a800 2024-09-16T01:51:23.972 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f86f01590e0 msgr2=0x7f86f01594c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f86f01590e0 0x7f86f01594c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f86f00c5950 msgr2=0x7f86f01f05b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f86f00c5950 0x7f86f01f05b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f86f00c5d30 msgr2=0x7f86f01f1000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f86f00c5d30 0x7f86f01f1000 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f86f0063110 msgr2=0x7f86f00ce270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f86f0063110 0x7f86f00ce270 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f86f002cb50 msgr2=0x7f86f002cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f86f002cb50 0x7f86f002cf90 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f86f00cea00 msgr2=0x7f86f0206f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f86f00cea00 0x7f86f0206f70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f86f02079c0 msgr2=0x7f86f0209db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f86f02079c0 0x7f86f0209db0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f86f020a800 msgr2=0x7f86f020cc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f86f020a800 0x7f86f020cc50 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86efe00640 1 -- 172.21.15.146:0/2976014684 reap_dead start 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86efe00640 1 -- 172.21.15.146:0/2976014684 reap_dead start 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86efe00640 1 -- 172.21.15.146:0/2976014684 reap_dead start 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86efe00640 1 -- 172.21.15.146:0/2976014684 reap_dead start 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86efe00640 1 -- 172.21.15.146:0/2976014684 reap_dead start 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f86d006c2b0 msgr2=0x7f86d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f86d006c2b0 0x7f86d006e770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86e0005880 tx=0x7f86e0009040 comp rx=0 tx=0).stop 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 msgr2=0x7f86f015cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:23.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 0x7f86f015cb90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00ebe0 tx=0x7f86dc00c520 comp rx=0 tx=0).stop 2024-09-16T01:51:23.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 shutdown_connections 2024-09-16T01:51:23.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f86d006c2b0 0x7f86d006e770 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 --2- 172.21.15.146:0/2976014684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86f00d0810 0x7f86f015cb90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:23.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 >> 172.21.15.146:0/2976014684 conn(0x7f86f006cb50 msgr2=0x7f86f00cfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:23.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.929+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 shutdown_connections 2024-09-16T01:51:23.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:23.930+0000 7f86f6200640 1 -- 172.21.15.146:0/2976014684 wait complete. 2024-09-16T01:51:23.976 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:23.961+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:24.062 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:24.062+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:24.162 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:24.163 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:24.163 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:24.163 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:24.162+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:24.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:24.264 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:24.263+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:24.365 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:24.364+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:24.374 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:24.374 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:24.374 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:24.374 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:24.374 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:24.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:24.466 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:24.465+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:24.567 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:24.566+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:24.587 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:24.587 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:24.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:24.587 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:24.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.017044552000002, 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.588 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:24.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.593 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.863498870000001, 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:24.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:24.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:24.599 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.642639189999997, 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:24.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:24.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:24.605 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.564508601, 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:24.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:24.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:24.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.992295714999997, 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:24.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.768139181000002, 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:24.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:24.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:24.623 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.470862158000003, 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:24.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.998054305, 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:24.629 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:24.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:24.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.635 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.810227719, 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:24.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:24.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:24.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.301411313999999, 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:24.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:24.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:24.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.9225654149999993, 2024-09-16T01:51:24.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:51:24.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.654 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:607 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2941786240000006, 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057026931000000003, 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 607 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290713+0000", 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6900000000000001e-06 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330961+0000", 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040248199999999998 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330992+0000", 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1251000000000003e-05 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331005+0000", 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3006999999999999e-05 2024-09-16T01:51:24.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332660+0000", 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016549589999999999 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332819+0000", 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159137 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332968+0000", 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148942 2024-09-16T01:51:24.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.342537+0000", 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095683910000000007 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347616+0000", 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005079793 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347682+0000", 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5622000000000003e-05 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347737+0000", 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4938999999999997e-05 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:634 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:24.660 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.0699902100000003, 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068283101999999998, 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 634 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:24.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514901+0000", 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5299999999999999e-06 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549207+0000", 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034306171000000003 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549224+0000", 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6976000000000001e-05 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549235+0000", 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0682999999999999e-05 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549568+0000", 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033294599999999998 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549714+0000", 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014617499999999999 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549858+0000", 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144356 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.566205+0000", 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01634683 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583078+0000", 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016872402000000002 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583135+0000", 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7286e-05 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583182+0000", 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6746999999999998e-05 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:688 3.26 3:642b2ad4:::100000001f7.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.5530227379999992, 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062867175999999997, 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 688 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031869+0000", 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5509999999999998e-06 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084368+0000", 2024-09-16T01:51:24.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052499410000000003 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084392+0000", 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084405+0000", 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2561000000000001e-05 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084491+0000", 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5945000000000005e-05 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084662+0000", 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017081299999999999 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084812+0000", 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993999999999999 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.091507+0000", 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066950079999999997 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094630+0000", 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031236570000000002 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094700+0000", 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9575999999999999e-05 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094733+0000", 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3260999999999999e-05 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:696 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.4577648740000004, 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071884475000000003, 2024-09-16T01:51:24.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 696 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127126+0000", 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9369999999999998e-06 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159227+0000", 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032101044000000002 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159250+0000", 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3332999999999998e-05 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159260+0000", 2024-09-16T01:51:24.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159343+0000", 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3373000000000003e-05 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159484+0000", 2024-09-16T01:51:24.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014100399999999999 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159646+0000", 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162044 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.186104+0000", 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026457794999999999 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198896+0000", 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012791749 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198959+0000", 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3387999999999997e-05 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.199008+0000", 2024-09-16T01:51:24.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9118e-05 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:776 3.11 e68/20 3:887bb7a4:::100000001f7.00000005:head v 68'37, mlcod=65'35)", 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.376845737, 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05614222, 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208044+0000", 2024-09-16T01:51:24.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219371+0000", 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011326688999999999 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219442+0000", 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0727000000000002e-05 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219483+0000", 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1165000000000003e-05 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219725+0000", 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024194200000000001 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219752+0000", 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7270000000000001e-05 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264133+0000", 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044381029000000002 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264185+0000", 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1879000000000003e-05 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:786 3.3a 3:5d44b367:::100000001f7.0000000f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:24.683 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.3412566439999996, 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055535233000000003, 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 786 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243634+0000", 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254427+0000", 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010792636 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254446+0000", 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9219999999999999e-05 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254458+0000", 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2387e-05 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254837+0000", 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00037878500000000003 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254934+0000", 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6915000000000003e-05 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.255090+0000", 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015594200000000001 2024-09-16T01:51:24.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.273598+0000", 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018507932000000001 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299039+0000", 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025441583 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299115+0000", 2024-09-16T01:51:24.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5319000000000002e-05 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299167+0000", 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2473999999999999e-05 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:785 3.11 e69/20 3:8b60708a:::100000001f7.0000000e:head v 69'38, mlcod=65'35)", 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.330024474, 2024-09-16T01:51:24.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082167670999999998, 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254865+0000", 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9799999999999999e-07 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286626+0000", 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031761664000000002 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286647+0000", 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268e-05 2024-09-16T01:51:24.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286659+0000", 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1276e-05 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286743+0000", 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4488999999999997e-05 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286772+0000", 2024-09-16T01:51:24.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9082999999999999e-05 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.336965+0000", 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050192716999999998 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337032+0000", 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6876000000000001e-05 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:790 3.16 3:69eaaa52:::100000001f7.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2822522349999996, 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061715643000000001, 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 790 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302638+0000", 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9410000000000001e-06 2024-09-16T01:51:24.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342651+0000", 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040012486 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342687+0000", 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5738999999999998e-05 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342701+0000", 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4255e-05 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343871+0000", 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001169977 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343975+0000", 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010435 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.344127+0000", 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015170400000000001 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.361011+0000", 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016884528999999999 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364224+0000", 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003212183 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364295+0000", 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0957000000000005e-05 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364352+0000", 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7522000000000003e-05 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:863 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.5331194520000002, 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055946002000000002, 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 863 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051772+0000", 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2620000000000001e-06 2024-09-16T01:51:24.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087118+0000", 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035346883000000003 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087186+0000", 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7743000000000005e-05 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087196+0000", 2024-09-16T01:51:24.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087342+0000", 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146625 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087445+0000", 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010255 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:24.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087572+0000", 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012764800000000001 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.101292+0000", 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013719543000000001 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107514+0000", 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062224120000000001 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107641+0000", 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012699 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107715+0000", 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3906999999999996e-05 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:24.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:24.709 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:24.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:24.711 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:24.709+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:24.812 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:24.811+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:24.869 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:24.869 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:24.869 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:24.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:24.927 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:24.926+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:24.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:51:25.020 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:25.019+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:25.043 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:25.043 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:25.043 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:25.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.131+0000 7fc99b000640 1 Processor -- start 2024-09-16T01:51:25.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.131+0000 7fc99b000640 1 -- start start 2024-09-16T01:51:25.132 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:25.131+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc99b000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9940c94d0 0x7fc99415a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc99b000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940c99f0 con 0x7fc994154d20 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc99b000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940c9c90 con 0x7fc99415eb60 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc99b000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc994155100 con 0x7fc9940c94d0 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc993e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc994154d20 0x7fc9940715a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44800/0 (socket says 172.21.15.146:44800) 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc993e00640 1 -- 172.21.15.146:0/1888015762 learned_addr learned my addr 172.21.15.146:0/1888015762 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.132+0000 7fc993400640 1 --2- 172.21.15.146:0/1888015762 >> v2:172.21.15.161:3300/0 conn(0x7fc9940c94d0 0x7fc99415a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.133+0000 7fc993400640 1 -- 172.21.15.146:0/1888015762 >> v1:172.21.15.161:6789/0 conn(0x7fc99415eb60 legacy=0x7fc9941652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.133+0000 7fc993400640 1 -- 172.21.15.146:0/1888015762 >> v1:172.21.15.146:6789/0 conn(0x7fc994154d20 legacy=0x7fc9940715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.133+0000 7fc992a00640 1 -- 172.21.15.146:0/1888015762 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3859595657 0 0) 0x7fc9940c99f0 con 0x7fc994154d20 2024-09-16T01:51:25.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.133+0000 7fc993400640 1 -- 172.21.15.146:0/1888015762 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99415e0c0 con 0x7fc9940c94d0 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.134+0000 7fc993400640 1 --2- 172.21.15.146:0/1888015762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940c94d0 0x7fc99415a930 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc9940c9e60 tx=0x7fc98002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6eac452e255afc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.134+0000 7fc992a00640 1 -- 172.21.15.146:0/1888015762 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980037070 con 0x7fc9940c94d0 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.134+0000 7fc992a00640 1 -- 172.21.15.146:0/1888015762 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc980032440 con 0x7fc9940c94d0 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.134+0000 7fc992a00640 1 -- 172.21.15.146:0/1888015762 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980032810 con 0x7fc9940c94d0 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 -- 172.21.15.146:0/1888015762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940c94d0 msgr2=0x7fc99415a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 --2- 172.21.15.146:0/1888015762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940c94d0 0x7fc99415a930 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc9940c9e60 tx=0x7fc98002a210 comp rx=0 tx=0).stop 2024-09-16T01:51:25.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 -- 172.21.15.146:0/1888015762 shutdown_connections 2024-09-16T01:51:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 --2- 172.21.15.146:0/1888015762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940c94d0 0x7fc99415a930 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 -- 172.21.15.146:0/1888015762 >> 172.21.15.146:0/1888015762 conn(0x7fc99406bc10 msgr2=0x7fc99406fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 -- 172.21.15.146:0/1888015762 shutdown_connections 2024-09-16T01:51:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.135+0000 7fc99b000640 1 -- 172.21.15.146:0/1888015762 wait complete. 2024-09-16T01:51:25.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.136+0000 7fc99b000640 1 Processor -- start 2024-09-16T01:51:25.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 -- start start 2024-09-16T01:51:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 0x7fc9940ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc994154d20 0x7fc9941e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415eb60 0x7fc9941e57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940ca1f0 con 0x7fc9940c94d0 2024-09-16T01:51:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940c5460 con 0x7fc99415eb60 2024-09-16T01:51:25.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.137+0000 7fc99b000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9941f10e0 con 0x7fc994154d20 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 0x7fc9940ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 0x7fc9940ce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39940/0 (socket says 172.21.15.146:39940) 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 -- 172.21.15.146:0/4007363221 learned_addr learned my addr 172.21.15.146:0/4007363221 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993400640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc994154d20 0x7fc9941e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc999000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415eb60 0x7fc9941e57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 -- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc994154d20 msgr2=0x7fc9941e42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc994154d20 0x7fc9941e42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 -- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415eb60 msgr2=0x7fc9941e57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415eb60 0x7fc9941e57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 -- 172.21.15.146:0/4007363221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9941e3e00 con 0x7fc9940c94d0 2024-09-16T01:51:25.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.138+0000 7fc993e00640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 0x7fc9940ce500 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c4d0 tx=0x7fc98400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:25.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.139+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984018070 con 0x7fc9940c94d0 2024-09-16T01:51:25.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.139+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc984007500 con 0x7fc9940c94d0 2024-09-16T01:51:25.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.139+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984004a20 con 0x7fc9940c94d0 2024-09-16T01:51:25.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.139+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9941f07b0 con 0x7fc9940c94d0 2024-09-16T01:51:25.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.139+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9941f0d70 con 0x7fc9940c94d0 2024-09-16T01:51:25.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.140+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99415e360 con 0x7fc9940c94d0 2024-09-16T01:51:25.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.141+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc98401b070 con 0x7fc9940c94d0 2024-09-16T01:51:25.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.142+0000 7fc98be00640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc97406c2b0 0x7fc97406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:25.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.142+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5741+0+0 (secure 0 0 0) 0x7fc98408a790 con 0x7fc9940c94d0 2024-09-16T01:51:25.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.142+0000 7fc993400640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc97406c2b0 0x7fc97406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:25.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.143+0000 7fc993400640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc97406c2b0 0x7fc97406e770 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc980034000 tx=0x7fc98002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:25.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.148+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc9840587f0 con 0x7fc9940c94d0 2024-09-16T01:51:25.216 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:25.215+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:25.267 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.006832064000001, 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.268 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:25.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:25.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.808459194000001, 2024-09-16T01:51:25.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:25.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:25.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:25.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.281 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.699656562999998, 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:25.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:25.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.288 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.380087930000002, 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:25.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:25.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:25.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:25.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:25.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.298 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.881425190000002, 2024-09-16T01:51:25.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:25.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:25.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.467852506, 2024-09-16T01:51:25.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:25.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:25.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.314 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.481479413999999, 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:25.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:25.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:25.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.320 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.413374773000001, 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.321 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:25.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:25.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:25.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:25.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:25.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:25.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:25.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:25.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.270915108000001, 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:25.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:25.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:25.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:25.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:25.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:25.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:25.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.923448884999999, 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.344 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:25.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:25.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:25.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:544 3.14 3:28ae9428:::100000001f7.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.618074983, 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064751230000000007, 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 544 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:25.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644039+0000", 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6619999999999999e-06 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665383+0000", 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021343964 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665408+0000", 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.561e-05 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665422+0000", 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3947999999999999e-05 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665607+0000", 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018433499999999999 2024-09-16T01:51:25.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665831+0000", 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022424399999999999 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665986+0000", 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155654 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.682208+0000", 2024-09-16T01:51:25.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016221920000000001 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708669+0000", 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026460513000000001 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708734+0000", 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4888000000000006e-05 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708788+0000", 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4491999999999999e-05 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:549 3.11 3:8ae93890:::100000001f7.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:25.358 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.575874962, 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081050590000000006, 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 549 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686239+0000", 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754451+0000", 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068212138000000005 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754474+0000", 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2733000000000001e-05 2024-09-16T01:51:25.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754483+0000", 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1349999999999998e-06 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754562+0000", 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8319999999999996e-05 2024-09-16T01:51:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754725+0000", 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016298699999999999 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754875+0000", 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149942 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.762160+0000", 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072859630000000003 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767169+0000", 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050088609999999999 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767231+0000", 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1158e-05 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767288+0000", 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7108000000000003e-05 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:606 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.9874609149999998, 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058221427999999999, 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274653+0000", 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322327+0000", 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047674080000000001 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322357+0000", 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9813000000000001e-05 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322369+0000", 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2074e-05 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322494+0000", 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012469 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322653+0000", 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158987 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322803+0000", 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150433 2024-09-16T01:51:25.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331674+0000", 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088701219999999994 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332772+0000", 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001098552 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332829+0000", 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6941000000000003e-05 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332873+0000", 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3516000000000001e-05 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:626 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:25.373 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7974690399999993, 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057812853999999997, 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 626 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464645+0000", 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9759999999999998e-06 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505192+0000", 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040546806999999997 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505211+0000", 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9386e-05 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505220+0000", 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4810000000000006e-06 2024-09-16T01:51:25.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505316+0000", 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6748000000000003e-05 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505479+0000", 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016267300000000001 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505668+0000", 2024-09-16T01:51:25.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018906700000000001 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.520824+0000", 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015155686999999999 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522336+0000", 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015126359999999999 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522404+0000", 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7174000000000004e-05 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522456+0000", 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2219000000000003e-05 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:629 3.13 3:ca345efb:::100000001f7.00000072:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.379 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7533203369999999, 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073672847999999999, 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 629 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508793+0000", 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5650000000000001e-06 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536849+0000", 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028055429999999999 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536888+0000", 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9128999999999998e-05 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536904+0000", 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6223e-05 2024-09-16T01:51:25.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537082+0000", 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017825600000000001 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537242+0000", 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159569 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537394+0000", 2024-09-16T01:51:25.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015228 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.548370+0000", 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010976324000000001 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582365+0000", 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994745999999999 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582423+0000", 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7357e-05 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582465+0000", 2024-09-16T01:51:25.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1968999999999997e-05 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:700 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.1284699049999993, 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074697384000000006, 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.386 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 700 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:25.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133644+0000", 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159468+0000", 2024-09-16T01:51:25.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025824323 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159551+0000", 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2286999999999993e-05 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159561+0000", 2024-09-16T01:51:25.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0307e-05 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159851+0000", 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029012100000000001 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160192+0000", 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034104600000000001 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160444+0000", 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025126400000000001 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.197238+0000", 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036794265999999999 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208261+0000", 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0110233 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208304+0000", 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2833999999999997e-05 2024-09-16T01:51:25.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208339+0000", 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5556000000000003e-05 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:776 3.11 3:887bb7a4:::100000001f7.00000005:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0654283509999996, 2024-09-16T01:51:25.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090355046999999994, 2024-09-16T01:51:25.394 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.394 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.394 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 776 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196686+0000", 2024-09-16T01:51:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5289999999999998e-06 2024-09-16T01:51:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206873+0000", 2024-09-16T01:51:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010186855999999999 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206895+0000", 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1549e-05 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206907+0000", 2024-09-16T01:51:25.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2884000000000001e-05 2024-09-16T01:51:25.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207056+0000", 2024-09-16T01:51:25.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014810499999999999 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207156+0000", 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010007 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207302+0000", 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014591499999999999 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263823+0000", 2024-09-16T01:51:25.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056521543 2024-09-16T01:51:25.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286896+0000", 2024-09-16T01:51:25.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023073327000000001 2024-09-16T01:51:25.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286980+0000", 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3869999999999995e-05 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.287039+0000", 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8399000000000001e-05 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0391575209999999, 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.404 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:25.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:51:25.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:51:25.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:788 3.2d 3:b61340bb:::100000001f7.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.966319736, 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058010644, 2024-09-16T01:51:25.411 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 788 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:25.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295795+0000", 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4509999999999999e-06 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338094+0000", 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042298997999999997 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338113+0000", 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9100999999999999e-05 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338121+0000", 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1370000000000002e-06 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338192+0000", 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1515999999999998e-05 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338283+0000", 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0276000000000001e-05 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338420+0000", 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013750000000000001 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.347191+0000", 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087705330000000005 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353679+0000", 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064878189999999997 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353751+0000", 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2193999999999994e-05 2024-09-16T01:51:25.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353803+0000", 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2119e-05 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:862 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.2252729569999996, 2024-09-16T01:51:25.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063151895999999999, 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 862 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:25.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036841+0000", 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.075e-06 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:25.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069065+0000", 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032224153999999998 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069093+0000", 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7797000000000002e-05 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069107+0000", 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3594000000000001e-05 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069222+0000", 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115179 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069335+0000", 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112748 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069508+0000", 2024-09-16T01:51:25.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173692 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.089364+0000", 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019855866 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099875+0000", 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511177 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:25.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099941+0000", 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5508999999999999e-05 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099991+0000", 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0105000000000002e-05 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:25.423 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:25.425 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:25.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.342+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fc994153380 con 0x7fc9940c94d0 2024-09-16T01:51:25.426 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:25.424+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:25.526 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:25.525+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:25.562 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:25.562 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:25.562 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:25.563 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:25.563 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:25.646 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:25.645+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:25.695 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:25.695 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 readers = 5094183.00 kB/sec 2024-09-16T01:51:25.695 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 readers = 5074311.08 kB/sec 2024-09-16T01:51:25.696 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 5094183.00 kB/sec 2024-09-16T01:51:25.696 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 5094183.00 kB/sec 2024-09-16T01:51:25.696 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 5094183.00 kB/sec 2024-09-16T01:51:25.697 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:25.707 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:25.707 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:25.707 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:25.707 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:25.707 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:25.716 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:25.757 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:25.756+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:25.829 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:25.828+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.862 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.309543312999999, 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:25.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:25.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.062509732000002, 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:25.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:25.874 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.003350646000001, 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.875 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:25.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:25.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.091772333999998, 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.882 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:25.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:25.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:25.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.047512103999999, 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.888 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:25.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:25.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:25.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.017012284000003, 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.894 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:25.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.899 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.975456762, 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:25.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:25.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.837410067999997, 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.906 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:25.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:25.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.718450341, 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.912 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:25.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:25.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:25.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.683654042000001, 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.920 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:25.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:25.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:25.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.930 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.600205088999999, 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.931 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:25.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:25.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:25.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:25.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.400902302, 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:51:25.937 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:51:25.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:51:25.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.256025363999999, 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.944 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.215380939999999, 2024-09-16T01:51:25.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:51:25.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:51:25.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.057438105999999, 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:51:25.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:51:25.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:51:25.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:51:25.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.932310845, 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.963 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:51:25.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:51:25.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.968 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:576 3.1b 3:d81af1ca:::100000001f7.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.823508158999999, 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052702763999999999, 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.969 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 576 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.970 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035658+0000", 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.03e-06 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075189+0000", 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039530957999999998 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075204+0000", 2024-09-16T01:51:25.971 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5071e-05 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075214+0000", 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075299+0000", 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4807000000000003e-05 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.972 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075401+0000", 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010220700000000001 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075495+0000", 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4006999999999996e-05 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.078964+0000", 2024-09-16T01:51:25.973 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003468741 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088273+0000", 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0093088089999999995 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088311+0000", 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8402000000000003e-05 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.974 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088359+0000", 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7920999999999997e-05 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:598 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.643428292999999, 2024-09-16T01:51:25.975 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054052254000000001, 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 598 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:25.976 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215738+0000", 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.526e-06 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.977 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250373+0000", 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034634160999999997 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250391+0000", 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7986e-05 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250400+0000", 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1439999999999992e-06 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.978 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250455+0000", 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5578999999999998e-05 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250562+0000", 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106717 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250665+0000", 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010307200000000001 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.979 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.253888+0000", 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032231019999999998 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269702+0000", 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015813473000000002 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269742+0000", 2024-09-16T01:51:25.980 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0154000000000002e-05 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269788+0000", 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.634e-05 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.8955109140000008, 2024-09-16T01:51:25.981 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.982 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.983 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.984 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:51:25.985 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.986 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:25.987 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.7324852719999999, 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:25.988 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.989 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:51:25.990 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:51:25.991 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.992 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:25.993 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:25.994 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:25.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:25.997 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:25.994+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:25.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.947+0000 7fc98be00640 1 -- 172.21.15.146:0/4007363221 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v71) ==== 101+0+0 (secure 0 0 0) 0x7fc98405caf0 con 0x7fc9940c94d0 2024-09-16T01:51:25.997 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:51:25.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc97406c2b0 msgr2=0x7fc97406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc97406c2b0 0x7fc97406e770 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc980034000 tx=0x7fc98002e000 comp rx=0 tx=0).stop 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 msgr2=0x7fc9940ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 0x7fc9940ce500 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c4d0 tx=0x7fc98400c990 comp rx=0 tx=0).stop 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 shutdown_connections 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc97406c2b0 0x7fc97406e770 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415eb60 0x7fc9941e57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc994154d20 0x7fc9941e42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 --2- 172.21.15.146:0/4007363221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9940c94d0 0x7fc9940ce500 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 >> 172.21.15.146:0/4007363221 conn(0x7fc99406bc10 msgr2=0x7fc994153b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 shutdown_connections 2024-09-16T01:51:25.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:25.950+0000 7fc99b000640 1 -- 172.21.15.146:0/4007363221 wait complete. 2024-09-16T01:51:26.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:51:26.044 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:51:26.123 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:26.122+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:26.142 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:26.143 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:26.143 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:26.143 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:26.151 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:26.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.189+0000 7fce38e00640 1 Processor -- start 2024-09-16T01:51:26.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.189+0000 7fce38e00640 1 -- start start 2024-09-16T01:51:26.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce38e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce340d0490 0x7fce340c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce38e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 0x7fce340c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce38e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce34154690 0x7fce34154b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce38e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fce340c6160 con 0x7fce340d0490 2024-09-16T01:51:26.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce38e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fce3415e220 con 0x7fce340d0870 2024-09-16T01:51:26.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce38e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fce3415e500 con 0x7fce34154690 2024-09-16T01:51:26.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce2aa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 0x7fce340c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce33400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce340d0490 0x7fce340c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce33e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce34154690 0x7fce34154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce2aa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 0x7fce340c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43556/0 (socket says 172.21.15.146:43556) 2024-09-16T01:51:26.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce33400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce340d0490 0x7fce340c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39954/0 (socket says 172.21.15.146:39954) 2024-09-16T01:51:26.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.190+0000 7fce2aa00640 1 -- 172.21.15.146:0/2078658578 learned_addr learned my addr 172.21.15.146:0/2078658578 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:26.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.191+0000 7fce33e00640 1 -- 172.21.15.146:0/2078658578 >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 msgr2=0x7fce340c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:26.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.191+0000 7fce33e00640 1 --2- 172.21.15.146:0/2078658578 >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 0x7fce340c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.191+0000 7fce33e00640 1 -- 172.21.15.146:0/2078658578 >> v2:172.21.15.146:3300/0 conn(0x7fce340d0490 msgr2=0x7fce340c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:26.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.191+0000 7fce33e00640 1 --2- 172.21.15.146:0/2078658578 >> v2:172.21.15.146:3300/0 conn(0x7fce340d0490 0x7fce340c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.191+0000 7fce33e00640 1 -- 172.21.15.146:0/2078658578 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3415e7e0 con 0x7fce34154690 2024-09-16T01:51:26.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.191+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2078658578 >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 0x7fce340c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.192+0000 7fce33e00640 1 --2- 172.21.15.146:0/2078658578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce34154690 0x7fce34154b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce340c8730 tx=0x7fce2c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d72c9ac7861c415f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.192+0000 7fce32a00640 1 -- 172.21.15.146:0/2078658578 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2c007740 con 0x7fce34154690 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.192+0000 7fce32a00640 1 -- 172.21.15.146:0/2078658578 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce2c0078e0 con 0x7fce34154690 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.192+0000 7fce32a00640 1 -- 172.21.15.146:0/2078658578 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2c007cb0 con 0x7fce34154690 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.192+0000 7fce38e00640 1 -- 172.21.15.146:0/2078658578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce34154690 msgr2=0x7fce34154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.192+0000 7fce38e00640 1 --2- 172.21.15.146:0/2078658578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce34154690 0x7fce34154b00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce340c8730 tx=0x7fce2c02a660 comp rx=0 tx=0).stop 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 -- 172.21.15.146:0/2078658578 shutdown_connections 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 --2- 172.21.15.146:0/2078658578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce34154690 0x7fce34154b00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 --2- 172.21.15.146:0/2078658578 >> v2:172.21.15.161:3300/0 conn(0x7fce340d0870 0x7fce340c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 --2- 172.21.15.146:0/2078658578 >> v2:172.21.15.146:3300/0 conn(0x7fce340d0490 0x7fce340c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 -- 172.21.15.146:0/2078658578 >> 172.21.15.146:0/2078658578 conn(0x7fce3406d6f0 msgr2=0x7fce3406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 -- 172.21.15.146:0/2078658578 shutdown_connections 2024-09-16T01:51:26.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.193+0000 7fce38e00640 1 -- 172.21.15.146:0/2078658578 wait complete. 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.194+0000 7fce38e00640 1 Processor -- start 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.194+0000 7fce38e00640 1 -- start start 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce38e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce340d0490 0x7fce3414e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce38e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 0x7fce3414edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce38e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce34154690 0x7fce340b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce38e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fce34155040 con 0x7fce340d0490 2024-09-16T01:51:26.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce38e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fce3415eda0 con 0x7fce34154690 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce38e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fce3415dec0 con 0x7fce340d0870 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce2aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 0x7fce3414edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce2aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 0x7fce3414edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52032/0 (socket says 172.21.15.146:52032) 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce2aa00640 1 -- 172.21.15.146:0/2114595143 learned_addr learned my addr 172.21.15.146:0/2114595143 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce34154690 0x7fce340b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.195+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce340d0490 0x7fce3414e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2aa00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce34154690 msgr2=0x7fce340b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce34154690 0x7fce340b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2aa00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce340d0490 msgr2=0x7fce3414e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce340d0490 0x7fce3414e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2aa00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3414f310 con 0x7fce340d0870 2024-09-16T01:51:26.199 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:26.196+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 0x7fce3414edd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00e980 tx=0x7fce1c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c017070 con 0x7fce340d0870 2024-09-16T01:51:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1c00ca00 con 0x7fce340d0870 2024-09-16T01:51:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c00cdd0 con 0x7fce340d0870 2024-09-16T01:51:26.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.196+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce341f46f0 con 0x7fce340d0870 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.197+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce3415d930 con 0x7fce340d0870 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.197+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fce34154b00 con 0x7fce340d0870 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.198+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fce1c01a070 con 0x7fce340d0870 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.199+0000 7fce2be00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce1006c2b0 0x7fce1006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.199+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 5741+0+0 (secure 0 0 0) 0x7fce1c08b3d0 con 0x7fce340d0870 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.200+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce1006c2b0 0x7fce1006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.200+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) ==== 54+0+16 (secure 0 0 0) 0x7fce1c08d970 con 0x7fce340d0870 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.200+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fce34151950 0x7fce34153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.200+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce340cc3f0 con 0x7fce34151950 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.200+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fce34151950 0x7fce34153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.200+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce1006c2b0 0x7fce1006e770 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce24009a20 tx=0x7fce24009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.201+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fce34151950 0x7fce34153170 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.201+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce340cc3f0 con 0x7fce34151950 2024-09-16T01:51:26.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.216+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce340c5d40 con 0x7fce34151950 2024-09-16T01:51:26.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.218+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce340c5d40 con 0x7fce34151950 2024-09-16T01:51:26.219 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:26.220 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:26.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.218+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fce3415cfe0 0x7fce3415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.218+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fce341e4140 con 0x7fce3415cfe0 2024-09-16T01:51:26.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.218+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fce3415cfe0 0x7fce3415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.219+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fce3415cfe0 0x7fce3415ba40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.219+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce341e4140 con 0x7fce3415cfe0 2024-09-16T01:51:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.234+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce341e3e60 con 0x7fce3415cfe0 2024-09-16T01:51:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.236+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce341e3e60 con 0x7fce3415cfe0 2024-09-16T01:51:26.237 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.236+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fce340b3690 0x7fce3415c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.236+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fce340cbef0 con 0x7fce340b3690 2024-09-16T01:51:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.236+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fce340b3690 0x7fce3415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.237+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fce340b3690 0x7fce3415c490 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.237+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce340cbef0 con 0x7fce340b3690 2024-09-16T01:51:26.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.251+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce34062630 con 0x7fce340b3690 2024-09-16T01:51:26.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.253+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce34062630 con 0x7fce340b3690 2024-09-16T01:51:26.254 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.253+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fce34062fa0 0x7fce341e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.253+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fce340633c0 con 0x7fce34062fa0 2024-09-16T01:51:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.253+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fce34062fa0 0x7fce341e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.254+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fce34062fa0 0x7fce341e6f40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.254+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce340633c0 con 0x7fce34062fa0 2024-09-16T01:51:26.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.268+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce34063600 con 0x7fce34062fa0 2024-09-16T01:51:26.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.270+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce34063600 con 0x7fce34062fa0 2024-09-16T01:51:26.271 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:26.272 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:26.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.270+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fce341e7c40 0x7fce3402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.270+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fce34206770 con 0x7fce341e7c40 2024-09-16T01:51:26.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.270+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fce341e7c40 0x7fce3402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.271+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fce341e7c40 0x7fce3402c640 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.272+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce34206770 con 0x7fce341e7c40 2024-09-16T01:51:26.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.285+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce3402cb80 con 0x7fce341e7c40 2024-09-16T01:51:26.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.287+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce3402cb80 con 0x7fce341e7c40 2024-09-16T01:51:26.288 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:26.289 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:26.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.287+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fce3402d200 0x7fce3402d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.287+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fce34209900 con 0x7fce3402d200 2024-09-16T01:51:26.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.288+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fce3402d200 0x7fce3402d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.288+0000 7fce2aa00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fce3402d200 0x7fce3402d660 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.289+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce34209900 con 0x7fce3402d200 2024-09-16T01:51:26.297 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:26.297 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:26.297 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:26.297 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:26.297 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:26.297 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:26.296+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:26.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.302+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce34207070 con 0x7fce3402d200 2024-09-16T01:51:26.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.304+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce34207070 con 0x7fce3402d200 2024-09-16T01:51:26.305 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:26.306 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:26.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.304+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fce342076f0 0x7fce34207b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.304+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fce3420cc90 con 0x7fce342076f0 2024-09-16T01:51:26.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.305+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fce342076f0 0x7fce34207b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.305+0000 7fce33e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fce342076f0 0x7fce34207b10 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.306+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce3420cc90 con 0x7fce342076f0 2024-09-16T01:51:26.308 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:26.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.319+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce342080c0 con 0x7fce342076f0 2024-09-16T01:51:26.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.321+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce342080c0 con 0x7fce342076f0 2024-09-16T01:51:26.322 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:26.322 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:26.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.321+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fce342087e0 0x7fce34208c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:26.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.321+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fce34209270 con 0x7fce342087e0 2024-09-16T01:51:26.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.321+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fce342087e0 0x7fce34208c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:26.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.322+0000 7fce33400640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fce342087e0 0x7fce34208c20 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:26.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.322+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce34209270 con 0x7fce342087e0 2024-09-16T01:51:26.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.336+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fce341e7500 con 0x7fce342087e0 2024-09-16T01:51:26.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce2be00640 1 -- 172.21.15.146:0/2114595143 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fce341e7500 con 0x7fce342087e0 2024-09-16T01:51:26.339 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:26.340 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:26.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fce34151950 msgr2=0x7fce34153170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fce34151950 0x7fce34153170 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fce3415cfe0 msgr2=0x7fce3415ba40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fce3415cfe0 0x7fce3415ba40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fce340b3690 msgr2=0x7fce3415c490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fce340b3690 0x7fce3415c490 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fce34062fa0 msgr2=0x7fce341e6f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fce34062fa0 0x7fce341e6f40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fce341e7c40 msgr2=0x7fce3402c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fce341e7c40 0x7fce3402c640 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fce3402d200 msgr2=0x7fce3402d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fce3402d200 0x7fce3402d660 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fce342076f0 msgr2=0x7fce34207b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fce342076f0 0x7fce34207b10 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fce342087e0 msgr2=0x7fce34208c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce33e00640 1 -- 172.21.15.146:0/2114595143 reap_dead start 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fce342087e0 0x7fce34208c20 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce33e00640 1 -- 172.21.15.146:0/2114595143 reap_dead start 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce33e00640 1 -- 172.21.15.146:0/2114595143 reap_dead start 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce33e00640 1 -- 172.21.15.146:0/2114595143 reap_dead start 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce1006c2b0 msgr2=0x7fce1006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce1006c2b0 0x7fce1006e770 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce24009a20 tx=0x7fce24009190 comp rx=0 tx=0).stop 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 msgr2=0x7fce3414edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.338+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 0x7fce3414edd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00e980 tx=0x7fce1c00ee40 comp rx=0 tx=0).stop 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 shutdown_connections 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fce342087e0 0x7fce34208c20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce1006c2b0 0x7fce1006e770 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 --2- 172.21.15.146:0/2114595143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce340d0870 0x7fce3414edd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:26.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 >> 172.21.15.146:0/2114595143 conn(0x7fce3406d6f0 msgr2=0x7fce340cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:26.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 shutdown_connections 2024-09-16T01:51:26.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:26.339+0000 7fce38e00640 1 -- 172.21.15.146:0/2114595143 wait complete. 2024-09-16T01:51:26.398 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:26.397+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:26.453 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:26.454 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:26.462 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:26.499 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:26.498+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:26.600 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:26.599+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:26.608 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:26.609 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:26.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:26.700 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:26.700+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:26.722 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:26.722 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:26.731 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:26.801 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:26.800+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:26.834 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:26.835 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:26.843 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:26.902 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:26.901+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:26.950 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:26.950 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:26.960 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:27.004 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:27.003+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:27.063 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:27.063 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:27.071 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:27.104 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:27.103+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:27.204 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:27.203+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:27.215 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:27.216 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:27.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:27.305 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:27.304+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:27.388 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:27.388 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:27.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:27.442 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:27.441+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:27.508 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:27.507+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:27.600 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:27.600 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:27.609 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:27.608+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:27.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:27.709 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:27.708+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:27.769 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:27.769 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:27.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:27.828 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:27.827+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:27.911 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:27.911+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:27.949 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:27.949 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:27.964 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:27.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:51:28.012 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:28.011+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:28.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:28.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:28.064 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.9341532869661907 2024-09-16T01:51:28.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.9341532869661907 2024-09-16T01:51:28.113 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:28.113+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 re-readers = 3272220.25 kB/sec 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 re-readers = 3264079.73 kB/sec 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 3272220.25 kB/sec 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 3272220.25 kB/sec 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 3272220.25 kB/sec 2024-09-16T01:51:28.136 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 1048576.00 kB 2024-09-16T01:51:28.176 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:28.177 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:28.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.175+0000 7f1bec000640 1 Processor -- start 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.175+0000 7f1bec000640 1 -- start start 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.176+0000 7f1bec000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1be40c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.176+0000 7f1bec000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be40c8ef0 con 0x7f1be4168200 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.176+0000 7f1bec000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be416ad60 con 0x7f1be4164d70 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.176+0000 7f1bec000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1be416b040 con 0x7f1be40c8ad0 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.177+0000 7f1be8c00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1be4168200 0x7f1be40c7430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44812/0 (socket says 172.21.15.146:44812) 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.177+0000 7f1be8c00640 1 -- 172.21.15.146:0/2796260975 learned_addr learned my addr 172.21.15.146:0/2796260975 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:28.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.177+0000 7f1be9600640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1be4164d70 0x7f1be4167af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:48736/0 (socket says 172.21.15.146:48736) 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.177+0000 7f1bea000640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1be40c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.177+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796363439 0 0) 0x7f1be416ad60 con 0x7f1be4164d70 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bc8003500 con 0x7f1be4164d70 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1855474116 0 0) 0x7f1be40c8ef0 con 0x7f1be4168200 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be416ad60 con 0x7f1be4168200 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3889068489 0 0) 0x7f1be416ad60 con 0x7f1be4168200 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 msgr2=0x7f1be40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1be40c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 >> v1:172.21.15.161:6789/0 conn(0x7f1be4164d70 legacy=0x7f1be4167af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.225 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be416b320 con 0x7f1be4168200 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f1bd0001670 con 0x7f1be4168200 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 >> v1:172.21.15.146:6789/0 conn(0x7f1be4168200 legacy=0x7f1be40c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1bc8003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.222+0000 7f1be3e00640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bc8006560 0x7f1bc80069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be3e00640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bc8006f20 0x7f1bc800fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1be416b040 con 0x7f1bc8006560 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc80100e0 con 0x7f1bc8006f20 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1bc8010260 con 0x7f1be40c8ad0 2024-09-16T01:51:28.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be9600640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1bc8003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be8c00640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bc8006560 0x7f1bc80069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1bea000640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bc8006f20 0x7f1bc800fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be9600640 1 -- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bc8006f20 msgr2=0x7f1bc800fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:28.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be9600640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bc8006f20 0x7f1bc800fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be9600640 1 -- 172.21.15.146:0/2796260975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bc8006560 msgr2=0x7f1bc80069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be9600640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bc8006560 0x7f1bc80069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.223+0000 7f1be9600640 1 -- 172.21.15.146:0/2796260975 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be416ad60 con 0x7f1be40c8ad0 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.224+0000 7f1be9600640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1bc8003d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4002000 tx=0x7f1bc80037f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84effce13bc88ff1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.224+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4001c50 con 0x7f1be40c8ad0 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd4004d60 con 0x7f1be40c8ad0 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1be3e00640 1 -- 172.21.15.146:0/2796260975 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40336a0 con 0x7f1be40c8ad0 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1bec000640 1 -- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 msgr2=0x7f1bc8003d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1bec000640 1 --2- 172.21.15.146:0/2796260975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be40c8ad0 0x7f1bc8003d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4002000 tx=0x7f1bc80037f0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1bea000640 1 -- 172.21.15.146:0/2796260975 reap_dead start 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1bec000640 1 -- 172.21.15.146:0/2796260975 shutdown_connections 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1bec000640 1 -- 172.21.15.146:0/2796260975 >> 172.21.15.146:0/2796260975 conn(0x7f1be406f3c0 msgr2=0x7f1be406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.225+0000 7f1bec000640 1 -- 172.21.15.146:0/2796260975 shutdown_connections 2024-09-16T01:51:28.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.226+0000 7f1bec000640 1 -- 172.21.15.146:0/2796260975 wait complete. 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.227+0000 7f1bec000640 1 Processor -- start 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.227+0000 7f1bec000640 1 -- start start 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bec000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1be40c8ad0 0x7f1be415d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bec000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 0x7f1be415c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bec000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 0x7f1be415e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bec000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1be40c3970 con 0x7f1be415eb20 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bec000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1be416b8e0 con 0x7f1be40c8ad0 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bec000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1be40c6bb0 con 0x7f1be415dc50 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1be8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 0x7f1be415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1bea000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 0x7f1be415e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1be9600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1be40c8ad0 0x7f1be415d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.228+0000 7f1be8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 0x7f1be415c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39988/0 (socket says 172.21.15.146:39988) 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 0x7f1be415e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52048/0 (socket says 172.21.15.146:52048) 2024-09-16T01:51:28.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1be8c00640 1 -- 172.21.15.146:0/3483632778 learned_addr learned my addr 172.21.15.146:0/3483632778 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 -- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1be40c8ad0 msgr2=0x7f1be415d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1be40c8ad0 0x7f1be415d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 -- 172.21.15.146:0/3483632778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 msgr2=0x7f1be415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 0x7f1be415c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 -- 172.21.15.146:0/3483632778 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be416a5f0 con 0x7f1be415dc50 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1be8c00640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 0x7f1be415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.229+0000 7f1bea000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 0x7f1be415e0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1be406d6b0 tx=0x7f1bdc011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.230+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc004030 con 0x7f1be415dc50 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.230+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bdc01b070 con 0x7f1be415dc50 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.230+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc004450 con 0x7f1be415dc50 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.230+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be416a8d0 con 0x7f1be415dc50 2024-09-16T01:51:28.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.230+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be4166ee0 con 0x7f1be415dc50 2024-09-16T01:51:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.231+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be40718c0 con 0x7f1be415dc50 2024-09-16T01:51:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.232+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1bdc00f7a0 con 0x7f1be415dc50 2024-09-16T01:51:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.232+0000 7fcec4600640 1 Processor -- start 2024-09-16T01:51:28.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.233+0000 7f1be1600640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bc406c2e0 0x7f1bc406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.233+0000 7fcec4600640 1 -- start start 2024-09-16T01:51:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.233+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 5741+0+0 (secure 0 0 0) 0x7f1bdc08b6b0 con 0x7f1be415dc50 2024-09-16T01:51:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.233+0000 7f1be9600640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bc406c2e0 0x7f1bc406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7f1be9600640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bc406c2e0 0x7f1bc406e7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4002000 tx=0x7f1bd4009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec4600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcebc0cc0a0 0x7fcebc0cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec4600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcebc15ed70 0x7fcebc0d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec4600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcebc15ba20 con 0x7fcebc0d0630 2024-09-16T01:51:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec4600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcebc15e1d0 con 0x7fcebc15ed70 2024-09-16T01:51:28.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec4600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcebc15aea0 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec1c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcebc0cc0a0 0x7fcebc0cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec1c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcebc0cc0a0 0x7fcebc0cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43582/0 (socket says 172.21.15.146:43582) 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec1c00640 1 -- 172.21.15.146:0/2962148554 learned_addr learned my addr 172.21.15.146:0/2962148554 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec1200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcebc0d0630 0x7fcebc15e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:48740/0 (socket says 172.21.15.146:48740) 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.234+0000 7fcec2600640 1 --2- 172.21.15.146:0/2962148554 >> v2:172.21.15.146:3300/0 conn(0x7fcebc15ed70 0x7fcebc0d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.235+0000 7fcec1c00640 1 -- 172.21.15.146:0/2962148554 >> v2:172.21.15.146:3300/0 conn(0x7fcebc15ed70 msgr2=0x7fcebc0d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.235+0000 7fcec1c00640 1 --2- 172.21.15.146:0/2962148554 >> v2:172.21.15.146:3300/0 conn(0x7fcebc15ed70 0x7fcebc0d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.235+0000 7fcec1c00640 1 -- 172.21.15.146:0/2962148554 >> v1:172.21.15.161:6789/0 conn(0x7fcebc0d0630 legacy=0x7fcebc15e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.235+0000 7fcec1c00640 1 -- 172.21.15.146:0/2962148554 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc15b180 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.235+0000 7fcebbe00640 1 -- 172.21.15.146:0/2962148554 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101979322 0 0) 0x7fcebc15ba20 con 0x7fcebc0d0630 2024-09-16T01:51:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.236+0000 7fcec1c00640 1 --2- 172.21.15.146:0/2962148554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0cc480 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0caca0 tx=0x7fceb002a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d6316f37363ff27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.236+0000 7fcebbe00640 1 -- 172.21.15.146:0/2962148554 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0037070 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.236+0000 7fcebbe00640 1 -- 172.21.15.146:0/2962148554 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb002ae50 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.236+0000 7fcebbe00640 1 -- 172.21.15.146:0/2962148554 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0032860 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.236+0000 7fcec4600640 1 -- 172.21.15.146:0/2962148554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 msgr2=0x7fcebc0cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.236+0000 7fcec4600640 1 --2- 172.21.15.146:0/2962148554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0cc480 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0caca0 tx=0x7fceb002a300 comp rx=0 tx=0).stop 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7fcec4600640 1 -- 172.21.15.146:0/2962148554 shutdown_connections 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7fcec4600640 1 --2- 172.21.15.146:0/2962148554 >> v2:172.21.15.146:3300/0 conn(0x7fcebc15ed70 0x7fcebc0d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7fcec4600640 1 --2- 172.21.15.146:0/2962148554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0cc480 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7fcec4600640 1 -- 172.21.15.146:0/2962148554 >> 172.21.15.146:0/2962148554 conn(0x7fcebc06d560 msgr2=0x7fcebc073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7fcec4600640 1 -- 172.21.15.146:0/2962148554 shutdown_connections 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7fcec4600640 1 -- 172.21.15.146:0/2962148554 wait complete. 2024-09-16T01:51:28.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.237+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1bdc010040 con 0x7f1be415dc50 2024-09-16T01:51:28.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.238+0000 7fcec4600640 1 Processor -- start 2024-09-16T01:51:28.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.239+0000 7fcec4600640 1 -- start start 2024-09-16T01:51:28.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.239+0000 7fcec4600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.239+0000 7fcec4600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc0d0630 0x7fcebc15d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec4600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcebc15ed70 0x7fcebc151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec4600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc0cbde0 con 0x7fcebc0d0630 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec4600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc15b740 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec4600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc15aaf0 con 0x7fcebc15ed70 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc0d0630 0x7fcebc15d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43586/0 (socket says 172.21.15.146:43586) 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 -- 172.21.15.146:0/430875313 learned_addr learned my addr 172.21.15.146:0/430875313 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec2600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcebc15ed70 0x7fcebc151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 -- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcebc15ed70 msgr2=0x7fcebc151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcebc15ed70 0x7fcebc151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 -- 172.21.15.146:0/430875313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc0d0630 msgr2=0x7fcebc15d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc0d0630 0x7fcebc15d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.240+0000 7fcec1c00640 1 -- 172.21.15.146:0/430875313 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc14f800 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.241+0000 7fcec1c00640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0c8290 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fceb0000c00 tx=0x7fceb00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.241+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0037040 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.241+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc1509d0 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.241+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb0004030 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.241+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0004400 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.241+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc150f90 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.243+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcebc1513c0 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.243+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fceb003a070 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.244+0000 7fceb9600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce9c06c2e0 0x7fce9c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.244+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5741+0+0 (secure 0 0 0) 0x7fceb00abac0 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.244+0000 7fcec1200640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce9c06c2e0 0x7fce9c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.245+0000 7fcec1200640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce9c06c2e0 0x7fce9c06e7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0ceca0 tx=0x7fcea8007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.249+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fceb0041020 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.315 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:28.314+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:28.357 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:28.389 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:28.389 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:28.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:28.415 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:28.414+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:28.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.450+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1be408cb90 con 0x7f1be415dc50 2024-09-16T01:51:28.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.466+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 0.9341532869661907} v 0) -- 0x7fcebc155140 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.517 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:28.516+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:28.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.518+0000 7f6a5e400640 1 Processor -- start 2024-09-16T01:51:28.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.518+0000 7f6a5e400640 1 -- start start 2024-09-16T01:51:28.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5e400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 0x7f6a58154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5e400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a580c6d00 0x7f6a58154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5e400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a5815e220 con 0x7f6a580d0490 2024-09-16T01:51:28.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5e400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6a580c6160 con 0x7f6a580c6d00 2024-09-16T01:51:28.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5e400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a5815e500 con 0x7f6a580d0870 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 0x7f6a58154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5da00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a580c6d00 0x7f6a58154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5d000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6a580d0490 0x7f6a580c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:48742/0 (socket says 172.21.15.146:48742) 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 0x7f6a58154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:52058/0 (socket says 172.21.15.146:52058) 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.520+0000 7f6a5d000640 1 -- 172.21.15.146:0/3494897084 learned_addr learned my addr 172.21.15.146:0/3494897084 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.521+0000 7f6a57e00640 1 -- 172.21.15.146:0/3494897084 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1898679495 0 0) 0x7f6a5815e220 con 0x7f6a580d0490 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.521+0000 7f6a5da00640 1 -- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 msgr2=0x7f6a58154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.521+0000 7f6a5da00640 1 --2- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 0x7f6a58154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.521+0000 7f6a5da00640 1 -- 172.21.15.146:0/3494897084 >> v1:172.21.15.161:6789/0 conn(0x7f6a580d0490 legacy=0x7f6a580c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.521+0000 7f6a5da00640 1 -- 172.21.15.146:0/3494897084 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5815e7e0 con 0x7f6a580c6d00 2024-09-16T01:51:28.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.521+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 0x7f6a58154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:28.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.522+0000 7f6a5da00640 1 --2- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580c6d00 0x7f6a58154bd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009d90 tx=0x7f6a5002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcb2380e7e37128f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.522+0000 7f6a57e00640 1 -- 172.21.15.146:0/3494897084 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50007740 con 0x7f6a580c6d00 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.522+0000 7f6a57e00640 1 -- 172.21.15.146:0/3494897084 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a500078e0 con 0x7f6a580c6d00 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.522+0000 7f6a57e00640 1 -- 172.21.15.146:0/3494897084 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50007cb0 con 0x7f6a580c6d00 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.522+0000 7f6a5e400640 1 -- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580c6d00 msgr2=0x7f6a58154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.522+0000 7f6a5e400640 1 --2- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580c6d00 0x7f6a58154bd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6a50009d90 tx=0x7f6a5002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.523+0000 7f6a5e400640 1 -- 172.21.15.146:0/3494897084 shutdown_connections 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.523+0000 7f6a5e400640 1 --2- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580c6d00 0x7f6a58154bd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.523+0000 7f6a5e400640 1 --2- 172.21.15.146:0/3494897084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580d0870 0x7f6a58154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.523+0000 7f6a5e400640 1 -- 172.21.15.146:0/3494897084 >> 172.21.15.146:0/3494897084 conn(0x7f6a5806d6f0 msgr2=0x7f6a5806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.523+0000 7f6a5e400640 1 -- 172.21.15.146:0/3494897084 shutdown_connections 2024-09-16T01:51:28.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.523+0000 7f6a5e400640 1 -- 172.21.15.146:0/3494897084 wait complete. 2024-09-16T01:51:28.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.524+0000 7f6a5e400640 1 Processor -- start 2024-09-16T01:51:28.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.524+0000 7f6a5e400640 1 -- start start 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.525+0000 7f6a5e400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580c6d00 0x7f6a58155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.525+0000 7f6a5e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a580d0490 0x7f6a581557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.525+0000 7f6a5e400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580d0870 0x7f6a581512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.525+0000 7f6a5e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5815dec0 con 0x7f6a580d0490 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.525+0000 7f6a5e400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5815eda0 con 0x7f6a580d0870 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.525+0000 7f6a5e400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a580c51f0 con 0x7f6a580c6d00 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a580d0490 0x7f6a581557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580c6d00 0x7f6a58155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5da00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580d0870 0x7f6a581512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a580d0490 0x7f6a581557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40012/0 (socket says 172.21.15.146:40012) 2024-09-16T01:51:28.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a4fe00640 1 -- 172.21.15.146:0/224663195 learned_addr learned my addr 172.21.15.146:0/224663195 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580d0870 msgr2=0x7f6a581512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a580d0870 0x7f6a581512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a580d0490 msgr2=0x7f6a581557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a580d0490 0x7f6a581557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a581f0750 con 0x7f6a580c6d00 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.526+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580c6d00 0x7f6a58155280 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800c480 tx=0x7f6a4800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.527+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48018070 con 0x7f6a580c6d00 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.527+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a480074e0 con 0x7f6a580c6d00 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.527+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48004a20 con 0x7f6a580c6d00 2024-09-16T01:51:28.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.527+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a580c9b30 con 0x7f6a580c6d00 2024-09-16T01:51:28.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.527+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a58156de0 con 0x7f6a580c6d00 2024-09-16T01:51:28.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.528+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6a580b6180 con 0x7f6a580c6d00 2024-09-16T01:51:28.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.529+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a48004bc0 con 0x7f6a580c6d00 2024-09-16T01:51:28.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.530+0000 7f6a55600640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a3406c2b0 0x7f6a3406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.530+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 5741+0+0 (secure 0 0 0) 0x7f6a4808a720 con 0x7f6a580c6d00 2024-09-16T01:51:28.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.530+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) ==== 54+0+16 (secure 0 0 0) 0x7f6a4808aad0 con 0x7f6a580c6d00 2024-09-16T01:51:28.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.530+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a3406c2b0 0x7f6a3406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.530+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a581e40d0 0x7f6a58156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.530+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a580cc3f0 con 0x7f6a581e40d0 2024-09-16T01:51:28.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.531+0000 7f6a5da00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a581e40d0 0x7f6a58156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.531+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a3406c2b0 0x7f6a3406e770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6a400095e0 tx=0x7f6a40009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.531+0000 7f6a5da00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a581e40d0 0x7f6a58156fe0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.532+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a580cc3f0 con 0x7f6a581e40d0 2024-09-16T01:51:28.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.547+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a580cca90 con 0x7f6a581e40d0 2024-09-16T01:51:28.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.548+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a580cca90 con 0x7f6a581e40d0 2024-09-16T01:51:28.550 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:28.550 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:28.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.549+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a5814fde0 0x7f6a581501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.549+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6a580c8d40 con 0x7f6a5814fde0 2024-09-16T01:51:28.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.549+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a5814fde0 0x7f6a581501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.550+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a5814fde0 0x7f6a581501c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.550+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a580c8d40 con 0x7f6a5814fde0 2024-09-16T01:51:28.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.566+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a581e7ae0 con 0x7f6a5814fde0 2024-09-16T01:51:28.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.567+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a581e7ae0 con 0x7f6a5814fde0 2024-09-16T01:51:28.568 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:28.569 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:28.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.568+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a581f4010 0x7f6a581f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.568+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6a580b3150 con 0x7f6a581f4010 2024-09-16T01:51:28.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.568+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a581f4010 0x7f6a581f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.568+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a581f4010 0x7f6a581f4430 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.569+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a580b3150 con 0x7f6a581f4010 2024-09-16T01:51:28.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.583+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a580b35d0 con 0x7f6a581f4010 2024-09-16T01:51:28.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.584+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a580b35d0 con 0x7f6a581f4010 2024-09-16T01:51:28.586 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:28.586 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:28.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.585+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a58062630 0x7f6a58062a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.585+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6a580b40b0 con 0x7f6a58062630 2024-09-16T01:51:28.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.585+0000 7f6a5da00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a58062630 0x7f6a58062a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.586+0000 7f6a5da00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a58062630 0x7f6a58062a90 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.586+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a580b40b0 con 0x7f6a58062630 2024-09-16T01:51:28.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.600+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a580632a0 con 0x7f6a58062630 2024-09-16T01:51:28.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.601+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a580632a0 con 0x7f6a58062630 2024-09-16T01:51:28.602 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:28.603 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:28.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.602+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a5802cb50 0x7f6a5802cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.602+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6a58063220 con 0x7f6a5802cb50 2024-09-16T01:51:28.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.602+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a5802cb50 0x7f6a5802cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.603+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a5802cb50 0x7f6a5802cf70 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.603+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a58063220 con 0x7f6a5802cb50 2024-09-16T01:51:28.608 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:28.608 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:28.608 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:28.618 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:28.617+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:28.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.617+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a58062fd0 con 0x7f6a5802cb50 2024-09-16T01:51:28.621 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:28.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.620+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a58062fd0 con 0x7f6a5802cb50 2024-09-16T01:51:28.622 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:28.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.620+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a58204d10 0x7f6a58207100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.620+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6a58063390 con 0x7f6a58204d10 2024-09-16T01:51:28.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.620+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a58204d10 0x7f6a58207100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.621+0000 7f6a4fe00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a58204d10 0x7f6a58207100 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.622+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a58063390 con 0x7f6a58204d10 2024-09-16T01:51:28.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.636+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a58063390 con 0x7f6a58204d10 2024-09-16T01:51:28.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.638+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a58063390 con 0x7f6a58204d10 2024-09-16T01:51:28.639 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:28.639 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:28.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.638+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a58207b50 0x7f6a58209f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.638+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6a58063500 con 0x7f6a58207b50 2024-09-16T01:51:28.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.638+0000 7f6a5da00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a58207b50 0x7f6a58209f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.639+0000 7f6a5da00640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a58207b50 0x7f6a58209f40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.640+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a58063500 con 0x7f6a58207b50 2024-09-16T01:51:28.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.657+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a58063500 con 0x7f6a58207b50 2024-09-16T01:51:28.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.659+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a58063500 con 0x7f6a58207b50 2024-09-16T01:51:28.660 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.659+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a5820a990 0x7f6a5820cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.659+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6a5802dac0 con 0x7f6a5820a990 2024-09-16T01:51:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.660+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a5820a990 0x7f6a5820cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.660+0000 7f6a5d000640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a5820a990 0x7f6a5820cd80 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:28.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.661+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a5802dac0 con 0x7f6a5820a990 2024-09-16T01:51:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.675+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6a5820d2c0 con 0x7f6a5820a990 2024-09-16T01:51:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a55600640 1 -- 172.21.15.146:0/224663195 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6a5820d2c0 con 0x7f6a5820a990 2024-09-16T01:51:28.679 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:28.680 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:28.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a581e40d0 msgr2=0x7f6a58156fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f6a581e40d0 0x7f6a58156fe0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a5814fde0 msgr2=0x7f6a581501c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f6a5814fde0 0x7f6a581501c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a581f4010 msgr2=0x7f6a581f4430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6a581f4010 0x7f6a581f4430 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a58062630 msgr2=0x7f6a58062a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6a58062630 0x7f6a58062a90 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a5802cb50 msgr2=0x7f6a5802cf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6a5802cb50 0x7f6a5802cf70 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a58204d10 msgr2=0x7f6a58207100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f6a58204d10 0x7f6a58207100 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a58207b50 msgr2=0x7f6a58209f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6a58207b50 0x7f6a58209f40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a5820a990 msgr2=0x7f6a5820cd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6a5820a990 0x7f6a5820cd80 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5da00640 1 -- 172.21.15.146:0/224663195 reap_dead start 2024-09-16T01:51:28.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5da00640 1 -- 172.21.15.146:0/224663195 reap_dead start 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5da00640 1 -- 172.21.15.146:0/224663195 reap_dead start 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5da00640 1 -- 172.21.15.146:0/224663195 reap_dead start 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5da00640 1 -- 172.21.15.146:0/224663195 reap_dead start 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a3406c2b0 msgr2=0x7f6a3406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a3406c2b0 0x7f6a3406e770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6a400095e0 tx=0x7f6a40009190 comp rx=0 tx=0).stop 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580c6d00 msgr2=0x7f6a58155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.678+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580c6d00 0x7f6a58155280 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800c480 tx=0x7f6a4800c940 comp rx=0 tx=0).stop 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.679+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 shutdown_connections 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.679+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a3406c2b0 0x7f6a3406e770 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.679+0000 7f6a5e400640 1 --2- 172.21.15.146:0/224663195 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a580c6d00 0x7f6a58155280 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.679+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 >> 172.21.15.146:0/224663195 conn(0x7f6a5806d6f0 msgr2=0x7f6a58154460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.679+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 shutdown_connections 2024-09-16T01:51:28.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.679+0000 7f6a5e400640 1 -- 172.21.15.146:0/224663195 wait complete. 2024-09-16T01:51:28.737 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:28.736+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:28.785 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:28.785 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:28.785 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:28.785 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:28.785 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:28.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:28.819 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:28.818+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:28.920 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:28.919+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:28.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.959+0000 7fceb9600640 1 -- 172.21.15.146:0/430875313 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 0.9341532869661907}]=0 reweighted osd.1 to 0.934153 (ef24) v72) ==== 134+0+0 (secure 0 0 0) 0x7fceb0079b30 con 0x7fcebc0cc0a0 2024-09-16T01:51:28.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.959+0000 7f1be1600640 1 -- 172.21.15.146:0/3483632778 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v72) ==== 87+0+0 (secure 0 0 0) 0x7f1bdc059710 con 0x7f1be415dc50 2024-09-16T01:51:28.961 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.1 to 0.934153 (ef24) 2024-09-16T01:51:28.961 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:51:28.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bc406c2e0 msgr2=0x7f1bc406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7f1bec000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bc406c2e0 0x7f1bc406e7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4002000 tx=0x7f1bd4009030 comp rx=0 tx=0).stop 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 msgr2=0x7f1be415e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7f1bec000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 0x7f1be415e0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1be406d6b0 tx=0x7f1bdc011dc0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce9c06c2e0 msgr2=0x7fce9c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7fcec4600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce9c06c2e0 0x7fce9c06e7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcebc0ceca0 tx=0x7fcea8007d50 comp rx=0 tx=0).stop 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 msgr2=0x7fcebc0c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.962+0000 7fcec4600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0c8290 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fceb0000c00 tx=0x7fceb00046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 shutdown_connections 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fce9c06c2e0 0x7fce9c06e7a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 shutdown_connections 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bc406c2e0 0x7f1bc406e7a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1be415dc50 0x7f1be415e0d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be415eb20 0x7f1be415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcebc15ed70 0x7fcebc151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc0d0630 0x7fcebc15d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 --2- 172.21.15.146:0/430875313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcebc0cc0a0 0x7fcebc0c8290 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fceb0000c00 tx=0x7fceb00046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 >> 172.21.15.146:0/430875313 conn(0x7fcebc06d560 msgr2=0x7fcebc0c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 --2- 172.21.15.146:0/3483632778 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1be40c8ad0 0x7f1be415d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 >> 172.21.15.146:0/3483632778 conn(0x7f1be406f3c0 msgr2=0x7f1be4161ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 shutdown_connections 2024-09-16T01:51:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7f1bec000640 1 -- 172.21.15.146:0/3483632778 wait complete. 2024-09-16T01:51:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 shutdown_connections 2024-09-16T01:51:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:28.963+0000 7fcec4600640 1 -- 172.21.15.146:0/430875313 wait complete. 2024-09-16T01:51:29.009 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.438245555999998, 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:29.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.284699873999998, 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:29.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:29.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:29.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.063840194000001, 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:29.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.985709604999997, 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:29.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:29.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.413496719000001, 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:29.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:29.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:29.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.189340184999999, 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:29.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:29.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.892063161999999, 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.046 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:29.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:29.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.419255309, 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:29.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:29.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.231428723000001, 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.058 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:29.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:29.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.722612317999999, 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.064 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:29.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.343766419, 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.070 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:51:29.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.075 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:607 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.715379628000001, 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057026931000000003, 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.076 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 607 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290713+0000", 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6900000000000001e-06 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330961+0000", 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040248199999999998 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330992+0000", 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1251000000000003e-05 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331005+0000", 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3006999999999999e-05 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332660+0000", 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016549589999999999 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332819+0000", 2024-09-16T01:51:29.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159137 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332968+0000", 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148942 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.342537+0000", 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095683910000000007 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347616+0000", 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005079793 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347682+0000", 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5622000000000003e-05 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347737+0000", 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4938999999999997e-05 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.081 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:634 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.491191214000001, 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068283101999999998, 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.082 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 634 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514901+0000", 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5299999999999999e-06 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549207+0000", 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034306171000000003 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549224+0000", 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6976000000000001e-05 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549235+0000", 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0682999999999999e-05 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549568+0000", 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033294599999999998 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549714+0000", 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014617499999999999 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549858+0000", 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144356 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.566205+0000", 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01634683 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583078+0000", 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016872402000000002 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583135+0000", 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7286e-05 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583182+0000", 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6746999999999998e-05 2024-09-16T01:51:29.087 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:688 3.26 3:642b2ad4:::100000001f7.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.974223742, 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062867175999999997, 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.088 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 688 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031869+0000", 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5509999999999998e-06 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084368+0000", 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052499410000000003 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084392+0000", 2024-09-16T01:51:29.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084405+0000", 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2561000000000001e-05 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084491+0000", 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5945000000000005e-05 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084662+0000", 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017081299999999999 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084812+0000", 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993999999999999 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.091507+0000", 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066950079999999997 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094630+0000", 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031236570000000002 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094700+0000", 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9575999999999999e-05 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094733+0000", 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3260999999999999e-05 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.093 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:696 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.878965878000001, 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071884475000000003, 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.094 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 696 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:29.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127126+0000", 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9369999999999998e-06 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159227+0000", 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032101044000000002 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159250+0000", 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3332999999999998e-05 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159260+0000", 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159343+0000", 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3373000000000003e-05 2024-09-16T01:51:29.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159484+0000", 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014100399999999999 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159646+0000", 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162044 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.186104+0000", 2024-09-16T01:51:29.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026457794999999999 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198896+0000", 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012791749 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198959+0000", 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3387999999999997e-05 2024-09-16T01:51:29.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.199008+0000", 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9118e-05 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:776 3.11 e68/20 3:887bb7a4:::100000001f7.00000005:head v 68'37, mlcod=65'35)", 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7980467410000003, 2024-09-16T01:51:29.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05614222, 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208044+0000", 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219371+0000", 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011326688999999999 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219442+0000", 2024-09-16T01:51:29.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0727000000000002e-05 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219483+0000", 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1165000000000003e-05 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219725+0000", 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024194200000000001 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219752+0000", 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7270000000000001e-05 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264133+0000", 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044381029000000002 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264185+0000", 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1879000000000003e-05 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.104 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:786 3.3a 3:5d44b367:::100000001f7.0000000f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7624576479999998, 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055535233000000003, 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 786 2024-09-16T01:51:29.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243634+0000", 2024-09-16T01:51:29.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254427+0000", 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010792636 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254446+0000", 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9219999999999999e-05 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254458+0000", 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2387e-05 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254837+0000", 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00037878500000000003 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254934+0000", 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6915000000000003e-05 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.255090+0000", 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015594200000000001 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.273598+0000", 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018507932000000001 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299039+0000", 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025441583 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299115+0000", 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5319000000000002e-05 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299167+0000", 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2473999999999999e-05 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.110 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:785 3.11 e69/20 3:8b60708a:::100000001f7.0000000e:head v 69'38, mlcod=65'35)", 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7512254780000003, 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082167670999999998, 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254865+0000", 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9799999999999999e-07 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286626+0000", 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031761664000000002 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286647+0000", 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268e-05 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286659+0000", 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1276e-05 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286743+0000", 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4488999999999997e-05 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286772+0000", 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9082999999999999e-05 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.336965+0000", 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050192716999999998 2024-09-16T01:51:29.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337032+0000", 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6876000000000001e-05 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:790 3.16 3:69eaaa52:::100000001f7.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7034532389999999, 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061715643000000001, 2024-09-16T01:51:29.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 790 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302638+0000", 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9410000000000001e-06 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342651+0000", 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040012486 2024-09-16T01:51:29.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342687+0000", 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5738999999999998e-05 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342701+0000", 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4255e-05 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343871+0000", 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001169977 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343975+0000", 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010435 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.344127+0000", 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015170400000000001 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.361011+0000", 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016884528999999999 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364224+0000", 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003212183 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364295+0000", 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0957000000000005e-05 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364352+0000", 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7522000000000003e-05 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:863 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.9543204559999996, 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055946002000000002, 2024-09-16T01:51:29.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 863 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051772+0000", 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2620000000000001e-06 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087118+0000", 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035346883000000003 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087186+0000", 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7743000000000005e-05 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087196+0000", 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087342+0000", 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146625 2024-09-16T01:51:29.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087445+0000", 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010255 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087572+0000", 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012764800000000001 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.101292+0000", 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013719543000000001 2024-09-16T01:51:29.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107514+0000", 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062224120000000001 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107641+0000", 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012699 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107715+0000", 2024-09-16T01:51:29.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3906999999999996e-05 2024-09-16T01:51:29.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.127 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:29.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:29.129 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:29.127+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:29.229 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:29.228+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:29.328 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:29.328 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:29.328 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:29.328 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:29.330 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:29.329+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:29.340 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:29.431 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:29.430+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:29.532 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:29.531+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:29.539 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:29.539 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:29.539 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:29.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:29.633 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:29.633+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:29.734 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:29.733+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:29.755 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.497487288000002, 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.756 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:29.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:29.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:29.762 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.299114418000002, 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:29.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:29.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:29.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:29.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.190311786999999, 2024-09-16T01:51:29.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:29.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.870743154000003, 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:29.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.372080414000003, 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.782 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:29.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.788 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.958507730000001, 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.789 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:29.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.972134638, 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:29.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:29.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:29.803 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.904029996999999, 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.804 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:29.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:29.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:29.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.761570332000002, 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.812 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:29.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:29.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.820 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.414104109, 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:29.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:29.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:29.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:29.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:29.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:544 3.14 3:28ae9428:::100000001f7.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.108730207000001, 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064751230000000007, 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 544 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644039+0000", 2024-09-16T01:51:29.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6619999999999999e-06 2024-09-16T01:51:29.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665383+0000", 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021343964 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665408+0000", 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.561e-05 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665422+0000", 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3947999999999999e-05 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665607+0000", 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018433499999999999 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665831+0000", 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022424399999999999 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665986+0000", 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155654 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.682208+0000", 2024-09-16T01:51:29.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016221920000000001 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708669+0000", 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026460513000000001 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708734+0000", 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4888000000000006e-05 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708788+0000", 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4491999999999999e-05 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:549 3.11 3:8ae93890:::100000001f7.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.066530186, 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081050590000000006, 2024-09-16T01:51:29.840 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 549 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686239+0000", 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754451+0000", 2024-09-16T01:51:29.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068212138000000005 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754474+0000", 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2733000000000001e-05 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754483+0000", 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1349999999999998e-06 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754562+0000", 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8319999999999996e-05 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754725+0000", 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016298699999999999 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754875+0000", 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149942 2024-09-16T01:51:29.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.762160+0000", 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072859630000000003 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767169+0000", 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050088609999999999 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767231+0000", 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1158e-05 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767288+0000", 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7108000000000003e-05 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:606 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:29.846 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.478116139000001, 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058221427999999999, 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274653+0000", 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322327+0000", 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047674080000000001 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322357+0000", 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9813000000000001e-05 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322369+0000", 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2074e-05 2024-09-16T01:51:29.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322494+0000", 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012469 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322653+0000", 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158987 2024-09-16T01:51:29.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322803+0000", 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150433 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331674+0000", 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088701219999999994 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332772+0000", 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001098552 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332829+0000", 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6941000000000003e-05 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332873+0000", 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3516000000000001e-05 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:626 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.288124264, 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057812853999999997, 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.853 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 626 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464645+0000", 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9759999999999998e-06 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505192+0000", 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040546806999999997 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505211+0000", 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9386e-05 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505220+0000", 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4810000000000006e-06 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505316+0000", 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6748000000000003e-05 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505479+0000", 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016267300000000001 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505668+0000", 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018906700000000001 2024-09-16T01:51:29.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.520824+0000", 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015155686999999999 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522336+0000", 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015126359999999999 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522404+0000", 2024-09-16T01:51:29.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7174000000000004e-05 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522456+0000", 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2219000000000003e-05 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:629 3.13 3:ca345efb:::100000001f7.00000072:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.859 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.243975560999999, 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073672847999999999, 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 629 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508793+0000", 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5650000000000001e-06 2024-09-16T01:51:29.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536849+0000", 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028055429999999999 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536888+0000", 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9128999999999998e-05 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536904+0000", 2024-09-16T01:51:29.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6223e-05 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537082+0000", 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017825600000000001 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537242+0000", 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159569 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537394+0000", 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015228 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.548370+0000", 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010976324000000001 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582365+0000", 2024-09-16T01:51:29.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994745999999999 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582423+0000", 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7357e-05 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582465+0000", 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1968999999999997e-05 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:700 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.619125129, 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074697384000000006, 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 700 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133644+0000", 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159468+0000", 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025824323 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159551+0000", 2024-09-16T01:51:29.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2286999999999993e-05 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159561+0000", 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0307e-05 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159851+0000", 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029012100000000001 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160192+0000", 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034104600000000001 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160444+0000", 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025126400000000001 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.197238+0000", 2024-09-16T01:51:29.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036794265999999999 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208261+0000", 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0110233 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208304+0000", 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2833999999999997e-05 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208339+0000", 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5556000000000003e-05 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:776 3.11 3:887bb7a4:::100000001f7.00000005:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5560835750000006, 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090355046999999994, 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 776 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196686+0000", 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5289999999999998e-06 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206873+0000", 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010186855999999999 2024-09-16T01:51:29.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206895+0000", 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1549e-05 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206907+0000", 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2884000000000001e-05 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207056+0000", 2024-09-16T01:51:29.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014810499999999999 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207156+0000", 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010007 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207302+0000", 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014591499999999999 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263823+0000", 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056521543 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286896+0000", 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023073327000000001 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286980+0000", 2024-09-16T01:51:29.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3869999999999995e-05 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.287039+0000", 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8399000000000001e-05 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5298127449999992, 2024-09-16T01:51:29.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:29.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:51:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:788 3.2d 3:b61340bb:::100000001f7.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.4569749600000002, 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058010644, 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 788 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:29.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295795+0000", 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4509999999999999e-06 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338094+0000", 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042298997999999997 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338113+0000", 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9100999999999999e-05 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338121+0000", 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1370000000000002e-06 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338192+0000", 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1515999999999998e-05 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338283+0000", 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0276000000000001e-05 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338420+0000", 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013750000000000001 2024-09-16T01:51:29.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.347191+0000", 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087705330000000005 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353679+0000", 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064878189999999997 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353751+0000", 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2193999999999994e-05 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353803+0000", 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2119e-05 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:862 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:29.890 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7159281810000007, 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063151895999999999, 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 862 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036841+0000", 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.075e-06 2024-09-16T01:51:29.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069065+0000", 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032224153999999998 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069093+0000", 2024-09-16T01:51:29.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7797000000000002e-05 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069107+0000", 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3594000000000001e-05 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069222+0000", 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115179 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069335+0000", 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112748 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069508+0000", 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173692 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.089364+0000", 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019855866 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099875+0000", 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511177 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099941+0000", 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5508999999999999e-05 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099991+0000", 2024-09-16T01:51:29.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0105000000000002e-05 2024-09-16T01:51:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:29.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:29.897 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:29.898 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:29.899 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:29.897+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:30.000 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:29.999+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:30.024 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:30.024 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:30.024 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:30.024 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:30.033 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:30.101 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:30.099+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:30.186 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:30.187 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:30.187 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:30.187 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:30.187 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:30.197 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:30.202 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:30.200+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:30.303 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.751255008999998, 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.304 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:30.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:30.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.504221428000001, 2024-09-16T01:51:30.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:30.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:30.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.316 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.445062342, 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:30.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:30.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.533484029999997, 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.323 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.489223799999998, 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:30.329 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:30.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:30.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.458723980000002, 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.336 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:30.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:30.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:30.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.417168457999999, 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.342 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:30.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:30.347 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.279121764000003, 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.348 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:30.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.350 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:30.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.160162036999999, 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.354 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:30.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:30.358 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.361 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.125365737999999, 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:30.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.364 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:30.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.041916785000002, 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.369 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:30.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:30.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.374 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.375 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.842613998000001, 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.376 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:51:30.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:51:30.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.381 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.69773706, 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.382 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.383 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:51:30.384 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.385 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.386 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.387 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.657092636, 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.388 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.389 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.390 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:51:30.391 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:51:30.392 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.393 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.499149802, 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.394 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.395 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.396 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:51:30.397 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:51:30.398 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.399 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.374022541, 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:51:30.400 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.401 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:51:30.402 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:51:30.403 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.404 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.405 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:576 3.1b 3:d81af1ca:::100000001f7.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.265219855, 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052702763999999999, 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.406 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 576 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.407 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035658+0000", 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.03e-06 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075189+0000", 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039530957999999998 2024-09-16T01:51:30.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075204+0000", 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5071e-05 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075214+0000", 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.409 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075299+0000", 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4807000000000003e-05 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075401+0000", 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010220700000000001 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075495+0000", 2024-09-16T01:51:30.410 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4006999999999996e-05 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.078964+0000", 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003468741 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088273+0000", 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0093088089999999995 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.411 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088311+0000", 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8402000000000003e-05 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088359+0000", 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7920999999999997e-05 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:598 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.412 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:30.413 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.085139989, 2024-09-16T01:51:30.413 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054052254000000001, 2024-09-16T01:51:30.413 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.413 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 598 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.416 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215738+0000", 2024-09-16T01:51:30.417 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.526e-06 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250373+0000", 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034634160999999997 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250391+0000", 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7986e-05 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.418 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250400+0000", 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1439999999999992e-06 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250455+0000", 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5578999999999998e-05 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250562+0000", 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106717 2024-09-16T01:51:30.419 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250665+0000", 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010307200000000001 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.253888+0000", 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032231019999999998 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.420 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269702+0000", 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015813473000000002 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269742+0000", 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0154000000000002e-05 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269788+0000", 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.634e-05 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.421 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.33722261, 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.422 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.423 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.424 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.425 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.426 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:51:30.427 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.428 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.1741969680000004, 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:51:30.429 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:30.430 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.431 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.432 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.433 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:51:30.434 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.435 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:30.436 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:30.302+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:30.437 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:30.478 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:30.477+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:30.539 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:30.538+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:30.582 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:30.583 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:30.583 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:30.583 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:30.592 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:30.641 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:30.639+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:30.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:30.698 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:30.698 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:30.698 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:30.699 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:30.699 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:30.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:30.750 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:30.748+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:30.812 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:19 3.1f 3:f8cab2eb:::10000000002.00000010:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 40.411684348999998, 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075358271000000004, 2024-09-16T01:51:30.813 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 19 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398086+0000", 2024-09-16T01:51:30.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2149999999999998e-06 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438214+0000", 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040127599999999999 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438231+0000", 2024-09-16T01:51:30.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7028e-05 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438238+0000", 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438293+0000", 2024-09-16T01:51:30.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5007999999999999e-05 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438402+0000", 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000109379 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438491+0000", 2024-09-16T01:51:30.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8774000000000001e-05 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.442442+0000", 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039511320000000004 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473344+0000", 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030902466999999999 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473394+0000", 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9264000000000001e-05 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473438+0000", 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4579000000000001e-05 2024-09-16T01:51:30.820 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:24 3.0 3:028eacff:::10000000002.00000015:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.821 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 40.367802955000002, 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.073041954000000006, 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 24 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.823 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441963+0000", 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7379999999999999e-06 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508113+0000", 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066149523000000002 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508127+0000", 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4678000000000001e-05 2024-09-16T01:51:30.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508135+0000", 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4959999999999999e-06 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508200+0000", 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5062999999999997e-05 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508374+0000", 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000174322 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508512+0000", 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013740099999999999 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514816+0000", 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063044199999999998 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514914+0000", 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7590000000000006e-05 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514951+0000", 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7348999999999998e-05 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.515003+0000", 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2373999999999997e-05 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.829 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 40.286564194, 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.830 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:30.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:51:30.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:51:30.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:51:30.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:51:30.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:51:30.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.838 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:50 3.17 3:eaf1bb0b:::10000000002.0000002f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 40.093108801, 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058893003999999999, 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.839 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 50 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716657+0000", 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9290000000000001e-06 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747413+0000", 2024-09-16T01:51:30.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030755714 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747428+0000", 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4452e-05 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747438+0000", 2024-09-16T01:51:30.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0436e-05 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747490+0000", 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1632000000000003e-05 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747608+0000", 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000117934 2024-09-16T01:51:30.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747716+0000", 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107953 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.751490+0000", 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037745750000000001 2024-09-16T01:51:30.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775454+0000", 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023963661000000001 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775504+0000", 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0003999999999998e-05 2024-09-16T01:51:30.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775549+0000", 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4713999999999999e-05 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:122 3.0 3:00eacacc:::10000000002.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.846 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.439850948, 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068671870999999995, 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 122 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.847 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369915+0000", 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5990000000000001e-06 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434232+0000", 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064317038000000007 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434250+0000", 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8028000000000001e-05 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434258+0000", 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6720000000000004e-06 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434311+0000", 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2914999999999997e-05 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434411+0000", 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100045 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434501+0000", 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494000000000005e-05 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.437130+0000", 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026284059999999998 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438521+0000", 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013915290000000001 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438553+0000", 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1992999999999997e-05 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438585+0000", 2024-09-16T01:51:30.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2152e-05 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:134 3.17 3:e85118c5:::10000000002.00000083:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:30.854 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.345147079, 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061709079999999999, 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 134 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464620+0000", 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8499999999999998e-06 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518730+0000", 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054109808000000002 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518748+0000", 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8289000000000001e-05 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518756+0000", 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8539999999999997e-06 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518809+0000", 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2879e-05 2024-09-16T01:51:30.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518914+0000", 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104719 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.519010+0000", 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6248000000000004e-05 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.522952+0000", 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039422440000000001 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526248+0000", 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003296003 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526290+0000", 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1708000000000001e-05 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526326+0000", 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6477999999999999e-05 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.861 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:472 3.31 3:8c3036ad:::10000000002.000000d5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.866087857, 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060972124000000003, 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.862 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 472 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943678+0000", 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5549999999999999e-06 2024-09-16T01:51:30.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974973+0000", 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031295264000000003 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974987+0000", 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3349e-05 2024-09-16T01:51:30.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974994+0000", 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3520000000000001e-06 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975074+0000", 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9864000000000006e-05 2024-09-16T01:51:30.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975140+0000", 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6377999999999994e-05 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975233+0000", 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2359000000000001e-05 2024-09-16T01:51:30.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.978088+0000", 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028552489999999998 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004578+0000", 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026490519000000001 2024-09-16T01:51:30.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004615+0000", 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6455999999999998e-05 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004649+0000", 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3779000000000003e-05 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.869 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:483 3.27 3:e6494e75:::10000000002.000000e0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.712983761, 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059372405000000003, 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.870 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 483 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096782+0000", 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137952+0000", 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041169532000000002 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137964+0000", 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1725999999999999e-05 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137988+0000", 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4020999999999999e-05 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138046+0000", 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8585000000000002e-05 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138124+0000", 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7312999999999997e-05 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138239+0000", 2024-09-16T01:51:30.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000115278 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.143288+0000", 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050496630000000002 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156064+0000", 2024-09-16T01:51:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012775269000000001 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156103+0000", 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8957e-05 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156153+0000", 2024-09-16T01:51:30.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0324000000000001e-05 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:545 3.38 3:1d8b64ca:::100000001f7.0000001e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.182109215000001, 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075800996999999995, 2024-09-16T01:51:30.878 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 545 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627656+0000", 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.176e-06 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655590+0000", 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027933824999999999 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655601+0000", 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1270999999999999e-05 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655608+0000", 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.55e-06 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655659+0000", 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0946999999999999e-05 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655756+0000", 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7448000000000006e-05 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655857+0000", 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100863 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.659273+0000", 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034161180000000001 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703393+0000", 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044119393 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703427+0000", 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4063000000000002e-05 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703456+0000", 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9343e-05 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:546 3.38 3:1d0ca575:::100000001f7.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.148844805, 2024-09-16T01:51:30.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068250515999999997, 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 546 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660921+0000", 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.539e-06 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709606+0000", 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048684573000000002 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709617+0000", 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1447999999999999e-05 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709622+0000", 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-06 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709691+0000", 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8879000000000002e-05 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709788+0000", 2024-09-16T01:51:30.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6768999999999995e-05 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709893+0000", 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010466799999999999 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.715292+0000", 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005399165 2024-09-16T01:51:30.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729109+0000", 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013817352 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729143+0000", 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4059e-05 2024-09-16T01:51:30.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729170+0000", 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6764e-05 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.894 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:574 3.27 3:e46e0c7f:::100000001f7.0000003b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.805108249, 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084267634999999994, 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 574 2024-09-16T01:51:30.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004659+0000", 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5129999999999999e-06 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032641+0000", 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027982080999999999 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032656+0000", 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5699999999999999e-05 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032666+0000", 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5599999999999999e-06 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032719+0000", 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3066999999999999e-05 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032827+0000", 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010841900000000001 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032929+0000", 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101128 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.036882+0000", 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039538949999999998 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088843+0000", 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051960682000000001 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088878+0000", 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.485e-05 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088924+0000", 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5739999999999999e-05 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:590 3.29 3:966924c4:::100000001f7.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.693597223999999, 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074607536000000002, 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.902 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 590 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116170+0000", 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162459+0000", 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046289624000000001 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162473+0000", 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3776e-05 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162481+0000", 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2889999999999998e-06 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162559+0000", 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7101000000000006e-05 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162675+0000", 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116933 2024-09-16T01:51:30.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162764+0000", 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8343000000000004e-05 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.167505+0000", 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047412519999999996 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190706+0000", 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023201366000000001 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190742+0000", 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5757999999999999e-05 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190775+0000", 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2484999999999998e-05 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:625 3.3d 3:bee2a002:::100000001f7.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.909 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.369120794000001, 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067884265999999999, 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 625 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.910 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440646+0000", 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0710000000000002e-06 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502844+0000", 2024-09-16T01:51:30.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062197927 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502866+0000", 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2580999999999999e-05 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502877+0000", 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0823e-05 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502939+0000", 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2360000000000006e-05 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.503095+0000", 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015614099999999999 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.503243+0000", 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014767400000000001 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.507276+0000", 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040332340000000001 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508460+0000", 2024-09-16T01:51:30.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011837740000000001 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508496+0000", 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5540999999999997e-05 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508528+0000", 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2140000000000001e-05 2024-09-16T01:51:30.916 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:631 3.23 3:c4d03f6f:::100000001f7.00000074:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.520808+0000", 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.288956353, 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.063082532999999996, 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.917 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 631 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.520808+0000", 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.520808+0000", 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.520809+0000", 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.229e-06 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539695+0000", 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.018885745999999998 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539714+0000", 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8668000000000002e-05 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539722+0000", 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9510000000000006e-06 2024-09-16T01:51:30.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539777+0000", 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5379e-05 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539901+0000", 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012439499999999999 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539997+0000", 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5222000000000004e-05 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.544982+0000", 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049858740000000004 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.583651+0000", 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038668181000000003 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.583820+0000", 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016971999999999999 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.583891+0000", 2024-09-16T01:51:30.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0167999999999996e-05 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:670 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.92906807, 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066678565999999995, 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.924 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 670 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:30.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880698+0000", 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2890000000000001e-06 2024-09-16T01:51:30.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903674+0000", 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022976465000000001 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903699+0000", 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4675e-05 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903712+0000", 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2935000000000001e-05 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903788+0000", 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6676999999999998e-05 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903928+0000", 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013916300000000001 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.904044+0000", 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011618200000000001 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.907487+0000", 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034430839999999999 2024-09-16T01:51:30.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947289+0000", 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039801925000000002 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947330+0000", 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.121e-05 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947375+0000", 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4960999999999999e-05 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:685 3.27 3:e74bd690:::100000001f7.000000aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.986565+0000", 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.823199237000001, 2024-09-16T01:51:30.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055646205999999997, 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 685 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.986565+0000", 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.986565+0000", 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.986567+0000", 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7770000000000001e-06 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015334+0000", 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028767478999999999 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015358+0000", 2024-09-16T01:51:30.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3340999999999999e-05 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015368+0000", 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0438999999999999e-05 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015432+0000", 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4321999999999998e-05 2024-09-16T01:51:30.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015550+0000", 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000117713 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015655+0000", 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010514900000000001 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019015+0000", 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033593020000000002 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.042112+0000", 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023097751999999999 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.042166+0000", 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3940999999999997e-05 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.042211+0000", 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4990999999999997e-05 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.938 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:687 3.29 3:95dcb5ee:::100000001f7.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.019206+0000", 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.790558158, 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057193637999999998, 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 687 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.939 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019206+0000", 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019206+0000", 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019209+0000", 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3910000000000001e-06 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066734+0000", 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047525322000000002 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066757+0000", 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2759e-05 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066766+0000", 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4979999999999995e-06 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066889+0000", 2024-09-16T01:51:30.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123125 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.067008+0000", 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118288 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.067124+0000", 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116496 2024-09-16T01:51:30.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.070545+0000", 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034212539999999999 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.076316+0000", 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057710089999999997 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.076358+0000", 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1724000000000002e-05 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.076400+0000", 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1771999999999999e-05 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:712 3.1e 3:7933c15e:::100000001f7.000000c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.523849258, 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067308615000000002, 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.946 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 712 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285917+0000", 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0370000000000001e-06 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330533+0000", 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044615783999999999 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330550+0000", 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6878999999999999e-05 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330558+0000", 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7000000000000008e-06 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.949 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330621+0000", 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3100000000000002e-05 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330773+0000", 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015215199999999999 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330932+0000", 2024-09-16T01:51:30.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015931100000000001 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.335841+0000", 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004909348 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353141+0000", 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017299833000000001 2024-09-16T01:51:30.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353182+0000", 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0849000000000001e-05 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353224+0000", 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1622000000000002e-05 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.952 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:861 3.31 3:8d161915:::100000001f7.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.034456+0000", 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.7753084549999993, 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058433348000000003, 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.953 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 861 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.034456+0000", 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.034456+0000", 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.034458+0000", 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1749999999999999e-06 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058502+0000", 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024044157 2024-09-16T01:51:30.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058517+0000", 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4609e-05 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058524+0000", 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9970000000000001e-06 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058570+0000", 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6400000000000003e-05 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058635+0000", 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4572000000000005e-05 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:30.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058727+0000", 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1694999999999995e-05 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.061959+0000", 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032324039999999999 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.092797+0000", 2024-09-16T01:51:30.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.03083851 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.092841+0000", 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3237e-05 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.092889+0000", 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8591999999999998e-05 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.959 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:872 3.18 3:19643448:::100000001f7.00000065:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.6601196030000001, 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.069066814000000004, 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:30.960 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 872 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149650+0000", 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8899999999999998e-06 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:30.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209584+0000", 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059934556 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209621+0000", 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6378999999999998e-05 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209629+0000", 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6410000000000008e-06 2024-09-16T01:51:30.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209715+0000", 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6013999999999993e-05 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209807+0000", 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2020000000000003e-05 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209966+0000", 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158889 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.216152+0000", 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006185467 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:30.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218614+0000", 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024625630000000001 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218669+0000", 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5104e-05 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218712+0000", 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2290999999999999e-05 2024-09-16T01:51:30.966 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.967 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.967 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.967 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:30.967 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:30.967 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:30.969 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:30.970 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:30.967+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:30.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.855+0000 7fbcee000640 1 Processor -- start 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.855+0000 7fbcee000640 1 -- start start 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbcee000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce80d0490 0x7fbce80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbcee000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbce8154da0 0x7fbce80c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbcee000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce815e220 con 0x7fbce80d0870 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbcee000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbce80c6160 con 0x7fbce8154da0 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbcee000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbce815e500 con 0x7fbce80d0490 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbcecc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce80d0490 0x7fbce80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbced600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbce8154da0 0x7fbce80c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbce7e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbce80d0870 0x7fbce8154690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44816/0 (socket says 172.21.15.146:44816) 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.856+0000 7fbce7e00640 1 -- 172.21.15.146:0/2215619798 learned_addr learned my addr 172.21.15.146:0/2215619798 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:30.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.857+0000 7fbced600640 1 -- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce80d0490 msgr2=0x7fbce80c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.857+0000 7fbced600640 1 --2- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce80d0490 0x7fbce80c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.857+0000 7fbced600640 1 -- 172.21.15.146:0/2215619798 >> v1:172.21.15.146:6789/0 conn(0x7fbce80d0870 legacy=0x7fbce8154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.857+0000 7fbced600640 1 -- 172.21.15.146:0/2215619798 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce815e7e0 con 0x7fbce8154da0 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.858+0000 7fbced600640 1 --2- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce8154da0 0x7fbce80c6b30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009d90 tx=0x7fbce002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccc415c307d475e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.858+0000 7fbce7400640 1 -- 172.21.15.146:0/2215619798 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce0007740 con 0x7fbce8154da0 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.858+0000 7fbce7400640 1 -- 172.21.15.146:0/2215619798 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce00078e0 con 0x7fbce8154da0 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.858+0000 7fbce7400640 1 -- 172.21.15.146:0/2215619798 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce0007cb0 con 0x7fbce8154da0 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 -- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce8154da0 msgr2=0x7fbce80c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 --2- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce8154da0 0x7fbce80c6b30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009d90 tx=0x7fbce002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 -- 172.21.15.146:0/2215619798 shutdown_connections 2024-09-16T01:51:30.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 --2- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce8154da0 0x7fbce80c6b30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 --2- 172.21.15.146:0/2215619798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce80d0490 0x7fbce80c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 -- 172.21.15.146:0/2215619798 >> 172.21.15.146:0/2215619798 conn(0x7fbce806d6f0 msgr2=0x7fbce806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 -- 172.21.15.146:0/2215619798 shutdown_connections 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.859+0000 7fbcee000640 1 -- 172.21.15.146:0/2215619798 wait complete. 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.861+0000 7fbcee000640 1 Processor -- start 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.861+0000 7fbcee000640 1 -- start start 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.861+0000 7fbcee000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 0x7fbce80b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcee000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce80d0870 0x7fbce80cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcee000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce8154da0 0x7fbce81f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcee000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbce815dec0 con 0x7fbce80d0490 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcee000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbce815eda0 con 0x7fbce80d0870 2024-09-16T01:51:30.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcee000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbce80c51f0 con 0x7fbce8154da0 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 0x7fbce80b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 0x7fbce80b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40026/0 (socket says 172.21.15.146:40026) 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 -- 172.21.15.146:0/2185764467 learned_addr learned my addr 172.21.15.146:0/2185764467 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce80d0870 0x7fbce80cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce8154da0 0x7fbce81f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce8154da0 msgr2=0x7fbce81f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce8154da0 0x7fbce81f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce80d0870 msgr2=0x7fbce80cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce80d0870 0x7fbce80cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce80b3ae0 con 0x7fbce80d0490 2024-09-16T01:51:30.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.862+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 0x7fbce80b5220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00e9b0 tx=0x7fbcdc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.863+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc017070 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.863+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc00c9a0 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.863+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce80c9260 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.863+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc00cd90 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.863+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce80b4310 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.864+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbce80c9f20 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.865+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbcdc01a070 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.866+0000 7fbce4c00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc806c2e0 0x7fbcc806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.866+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5741+0+0 (secure 0 0 0) 0x7fbcdc08b3a0 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.866+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc806c2e0 0x7fbcc806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.866+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v73) ==== 54+0+16 (secure 0 0 0) 0x7fbce80c9f20 con 0x7fbce80d0490 2024-09-16T01:51:30.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.866+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbce81f4660 0x7fbce81e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.866+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbce80d0ff0 con 0x7fbce81f4660 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.867+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc806c2e0 0x7fbcc806e7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbce81e5820 tx=0x7fbcd4007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.867+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbce81f4660 0x7fbce81e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.867+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbce81f4660 0x7fbce81e7030 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.868+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce80d0ff0 con 0x7fbce81f4660 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.882+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce8154690 con 0x7fbce81f4660 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.884+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce8154690 con 0x7fbce81f4660 2024-09-16T01:51:30.980 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:30.981 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:30.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.884+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbce814ddc0 0x7fbce814e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.884+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbce814e6e0 con 0x7fbce814ddc0 2024-09-16T01:51:30.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.885+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbce814ddc0 0x7fbce814e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.885+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbce814ddc0 0x7fbce814e1a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.886+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce814e6e0 con 0x7fbce814ddc0 2024-09-16T01:51:30.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.901+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce80b3e00 con 0x7fbce814ddc0 2024-09-16T01:51:30.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.903+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce80b3e00 con 0x7fbce814ddc0 2024-09-16T01:51:30.982 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:30.983 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:30.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.903+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbce81f0de0 0x7fbce81f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.903+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbce80cb7b0 con 0x7fbce81f0de0 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.903+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbce81f0de0 0x7fbce81f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.904+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbce81f0de0 0x7fbce81f11c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.905+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce80cb7b0 con 0x7fbce81f0de0 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.918+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce80b6240 con 0x7fbce81f0de0 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.920+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce80b6240 con 0x7fbce81f0de0 2024-09-16T01:51:30.984 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:30.985 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:30.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.920+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbce8062cb0 0x7fbce80630f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.920+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbce82034b0 con 0x7fbce8062cb0 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.921+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbce8062cb0 0x7fbce80630f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.921+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbce8062cb0 0x7fbce80630f0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.922+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce82034b0 con 0x7fbce8062cb0 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.935+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce81e4090 con 0x7fbce8062cb0 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.937+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce81e4090 con 0x7fbce8062cb0 2024-09-16T01:51:30.986 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:30.987 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:30.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.937+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbce81e47b0 0x7fbce802c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.937+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbce81e4c10 con 0x7fbce81e47b0 2024-09-16T01:51:30.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.938+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbce81e47b0 0x7fbce802c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.938+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbce81e47b0 0x7fbce802c640 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.939+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce81e4c10 con 0x7fbce81e47b0 2024-09-16T01:51:30.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.952+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce81e4c10 con 0x7fbce81e47b0 2024-09-16T01:51:30.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.954+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce81e4c10 con 0x7fbce81e47b0 2024-09-16T01:51:30.988 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:30.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.954+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbce802d090 0x7fbce802d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.955+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbce81e4c10 con 0x7fbce802d090 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.955+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbce802d090 0x7fbce802d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.955+0000 7fbce7e00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbce802d090 0x7fbce802d4f0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.956+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce81e4c10 con 0x7fbce802d090 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.970+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce81e4c10 con 0x7fbce802d090 2024-09-16T01:51:30.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.972+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce81e4c10 con 0x7fbce802d090 2024-09-16T01:51:30.991 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.972+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbce8207b90 0x7fbce8209f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.972+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbce81e4ce0 con 0x7fbce8207b90 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.973+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbce8207b90 0x7fbce8209f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.973+0000 7fbced600640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbce8207b90 0x7fbce8209f80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.974+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce81e4ce0 con 0x7fbce8207b90 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.987+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce820a4c0 con 0x7fbce8207b90 2024-09-16T01:51:30.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.989+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce820a4c0 con 0x7fbce8207b90 2024-09-16T01:51:30.993 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:30.993 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:30.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.989+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbce820a9d0 0x7fbce820cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:30.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.989+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbce820fd50 con 0x7fbce820a9d0 2024-09-16T01:51:30.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.990+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbce820a9d0 0x7fbce820cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:30.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.990+0000 7fbcecc00640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbce820a9d0 0x7fbce820cdc0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:30.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:30.991+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce820fd50 con 0x7fbce820a9d0 2024-09-16T01:51:31.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.005+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbce820ea90 con 0x7fbce820a9d0 2024-09-16T01:51:31.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.006+0000 7fbce4c00640 1 -- 172.21.15.146:0/2185764467 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbce820ea90 con 0x7fbce820a9d0 2024-09-16T01:51:31.008 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbce81f4660 msgr2=0x7fbce81e7030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbce81f4660 0x7fbce81e7030 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbce814ddc0 msgr2=0x7fbce814e1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbce814ddc0 0x7fbce814e1a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbce81f0de0 msgr2=0x7fbce81f11c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbce81f0de0 0x7fbce81f11c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbce8062cb0 msgr2=0x7fbce80630f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbce8062cb0 0x7fbce80630f0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbce81e47b0 msgr2=0x7fbce802c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbce81e47b0 0x7fbce802c640 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbce802d090 msgr2=0x7fbce802d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbce802d090 0x7fbce802d4f0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbced600640 1 -- 172.21.15.146:0/2185764467 reap_dead start 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbce8207b90 msgr2=0x7fbce8209f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbce8207b90 0x7fbce8209f80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbced600640 1 -- 172.21.15.146:0/2185764467 reap_dead start 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbce820a9d0 msgr2=0x7fbce820cdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbce820a9d0 0x7fbce820cdc0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbced600640 1 -- 172.21.15.146:0/2185764467 reap_dead start 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbced600640 1 -- 172.21.15.146:0/2185764467 reap_dead start 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc806c2e0 msgr2=0x7fbcc806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc806c2e0 0x7fbcc806e7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbce81e5820 tx=0x7fbcd4007e80 comp rx=0 tx=0).stop 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 msgr2=0x7fbce80b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 0x7fbce80b5220 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00e9b0 tx=0x7fbcdc00ee70 comp rx=0 tx=0).stop 2024-09-16T01:51:31.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 shutdown_connections 2024-09-16T01:51:31.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc806c2e0 0x7fbcc806e7a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 --2- 172.21.15.146:0/2185764467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce80d0490 0x7fbce80b5220 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 >> 172.21.15.146:0/2185764467 conn(0x7fbce806d6f0 msgr2=0x7fbce80c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:31.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 shutdown_connections 2024-09-16T01:51:31.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.007+0000 7fbcee000640 1 -- 172.21.15.146:0/2185764467 wait complete. 2024-09-16T01:51:31.069 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:31.068+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:31.117 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:31.117 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:31.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:31.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.145+0000 7fb210a00640 1 Processor -- start 2024-09-16T01:51:31.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.145+0000 7fb210a00640 1 -- start start 2024-09-16T01:51:31.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb210a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c0c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:31.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb210a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb20c072bf0 0x7fb20c157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:31.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb210a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb20c0c1150 con 0x7fb20c1515f0 2024-09-16T01:51:31.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb210a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb20c073630 con 0x7fb20c072bf0 2024-09-16T01:51:31.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb210a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb20c073350 con 0x7fb20c1519d0 2024-09-16T01:51:31.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb20a800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb20c1515f0 0x7fb20c0c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44826/0 (socket says 172.21.15.146:44826) 2024-09-16T01:51:31.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb20a800640 1 -- 172.21.15.146:0/2351052343 learned_addr learned my addr 172.21.15.146:0/2351052343 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:31.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb20b200640 1 --2- 172.21.15.146:0/2351052343 >> v2:172.21.15.146:3300/0 conn(0x7fb20c072bf0 0x7fb20c157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:31.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.146+0000 7fb201e00640 1 --2- 172.21.15.146:0/2351052343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c0c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:31.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.147+0000 7fb201e00640 1 -- 172.21.15.146:0/2351052343 >> v2:172.21.15.146:3300/0 conn(0x7fb20c072bf0 msgr2=0x7fb20c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:31.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.147+0000 7fb201e00640 1 --2- 172.21.15.146:0/2351052343 >> v2:172.21.15.146:3300/0 conn(0x7fb20c072bf0 0x7fb20c157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.147+0000 7fb201e00640 1 -- 172.21.15.146:0/2351052343 >> v1:172.21.15.146:6789/0 conn(0x7fb20c1515f0 legacy=0x7fb20c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.147+0000 7fb209e00640 1 -- 172.21.15.146:0/2351052343 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270076325 0 0) 0x7fb20c0c1150 con 0x7fb20c1515f0 2024-09-16T01:51:31.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.147+0000 7fb201e00640 1 -- 172.21.15.146:0/2351052343 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c0c13f0 con 0x7fb20c1519d0 2024-09-16T01:51:31.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.148+0000 7fb201e00640 1 --2- 172.21.15.146:0/2351052343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c0c8060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40097a0 tx=0x7fb1f402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a3c2bd33fd3fd54 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:31.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.148+0000 7fb209e00640 1 -- 172.21.15.146:0/2351052343 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4037070 con 0x7fb20c1519d0 2024-09-16T01:51:31.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.148+0000 7fb209e00640 1 -- 172.21.15.146:0/2351052343 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1f4032440 con 0x7fb20c1519d0 2024-09-16T01:51:31.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.148+0000 7fb209e00640 1 -- 172.21.15.146:0/2351052343 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4032810 con 0x7fb20c1519d0 2024-09-16T01:51:31.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 -- 172.21.15.146:0/2351052343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 msgr2=0x7fb20c0c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 --2- 172.21.15.146:0/2351052343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c0c8060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40097a0 tx=0x7fb1f402a230 comp rx=0 tx=0).stop 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 -- 172.21.15.146:0/2351052343 shutdown_connections 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 --2- 172.21.15.146:0/2351052343 >> v2:172.21.15.146:3300/0 conn(0x7fb20c072bf0 0x7fb20c157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 --2- 172.21.15.146:0/2351052343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c0c8060 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 -- 172.21.15.146:0/2351052343 >> 172.21.15.146:0/2351052343 conn(0x7fb20c067950 msgr2=0x7fb20c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 -- 172.21.15.146:0/2351052343 shutdown_connections 2024-09-16T01:51:31.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.149+0000 7fb210a00640 1 -- 172.21.15.146:0/2351052343 wait complete. 2024-09-16T01:51:31.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.150+0000 7fb210a00640 1 Processor -- start 2024-09-16T01:51:31.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.150+0000 7fb210a00640 1 -- start start 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.151+0000 7fb210a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 0x7fb20c13eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.151+0000 7fb210a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb20c1515f0 0x7fb20c13a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.151+0000 7fb210a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c161e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.151+0000 7fb210a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c073000 con 0x7fb20c072bf0 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.151+0000 7fb210a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb20c0c19b0 con 0x7fb20c1515f0 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.151+0000 7fb210a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb20c074060 con 0x7fb20c1519d0 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 0x7fb20c13eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 0x7fb20c13eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40042/0 (socket says 172.21.15.146:40042) 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 -- 172.21.15.146:0/1277568699 learned_addr learned my addr 172.21.15.146:0/1277568699 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:31.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20b200640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb201e00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb20c1515f0 0x7fb20c13a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 -- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 msgr2=0x7fb20c161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 -- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb20c1515f0 msgr2=0x7fb20c13a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb20c1515f0 0x7fb20c13a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 -- 172.21.15.146:0/1277568699 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20c14c2d0 con 0x7fb20c072bf0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.152+0000 7fb20a800640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 0x7fb20c13eb00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c500 tx=0x7fb1fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.153+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc018070 con 0x7fb20c072bf0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.153+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1fc007580 con 0x7fb20c072bf0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.153+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc004a20 con 0x7fb20c072bf0 2024-09-16T01:51:31.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.153+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20c0b1450 con 0x7fb20c072bf0 2024-09-16T01:51:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.153+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20c14d5c0 con 0x7fb20c072bf0 2024-09-16T01:51:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.154+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb20c073e40 con 0x7fb20c072bf0 2024-09-16T01:51:31.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.155+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb1fc01b070 con 0x7fb20c072bf0 2024-09-16T01:51:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.155+0000 7fb203400640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb1e806c2e0 0x7fb1e806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.156+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5741+0+0 (secure 0 0 0) 0x7fb1fc08a6b0 con 0x7fb20c072bf0 2024-09-16T01:51:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.156+0000 7fb201e00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb1e806c2e0 0x7fb1e806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:31.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.156+0000 7fb201e00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb1e806c2e0 0x7fb1e806e7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4034000 tx=0x7fb1f402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:31.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.160+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb1fc0587a0 con 0x7fb20c072bf0 2024-09-16T01:51:31.210 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:31.209+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:31.270 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:31.270 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:31.271 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:31.269+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:31.280 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:31.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.344+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb20c163720 con 0x7fb20c072bf0 2024-09-16T01:51:31.371 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:31.370+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:31.424 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:31.424 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:31.433 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:31.474 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:31.473+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:31.573 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:31.572+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:31.578 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:31.578 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:31.587 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:31.674 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:31.672+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:31.731 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:31.731 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:31.739 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:31.783 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:31.782+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:31.842 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:31.842 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:31.851 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:31.897 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:31.895+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:31.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.968+0000 7fb203400640 1 -- 172.21.15.146:0/1277568699 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v74) ==== 97+0+0 (secure 0 0 0) 0x7fb1fc05caa0 con 0x7fb20c072bf0 2024-09-16T01:51:31.971 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb1e806c2e0 msgr2=0x7fb1e806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb1e806c2e0 0x7fb1e806e7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4034000 tx=0x7fb1f402e000 comp rx=0 tx=0).stop 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 msgr2=0x7fb20c13eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 0x7fb20c13eb00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc00c500 tx=0x7fb1fc00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 shutdown_connections 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb1e806c2e0 0x7fb1e806e7a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb20c1519d0 0x7fb20c161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb20c1515f0 0x7fb20c13a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 --2- 172.21.15.146:0/1277568699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20c072bf0 0x7fb20c13eb00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:31.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 >> 172.21.15.146:0/1277568699 conn(0x7fb20c067950 msgr2=0x7fb20c067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:31.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 shutdown_connections 2024-09-16T01:51:31.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:31.972+0000 7fb210a00640 1 -- 172.21.15.146:0/1277568699 wait complete. 2024-09-16T01:51:31.976 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:31.975+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:32.016 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:32.016 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:32.028 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:32.077 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:32.077+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:32.178 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:32.176+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:32.228 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:32.228 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:32.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:32.282 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:32.280+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:32.379 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:32.379+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:32.441 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:32.441 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:32.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:32.480 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:32.479+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:32.581 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:32.579+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:32.613 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:32.613 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:32.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:32.712 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:32.710+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:32.782 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:32.780+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:32.824 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:32.824 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:32.836 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:32.923 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:32.921+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:32.984 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:32.982+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:33.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:33.088 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:33.084+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:33.097 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:33.097 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:33.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:33.186 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:33.185+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:33.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.247+0000 7fe3f3200640 1 Processor -- start 2024-09-16T01:51:33.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.247+0000 7fe3f3200640 1 -- start start 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40c6d00 0x7fe3f40c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3f415e220 con 0x7fe3f40d0870 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f3200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3f415e500 con 0x7fe3f40d0490 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f40c6160 con 0x7fe3f40c6d00 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f1e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe3f40d0490 0x7fe3f40c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:49670/0 (socket says 172.21.15.146:49670) 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3e9400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe3f40d0870 0x7fe3f4154690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54308/0 (socket says 172.21.15.146:54308) 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f1e00640 1 -- 172.21.15.146:0/3279069757 learned_addr learned my addr 172.21.15.146:0/3279069757 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:33.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40c6d00 0x7fe3f40c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f1400640 1 -- 172.21.15.146:0/3279069757 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231474024 0 0) 0x7fe3f415e500 con 0x7fe3f40d0490 2024-09-16T01:51:33.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f1400640 1 -- 172.21.15.146:0/3279069757 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3d8003520 con 0x7fe3f40d0490 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f2800640 1 -- 172.21.15.146:0/3279069757 >> v1:172.21.15.161:6789/0 conn(0x7fe3f40d0490 legacy=0x7fe3f40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f2800640 1 -- 172.21.15.146:0/3279069757 >> v1:172.21.15.146:6789/0 conn(0x7fe3f40d0870 legacy=0x7fe3f4154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.249+0000 7fe3f2800640 1 -- 172.21.15.146:0/3279069757 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f415e7e0 con 0x7fe3f40c6d00 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.250+0000 7fe3f2800640 1 --2- 172.21.15.146:0/3279069757 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40c6d00 0x7fe3f40c51f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0097d0 tx=0x7fe3ec02a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5fb44824eb3af25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f1400640 1 -- 172.21.15.146:0/3279069757 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec037070 con 0x7fe3f40c6d00 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f1400640 1 -- 172.21.15.146:0/3279069757 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3ec032440 con 0x7fe3f40c6d00 2024-09-16T01:51:33.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f1400640 1 -- 172.21.15.146:0/3279069757 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec032810 con 0x7fe3f40c6d00 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 -- 172.21.15.146:0/3279069757 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40c6d00 msgr2=0x7fe3f40c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 --2- 172.21.15.146:0/3279069757 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40c6d00 0x7fe3f40c51f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0097d0 tx=0x7fe3ec02a2a0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 -- 172.21.15.146:0/3279069757 shutdown_connections 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 --2- 172.21.15.146:0/3279069757 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40c6d00 0x7fe3f40c51f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 -- 172.21.15.146:0/3279069757 >> 172.21.15.146:0/3279069757 conn(0x7fe3f406d6f0 msgr2=0x7fe3f406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 -- 172.21.15.146:0/3279069757 shutdown_connections 2024-09-16T01:51:33.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.251+0000 7fe3f3200640 1 -- 172.21.15.146:0/3279069757 wait complete. 2024-09-16T01:51:33.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.253+0000 7fe3f3200640 1 Processor -- start 2024-09-16T01:51:33.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.253+0000 7fe3f3200640 1 -- start start 2024-09-16T01:51:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3f40c6d00 0x7fe3f41f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3f40d0490 0x7fe3f4153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 0x7fe3f414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f415dec0 con 0x7fe3f40c6d00 2024-09-16T01:51:33.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f415eda0 con 0x7fe3f40d0490 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f4154da0 con 0x7fe3f40d0870 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 0x7fe3f414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3e9400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3f40d0490 0x7fe3f4153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 0x7fe3f414dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43910/0 (socket says 172.21.15.146:43910) 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 learned_addr learned my addr 172.21.15.146:0/2085546674 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.254+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3f40c6d00 0x7fe3f41f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3f40d0490 msgr2=0x7fe3f4153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3f40d0490 0x7fe3f4153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3f40c6d00 msgr2=0x7fe3f41f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3f40c6d00 0x7fe3f41f0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f40b3de0 con 0x7fe3f40d0870 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 0x7fe3f414dce0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec035040 tx=0x7fe3ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec037040 con 0x7fe3f40d0870 2024-09-16T01:51:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3ec042070 con 0x7fe3f40d0870 2024-09-16T01:51:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec03d8a0 con 0x7fe3f40d0870 2024-09-16T01:51:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f41f11f0 con 0x7fe3f40d0870 2024-09-16T01:51:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.255+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f40b3b00 con 0x7fe3f40d0870 2024-09-16T01:51:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.256+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe3f40c65f0 con 0x7fe3f40d0870 2024-09-16T01:51:33.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.257+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe3ec03da40 con 0x7fe3f40d0870 2024-09-16T01:51:33.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.258+0000 7fe3eaa00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3d006c2e0 0x7fe3d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.258+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 5741+0+0 (secure 0 0 0) 0x7fe3ec0abad0 con 0x7fe3f40d0870 2024-09-16T01:51:33.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.258+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3d006c2e0 0x7fe3d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.258+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v74) ==== 54+0+16 (secure 0 0 0) 0x7fe3ec0b0220 con 0x7fe3f40d0870 2024-09-16T01:51:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.259+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe3f40b4a30 0x7fe3f41f0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.259+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3f414e730 con 0x7fe3f40b4a30 2024-09-16T01:51:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.259+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3d006c2e0 0x7fe3d006e7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4001560 tx=0x7fe3e4001a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.259+0000 7fe3e9400640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe3f40b4a30 0x7fe3f41f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.260+0000 7fe3e9400640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe3f40b4a30 0x7fe3f41f0390 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.260+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3dc004aa0 con 0x7fe3f40b4a30 2024-09-16T01:51:33.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.275+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f41f08d0 con 0x7fe3f40b4a30 2024-09-16T01:51:33.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.277+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f41f08d0 con 0x7fe3f40b4a30 2024-09-16T01:51:33.279 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:33.279 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:33.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.277+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe3f41e4150 0x7fe3f415cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.277+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe3f406e680 con 0x7fe3f41e4150 2024-09-16T01:51:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.277+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe3f41e4150 0x7fe3f415cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.278+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe3f41e4150 0x7fe3f415cae0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.279+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f406e680 con 0x7fe3f41e4150 2024-09-16T01:51:33.287 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:33.285+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:33.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.294+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f40cfbe0 con 0x7fe3f41e4150 2024-09-16T01:51:33.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.295+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f40cfbe0 con 0x7fe3f41e4150 2024-09-16T01:51:33.297 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:33.298 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:33.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.296+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe3f40b5d60 0x7fe3f415d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.296+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe3f40b3120 con 0x7fe3f40b5d60 2024-09-16T01:51:33.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.296+0000 7fe3e9400640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe3f40b5d60 0x7fe3f415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.297+0000 7fe3e9400640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe3f40b5d60 0x7fe3f415d530 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.297+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f40b3120 con 0x7fe3f40b5d60 2024-09-16T01:51:33.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.311+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f4062880 con 0x7fe3f40b5d60 2024-09-16T01:51:33.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.313+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f4062880 con 0x7fe3f40b5d60 2024-09-16T01:51:33.315 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:33.315 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:33.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.313+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe3f4063360 0x7fe3f4152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.313+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe3f4152660 con 0x7fe3f4063360 2024-09-16T01:51:33.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.313+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe3f4063360 0x7fe3f4152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.314+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe3f4063360 0x7fe3f4152120 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.314+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f4152660 con 0x7fe3f4063360 2024-09-16T01:51:33.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.328+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f4152b70 con 0x7fe3f4063360 2024-09-16T01:51:33.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.330+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f4152b70 con 0x7fe3f4063360 2024-09-16T01:51:33.332 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:33.332 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:33.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.330+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe3f402c640 0x7fe3f402caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.330+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe3f4152660 con 0x7fe3f402c640 2024-09-16T01:51:33.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.330+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe3f402c640 0x7fe3f402caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.331+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe3f402c640 0x7fe3f402caa0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.332+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f4152660 con 0x7fe3f402c640 2024-09-16T01:51:33.336 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:33.336 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:33.336 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:33.336 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:33.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.346+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f4152660 con 0x7fe3f402c640 2024-09-16T01:51:33.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:33.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.347+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f4152660 con 0x7fe3f402c640 2024-09-16T01:51:33.350 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.348+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe3f402d4f0 0x7fe3f4206c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.348+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe3f402d930 con 0x7fe3f402d4f0 2024-09-16T01:51:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.348+0000 7fe3e9400640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe3f402d4f0 0x7fe3f4206c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.348+0000 7fe3e9400640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe3f402d4f0 0x7fe3f4206c20 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.349+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f402d930 con 0x7fe3f402d4f0 2024-09-16T01:51:33.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.363+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f402d930 con 0x7fe3f402d4f0 2024-09-16T01:51:33.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.365+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f402d930 con 0x7fe3f402d4f0 2024-09-16T01:51:33.367 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.365+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe3f4207670 0x7fe3f4209a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.365+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe3f420c930 con 0x7fe3f4207670 2024-09-16T01:51:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.366+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe3f4207670 0x7fe3f4209a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.366+0000 7fe3f2800640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe3f4207670 0x7fe3f4209a60 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.367+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f420c930 con 0x7fe3f4207670 2024-09-16T01:51:33.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.383+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f420c880 con 0x7fe3f4207670 2024-09-16T01:51:33.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.385+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f420c880 con 0x7fe3f4207670 2024-09-16T01:51:33.387 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:33.388 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:33.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.385+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe3f420a4b0 0x7fe3f420a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:33.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.385+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe3f420fd00 con 0x7fe3f420a4b0 2024-09-16T01:51:33.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.385+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe3f420a4b0 0x7fe3f420a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:33.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.386+0000 7fe3f1e00640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe3f420a4b0 0x7fe3f420a910 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:33.389 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:33.386+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:33.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.387+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3f420fd00 con 0x7fe3f420a4b0 2024-09-16T01:51:33.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.401+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3f420af60 con 0x7fe3f420a4b0 2024-09-16T01:51:33.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.402+0000 7fe3eaa00640 1 -- 172.21.15.146:0/2085546674 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3f420af60 con 0x7fe3f420a4b0 2024-09-16T01:51:33.404 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe3f40b4a30 msgr2=0x7fe3f41f0390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe3f40b4a30 0x7fe3f41f0390 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe3f41e4150 msgr2=0x7fe3f415cae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe3f41e4150 0x7fe3f415cae0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe3f40b5d60 msgr2=0x7fe3f415d530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe3f40b5d60 0x7fe3f415d530 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe3f4063360 msgr2=0x7fe3f4152120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe3f4063360 0x7fe3f4152120 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe3f402c640 msgr2=0x7fe3f402caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe3f402c640 0x7fe3f402caa0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe3f402d4f0 msgr2=0x7fe3f4206c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe3f402d4f0 0x7fe3f4206c20 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 reap_dead start 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe3f4207670 msgr2=0x7fe3f4209a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe3f4207670 0x7fe3f4209a60 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 reap_dead start 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe3f420a4b0 msgr2=0x7fe3f420a910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe3f420a4b0 0x7fe3f420a910 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 reap_dead start 2024-09-16T01:51:33.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f2800640 1 -- 172.21.15.146:0/2085546674 reap_dead start 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3d006c2e0 msgr2=0x7fe3d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3d006c2e0 0x7fe3d006e7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4001560 tx=0x7fe3e4001a90 comp rx=0 tx=0).stop 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 msgr2=0x7fe3f414dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 0x7fe3f414dce0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec035040 tx=0x7fe3ec0046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 shutdown_connections 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe3d006c2e0 0x7fe3d006e7a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 --2- 172.21.15.146:0/2085546674 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3f40d0870 0x7fe3f414dce0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 >> 172.21.15.146:0/2085546674 conn(0x7fe3f406d6f0 msgr2=0x7fe3f40cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:33.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 shutdown_connections 2024-09-16T01:51:33.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:33.403+0000 7fe3f3200640 1 -- 172.21.15.146:0/2085546674 wait complete. 2024-09-16T01:51:33.488 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:33.488+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:33.511 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:33.511 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:33.511 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:33.511 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:33.511 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:33.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:33.589 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:33.588+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.113959098000002, 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:33.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:33.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.960413416000002, 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:33.696 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.739553735999998, 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.697 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:33.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.661423147000001, 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.089210260999998, 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.708 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:33.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.713 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.865053727000003, 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:33.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:33.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:33.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:33.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.567776704000003, 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:33.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:33.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:33.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:33.725 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.094968850999997, 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:33.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:33.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:33.731 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.907142264999997, 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:33.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:33.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:33.737 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.39832586, 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:33.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:33.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.019479960999998, 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:51:33.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:607 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.391093170000001, 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057026931000000003, 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 607 2024-09-16T01:51:33.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290713+0000", 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6900000000000001e-06 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330961+0000", 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040248199999999998 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330992+0000", 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1251000000000003e-05 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331005+0000", 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3006999999999999e-05 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332660+0000", 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016549589999999999 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332819+0000", 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159137 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332968+0000", 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148942 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.342537+0000", 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095683910000000007 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347616+0000", 2024-09-16T01:51:33.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005079793 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347682+0000", 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5622000000000003e-05 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347737+0000", 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4938999999999997e-05 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.756 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:634 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.166904756000001, 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068283101999999998, 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 634 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514901+0000", 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5299999999999999e-06 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549207+0000", 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034306171000000003 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549224+0000", 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6976000000000001e-05 2024-09-16T01:51:33.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549235+0000", 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0682999999999999e-05 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549568+0000", 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033294599999999998 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549714+0000", 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014617499999999999 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549858+0000", 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144356 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.566205+0000", 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01634683 2024-09-16T01:51:33.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583078+0000", 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016872402000000002 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583135+0000", 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7286e-05 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583182+0000", 2024-09-16T01:51:33.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6746999999999998e-05 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:688 3.26 3:642b2ad4:::100000001f7.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.649937284, 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062867175999999997, 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.763 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 688 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031869+0000", 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5509999999999998e-06 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084368+0000", 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052499410000000003 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084392+0000", 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084405+0000", 2024-09-16T01:51:33.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2561000000000001e-05 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084491+0000", 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5945000000000005e-05 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084662+0000", 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017081299999999999 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084812+0000", 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993999999999999 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.091507+0000", 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066950079999999997 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094630+0000", 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031236570000000002 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094700+0000", 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9575999999999999e-05 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094733+0000", 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3260999999999999e-05 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.769 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:696 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.554679419999999, 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071884475000000003, 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 696 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127126+0000", 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9369999999999998e-06 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159227+0000", 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032101044000000002 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159250+0000", 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3332999999999998e-05 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159260+0000", 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159343+0000", 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3373000000000003e-05 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159484+0000", 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014100399999999999 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159646+0000", 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162044 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.186104+0000", 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026457794999999999 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198896+0000", 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012791749 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198959+0000", 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3387999999999997e-05 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.199008+0000", 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9118e-05 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:776 3.11 e68/20 3:887bb7a4:::100000001f7.00000005:head v 68'37, mlcod=65'35)", 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.473760283000001, 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05614222, 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208044+0000", 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219371+0000", 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011326688999999999 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219442+0000", 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0727000000000002e-05 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219483+0000", 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1165000000000003e-05 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219725+0000", 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024194200000000001 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219752+0000", 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7270000000000001e-05 2024-09-16T01:51:33.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264133+0000", 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044381029000000002 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264185+0000", 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1879000000000003e-05 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.779 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:786 3.3a 3:5d44b367:::100000001f7.0000000f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.43817119, 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055535233000000003, 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 786 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243634+0000", 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254427+0000", 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010792636 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254446+0000", 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9219999999999999e-05 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254458+0000", 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2387e-05 2024-09-16T01:51:33.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254837+0000", 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00037878500000000003 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254934+0000", 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6915000000000003e-05 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.255090+0000", 2024-09-16T01:51:33.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015594200000000001 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.273598+0000", 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018507932000000001 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299039+0000", 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025441583 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299115+0000", 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5319000000000002e-05 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299167+0000", 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2473999999999999e-05 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:785 3.11 e69/20 3:8b60708a:::100000001f7.0000000e:head v 69'38, mlcod=65'35)", 2024-09-16T01:51:33.785 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.426939020000001, 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082167670999999998, 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254865+0000", 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9799999999999999e-07 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286626+0000", 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031761664000000002 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286647+0000", 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268e-05 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286659+0000", 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1276e-05 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286743+0000", 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4488999999999997e-05 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286772+0000", 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9082999999999999e-05 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.336965+0000", 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050192716999999998 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337032+0000", 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6876000000000001e-05 2024-09-16T01:51:33.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:790 3.16 3:69eaaa52:::100000001f7.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.379166781, 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061715643000000001, 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 790 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302638+0000", 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9410000000000001e-06 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342651+0000", 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040012486 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342687+0000", 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5738999999999998e-05 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342701+0000", 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4255e-05 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343871+0000", 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001169977 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343975+0000", 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010435 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.344127+0000", 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015170400000000001 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.361011+0000", 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016884528999999999 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364224+0000", 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003212183 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364295+0000", 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0957000000000005e-05 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364352+0000", 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7522000000000003e-05 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.795 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:863 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.630033998, 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055946002000000002, 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 863 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:33.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051772+0000", 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2620000000000001e-06 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087118+0000", 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035346883000000003 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087186+0000", 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7743000000000005e-05 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087196+0000", 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087342+0000", 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146625 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087445+0000", 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010255 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087572+0000", 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012764800000000001 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.101292+0000", 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013719543000000001 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107514+0000", 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062224120000000001 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107641+0000", 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012699 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:33.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107715+0000", 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3906999999999996e-05 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:33.801 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:33.803 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:33.803 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:33.801+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:33.903 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:33.902+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:33.984 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:33.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:33.984 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T01:51:33.984 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-16T01:51:33.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T01:51:33.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:51:34.004 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:34.004 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:34.004 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:34.005 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:34.005 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:34.003+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:34.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:34.104 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:34.102+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:34.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.143+0000 7ff1f4600640 1 Processor -- start 2024-09-16T01:51:34.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.143+0000 7ff1f4600640 1 -- start start 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f4600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1ec0c92f0 0x7ff1ec06ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f4600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec0c47d0 con 0x7ff1ec0c7bc0 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f4600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec0c4ab0 con 0x7ff1ec0c96d0 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f4600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ec0c9c50 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f1c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1ec0c92f0 0x7ff1ec06ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f2600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff1ec0c7bc0 0x7ff1ec151440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54322/0 (socket says 172.21.15.146:54322) 2024-09-16T01:51:34.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f2600640 1 -- 172.21.15.146:0/226784868 learned_addr learned my addr 172.21.15.146:0/226784868 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:34.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1f1c00640 1 -- 172.21.15.146:0/226784868 >> v1:172.21.15.161:6789/0 conn(0x7ff1ec0c96d0 legacy=0x7ff1ec06f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.146+0000 7ff1f1c00640 1 -- 172.21.15.146:0/226784868 >> v1:172.21.15.146:6789/0 conn(0x7ff1ec0c7bc0 legacy=0x7ff1ec151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.145+0000 7ff1ebe00640 1 -- 172.21.15.146:0/226784868 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 735666997 0 0) 0x7ff1ec0c4ab0 con 0x7ff1ec0c96d0 2024-09-16T01:51:34.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.146+0000 7ff1f1c00640 1 -- 172.21.15.146:0/226784868 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec0c9ea0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.147+0000 7ff1f1c00640 1 --2- 172.21.15.146:0/226784868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec06ef90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0c9dd0 tx=0x7ff1e002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72375b0c57997484 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:34.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.147+0000 7ff1ebe00640 1 -- 172.21.15.146:0/226784868 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0037070 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.147+0000 7ff1ebe00640 1 -- 172.21.15.146:0/226784868 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e002ae30 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.147+0000 7ff1ebe00640 1 -- 172.21.15.146:0/226784868 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0032640 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.148+0000 7ff1f4600640 1 -- 172.21.15.146:0/226784868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 msgr2=0x7ff1ec06ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.148+0000 7ff1f4600640 1 --2- 172.21.15.146:0/226784868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec06ef90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0c9dd0 tx=0x7ff1e002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.149+0000 7ff1f4600640 1 -- 172.21.15.146:0/226784868 shutdown_connections 2024-09-16T01:51:34.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.149+0000 7ff1f4600640 1 --2- 172.21.15.146:0/226784868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec06ef90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.149+0000 7ff1f4600640 1 -- 172.21.15.146:0/226784868 >> 172.21.15.146:0/226784868 conn(0x7ff1ec0680e0 msgr2=0x7ff1ec070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:34.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.149+0000 7ff1f4600640 1 -- 172.21.15.146:0/226784868 shutdown_connections 2024-09-16T01:51:34.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.149+0000 7ff1f4600640 1 -- 172.21.15.146:0/226784868 wait complete. 2024-09-16T01:51:34.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.151+0000 7ff1f4600640 1 Processor -- start 2024-09-16T01:51:34.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.151+0000 7ff1f4600640 1 -- start start 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1ec0c7bc0 0x7ff1ec151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff1ec0c96d0 0x7ff1ec1f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f4600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec06fe70 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f4600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec0ca460 con 0x7ff1ec0c7bc0 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f4600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ec06fff0 con 0x7ff1ec0c96d0 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54924/0 (socket says 172.21.15.146:54924) 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f1200640 1 -- 172.21.15.146:0/3121690008 learned_addr learned my addr 172.21.15.146:0/3121690008 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:34.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f1c00640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1ec0c7bc0 0x7ff1ec151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.153+0000 7ff1f2600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff1ec0c96d0 0x7ff1ec1f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1200640 1 -- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff1ec0c96d0 msgr2=0x7ff1ec1f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1200640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff1ec0c96d0 0x7ff1ec1f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1200640 1 -- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1ec0c7bc0 msgr2=0x7ff1ec151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1200640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1ec0c7bc0 0x7ff1ec151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1200640 1 -- 172.21.15.146:0/3121690008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec1521c0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1c00640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1ec0c7bc0 0x7ff1ec151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:34.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f1200640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec152d70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800c410 tx=0x7ff1d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:34.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8017070 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d80035f0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80039e0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.154+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec0cb210 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.155+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec0cb7d0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.156+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ec06ef90 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.157+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff1d801a070 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.158+0000 7ff1e9600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff1cc06c2b0 0x7ff1cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.158+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5741+0+0 (secure 0 0 0) 0x7ff1d808a7c0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.158+0000 7ff1f1c00640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff1cc06c2b0 0x7ff1cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.159+0000 7ff1f1c00640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff1cc06c2b0 0x7ff1cc06e770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0034000 tx=0x7ff1e002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:34.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.166+0000 7fc35a200640 1 Processor -- start 2024-09-16T01:51:34.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.166+0000 7fc35a200640 1 -- start start 2024-09-16T01:51:34.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.166+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff1d80588c0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.167+0000 7fc35a200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc354151a90 0x7fc3540c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.167+0000 7fc35a200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3540c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.167+0000 7fc35a200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc354151e70 con 0x7fc3540c6410 2024-09-16T01:51:34.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.167+0000 7fc35a200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc3540c8130 con 0x7fc354151a90 2024-09-16T01:51:34.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.167+0000 7fc35a200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc3540c8410 con 0x7fc3540c5280 2024-09-16T01:51:34.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc353400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc354151a90 0x7fc3540c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc353e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3540c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc352a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc3540c6410 0x7fc354158040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:49700/0 (socket says 172.21.15.146:49700) 2024-09-16T01:51:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc352a00640 1 -- 172.21.15.146:0/427500030 learned_addr learned my addr 172.21.15.146:0/427500030 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc353400640 1 -- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 msgr2=0x7fc3540c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc353400640 1 --2- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3540c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc353400640 1 -- 172.21.15.146:0/427500030 >> v1:172.21.15.161:6789/0 conn(0x7fc3540c6410 legacy=0x7fc354158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc353400640 1 -- 172.21.15.146:0/427500030 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3540c86f0 con 0x7fc354151a90 2024-09-16T01:51:34.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.168+0000 7fc352000640 1 -- 172.21.15.146:0/427500030 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695376746 0 0) 0x7fc354151e70 con 0x7fc3540c6410 2024-09-16T01:51:34.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.169+0000 7fc353400640 1 --2- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc3540c36b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc3440097a0 tx=0x7fc34402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf381de69d88fd2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:34.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.169+0000 7fc352000640 1 -- 172.21.15.146:0/427500030 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc344037070 con 0x7fc354151a90 2024-09-16T01:51:34.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc352000640 1 -- 172.21.15.146:0/427500030 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc344032440 con 0x7fc354151a90 2024-09-16T01:51:34.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc352000640 1 -- 172.21.15.146:0/427500030 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc344032810 con 0x7fc354151a90 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 -- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 msgr2=0x7fc3540c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 --2- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc3540c36b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc3440097a0 tx=0x7fc34402a180 comp rx=0 tx=0).stop 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 -- 172.21.15.146:0/427500030 shutdown_connections 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 --2- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3540c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 --2- 172.21.15.146:0/427500030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc3540c36b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 -- 172.21.15.146:0/427500030 >> 172.21.15.146:0/427500030 conn(0x7fc35406f3c0 msgr2=0x7fc35406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 -- 172.21.15.146:0/427500030 shutdown_connections 2024-09-16T01:51:34.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.170+0000 7fc35a200640 1 -- 172.21.15.146:0/427500030 wait complete. 2024-09-16T01:51:34.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.173+0000 7fc35a200640 1 Processor -- start 2024-09-16T01:51:34.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.173+0000 7fc35a200640 1 -- start start 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.174+0000 7fc35a200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3541f0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc35a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3540c6410 0x7fc354160f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc35a200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc354153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc35a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc354157b20 con 0x7fc3540c6410 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc35a200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc354157840 con 0x7fc354151a90 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc35a200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc3540c3bf0 con 0x7fc3540c5280 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc354153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3541f0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc352a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3540c6410 0x7fc354160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc354153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60244/0 (socket says 172.21.15.146:60244) 2024-09-16T01:51:34.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3541f0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43944/0 (socket says 172.21.15.146:43944) 2024-09-16T01:51:34.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 -- 172.21.15.146:0/227823262 learned_addr learned my addr 172.21.15.146:0/227823262 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 -- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 msgr2=0x7fc3541f0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3541f0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 -- 172.21.15.146:0/227823262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3540c6410 msgr2=0x7fc354160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3540c6410 0x7fc354160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353e00640 1 -- 172.21.15.146:0/227823262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3540c4a60 con 0x7fc354151a90 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.175+0000 7fc353400640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3541f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.176+0000 7fc353e00640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc354153620 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00efc0 tx=0x7fc34c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.176+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c018020 con 0x7fc354151a90 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.176+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc34c014480 con 0x7fc354151a90 2024-09-16T01:51:34.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.176+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c014870 con 0x7fc354151a90 2024-09-16T01:51:34.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.176+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35415af60 con 0x7fc354151a90 2024-09-16T01:51:34.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.176+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3541f0320 con 0x7fc354151a90 2024-09-16T01:51:34.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.178+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3540c71f0 con 0x7fc354151a90 2024-09-16T01:51:34.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.179+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc34c005d10 con 0x7fc354151a90 2024-09-16T01:51:34.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.180+0000 7fc34b400640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc33406c2b0 0x7fc33406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:34.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.180+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5741+0+0 (secure 0 0 0) 0x7fc34c08c090 con 0x7fc354151a90 2024-09-16T01:51:34.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.180+0000 7fc353400640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc33406c2b0 0x7fc33406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:34.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.181+0000 7fc353400640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc33406c2b0 0x7fc33406e770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc344034000 tx=0x7fc34402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:34.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.185+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc34c022030 con 0x7fc354151a90 2024-09-16T01:51:34.205 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:34.203+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:34.230 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:34.230 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:34.230 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:34.230 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:34.230 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:34.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:34.305 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:34.305+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.366+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7ff1ec0c6ae0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.377+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fc3540723f0 con 0x7fc354151a90 2024-09-16T01:51:34.406 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:34.404+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:34.444 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.186579623999997, 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:34.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:34.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.988206753999997, 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.452 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:34.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.879404123, 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.458 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:34.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:34.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:34.463 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.559835489999998, 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.464 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:34.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:34.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.061172749999997, 2024-09-16T01:51:34.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:34.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:163 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.647600066000003, 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063748808000000004, 2024-09-16T01:51:34.476 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 163 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:34.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794259+0000", 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.794262+0000", 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1569999999999998e-06 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822020+0000", 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027758036999999999 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822064+0000", 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3903999999999998e-05 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822080+0000", 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6368e-05 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822251+0000", 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017151499999999999 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822423+0000", 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017141199999999999 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.822573+0000", 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993299999999999 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.835371+0000", 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012797792000000001 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857900+0000", 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022529779 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.857966+0000", 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5825000000000001e-05 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.858008+0000", 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2086000000000003e-05 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:34.482 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.661226974000002, 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:34.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.593122332999997, 2024-09-16T01:51:34.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:34.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:34.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:34.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.450662668, 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:34.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:34.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:34.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.103196445000002, 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:34.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:34.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:544 3.14 3:28ae9428:::100000001f7.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.797822542999999, 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064751230000000007, 2024-09-16T01:51:34.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 544 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644039+0000", 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6619999999999999e-06 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665383+0000", 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021343964 2024-09-16T01:51:34.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665408+0000", 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.561e-05 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665422+0000", 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3947999999999999e-05 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665607+0000", 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018433499999999999 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665831+0000", 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022424399999999999 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665986+0000", 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155654 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.682208+0000", 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016221920000000001 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708669+0000", 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026460513000000001 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708734+0000", 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4888000000000006e-05 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708788+0000", 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4491999999999999e-05 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:549 3.11 3:8ae93890:::100000001f7.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.755622521999999, 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081050590000000006, 2024-09-16T01:51:34.517 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 549 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686239+0000", 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754451+0000", 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068212138000000005 2024-09-16T01:51:34.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754474+0000", 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2733000000000001e-05 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754483+0000", 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1349999999999998e-06 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754562+0000", 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8319999999999996e-05 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754725+0000", 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016298699999999999 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754875+0000", 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149942 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.762160+0000", 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072859630000000003 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767169+0000", 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050088609999999999 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767231+0000", 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1158e-05 2024-09-16T01:51:34.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767288+0000", 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7108000000000003e-05 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:606 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.167208474999999, 2024-09-16T01:51:34.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058221427999999999, 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 606 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274651+0000", 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.274653+0000", 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2199999999999999e-06 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322327+0000", 2024-09-16T01:51:34.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047674080000000001 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322357+0000", 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9813000000000001e-05 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322369+0000", 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2074e-05 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322494+0000", 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012469 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322653+0000", 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158987 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.322803+0000", 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150433 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331674+0000", 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0088701219999999994 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332772+0000", 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001098552 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332829+0000", 2024-09-16T01:51:34.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6941000000000003e-05 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332873+0000", 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3516000000000001e-05 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:626 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.977216599999998, 2024-09-16T01:51:34.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057812853999999997, 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 626 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464643+0000", 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.464645+0000", 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9759999999999998e-06 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505192+0000", 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040546806999999997 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505211+0000", 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9386e-05 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505220+0000", 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4810000000000006e-06 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505316+0000", 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6748000000000003e-05 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505479+0000", 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016267300000000001 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.505668+0000", 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018906700000000001 2024-09-16T01:51:34.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.520824+0000", 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015155686999999999 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522336+0000", 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015126359999999999 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522404+0000", 2024-09-16T01:51:34.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7174000000000004e-05 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.522456+0000", 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2219000000000003e-05 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:629 3.13 3:ca345efb:::100000001f7.00000072:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:34.535 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.933067897000001, 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073672847999999999, 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 629 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:34.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508793+0000", 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5650000000000001e-06 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536849+0000", 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028055429999999999 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536888+0000", 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9128999999999998e-05 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536904+0000", 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6223e-05 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537082+0000", 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017825600000000001 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537242+0000", 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159569 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537394+0000", 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015228 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.548370+0000", 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010976324000000001 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582365+0000", 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994745999999999 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582423+0000", 2024-09-16T01:51:34.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7357e-05 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582465+0000", 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1968999999999997e-05 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:700 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.308217464999998, 2024-09-16T01:51:34.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074697384000000006, 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 700 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133644+0000", 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159468+0000", 2024-09-16T01:51:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025824323 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159551+0000", 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2286999999999993e-05 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159561+0000", 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0307e-05 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159851+0000", 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029012100000000001 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160192+0000", 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034104600000000001 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160444+0000", 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025126400000000001 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.197238+0000", 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036794265999999999 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208261+0000", 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0110233 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208304+0000", 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2833999999999997e-05 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208339+0000", 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5556000000000003e-05 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:776 3.11 3:887bb7a4:::100000001f7.00000005:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.245175911, 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090355046999999994, 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.547 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 776 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196686+0000", 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5289999999999998e-06 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206873+0000", 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010186855999999999 2024-09-16T01:51:34.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206895+0000", 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1549e-05 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206907+0000", 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2884000000000001e-05 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207056+0000", 2024-09-16T01:51:34.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014810499999999999 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207156+0000", 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010007 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207302+0000", 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014591499999999999 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263823+0000", 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056521543 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286896+0000", 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023073327000000001 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286980+0000", 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3869999999999995e-05 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.287039+0000", 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8399000000000001e-05 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.218905081000001, 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.553 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:788 3.2d 3:b61340bb:::100000001f7.00000011:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.146067296, 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058010644, 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 788 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295792+0000", 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.295795+0000", 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4509999999999999e-06 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338094+0000", 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042298997999999997 2024-09-16T01:51:34.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338113+0000", 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9100999999999999e-05 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338121+0000", 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1370000000000002e-06 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338192+0000", 2024-09-16T01:51:34.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1515999999999998e-05 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338283+0000", 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0276000000000001e-05 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.338420+0000", 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013750000000000001 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.347191+0000", 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087705330000000005 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353679+0000", 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064878189999999997 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353751+0000", 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2193999999999994e-05 2024-09-16T01:51:34.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.353803+0000", 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2119e-05 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:862 3.9 3:93def555:::100000001f7.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.405020517000001, 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063151895999999999, 2024-09-16T01:51:34.565 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 862 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036839+0000", 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.036841+0000", 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.075e-06 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:34.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069065+0000", 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032224153999999998 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069093+0000", 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7797000000000002e-05 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069107+0000", 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3594000000000001e-05 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069222+0000", 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115179 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069335+0000", 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112748 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.069508+0000", 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173692 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.089364+0000", 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019855866 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099875+0000", 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511177 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099941+0000", 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5508999999999999e-05 2024-09-16T01:51:34.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.099991+0000", 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0105000000000002e-05 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:34.571 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:34.573 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:34.573 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:34.571+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:34.674 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:34.673+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:34.688 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:34.688 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:34.688 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:34.689 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:34.698 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:34.782 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:34.781+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:34.844 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:34.844 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:34.844 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:34.844 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:34.844 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:34.852 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:34.893 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:34.892+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:34.977 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:34.975+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:34.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.986+0000 7ff1e9600640 1 -- 172.21.15.146:0/3121690008 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v75) ==== 133+0+0 (secure 0 0 0) 0x7ff1d805cbc0 con 0x7ff1ec0c92f0 2024-09-16T01:51:34.988 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T01:51:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.988+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff1cc06c2b0 msgr2=0x7ff1cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.988+0000 7ff1f4600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff1cc06c2b0 0x7ff1cc06e770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0034000 tx=0x7ff1e002e000 comp rx=0 tx=0).stop 2024-09-16T01:51:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.988+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 msgr2=0x7ff1ec152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.988+0000 7ff1f4600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec152d70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800c410 tx=0x7ff1d800c8d0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 shutdown_connections 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff1cc06c2b0 0x7ff1cc06e770 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff1ec0c96d0 0x7ff1ec1f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1ec0c92f0 0x7ff1ec152d70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 --2- 172.21.15.146:0/3121690008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1ec0c7bc0 0x7ff1ec151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 >> 172.21.15.146:0/3121690008 conn(0x7ff1ec0680e0 msgr2=0x7ff1ec0c84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 shutdown_connections 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7ff1f4600640 1 -- 172.21.15.146:0/3121690008 wait complete. 2024-09-16T01:51:34.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.989+0000 7fc34b400640 1 -- 172.21.15.146:0/227823262 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v75) ==== 91+0+0 (secure 0 0 0) 0x7fc34c010040 con 0x7fc354151a90 2024-09-16T01:51:34.993 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc33406c2b0 msgr2=0x7fc33406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc33406c2b0 0x7fc33406e770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc344034000 tx=0x7fc34402e000 comp rx=0 tx=0).stop 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 msgr2=0x7fc354153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc354153620 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00efc0 tx=0x7fc34c00c490 comp rx=0 tx=0).stop 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 shutdown_connections 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc33406c2b0 0x7fc33406e770 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc354151a90 0x7fc354153620 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3540c6410 0x7fc354160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 --2- 172.21.15.146:0/227823262 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc3540c5280 0x7fc3541f0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:34.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 >> 172.21.15.146:0/227823262 conn(0x7fc35406f3c0 msgr2=0x7fc3540718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 shutdown_connections 2024-09-16T01:51:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:34.992+0000 7fc35a200640 1 -- 172.21.15.146:0/227823262 wait complete. 2024-09-16T01:51:35.002 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:35.002 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:35.002 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:35.002 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:35.002 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.002 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.450264376, 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.004 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.010 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.203230795000003, 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.011 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.012 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:35.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:35.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.144071709000002, 2024-09-16T01:51:35.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:35.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:35.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.232493396999999, 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.026 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:35.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.032 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.033 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.188233167, 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.034 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:35.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:35.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.040 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.157733346999997, 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.041 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:35.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.046 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:35.047 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:35.047 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.047 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.047 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.047 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:35.047 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.048 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.116177825000001, 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:35.049 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.050 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.051 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:35.052 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.053 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.054 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:35.055 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:187 3.1b 3:da9dcad1:::10000000002.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.978131130999998, 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051746425999999998, 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.056 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 187 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.057 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021754+0000", 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.021757+0000", 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.368e-06 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057249+0000", 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035492903999999999 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057279+0000", 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.995e-05 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057293+0000", 2024-09-16T01:51:35.059 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3890000000000001e-05 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057365+0000", 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1260000000000006e-05 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057504+0000", 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000138959 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.060 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.057631+0000", 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000127236 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.062710+0000", 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050794109999999998 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.061 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073372+0000", 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010662321000000001 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073439+0000", 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6614999999999998e-05 2024-09-16T01:51:35.062 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:52.073501+0000", 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1512000000000003e-05 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:35.063 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.859171404000001, 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.064 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:35.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:35.068 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:35.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.070 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.071 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.824375105000001, 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.072 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:35.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:35.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.077 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.079 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.740926152, 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.080 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:35.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:35.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.541623365, 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.086 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:51:35.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:51:35.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.396746427, 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.092 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:51:35.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.094 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:51:35.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.097 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.356102003, 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.098 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:51:35.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:51:35.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.103 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.198159169, 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.104 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:51:35.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:51:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.109 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.073031908000001, 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.110 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:51:35.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:51:35.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:576 3.1b 3:d81af1ca:::100000001f7.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.115 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.964229222, 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052702763999999999, 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 576 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035656+0000", 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.035658+0000", 2024-09-16T01:51:35.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.03e-06 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075189+0000", 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039530957999999998 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075204+0000", 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5071e-05 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075214+0000", 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8109999999999997e-06 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075299+0000", 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4807000000000003e-05 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075401+0000", 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010220700000000001 2024-09-16T01:51:35.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.075495+0000", 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4006999999999996e-05 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.078964+0000", 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003468741 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088273+0000", 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0093088089999999995 2024-09-16T01:51:35.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088311+0000", 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8402000000000003e-05 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088359+0000", 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7920999999999997e-05 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:598 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.784149356, 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054052254000000001, 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.124 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 598 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215738+0000", 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.526e-06 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250373+0000", 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034634160999999997 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250391+0000", 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7986e-05 2024-09-16T01:51:35.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250400+0000", 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1439999999999992e-06 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250455+0000", 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5578999999999998e-05 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250562+0000", 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106717 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250665+0000", 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010307200000000001 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.253888+0000", 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032231019999999998 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269702+0000", 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015813473000000002 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269742+0000", 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0154000000000002e-05 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269788+0000", 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.634e-05 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.036231977, 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.130 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:51:35.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:35.136 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 13.873206335000001, 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:51:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.142 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:35.144 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:35.145 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:35.145 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:35.145 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:35.145 INFO:tasks.workunit.client.0.smithi146.stdout:iozone test complete. 2024-09-16T01:51:35.146 INFO:tasks.workunit.client.0.smithi146.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-09-16T01:51:35.146 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:35.142+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:35.205 INFO:tasks.workunit.client.0.smithi146.stdout: Iozone: Performance Test of File I/O 2024-09-16T01:51:35.205 INFO:tasks.workunit.client.0.smithi146.stdout: Version $Revision: 3.494 $ 2024-09-16T01:51:35.205 INFO:tasks.workunit.client.0.smithi146.stdout: Compiled for 64 bit mode. 2024-09-16T01:51:35.205 INFO:tasks.workunit.client.0.smithi146.stdout: Build: linux-AMD64 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Alexey Skidanov, Sudhir Kumar. 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Run began: Mon Sep 16 01:51:35 2024 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Include close in write timing 2024-09-16T01:51:35.206 INFO:tasks.workunit.client.0.smithi146.stdout: Include fsync in write timing 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: File size set to 10485760 kB 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Record Size 1024 kB 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Output is in kBytes/sec 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Time Resolution = 0.000001 seconds. 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Processor cache size set to 1024 kBytes. 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Processor cache line size set to 32 bytes. 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: File stride size set to 17 * record size. 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Throughput test with 1 process 2024-09-16T01:51:35.207 INFO:tasks.workunit.client.0.smithi146.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-09-16T01:51:35.244 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:35.242+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:35.249 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:35.250 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:35.250 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:35.250 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:35.259 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:35.345 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:35.344+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:35.363 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:35.363 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:35.364 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:35.364 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:35.364 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:35.372 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:35.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:35.456 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:35.455+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:35.518 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:35.518 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:35.518 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:35.518 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:35.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:19 3.1f 3:f8cab2eb:::10000000002.00000010:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.117675038000002, 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075358271000000004, 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 19 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398086+0000", 2024-09-16T01:51:35.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2149999999999998e-06 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438214+0000", 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040127599999999999 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438231+0000", 2024-09-16T01:51:35.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7028e-05 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438238+0000", 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438293+0000", 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5007999999999999e-05 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.522 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438402+0000", 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000109379 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438491+0000", 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8774000000000001e-05 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.442442+0000", 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039511320000000004 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473344+0000", 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030902466999999999 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473394+0000", 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9264000000000001e-05 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473438+0000", 2024-09-16T01:51:35.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4579000000000001e-05 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:24 3.0 3:028eacff:::10000000002.00000015:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.528 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.073793643999998, 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.073041954000000006, 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 24 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.529 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441963+0000", 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7379999999999999e-06 2024-09-16T01:51:35.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508113+0000", 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066149523000000002 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508127+0000", 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4678000000000001e-05 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508135+0000", 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4959999999999999e-06 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508200+0000", 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5062999999999997e-05 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508374+0000", 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000174322 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508512+0000", 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013740099999999999 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514816+0000", 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063044199999999998 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514914+0000", 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7590000000000006e-05 2024-09-16T01:51:35.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514951+0000", 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7348999999999998e-05 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.515003+0000", 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2373999999999997e-05 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.992554882999997, 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.535 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:51:35.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.540 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:50 3.17 3:eaf1bb0b:::10000000002.0000002f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.799099490000003, 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058893003999999999, 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.541 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 50 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716656+0000", 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.716657+0000", 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9290000000000001e-06 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747413+0000", 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030755714 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747428+0000", 2024-09-16T01:51:35.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4452e-05 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747438+0000", 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0436e-05 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747490+0000", 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1632000000000003e-05 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747608+0000", 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000117934 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.747716+0000", 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107953 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.751490+0000", 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037745750000000001 2024-09-16T01:51:35.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775454+0000", 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023963661000000001 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775504+0000", 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0003999999999998e-05 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.775549+0000", 2024-09-16T01:51:35.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4713999999999999e-05 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:122 3.0 3:00eacacc:::10000000002.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.145841636999997, 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068671870999999995, 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.547 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 122 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.548 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369915+0000", 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5990000000000001e-06 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434232+0000", 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064317038000000007 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434250+0000", 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8028000000000001e-05 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434258+0000", 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6720000000000004e-06 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434311+0000", 2024-09-16T01:51:35.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2914999999999997e-05 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434411+0000", 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100045 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434501+0000", 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494000000000005e-05 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.437130+0000", 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026284059999999998 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438521+0000", 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013915290000000001 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438553+0000", 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1992999999999997e-05 2024-09-16T01:51:35.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438585+0000", 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2152e-05 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:134 3.17 3:e85118c5:::10000000002.00000083:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.051137767999997, 2024-09-16T01:51:35.553 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061709079999999999, 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 134 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464617+0000", 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.464620+0000", 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8499999999999998e-06 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518730+0000", 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054109808000000002 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518748+0000", 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8289000000000001e-05 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518756+0000", 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8539999999999997e-06 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518809+0000", 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2879e-05 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.518914+0000", 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104719 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.519010+0000", 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6248000000000004e-05 2024-09-16T01:51:35.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.522952+0000", 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039422440000000001 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526248+0000", 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003296003 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526290+0000", 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1708000000000001e-05 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.526326+0000", 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6477999999999999e-05 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:472 3.31 3:8c3036ad:::10000000002.000000d5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:35.559 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.572078546, 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060972124000000003, 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 472 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:35.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943677+0000", 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.943678+0000", 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5549999999999999e-06 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974973+0000", 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031295264000000003 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974987+0000", 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3349e-05 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.974994+0000", 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3520000000000001e-06 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975074+0000", 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9864000000000006e-05 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975140+0000", 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6377999999999994e-05 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.975233+0000", 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2359000000000001e-05 2024-09-16T01:51:35.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:59.978088+0000", 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028552489999999998 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004578+0000", 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026490519000000001 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004615+0000", 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6455999999999998e-05 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.004649+0000", 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3779000000000003e-05 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.565 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:483 3.27 3:e6494e75:::10000000002.000000e0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 35.41897445, 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059372405000000003, 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 483 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096781+0000", 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.096782+0000", 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7370000000000001e-06 2024-09-16T01:51:35.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137952+0000", 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041169532000000002 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137964+0000", 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1725999999999999e-05 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.137988+0000", 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4020999999999999e-05 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138046+0000", 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8585000000000002e-05 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138124+0000", 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7312999999999997e-05 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.138239+0000", 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000115278 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.143288+0000", 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050496630000000002 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156064+0000", 2024-09-16T01:51:35.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012775269000000001 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156103+0000", 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8957e-05 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.156153+0000", 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0324000000000001e-05 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.571 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:545 3.38 3:1d8b64ca:::100000001f7.0000001e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.888099904000001, 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075800996999999995, 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 545 2024-09-16T01:51:35.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627656+0000", 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.176e-06 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655590+0000", 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027933824999999999 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655601+0000", 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1270999999999999e-05 2024-09-16T01:51:35.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655608+0000", 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.55e-06 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655659+0000", 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0946999999999999e-05 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655756+0000", 2024-09-16T01:51:35.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7448000000000006e-05 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655857+0000", 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100863 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.659273+0000", 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034161180000000001 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703393+0000", 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044119393 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703427+0000", 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4063000000000002e-05 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703456+0000", 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9343e-05 2024-09-16T01:51:35.577 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:546 3.38 3:1d0ca575:::100000001f7.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.854835494, 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068250515999999997, 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.578 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 546 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660921+0000", 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.539e-06 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709606+0000", 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048684573000000002 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709617+0000", 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1447999999999999e-05 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709622+0000", 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-06 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709691+0000", 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8879000000000002e-05 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709788+0000", 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6768999999999995e-05 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709893+0000", 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010466799999999999 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.715292+0000", 2024-09-16T01:51:35.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005399165 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729109+0000", 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013817352 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729143+0000", 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4059e-05 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729170+0000", 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6764e-05 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:574 3.27 3:e46e0c7f:::100000001f7.0000003b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.511098938, 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084267634999999994, 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.584 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 574 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004659+0000", 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5129999999999999e-06 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032641+0000", 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027982080999999999 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032656+0000", 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5699999999999999e-05 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032666+0000", 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5599999999999999e-06 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032719+0000", 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3066999999999999e-05 2024-09-16T01:51:35.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032827+0000", 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010841900000000001 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032929+0000", 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101128 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.036882+0000", 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039538949999999998 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088843+0000", 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051960682000000001 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088878+0000", 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.485e-05 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088924+0000", 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5739999999999999e-05 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:590 3.29 3:966924c4:::100000001f7.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.399587913000001, 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074607536000000002, 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 590 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116170+0000", 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162459+0000", 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046289624000000001 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162473+0000", 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3776e-05 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162481+0000", 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2889999999999998e-06 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162559+0000", 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7101000000000006e-05 2024-09-16T01:51:35.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162675+0000", 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116933 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162764+0000", 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8343000000000004e-05 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.167505+0000", 2024-09-16T01:51:35.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047412519999999996 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190706+0000", 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023201366000000001 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190742+0000", 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5757999999999999e-05 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190775+0000", 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2484999999999998e-05 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:625 3.3d 3:bee2a002:::100000001f7.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.075111483000001, 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067884265999999999, 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.596 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 625 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440646+0000", 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0710000000000002e-06 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502844+0000", 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062197927 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502866+0000", 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2580999999999999e-05 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502877+0000", 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0823e-05 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502939+0000", 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2360000000000006e-05 2024-09-16T01:51:35.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.503095+0000", 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015614099999999999 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.503243+0000", 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014767400000000001 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.507276+0000", 2024-09-16T01:51:35.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040332340000000001 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508460+0000", 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011837740000000001 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508496+0000", 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5540999999999997e-05 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508528+0000", 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2140000000000001e-05 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:631 3.23 3:c4d03f6f:::100000001f7.00000074:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.520808+0000", 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.994947042, 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.063082532999999996, 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.602 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 631 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.520808+0000", 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.520808+0000", 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.520809+0000", 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.229e-06 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539695+0000", 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.018885745999999998 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539714+0000", 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8668000000000002e-05 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539722+0000", 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9510000000000006e-06 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539777+0000", 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5379e-05 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539901+0000", 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012439499999999999 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.539997+0000", 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5222000000000004e-05 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.544982+0000", 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049858740000000004 2024-09-16T01:51:35.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.583651+0000", 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038668181000000003 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.583820+0000", 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016971999999999999 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.583891+0000", 2024-09-16T01:51:35.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0167999999999996e-05 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:670 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.635058759, 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066678565999999995, 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.608 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 670 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880698+0000", 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2890000000000001e-06 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903674+0000", 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022976465000000001 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903699+0000", 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4675e-05 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903712+0000", 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2935000000000001e-05 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903788+0000", 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6676999999999998e-05 2024-09-16T01:51:35.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903928+0000", 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013916300000000001 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.904044+0000", 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011618200000000001 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.907487+0000", 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034430839999999999 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947289+0000", 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039801925000000002 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947330+0000", 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.121e-05 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947375+0000", 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4960999999999999e-05 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:685 3.27 3:e74bd690:::100000001f7.000000aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.986565+0000", 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.529189926000001, 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055646205999999997, 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.614 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 685 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.986565+0000", 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.986565+0000", 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.986567+0000", 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7770000000000001e-06 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015334+0000", 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028767478999999999 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015358+0000", 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3340999999999999e-05 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015368+0000", 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0438999999999999e-05 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015432+0000", 2024-09-16T01:51:35.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4321999999999998e-05 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015550+0000", 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000117713 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.015655+0000", 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010514900000000001 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019015+0000", 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033593020000000002 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.042112+0000", 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023097751999999999 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.042166+0000", 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3940999999999997e-05 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.042211+0000", 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4990999999999997e-05 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:687 3.29 3:95dcb5ee:::100000001f7.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.019206+0000", 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.496548847, 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057193637999999998, 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.620 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 687 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019206+0000", 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019206+0000", 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.019209+0000", 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3910000000000001e-06 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066734+0000", 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047525322000000002 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066757+0000", 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2759e-05 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066766+0000", 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4979999999999995e-06 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.066889+0000", 2024-09-16T01:51:35.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000123125 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.067008+0000", 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118288 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.067124+0000", 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116496 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.070545+0000", 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034212539999999999 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.076316+0000", 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0057710089999999997 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.076358+0000", 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1724000000000002e-05 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.076400+0000", 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1771999999999999e-05 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:712 3.1e 3:7933c15e:::100000001f7.000000c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.229839946999999, 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067308615000000002, 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.626 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 712 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285917+0000", 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0370000000000001e-06 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330533+0000", 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044615783999999999 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330550+0000", 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6878999999999999e-05 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330558+0000", 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7000000000000008e-06 2024-09-16T01:51:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330621+0000", 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3100000000000002e-05 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330773+0000", 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015215199999999999 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:35.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330932+0000", 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015931100000000001 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.335841+0000", 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004909348 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353141+0000", 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017299833000000001 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353182+0000", 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0849000000000001e-05 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353224+0000", 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1622000000000002e-05 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.632 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:861 3.31 3:8d161915:::100000001f7.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.034456+0000", 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.481299143999999, 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058433348000000003, 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 861 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.034456+0000", 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.034456+0000", 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.034458+0000", 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1749999999999999e-06 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058502+0000", 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024044157 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058517+0000", 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4609e-05 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058524+0000", 2024-09-16T01:51:35.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9970000000000001e-06 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058570+0000", 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6400000000000003e-05 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058635+0000", 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4572000000000005e-05 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:35.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.058727+0000", 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1694999999999995e-05 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.061959+0000", 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032324039999999999 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.092797+0000", 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.03083851 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.092841+0000", 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3237e-05 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.092889+0000", 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8591999999999998e-05 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:872 3.18 3:19643448:::100000001f7.00000065:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.366110292, 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.069066814000000004, 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 872 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:35.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149650+0000", 2024-09-16T01:51:35.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8899999999999998e-06 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209584+0000", 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059934556 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209621+0000", 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6378999999999998e-05 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:35.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209629+0000", 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6410000000000008e-06 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209715+0000", 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6013999999999993e-05 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209807+0000", 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2020000000000003e-05 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209966+0000", 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158889 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.216152+0000", 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006185467 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218614+0000", 2024-09-16T01:51:35.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024625630000000001 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218669+0000", 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5104e-05 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218712+0000", 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2290999999999999e-05 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.644 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:35.645 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:35.645 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:35.647 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:35.647 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:35.645+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:35.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.578+0000 7f7d81c00640 1 Processor -- start 2024-09-16T01:51:35.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.578+0000 7f7d81c00640 1 -- start start 2024-09-16T01:51:35.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d81c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c0c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d81c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c0702a0 con 0x7f7d7c154bc0 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d81c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c070580 con 0x7f7d7c0c5280 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d81c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7c070ad0 con 0x7f7d7c070eb0 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d7aa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7d7c154bc0 0x7f7d7c0ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54332/0 (socket says 172.21.15.146:54332) 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d7aa00640 1 -- 172.21.15.146:0/3527840927 learned_addr learned my addr 172.21.15.146:0/3527840927 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d7a000640 1 --2- 172.21.15.146:0/3527840927 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c0c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d79600640 1 -- 172.21.15.146:0/3527840927 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048074027 0 0) 0x7f7d7c070580 con 0x7f7d7c0c5280 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d7a000640 1 -- 172.21.15.146:0/3527840927 >> v1:172.21.15.161:6789/0 conn(0x7f7d7c0c5280 legacy=0x7f7d7c165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d7a000640 1 -- 172.21.15.146:0/3527840927 >> v1:172.21.15.146:6789/0 conn(0x7f7d7c154bc0 legacy=0x7f7d7c0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.580+0000 7f7d7a000640 1 -- 172.21.15.146:0/3527840927 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c0ca6c0 con 0x7f7d7c070eb0 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.581+0000 7f7d7a000640 1 --2- 172.21.15.146:0/3527840927 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c0c96e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7d680097d0 tx=0x7f7d6802a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc9483d322667654 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.581+0000 7f7d79600640 1 -- 172.21.15.146:0/3527840927 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68037070 con 0x7f7d7c070eb0 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d79600640 1 -- 172.21.15.146:0/3527840927 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d68032440 con 0x7f7d7c070eb0 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d79600640 1 -- 172.21.15.146:0/3527840927 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68032810 con 0x7f7d7c070eb0 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d81c00640 1 -- 172.21.15.146:0/3527840927 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 msgr2=0x7f7d7c0c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d81c00640 1 --2- 172.21.15.146:0/3527840927 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c0c96e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7d680097d0 tx=0x7f7d6802a200 comp rx=0 tx=0).stop 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d81c00640 1 -- 172.21.15.146:0/3527840927 shutdown_connections 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d81c00640 1 --2- 172.21.15.146:0/3527840927 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c0c96e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d81c00640 1 -- 172.21.15.146:0/3527840927 >> 172.21.15.146:0/3527840927 conn(0x7f7d7c06c2a0 msgr2=0x7f7d7c067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.582+0000 7f7d81c00640 1 -- 172.21.15.146:0/3527840927 shutdown_connections 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.583+0000 7f7d81c00640 1 -- 172.21.15.146:0/3527840927 wait complete. 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.584+0000 7f7d81c00640 1 Processor -- start 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.584+0000 7f7d81c00640 1 -- start start 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d81c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d81c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 0x7f7d7c146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d81c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d7c154bc0 0x7f7d7c152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d81c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c0ca470 con 0x7f7d7c154bc0 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d81c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c0c8fe0 con 0x7f7d7c0c5280 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d81c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7c0caa30 con 0x7f7d7c070eb0 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.585+0000 7f7d7aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 0x7f7d7c146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c1538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43958/0 (socket says 172.21.15.146:43958) 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 0x7f7d7c146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60250/0 (socket says 172.21.15.146:60250) 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7aa00640 1 -- 172.21.15.146:0/2822157078 learned_addr learned my addr 172.21.15.146:0/2822157078 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d7c154bc0 0x7f7d7c152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 msgr2=0x7f7d7c1538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d7c070eb0 0x7f7d7c1538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d7c154bc0 msgr2=0x7f7d7c152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d7c154bc0 0x7f7d7c152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.586+0000 7f7d7a000640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c0cdfb0 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.587+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 0x7f7d7c146080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7d68031000 tx=0x7f7d680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.587+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68032e60 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.587+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d680307b0 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.587+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68037040 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.587+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7c1e7a40 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.587+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7c0cc9b0 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.588+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7d7c0c8cf0 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.589+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7d68044020 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.590+0000 7f7d72a00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d5c06c2e0 0x7f7d5c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.590+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5741+0+0 (secure 0 0 0) 0x7f7d680ab670 con 0x7f7d7c0c5280 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.590+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d5c06c2e0 0x7f7d5c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.590+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v75) ==== 54+0+16 (secure 0 0 0) 0x7f7d680b1170 con 0x7f7d7c0c5280 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.590+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7d7c1465c0 0x7f7d7c1481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.590+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c0cc760 con 0x7f7d7c1465c0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.591+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d5c06c2e0 0x7f7d5c06e7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c005880 tx=0x7f7d6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.591+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7d7c1465c0 0x7f7d7c1481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.591+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7d7c1465c0 0x7f7d7c1481a0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.592+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c0cc760 con 0x7f7d7c1465c0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.621+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c1530a0 con 0x7f7d7c1465c0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.623+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c1530a0 con 0x7f7d7c1465c0 2024-09-16T01:51:35.656 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:35.657 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:35.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.623+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7d7c141650 0x7f7d7c1f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.623+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c070c90 con 0x7f7d7c141650 2024-09-16T01:51:35.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.624+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7d7c141650 0x7f7d7c1f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.624+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7d7c141650 0x7f7d7c1f38c0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.625+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c070c90 con 0x7f7d7c141650 2024-09-16T01:51:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.654+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c067af0 con 0x7f7d7c141650 2024-09-16T01:51:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.656+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c067af0 con 0x7f7d7c141650 2024-09-16T01:51:35.658 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.656+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7d7c1f4310 0x7f7d7c062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.656+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c0cdc90 con 0x7f7d7c1f4310 2024-09-16T01:51:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.656+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7d7c1f4310 0x7f7d7c062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.657+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7d7c1f4310 0x7f7d7c062630 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.658+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c0cdc90 con 0x7f7d7c1f4310 2024-09-16T01:51:35.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.681+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c062b70 con 0x7f7d7c1f4310 2024-09-16T01:51:35.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.683+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c062b70 con 0x7f7d7c1f4310 2024-09-16T01:51:35.685 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:35.686 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:35.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.684+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7d7c1584e0 0x7f7d7c158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.684+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c062b70 con 0x7f7d7c1584e0 2024-09-16T01:51:35.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.684+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7d7c1584e0 0x7f7d7c158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.684+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7d7c1584e0 0x7f7d7c158920 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.685+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c062b70 con 0x7f7d7c1584e0 2024-09-16T01:51:35.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.699+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c062b70 con 0x7f7d7c1584e0 2024-09-16T01:51:35.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.701+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c062b70 con 0x7f7d7c1584e0 2024-09-16T01:51:35.703 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:35.703 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:35.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.701+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7d7c159190 0x7f7d7c02cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.701+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c062b70 con 0x7f7d7c159190 2024-09-16T01:51:35.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.701+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7d7c159190 0x7f7d7c02cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.702+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7d7c159190 0x7f7d7c02cb50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.703+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c062b70 con 0x7f7d7c159190 2024-09-16T01:51:35.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.716+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c062b70 con 0x7f7d7c159190 2024-09-16T01:51:35.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.718+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c062b70 con 0x7f7d7c159190 2024-09-16T01:51:35.721 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:35.721 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:35.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.719+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7d7c02d5a0 0x7f7d7c206c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.719+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c02d9e0 con 0x7f7d7c02d5a0 2024-09-16T01:51:35.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.719+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7d7c02d5a0 0x7f7d7c206c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.720+0000 7f7d7a000640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7d7c02d5a0 0x7f7d7c206c50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.720+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c02d9e0 con 0x7f7d7c02d5a0 2024-09-16T01:51:35.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.734+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c02d9e0 con 0x7f7d7c02d5a0 2024-09-16T01:51:35.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.736+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c02d9e0 con 0x7f7d7c02d5a0 2024-09-16T01:51:35.738 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:35.738 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:35.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.736+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7d7c2076a0 0x7f7d7c209a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.736+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c20c970 con 0x7f7d7c2076a0 2024-09-16T01:51:35.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.737+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7d7c2076a0 0x7f7d7c209a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.737+0000 7f7d7b400640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7d7c2076a0 0x7f7d7c209a90 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.738+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c20c970 con 0x7f7d7c2076a0 2024-09-16T01:51:35.747 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:35.746+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:35.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.752+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c20c970 con 0x7f7d7c2076a0 2024-09-16T01:51:35.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.754+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c20c970 con 0x7f7d7c2076a0 2024-09-16T01:51:35.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:35.757 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:35.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.755+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7d7c20a540 0x7f7d7c20a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:35.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.755+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7d7c20fd10 con 0x7f7d7c20a540 2024-09-16T01:51:35.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.755+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7d7c20a540 0x7f7d7c20a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:35.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.755+0000 7f7d7aa00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7d7c20a540 0x7f7d7c20a980 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:35.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.756+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d7c20fd10 con 0x7f7d7c20a540 2024-09-16T01:51:35.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.770+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7d7c20af30 con 0x7f7d7c20a540 2024-09-16T01:51:35.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d72a00640 1 -- 172.21.15.146:0/2822157078 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7d7c20af30 con 0x7f7d7c20a540 2024-09-16T01:51:35.774 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:35.775 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:35.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7d7c1465c0 msgr2=0x7f7d7c1481a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7d7c1465c0 0x7f7d7c1481a0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7d7c141650 msgr2=0x7f7d7c1f38c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7d7c141650 0x7f7d7c1f38c0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7d7c1f4310 msgr2=0x7f7d7c062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7d7c1f4310 0x7f7d7c062630 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7d7c1584e0 msgr2=0x7f7d7c158920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7d7c1584e0 0x7f7d7c158920 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7d7c159190 msgr2=0x7f7d7c02cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7d7c159190 0x7f7d7c02cb50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7d7c02d5a0 msgr2=0x7f7d7c206c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7d7c02d5a0 0x7f7d7c206c50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7d7c2076a0 msgr2=0x7f7d7c209a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7d7c2076a0 0x7f7d7c209a90 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7d7c20a540 msgr2=0x7f7d7c20a980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7d7c20a540 0x7f7d7c20a980 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d7b400640 1 -- 172.21.15.146:0/2822157078 reap_dead start 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d7b400640 1 -- 172.21.15.146:0/2822157078 reap_dead start 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d7b400640 1 -- 172.21.15.146:0/2822157078 reap_dead start 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d7b400640 1 -- 172.21.15.146:0/2822157078 reap_dead start 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d7b400640 1 -- 172.21.15.146:0/2822157078 reap_dead start 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.772+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d5c06c2e0 msgr2=0x7f7d5c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d5c06c2e0 0x7f7d5c06e7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c005880 tx=0x7f7d6c009040 comp rx=0 tx=0).stop 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 msgr2=0x7f7d7c146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 0x7f7d7c146080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7d68031000 tx=0x7f7d680046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 shutdown_connections 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7d5c06c2e0 0x7f7d5c06e7a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 --2- 172.21.15.146:0/2822157078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d7c0c5280 0x7f7d7c146080 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:35.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 >> 172.21.15.146:0/2822157078 conn(0x7f7d7c06c2a0 msgr2=0x7f7d7c0c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:35.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 shutdown_connections 2024-09-16T01:51:35.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:35.773+0000 7f7d81c00640 1 -- 172.21.15.146:0/2822157078 wait complete. 2024-09-16T01:51:35.793 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:35.794 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:35.803 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:35.888 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:35.887+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:35.948 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:35.948 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:35.949 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:35.947+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:35.957 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:36.050 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:36.048+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:36.100 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:36.100 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:36.109 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:36.150 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:36.149+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:36.213 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:36.214 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:36.222 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:36.266 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:36.265+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:36.326 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:36.326 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:36.335 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:36.353 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:36.351+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:36.454 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:36.453+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:36.480 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:36.481 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:36.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:36.554 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:36.553+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:36.655 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:36.655 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:36.656 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:36.655+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:36.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:36.756 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:36.754+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:36.836 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:36.836 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:36.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:36.857 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:36.856+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:36.957 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:36.956+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:37.055 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:37.055 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:37.059 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:37.058+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:37.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:37.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:51:37.160 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:37.158+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:37.261 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:37.260+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:37.275 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:37.276 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:37.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:37.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.317+0000 7f714a800640 1 Processor -- start 2024-09-16T01:51:37.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.317+0000 7f714a800640 1 -- start start 2024-09-16T01:51:37.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f714a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f71440c6410 0x7f7144158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f714a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7144151e70 con 0x7f71440c5280 2024-09-16T01:51:37.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f714a800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71440c8130 con 0x7f7144151a90 2024-09-16T01:51:37.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f714a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f71440c8410 con 0x7f71440c6410 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f7143e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f71440c5280 0x7f7144161fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54344/0 (socket says 172.21.15.146:54344) 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f7143400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7144151a90 0x7f71440c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:49708/0 (socket says 172.21.15.146:49708) 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f7143e00640 1 -- 172.21.15.146:0/1544637737 learned_addr learned my addr 172.21.15.146:0/1544637737 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f7142a00640 1 --2- 172.21.15.146:0/1544637737 >> v2:172.21.15.146:3300/0 conn(0x7f71440c6410 0x7f7144158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f7142000640 1 -- 172.21.15.146:0/1544637737 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689698707 0 0) 0x7f71440c8130 con 0x7f7144151a90 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.319+0000 7f7142000640 1 -- 172.21.15.146:0/1544637737 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7128005730 con 0x7f7144151a90 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.320+0000 7f7142a00640 1 -- 172.21.15.146:0/1544637737 >> v1:172.21.15.161:6789/0 conn(0x7f7144151a90 legacy=0x7f71440c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.320+0000 7f7142a00640 1 -- 172.21.15.146:0/1544637737 >> v1:172.21.15.146:6789/0 conn(0x7f71440c5280 legacy=0x7f7144161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.320+0000 7f7142000640 1 -- 172.21.15.146:0/1544637737 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742655121 0 0) 0x7f7144151e70 con 0x7f71440c5280 2024-09-16T01:51:37.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.320+0000 7f7142a00640 1 -- 172.21.15.146:0/1544637737 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71440c86f0 con 0x7f71440c6410 2024-09-16T01:51:37.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.321+0000 7f7142a00640 1 --2- 172.21.15.146:0/1544637737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f7144158040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f71300097a0 tx=0x7f713002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fc882436acac1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.321+0000 7f7142000640 1 -- 172.21.15.146:0/1544637737 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7130037070 con 0x7f71440c6410 2024-09-16T01:51:37.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.321+0000 7f7142000640 1 -- 172.21.15.146:0/1544637737 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7130032440 con 0x7f71440c6410 2024-09-16T01:51:37.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.321+0000 7f7142000640 1 -- 172.21.15.146:0/1544637737 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71300327f0 con 0x7f71440c6410 2024-09-16T01:51:37.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 -- 172.21.15.146:0/1544637737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 msgr2=0x7f7144158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 --2- 172.21.15.146:0/1544637737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f7144158040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f71300097a0 tx=0x7f713002a230 comp rx=0 tx=0).stop 2024-09-16T01:51:37.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 -- 172.21.15.146:0/1544637737 shutdown_connections 2024-09-16T01:51:37.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 --2- 172.21.15.146:0/1544637737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f7144158040 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 -- 172.21.15.146:0/1544637737 >> 172.21.15.146:0/1544637737 conn(0x7f714406f3c0 msgr2=0x7f714406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:37.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 -- 172.21.15.146:0/1544637737 shutdown_connections 2024-09-16T01:51:37.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.322+0000 7f714a800640 1 -- 172.21.15.146:0/1544637737 wait complete. 2024-09-16T01:51:37.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.324+0000 7f714a800640 1 Processor -- start 2024-09-16T01:51:37.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.324+0000 7f714a800640 1 -- start start 2024-09-16T01:51:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f714a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 0x7f71441f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f714a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f71441f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f714a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7144151a90 0x7f7144158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f714a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7144157b20 con 0x7f71440c6410 2024-09-16T01:51:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f714a800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7144157840 con 0x7f7144151a90 2024-09-16T01:51:37.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f714a800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7144158580 con 0x7f71440c5280 2024-09-16T01:51:37.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f7143400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 0x7f71441f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f7143e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7144151a90 0x7f7144158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f7143400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 0x7f71441f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43972/0 (socket says 172.21.15.146:43972) 2024-09-16T01:51:37.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f7143400640 1 -- 172.21.15.146:0/397041124 learned_addr learned my addr 172.21.15.146:0/397041124 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:37.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.325+0000 7f7142a00640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f71441f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143400640 1 -- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7144151a90 msgr2=0x7f7144158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143400640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7144151a90 0x7f7144158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143400640 1 -- 172.21.15.146:0/397041124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 msgr2=0x7f71441f3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143400640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f71441f3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143400640 1 -- 172.21.15.146:0/397041124 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71441f3290 con 0x7f71440c5280 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143e00640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7144151a90 0x7f7144158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:37.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f7143400640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 0x7f71441f3760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f713400c440 tx=0x7f713400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134018070 con 0x7f71440c5280 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7134007480 con 0x7f71440c5280 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71340047e0 con 0x7f71440c5280 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71440ccbd0 con 0x7f71440c5280 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.326+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71440cce70 con 0x7f71440c5280 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.328+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71440c47c0 con 0x7f71440c5280 2024-09-16T01:51:37.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.328+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f71340049a0 con 0x7f71440c5280 2024-09-16T01:51:37.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.329+0000 7f713b400640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f712406c2b0 0x7f712406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.329+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(75..75 src has 1..75) ==== 5741+0+0 (secure 0 0 0) 0x7f713408a640 con 0x7f71440c5280 2024-09-16T01:51:37.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.330+0000 7f7142a00640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f712406c2b0 0x7f712406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.330+0000 7f7142a00640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f712406c2b0 0x7f712406e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7130034000 tx=0x7f713002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.334+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f71440c47c0 con 0x7f71440c5280 2024-09-16T01:51:37.361 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:37.359+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:37.458 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:37.458 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:37.462 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:37.460+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:37.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:37.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.527+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f71440c71f0 con 0x7f71440c5280 2024-09-16T01:51:37.563 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:37.561+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:37.665 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:37.663+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:37.678 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:37.691 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:37.765 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:37.764+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:37.791 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:37.866 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:37.864+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:37.899 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:37.899 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:37.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:37.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.949+0000 7f7673400640 1 Processor -- start 2024-09-16T01:51:37.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.949+0000 7f7673400640 1 -- start start 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f7673400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c152c30 0x7f766c0c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f7673400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766c0c4240 con 0x7f766c0c6ab0 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f7673400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766c0c4500 con 0x7f766c0c6e90 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f7673400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f766c0c4da0 con 0x7f766c152c30 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f766be00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f766c0c6ab0 0x7f766c0c8c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54360/0 (socket says 172.21.15.146:54360) 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f766be00640 1 -- 172.21.15.146:0/480599798 learned_addr learned my addr 172.21.15.146:0/480599798 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:37.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.951+0000 7f7670a00640 1 --2- 172.21.15.146:0/480599798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c152c30 0x7f766c0c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.952+0000 7f7670a00640 1 -- 172.21.15.146:0/480599798 >> v1:172.21.15.161:6789/0 conn(0x7f766c0c6e90 legacy=0x7f766c0c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.952+0000 7f7670a00640 1 -- 172.21.15.146:0/480599798 >> v1:172.21.15.146:6789/0 conn(0x7f766c0c6ab0 legacy=0x7f766c0c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.952+0000 7f766b400640 1 -- 172.21.15.146:0/480599798 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1822636294 0 0) 0x7f766c0c4240 con 0x7f766c0c6ab0 2024-09-16T01:51:37.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.952+0000 7f7670a00640 1 -- 172.21.15.146:0/480599798 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c0c47e0 con 0x7f766c152c30 2024-09-16T01:51:37.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.953+0000 7f7670a00640 1 --2- 172.21.15.146:0/480599798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c152c30 0x7f766c0c36b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7660009750 tx=0x7f766002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab53dac2ef31c24c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.953+0000 7f766b400640 1 -- 172.21.15.146:0/480599798 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660037070 con 0x7f766c152c30 2024-09-16T01:51:37.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.953+0000 7f766b400640 1 -- 172.21.15.146:0/480599798 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f766002adb0 con 0x7f766c152c30 2024-09-16T01:51:37.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.953+0000 7f766b400640 1 -- 172.21.15.146:0/480599798 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660032660 con 0x7f766c152c30 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 -- 172.21.15.146:0/480599798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c152c30 msgr2=0x7f766c0c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 --2- 172.21.15.146:0/480599798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c152c30 0x7f766c0c36b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7660009750 tx=0x7f766002a0a0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 -- 172.21.15.146:0/480599798 shutdown_connections 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 --2- 172.21.15.146:0/480599798 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c152c30 0x7f766c0c36b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 -- 172.21.15.146:0/480599798 >> 172.21.15.146:0/480599798 conn(0x7f766c06bc10 msgr2=0x7f766c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 -- 172.21.15.146:0/480599798 shutdown_connections 2024-09-16T01:51:37.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.954+0000 7f7673400640 1 -- 172.21.15.146:0/480599798 wait complete. 2024-09-16T01:51:37.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.956+0000 7f7673400640 1 Processor -- start 2024-09-16T01:51:37.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.956+0000 7f7673400640 1 -- start start 2024-09-16T01:51:37.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7673400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c0c6ab0 0x7f766c0bdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7673400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f766c0c6e90 0x7f766c06f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7673400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f766c152c30 0x7f766c153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7673400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f766c0c9350 con 0x7f766c152c30 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7673400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f766c0c6630 con 0x7f766c0c6e90 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7673400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f766c0c94d0 con 0x7f766c0c6ab0 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7670a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c0c6ab0 0x7f766c0bdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f766be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f766c0c6e90 0x7f766c06f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7670a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c0c6ab0 0x7f766c0bdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44002/0 (socket says 172.21.15.146:44002) 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7670a00640 1 -- 172.21.15.146:0/3513677052 learned_addr learned my addr 172.21.15.146:0/3513677052 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:37.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.957+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f766c152c30 0x7f766c153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7670a00640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f766c0c6e90 msgr2=0x7f766c06f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f766c0c6e90 0x7f766c06f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7670a00640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f766c152c30 msgr2=0x7f766c153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f766c152c30 0x7f766c153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7670a00640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c06fc30 con 0x7f766c0c6ab0 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c0c6ab0 0x7f766c0bdc30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7660009870 tx=0x7f76600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660037040 con 0x7f766c0c6ab0 2024-09-16T01:51:37.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.958+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7660004030 con 0x7f766c0c6ab0 2024-09-16T01:51:37.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.959+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660004200 con 0x7f766c0c6ab0 2024-09-16T01:51:37.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.959+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766c0be2c0 con 0x7f766c0c6ab0 2024-09-16T01:51:37.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.959+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f766c0be860 con 0x7f766c0c6ab0 2024-09-16T01:51:37.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.960+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f766c0c36b0 con 0x7f766c0c6ab0 2024-09-16T01:51:37.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.961+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f766003a070 con 0x7f766c0c6ab0 2024-09-16T01:51:37.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.962+0000 7f7668c00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f764c06c2b0 0x7f764c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.962+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(75..75 src has 1..75) ==== 5741+0+0 (secure 0 0 0) 0x7f76600aba60 con 0x7f766c0c6ab0 2024-09-16T01:51:37.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.963+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v75) ==== 54+0+16 (secure 0 0 0) 0x7f7660041020 con 0x7f766c0c6ab0 2024-09-16T01:51:37.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.963+0000 7f766be00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f764c06c2b0 0x7f764c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.963+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f766c1543b0 0x7f766c0bea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.963+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f766c0befa0 con 0x7f766c1543b0 2024-09-16T01:51:37.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.963+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f766c1543b0 0x7f766c0bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.963+0000 7f766be00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f764c06c2b0 0x7f764c06e770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7658007bd0 tx=0x7f7658007b20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.964+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f766c1543b0 0x7f766c0bea60 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.966 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:37.964+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:37.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.965+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c0befa0 con 0x7f766c1543b0 2024-09-16T01:51:37.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.979+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c0c7530 con 0x7f766c1543b0 2024-09-16T01:51:37.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.982+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c0c7530 con 0x7f766c1543b0 2024-09-16T01:51:37.984 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:37.985 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:37.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.983+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f766c147b40 0x7f766c147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:37.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.983+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f766c150960 con 0x7f766c147b40 2024-09-16T01:51:37.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.983+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f766c147b40 0x7f766c147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:37.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.984+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f766c147b40 0x7f766c147f20 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:37.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.985+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c150960 con 0x7f766c147b40 2024-09-16T01:51:37.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.994+0000 7f713b400640 1 -- 172.21.15.146:0/397041124 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v76) ==== 101+0+0 (secure 0 0 0) 0x7f71340587f0 con 0x7f71440c5280 2024-09-16T01:51:37.996 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:51:37.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.997+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f712406c2b0 msgr2=0x7f712406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.997+0000 7f714a800640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f712406c2b0 0x7f712406e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7130034000 tx=0x7f713002e000 comp rx=0 tx=0).stop 2024-09-16T01:51:37.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.997+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 msgr2=0x7f71441f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:37.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.997+0000 7f714a800640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 0x7f71441f3760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f713400c440 tx=0x7f713400c900 comp rx=0 tx=0).stop 2024-09-16T01:51:38.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 shutdown_connections 2024-09-16T01:51:38.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f712406c2b0 0x7f712406e770 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7144151a90 0x7f7144158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71440c6410 0x7f71441f3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 --2- 172.21.15.146:0/397041124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71440c5280 0x7f71441f3760 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 >> 172.21.15.146:0/397041124 conn(0x7f714406f3c0 msgr2=0x7f7144155b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:38.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 shutdown_connections 2024-09-16T01:51:38.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:37.999+0000 7f714a800640 1 -- 172.21.15.146:0/397041124 wait complete. 2024-09-16T01:51:38.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.004+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c151de0 con 0x7f766c147b40 2024-09-16T01:51:38.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.005+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c151de0 con 0x7f766c147b40 2024-09-16T01:51:38.008 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:38.008 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:38.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.006+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f766c0730e0 0x7f766c073520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:38.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.006+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f766c14efe0 con 0x7f766c0730e0 2024-09-16T01:51:38.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.006+0000 7f766be00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f766c0730e0 0x7f766c073520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:38.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.007+0000 7f766be00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f766c0730e0 0x7f766c073520 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:38.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.008+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c14efe0 con 0x7f766c0730e0 2024-09-16T01:51:38.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.023+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c14f4f0 con 0x7f766c0730e0 2024-09-16T01:51:38.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.026+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c14f4f0 con 0x7f766c0730e0 2024-09-16T01:51:38.028 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:38.028 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:38.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.026+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f766c062630 0x7f766c062a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:38.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.026+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f766c14efe0 con 0x7f766c062630 2024-09-16T01:51:38.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.026+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f766c062630 0x7f766c062a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:38.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.027+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f766c062630 0x7f766c062a90 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:38.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.028+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c14efe0 con 0x7f766c062630 2024-09-16T01:51:38.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.041+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c14efe0 con 0x7f766c062630 2024-09-16T01:51:38.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.043+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c14efe0 con 0x7f766c062630 2024-09-16T01:51:38.045 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:38.046 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:38.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.043+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f766c02c640 0x7f766c02ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:38.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.043+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f766c14efe0 con 0x7f766c02c640 2024-09-16T01:51:38.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.044+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f766c02c640 0x7f766c02ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:38.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.045+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f766c02c640 0x7f766c02ca80 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:38.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.045+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c14efe0 con 0x7f766c02c640 2024-09-16T01:51:38.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.059+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c14efe0 con 0x7f766c02c640 2024-09-16T01:51:38.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.062+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c14efe0 con 0x7f766c02c640 2024-09-16T01:51:38.064 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:38.064 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:38.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.062+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f766c02d4d0 0x7f766c206f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:38.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.062+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f766c062fd0 con 0x7f766c02d4d0 2024-09-16T01:51:38.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.062+0000 7f766be00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f766c02d4d0 0x7f766c206f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.063+0000 7f766be00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f766c02d4d0 0x7f766c206f40 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.063+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c062fd0 con 0x7f766c02d4d0 2024-09-16T01:51:38.067 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:38.066+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:38.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.077+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c062fd0 con 0x7f766c02d4d0 2024-09-16T01:51:38.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.079+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c062fd0 con 0x7f766c02d4d0 2024-09-16T01:51:38.081 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:38.082 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:38.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.080+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f766c207990 0x7f766c209d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:38.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.080+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f766c02d980 con 0x7f766c207990 2024-09-16T01:51:38.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.080+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f766c207990 0x7f766c209d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:38.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.081+0000 7f7671400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f766c207990 0x7f766c209d80 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:38.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.082+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c02d980 con 0x7f766c207990 2024-09-16T01:51:38.088 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:38.088 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:38.088 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:38.088 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:38.088 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:38.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.095+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c02d980 con 0x7f766c207990 2024-09-16T01:51:38.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.097+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c02d980 con 0x7f766c207990 2024-09-16T01:51:38.099 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:38.100 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:38.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.098+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f766c20a7d0 0x7f766c20cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:38.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.098+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f766c20fa40 con 0x7f766c20a7d0 2024-09-16T01:51:38.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.098+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f766c20a7d0 0x7f766c20cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:38.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.099+0000 7f7670a00640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f766c20a7d0 0x7f766c20cbc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:38.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.099+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f766c20fa40 con 0x7f766c20a7d0 2024-09-16T01:51:38.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:38.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.113+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f766c20d3d0 con 0x7f766c20a7d0 2024-09-16T01:51:38.120 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.115+0000 7f7668c00640 1 -- 172.21.15.146:0/3513677052 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f766c20d3d0 con 0x7f766c20a7d0 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f766c1543b0 msgr2=0x7f766c0bea60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f766c1543b0 0x7f766c0bea60 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f766c147b40 msgr2=0x7f766c147f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f766c147b40 0x7f766c147f20 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f766c0730e0 msgr2=0x7f766c073520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f766c0730e0 0x7f766c073520 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f766c062630 msgr2=0x7f766c062a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f766c062630 0x7f766c062a90 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f766c02c640 msgr2=0x7f766c02ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f766c02c640 0x7f766c02ca80 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f766c02d4d0 msgr2=0x7f766c206f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f766c02d4d0 0x7f766c206f40 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f766c207990 msgr2=0x7f766c209d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f766c207990 0x7f766c209d80 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f766c20a7d0 msgr2=0x7f766c20cbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f766c20a7d0 0x7f766c20cbc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7671400640 1 -- 172.21.15.146:0/3513677052 reap_dead start 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7671400640 1 -- 172.21.15.146:0/3513677052 reap_dead start 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7671400640 1 -- 172.21.15.146:0/3513677052 reap_dead start 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7671400640 1 -- 172.21.15.146:0/3513677052 reap_dead start 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f764c06c2b0 msgr2=0x7f764c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f764c06c2b0 0x7f764c06e770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7658007bd0 tx=0x7f7658007b20 comp rx=0 tx=0).stop 2024-09-16T01:51:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c0c6ab0 msgr2=0x7f766c0bdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 --2- 172.21.15.146:0/3513677052 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f766c0c6ab0 0x7f766c0bdc30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7660009870 tx=0x7f76600046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7671400640 1 -- 172.21.15.146:0/3513677052 reap_dead start 2024-09-16T01:51:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 shutdown_connections 2024-09-16T01:51:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 >> 172.21.15.146:0/3513677052 conn(0x7f766c06bc10 msgr2=0x7f766c14e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 shutdown_connections 2024-09-16T01:51:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:38.116+0000 7f7673400640 1 -- 172.21.15.146:0/3513677052 wait complete. 2024-09-16T01:51:38.178 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:38.176+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:38.269 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:38.267+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:38.370 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:38.370 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:38.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:38.370 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:38.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.370 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:39 3.2e 3:75eea580:::10000000002.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.797967116000002, 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059473440000000002, 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.371 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 39 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567844+0000", 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.567846+0000", 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608837+0000", 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040991 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608871+0000", 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4e-05 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608885+0000", 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3994e-05 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.608979+0000", 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3642999999999998e-05 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609134+0000", 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015512100000000001 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.609277+0000", 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143344 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.624326+0000", 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015048977 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.626756+0000", 2024-09-16T01:51:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024300290000000002 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627233+0000", 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00047636500000000002 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627318+0000", 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4773999999999998e-05 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:52 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.644421434000002, 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.06311456, 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 52 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.379 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721390+0000", 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.721392+0000", 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4169999999999999e-06 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774787+0000", 2024-09-16T01:51:38.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053395104999999998 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774815+0000", 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7322e-05 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774826+0000", 2024-09-16T01:51:38.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1443999999999999e-05 2024-09-16T01:51:38.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.774959+0000", 2024-09-16T01:51:38.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013309300000000001 2024-09-16T01:51:38.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775100+0000", 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014046100000000001 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.775233+0000", 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013334499999999999 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784155+0000", 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089218490000000008 2024-09-16T01:51:38.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784387+0000", 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023232799999999999 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784447+0000", 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9413000000000002e-05 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.784504+0000", 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7782999999999999e-05 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:79 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.423561753999998, 2024-09-16T01:51:38.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055130523000000001, 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 79 2024-09-16T01:51:38.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942250+0000", 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.942252+0000", 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1789999999999998e-06 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977377+0000", 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035124872000000001 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977396+0000", 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9582e-05 2024-09-16T01:51:38.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977408+0000", 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1537e-05 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.977687+0000", 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00027904200000000002 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.980431+0000", 2024-09-16T01:51:38.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0027442479999999999 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.981286+0000", 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00085455600000000002 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.996467+0000", 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015181940999999999 2024-09-16T01:51:38.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997278+0000", 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081073899999999999 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997338+0000", 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9321999999999999e-05 2024-09-16T01:51:38.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.997380+0000", 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2505e-05 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.394 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.345431165000001, 2024-09-16T01:51:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.396 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:38.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:38.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:38.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:38.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:38.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:38.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:38.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:38.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.407 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.408 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:143 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.408 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:38.408 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.773218278999998, 2024-09-16T01:51:38.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057181145000000003, 2024-09-16T01:51:38.408 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.409 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.409 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.409 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.409 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.409 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 143 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592593+0000", 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.592596+0000", 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.649e-06 2024-09-16T01:51:38.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615846+0000", 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023250214000000002 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615868+0000", 2024-09-16T01:51:38.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1861e-05 2024-09-16T01:51:38.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615880+0000", 2024-09-16T01:51:38.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1980999999999999e-05 2024-09-16T01:51:38.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.615970+0000", 2024-09-16T01:51:38.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9854999999999999e-05 2024-09-16T01:51:38.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616129+0000", 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159359 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.616292+0000", 2024-09-16T01:51:38.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016279599999999999 2024-09-16T01:51:38.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.621924+0000", 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0056320600000000004 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649646+0000", 2024-09-16T01:51:38.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027722323 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649722+0000", 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6280999999999997e-05 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.649774+0000", 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1765999999999999e-05 2024-09-16T01:51:38.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:170 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.549061745000003, 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05546748, 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 170 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816750+0000", 2024-09-16T01:51:38.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.816751+0000", 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9850000000000001e-06 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850141+0000", 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033389689 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850173+0000", 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1523999999999998e-05 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850186+0000", 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3464e-05 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850347+0000", 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034600000000001 2024-09-16T01:51:38.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850497+0000", 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000150187 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.850658+0000", 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161395 2024-09-16T01:51:38.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.867862+0000", 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017204224000000001 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872094+0000", 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042320070000000003 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872161+0000", 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6880999999999999e-05 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.872217+0000", 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5778e-05 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:201 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:38.425 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.251784721999996, 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061836125999999998, 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 201 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:38.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114027+0000", 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.114028+0000", 2024-09-16T01:51:38.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.827e-06 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159654+0000", 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045625813000000001 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159682+0000", 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7823000000000001e-05 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159695+0000", 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867000000000001e-05 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159776+0000", 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0808000000000003e-05 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.159945+0000", 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016909999999999999 2024-09-16T01:51:38.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.160094+0000", 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014941800000000001 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.169627+0000", 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095324639999999992 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175745+0000", 2024-09-16T01:51:38.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0061183490000000004 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175807+0000", 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1950000000000001e-05 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:52.175863+0000", 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5707e-05 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.431 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.778976868999997, 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.432 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:38.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:38.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:38.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.591150282999998, 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.438 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:505 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:38.444 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.082333878, 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060008988999999999, 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 505 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:38.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283477+0000", 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.283491+0000", 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.346e-05 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330858+0000", 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047367419000000001 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330883+0000", 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4847000000000001e-05 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330895+0000", 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2233e-05 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.330981+0000", 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5752000000000002e-05 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331086+0000", 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010455600000000001 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.331237+0000", 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015141199999999999 2024-09-16T01:51:38.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.340187+0000", 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089496609999999994 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343394+0000", 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032074719999999998 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343448+0000", 2024-09-16T01:51:38.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3677999999999997e-05 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.343486+0000", 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8498999999999999e-05 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.703487978999998, 2024-09-16T01:51:38.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:51:38.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:607 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.075101188000001, 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057026931000000003, 2024-09-16T01:51:38.456 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 607 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290710+0000", 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.290713+0000", 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6900000000000001e-06 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330961+0000", 2024-09-16T01:51:38.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040248199999999998 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.330992+0000", 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1251000000000003e-05 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.331005+0000", 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3006999999999999e-05 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332660+0000", 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016549589999999999 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332819+0000", 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159137 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.332968+0000", 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148942 2024-09-16T01:51:38.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.342537+0000", 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095683910000000007 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347616+0000", 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005079793 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347682+0000", 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5622000000000003e-05 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.347737+0000", 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4938999999999997e-05 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:634 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:38.462 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.850912774000001, 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068283101999999998, 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 634 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514901+0000", 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5299999999999999e-06 2024-09-16T01:51:38.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549207+0000", 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034306171000000003 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549224+0000", 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6976000000000001e-05 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549235+0000", 2024-09-16T01:51:38.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0682999999999999e-05 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549568+0000", 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033294599999999998 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549714+0000", 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014617499999999999 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:38.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549858+0000", 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144356 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.566205+0000", 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01634683 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583078+0000", 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016872402000000002 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583135+0000", 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7286e-05 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583182+0000", 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6746999999999998e-05 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.468 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:688 3.26 3:642b2ad4:::100000001f7.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.333945302, 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.062867175999999997, 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 688 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031866+0000", 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.031869+0000", 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5509999999999998e-06 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084368+0000", 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052499410000000003 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084392+0000", 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084405+0000", 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2561000000000001e-05 2024-09-16T01:51:38.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084491+0000", 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5945000000000005e-05 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084662+0000", 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017081299999999999 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.084812+0000", 2024-09-16T01:51:38.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014993999999999999 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.091507+0000", 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066950079999999997 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094630+0000", 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031236570000000002 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094700+0000", 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9575999999999999e-05 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.094733+0000", 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3260999999999999e-05 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:696 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:38.474 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.238687437999999, 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071884475000000003, 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 696 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127126+0000", 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9369999999999998e-06 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159227+0000", 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032101044000000002 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159250+0000", 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3332999999999998e-05 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159260+0000", 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:51:38.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159343+0000", 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3373000000000003e-05 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159484+0000", 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014100399999999999 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159646+0000", 2024-09-16T01:51:38.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162044 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.186104+0000", 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026457794999999999 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198896+0000", 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012791749 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198959+0000", 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3387999999999997e-05 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.199008+0000", 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9118e-05 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:776 3.11 e68/20 3:887bb7a4:::100000001f7.00000005:head v 68'37, mlcod=65'35)", 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:38.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.157768301000001, 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.05614222, 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208043+0000", 2024-09-16T01:51:38.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.208044+0000", 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219371+0000", 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011326688999999999 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219442+0000", 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0727000000000002e-05 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219483+0000", 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1165000000000003e-05 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219725+0000", 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024194200000000001 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.219752+0000", 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7270000000000001e-05 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264133+0000", 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044381029000000002 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.264185+0000", 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1879000000000003e-05 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:786 3.3a 3:5d44b367:::100000001f7.0000000f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.122179207999999, 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055535233000000003, 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 786 2024-09-16T01:51:38.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243632+0000", 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.243634+0000", 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.04e-06 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254427+0000", 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010792636 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254446+0000", 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9219999999999999e-05 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254458+0000", 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2387e-05 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254837+0000", 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00037878500000000003 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254934+0000", 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6915000000000003e-05 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.255090+0000", 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015594200000000001 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.273598+0000", 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018507932000000001 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299039+0000", 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025441583 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299115+0000", 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5319000000000002e-05 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.299167+0000", 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2473999999999999e-05 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.490 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:785 3.11 e69/20 3:8b60708a:::100000001f7.0000000e:head v 69'38, mlcod=65'35)", 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.110947037999999, 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082167670999999998, 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254865+0000", 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9799999999999999e-07 2024-09-16T01:51:38.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286626+0000", 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031761664000000002 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286647+0000", 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268e-05 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286659+0000", 2024-09-16T01:51:38.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1276e-05 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286743+0000", 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4488999999999997e-05 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286772+0000", 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9082999999999999e-05 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.336965+0000", 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050192716999999998 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337032+0000", 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6876000000000001e-05 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:790 3.16 3:69eaaa52:::100000001f7.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:38.495 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.063174798999999, 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061715643000000001, 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 790 2024-09-16T01:51:38.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302636+0000", 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.302638+0000", 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9410000000000001e-06 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342651+0000", 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040012486 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342687+0000", 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5738999999999998e-05 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.342701+0000", 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4255e-05 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343871+0000", 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001169977 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.343975+0000", 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010435 2024-09-16T01:51:38.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.344127+0000", 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015170400000000001 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.361011+0000", 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016884528999999999 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364224+0000", 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003212183 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364295+0000", 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0957000000000005e-05 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.364352+0000", 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7522000000000003e-05 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:863 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.314042015999998, 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055946002000000002, 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 863 2024-09-16T01:51:38.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051769+0000", 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:38.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.051772+0000", 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2620000000000001e-06 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087118+0000", 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035346883000000003 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087186+0000", 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7743000000000005e-05 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087196+0000", 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4390000000000006e-06 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087342+0000", 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146625 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087445+0000", 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010255 2024-09-16T01:51:38.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.087572+0000", 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012764800000000001 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.101292+0000", 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013719543000000001 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107514+0000", 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0062224120000000001 2024-09-16T01:51:38.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107641+0000", 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012699 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:21.107715+0000", 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3906999999999996e-05 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:38.507 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:38.508 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:38.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:38.509 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:38.507+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:38.610 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:38.608+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:38.710 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:38.708+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:38.808 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:38.808 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:38.808 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:38.808 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:38.811 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:38.809+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:38.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:38.912 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:38.910+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:39.013 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:39.011+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:39.113 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:39.111+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:39.169 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:39.169 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:39.169 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:39.169 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:39.169 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:39.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:39.215 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:39.213+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:39.316 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:39.314+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:39.419 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:39.417+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:39.479 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 49.219993119000002, 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089626530999999995, 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255280+0000", 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.255282+0000", 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.824e-06 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328787+0000", 2024-09-16T01:51:39.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073505557999999999 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328819+0000", 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2011999999999998e-05 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.328834+0000", 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4429e-05 2024-09-16T01:51:39.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329011+0000", 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017733700000000001 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329176+0000", 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000164699 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.329305+0000", 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129677 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.341714+0000", 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012408265 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344793+0000", 2024-09-16T01:51:39.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0030787779999999999 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344854+0000", 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.198e-05 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.344906+0000", 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1972000000000003e-05 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:26 3.28 3:16173d1c:::10000000002.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 49.021620249000001, 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066384046000000002, 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 26 2024-09-16T01:51:39.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453653+0000", 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.453655+0000", 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.198e-06 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505446+0000", 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051791112 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505469+0000", 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3095000000000002e-05 2024-09-16T01:51:39.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505490+0000", 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0882000000000001e-05 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505631+0000", 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014133700000000001 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505769+0000", 2024-09-16T01:51:39.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013709200000000001 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.505903+0000", 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134859 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.513339+0000", 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074353020000000004 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519735+0000", 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0063959419999999999 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.519874+0000", 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138911 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.520037+0000", 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016331599999999999 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.492 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:34 3.2f 3:f540378d:::10000000002.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.912817617999998, 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072279182999999997, 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.493 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 34 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562455+0000", 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.562457+0000", 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.95e-06 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597176+0000", 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034718907 2024-09-16T01:51:39.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597208+0000", 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1692000000000001e-05 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597222+0000", 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4156e-05 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597509+0000", 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028733900000000001 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597698+0000", 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018843899999999999 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.597830+0000", 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000132158 2024-09-16T01:51:39.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.627844+0000", 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030013486999999998 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634635+0000", 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0067911300000000003 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634687+0000", 2024-09-16T01:51:39.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2064999999999997e-05 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634735+0000", 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7859999999999999e-05 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:72 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:39.499 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.593248985000002, 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071503988000000004, 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 72 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882024+0000", 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.882026+0000", 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9250000000000002e-06 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925408+0000", 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043381722999999997 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925433+0000", 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4797999999999999e-05 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925445+0000", 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.289e-05 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925541+0000", 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5186999999999995e-05 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925827+0000", 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00028608699999999997 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.925979+0000", 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152671 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.948694+0000", 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022714709 2024-09-16T01:51:39.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953462+0000", 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0047676050000000003 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953495+0000", 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3600999999999999e-05 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.953528+0000", 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2792e-05 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:123 3.a 3:51af04f2:::10000000002.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.094586245000002, 2024-09-16T01:51:39.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066760945000000002, 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 123 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380687+0000", 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.380690+0000", 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0680000000000001e-06 2024-09-16T01:51:39.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436239+0000", 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055549065000000002 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436260+0000", 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0562999999999999e-05 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436271+0000", 2024-09-16T01:51:39.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1491999999999999e-05 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436361+0000", 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0367999999999999e-05 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436463+0000", 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101674 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.436558+0000", 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4512000000000006e-05 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.446784+0000", 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010226727999999999 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447343+0000", 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861300000000001 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447397+0000", 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4435999999999999e-05 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.447448+0000", 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0426000000000001e-05 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.512 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:459 3.14 3:2878c2ad:::10000000002.000000c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.694640468999999, 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065187593000000002, 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.513 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 459 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780633+0000", 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.780634+0000", 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5939999999999999e-06 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802137+0000", 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021503069999999999 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802165+0000", 2024-09-16T01:51:39.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7787e-05 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802177+0000", 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2170999999999999e-05 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802585+0000", 2024-09-16T01:51:39.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00040773300000000002 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802690+0000", 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105072 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.802842+0000", 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151743 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.831317+0000", 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028475495999999999 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845705+0000", 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014387341 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845769+0000", 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4418000000000006e-05 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.845820+0000", 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1168000000000002e-05 2024-09-16T01:51:39.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.626535828000002, 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:51:39.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:51:39.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:51:39.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:51:39.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:51:39.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.484076162999997, 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.529 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:51:39.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:51:39.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.13660994, 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.537 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:51:39.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:544 3.14 3:28ae9428:::100000001f7.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.831236038, 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064751230000000007, 2024-09-16T01:51:39.545 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 544 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644037+0000", 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.644039+0000", 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6619999999999999e-06 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665383+0000", 2024-09-16T01:51:39.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021343964 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665408+0000", 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.561e-05 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665422+0000", 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3947999999999999e-05 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665607+0000", 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018433499999999999 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665831+0000", 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022424399999999999 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.665986+0000", 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155654 2024-09-16T01:51:39.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.682208+0000", 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016221920000000001 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708669+0000", 2024-09-16T01:51:39.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026460513000000001 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708734+0000", 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4888000000000006e-05 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.708788+0000", 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4491999999999999e-05 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:549 3.11 3:8ae93890:::100000001f7.00000022:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.789036017000001, 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081050590000000006, 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 549 2024-09-16T01:51:39.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686237+0000", 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.686239+0000", 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2450000000000001e-06 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754451+0000", 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068212138000000005 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754474+0000", 2024-09-16T01:51:39.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2733000000000001e-05 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754483+0000", 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1349999999999998e-06 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754562+0000", 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8319999999999996e-05 2024-09-16T01:51:39.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754725+0000", 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016298699999999999 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.754875+0000", 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149942 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.762160+0000", 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072859630000000003 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767169+0000", 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050088609999999999 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767231+0000", 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1158e-05 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.767288+0000", 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7108000000000003e-05 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:629 3.13 3:ca345efb:::100000001f7.00000072:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.966481391999999, 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073672847999999999, 2024-09-16T01:51:39.558 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 629 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508792+0000", 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.508793+0000", 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5650000000000001e-06 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536849+0000", 2024-09-16T01:51:39.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028055429999999999 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536888+0000", 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9128999999999998e-05 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.536904+0000", 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6223e-05 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537082+0000", 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017825600000000001 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537242+0000", 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159569 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.537394+0000", 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015228 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.548370+0000", 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010976324000000001 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582365+0000", 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033994745999999999 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582423+0000", 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7357e-05 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.582465+0000", 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1968999999999997e-05 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:700 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.34163096, 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074697384000000006, 2024-09-16T01:51:39.564 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 700 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133642+0000", 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.133644+0000", 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159468+0000", 2024-09-16T01:51:39.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025824323 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159551+0000", 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2286999999999993e-05 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159561+0000", 2024-09-16T01:51:39.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0307e-05 2024-09-16T01:51:39.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159851+0000", 2024-09-16T01:51:39.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029012100000000001 2024-09-16T01:51:39.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160192+0000", 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034104600000000001 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.160444+0000", 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025126400000000001 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.197238+0000", 2024-09-16T01:51:39.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036794265999999999 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208261+0000", 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0110233 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208304+0000", 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2833999999999997e-05 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.208339+0000", 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5556000000000003e-05 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:776 3.11 3:887bb7a4:::100000001f7.00000005:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:39.571 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.278589405999998, 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090355046999999994, 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 776 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.572 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196684+0000", 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.196686+0000", 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5289999999999998e-06 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206873+0000", 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010186855999999999 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206895+0000", 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1549e-05 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.206907+0000", 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2884000000000001e-05 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207056+0000", 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014810499999999999 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207156+0000", 2024-09-16T01:51:39.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010007 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.207302+0000", 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014591499999999999 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263823+0000", 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056521543 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286896+0000", 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023073327000000001 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286980+0000", 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3869999999999995e-05 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.287039+0000", 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8399000000000001e-05 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.577 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.252318576, 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:51:39.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1028 3.11 3:893be98d:::100000001f8.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:38.156760+0000", 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.318512688, 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065071496000000006, 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1028 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.156760+0000", 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.156760+0000", 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.156763+0000", 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2249999999999999e-06 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.173241+0000", 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016478599 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.173282+0000", 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0698000000000002e-05 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.173297+0000", 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4959e-05 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.177078+0000", 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0037816109999999998 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.177303+0000", 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022409400000000001 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.177467+0000", 2024-09-16T01:51:39.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016467200000000001 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.200423+0000", 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02295587 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.221700+0000", 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021277252 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.221777+0000", 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6848999999999996e-05 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.221832+0000", 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4666999999999997e-05 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.590 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1032 3.11 3:8be3ecac:::100000001f8.00000005:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:38.203724+0000", 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2715491059999999, 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065471911999999993, 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1032 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.203724+0000", 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.203724+0000", 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.203727+0000", 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4970000000000001e-06 2024-09-16T01:51:39.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.233935+0000", 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030207910000000001 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.233953+0000", 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7601e-05 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.233964+0000", 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0964999999999999e-05 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.234310+0000", 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034594200000000002 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.234461+0000", 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015126399999999999 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.235670+0000", 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012091700000000001 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.267555+0000", 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031884728000000001 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.269110+0000", 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015547530000000001 2024-09-16T01:51:39.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.269157+0000", 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7033999999999997e-05 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.269196+0000", 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9048000000000003e-05 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.21457907300000001, 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.597 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:51:39.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:51:39.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1120 3.2f 3:f55c0805:::100000001f8.0000005d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.314624+0000", 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.160649509, 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072418724000000004, 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.603 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.604 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:39.602+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1120 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.314624+0000", 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.314624+0000", 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.314625+0000", 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7799999999999999e-06 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.335965+0000", 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021339565000000001 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336795+0000", 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000829827 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.337157+0000", 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036198099999999999 2024-09-16T01:51:39.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.337252+0000", 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5697000000000002e-05 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.337412+0000", 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159739 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.337567+0000", 2024-09-16T01:51:39.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015437299999999999 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.360782+0000", 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023215737 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386959+0000", 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026176985999999999 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.387006+0000", 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6328000000000001e-05 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.387042+0000", 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6711000000000001e-05 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.610 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1136 3.d 3:b30cb2b3:::100000001f8.0000006d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.372405+0000", 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.102867813, 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079041970000000003, 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1136 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.372405+0000", 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.372405+0000", 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.372407+0000", 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.855e-06 2024-09-16T01:51:39.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.400128+0000", 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027721328999999999 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.400184+0000", 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5819000000000002e-05 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:39.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.400226+0000", 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1936e-05 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.400367+0000", 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014033200000000001 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.400538+0000", 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171368 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.400703+0000", 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016537 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.449464+0000", 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048760349000000001 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451328+0000", 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001864178 2024-09-16T01:51:39.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451392+0000", 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4404000000000002e-05 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451447+0000", 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.503e-05 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.616 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:39.617 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:39.617 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:39.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:39.705 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:39.703+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:39.746 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:39.747 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:39.747 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:39.747 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:39.756 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:39.840 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:39.838+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:39.906 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:39.904+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:39.915 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:39.915 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:39.915 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:39.915 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:39.915 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:39.926 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:40.010 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:40.008+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:40.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:51:40.088 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:40.088 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:35 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.535772848000001, 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.05616119, 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.089 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 35 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549621+0000", 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.549622+0000", 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.094e-06 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587563+0000", 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037940848999999999 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587581+0000", 2024-09-16T01:51:40.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.859e-05 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587593+0000", 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1085e-05 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587650+0000", 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7088e-05 2024-09-16T01:51:40.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587749+0000", 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9407999999999994e-05 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.587844+0000", 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4833999999999994e-05 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.592432+0000", 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045886039999999996 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605689+0000", 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013256733999999999 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605730+0000", 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1239999999999998e-05 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.605782+0000", 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1663999999999999e-05 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:64 3.2 3:40abce4f:::10000000002.0000003d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.288739266999997, 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055151562000000001, 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.095 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 64 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796655+0000", 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.796657+0000", 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1940000000000001e-06 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826854+0000", 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030197750999999998 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826870+0000", 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291e-05 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826877+0000", 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5399999999999998e-06 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.826925+0000", 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8211999999999999e-05 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827026+0000", 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100973 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.827121+0000", 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4197000000000006e-05 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.829604+0000", 2024-09-16T01:51:40.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002482981 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851732+0000", 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022128302999999998 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851767+0000", 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4928999999999997e-05 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.851806+0000", 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9190999999999998e-05 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.229580181000003, 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.101 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:51:40.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:51:40.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.318001869, 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.107 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:51:40.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.273741639000001, 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:51:40.113 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:51:40.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:51:40.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:172 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.243241818999998, 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057185613000000003, 2024-09-16T01:51:40.119 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 172 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842152+0000", 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.842153+0000", 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.271e-06 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872134+0000", 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029980946000000001 2024-09-16T01:51:40.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872156+0000", 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1364000000000001e-05 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872165+0000", 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4870000000000008e-06 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872281+0000", 2024-09-16T01:51:40.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011554500000000001 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872529+0000", 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000248626 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.872773+0000", 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00024416999999999999 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.877034+0000", 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042608489999999997 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899226+0000", 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022191517000000001 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899285+0000", 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9762000000000001e-05 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.899338+0000", 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2076000000000001e-05 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.201686297000002, 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.125 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:51:40.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.944679876000002, 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:51:40.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:40.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:51:40.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.909883577000002, 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:51:40.138 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:51:40.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:51:40.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:51:40.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:51:40.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:40.145 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.826434624000001, 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:51:40.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.627131837, 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:51:40.151 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:51:40.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:51:40.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.482254899000001, 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.157 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:51:40.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.441610475000001, 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.163 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:51:40.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:51:40.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:40.169 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.283667641000001, 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:51:40.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.175 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.158540380000002, 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:51:40.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:51:40.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:598 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:40.181 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.869657828000001, 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.054052254000000001, 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 598 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:40.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215736+0000", 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.215738+0000", 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.526e-06 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250373+0000", 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034634160999999997 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250391+0000", 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7986e-05 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250400+0000", 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1439999999999992e-06 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250455+0000", 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5578999999999998e-05 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250562+0000", 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106717 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.250665+0000", 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010307200000000001 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.253888+0000", 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032231019999999998 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269702+0000", 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015813473000000002 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269742+0000", 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0154000000000002e-05 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.269788+0000", 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.634e-05 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.187 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.121740449000001, 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:51:40.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:51:40.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:40.193 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 18.958714807, 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:51:40.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:51:40.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.82497701800000001, 2024-09-16T01:51:40.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:51:40.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:51:40.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.130586904, 2024-09-16T01:51:40.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:51:40.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:51:40.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.211 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:40.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:40.212 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:40.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:40.212 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T01:51:40.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T01:51:40.214 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:40.215 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:40.210+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:40.313 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:40.311+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:40.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.331+0000 7fdfdfe00640 1 Processor -- start 2024-09-16T01:51:40.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.331+0000 7fdfdfe00640 1 -- start start 2024-09-16T01:51:40.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfdfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdfd80d65a0 0x7fdfd8044b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfdfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdfd8050900 0x7fdfd8045070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfdfe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd8035e40 con 0x7fdfd804b960 2024-09-16T01:51:40.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfdfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd80dc4e0 con 0x7fdfd80d65a0 2024-09-16T01:51:40.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfdfe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd804ad60 con 0x7fdfd8050900 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfde000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdfd8050900 0x7fdfd8045070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.333+0000 7fdfde000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdfd8050900 0x7fdfd8045070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60274/0 (socket says 172.21.15.146:60274) 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.334+0000 7fdfde000640 1 -- 172.21.15.146:0/433154175 learned_addr learned my addr 172.21.15.146:0/433154175 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.334+0000 7fdfdea00640 1 --2- 172.21.15.146:0/433154175 >> v2:172.21.15.146:3300/0 conn(0x7fdfd80d65a0 0x7fdfd8044b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.334+0000 7fdfde000640 1 -- 172.21.15.146:0/433154175 >> v2:172.21.15.146:3300/0 conn(0x7fdfd80d65a0 msgr2=0x7fdfd8044b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.334+0000 7fdfde000640 1 --2- 172.21.15.146:0/433154175 >> v2:172.21.15.146:3300/0 conn(0x7fdfd80d65a0 0x7fdfd8044b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.334+0000 7fdfde000640 1 -- 172.21.15.146:0/433154175 >> v1:172.21.15.146:6789/0 conn(0x7fdfd804b960 legacy=0x7fdfd80dc970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:40.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.334+0000 7fdfde000640 1 -- 172.21.15.146:0/433154175 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd804b040 con 0x7fdfd8050900 2024-09-16T01:51:40.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.336+0000 7fdfde000640 1 --2- 172.21.15.146:0/433154175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd8050900 0x7fdfd8045070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8045e10 tx=0x7fdfcc02a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fead915fef93fd1f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.336+0000 7fdfdd600640 1 -- 172.21.15.146:0/433154175 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc037070 con 0x7fdfd8050900 2024-09-16T01:51:40.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.336+0000 7fdfdd600640 1 -- 172.21.15.146:0/433154175 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc032440 con 0x7fdfd8050900 2024-09-16T01:51:40.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.336+0000 7fdfdd600640 1 -- 172.21.15.146:0/433154175 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc032810 con 0x7fdfd8050900 2024-09-16T01:51:40.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 -- 172.21.15.146:0/433154175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd8050900 msgr2=0x7fdfd8045070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/433154175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd8050900 0x7fdfd8045070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8045e10 tx=0x7fdfcc02a2c0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 -- 172.21.15.146:0/433154175 shutdown_connections 2024-09-16T01:51:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/433154175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd8050900 0x7fdfd8045070 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/433154175 >> v2:172.21.15.146:3300/0 conn(0x7fdfd80d65a0 0x7fdfd8044b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 -- 172.21.15.146:0/433154175 >> 172.21.15.146:0/433154175 conn(0x7fdfd804fc50 msgr2=0x7fdfd8050060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 -- 172.21.15.146:0/433154175 shutdown_connections 2024-09-16T01:51:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.338+0000 7fdfdfe00640 1 -- 172.21.15.146:0/433154175 wait complete. 2024-09-16T01:51:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.341+0000 7fdfdfe00640 1 Processor -- start 2024-09-16T01:51:40.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.341+0000 7fdfdfe00640 1 -- start start 2024-09-16T01:51:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.343+0000 7fdfdfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 0x7fdfd8049c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.344+0000 7fdfdfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8050900 0x7fdfd8048c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.344+0000 7fdfdea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 0x7fdfd8049c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.344+0000 7fdfdea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 0x7fdfd8049c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60280/0 (socket says 172.21.15.146:60280) 2024-09-16T01:51:40.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.344+0000 7fdfdea00640 1 -- 172.21.15.146:0/1394447596 learned_addr learned my addr 172.21.15.146:0/1394447596 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:40.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.345+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfd80d65a0 0x7fdfd8044da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.345+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd80d6980 con 0x7fdfd8050900 2024-09-16T01:51:40.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.345+0000 7fdfdf400640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfd80d65a0 0x7fdfd8044da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.346+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd804c390 con 0x7fdfd804b960 2024-09-16T01:51:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.346+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd80d6b00 con 0x7fdfd80d65a0 2024-09-16T01:51:40.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.346+0000 7fdfde000640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8050900 0x7fdfd8048c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdf400640 1 -- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 msgr2=0x7fdfd8049c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdf400640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 0x7fdfd8049c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdf400640 1 -- 172.21.15.146:0/1394447596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8050900 msgr2=0x7fdfd8048c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdf400640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8050900 0x7fdfd8048c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdf400640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd80ce6a0 con 0x7fdfd80d65a0 2024-09-16T01:51:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdea00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 0x7fdfd8049c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:51:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.347+0000 7fdfdf400640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfd80d65a0 0x7fdfd8044da0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0051790 tx=0x7fdfe0069890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.348+0000 7fdfde000640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8050900 0x7fdfd8048c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:51:40.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.349+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0091070 con 0x7fdfd80d65a0 2024-09-16T01:51:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.349+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe00718a0 con 0x7fdfd80d65a0 2024-09-16T01:51:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.349+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0071d10 con 0x7fdfd80d65a0 2024-09-16T01:51:40.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.349+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd80cdfb0 con 0x7fdfd80d65a0 2024-09-16T01:51:40.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.351+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd80491a0 con 0x7fdfd80d65a0 2024-09-16T01:51:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.352+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdfe0094070 con 0x7fdfd80d65a0 2024-09-16T01:51:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.354+0000 7fdfd6a00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfc006c330 0x7fdfc006e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.354+0000 7fdfdea00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfc006c330 0x7fdfc006e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.355+0000 7fdfdea00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfc006c330 0x7fdfc006e7f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0006fd0 tx=0x7fdfd0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.357+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd80d0010 con 0x7fdfd80d65a0 2024-09-16T01:51:40.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.376+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(77..77 src has 1..77) ==== 5799+0+0 (secure 0 0 0) 0x7fdfe010b060 con 0x7fdfd80d65a0 2024-09-16T01:51:40.380 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:40.380 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:40.380 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:40.380 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:40.390 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:40.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.395+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdfd80d0010 con 0x7fdfd80d65a0 2024-09-16T01:51:40.414 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:40.412+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:40.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.477+0000 7f8417200640 1 Processor -- start 2024-09-16T01:51:40.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.477+0000 7f8417200640 1 -- start start 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.478+0000 7f8417200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f841815b340 0x7f8418166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.478+0000 7f8417200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841806d140 con 0x7f84180c75a0 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.478+0000 7f8417200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841806d420 con 0x7f84180c6b70 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.478+0000 7f8417200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f84180cab20 con 0x7f841815b340 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8415400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f84180c75a0 0x7f84180cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54374/0 (socket says 172.21.15.146:54374) 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8415400640 1 -- 172.21.15.146:0/1261536001 learned_addr learned my addr 172.21.15.146:0/1261536001 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:40.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8416800640 1 --2- 172.21.15.146:0/1261536001 >> v2:172.21.15.161:3300/0 conn(0x7f841815b340 0x7f8418166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8416800640 1 -- 172.21.15.146:0/1261536001 >> v1:172.21.15.161:6789/0 conn(0x7f84180c6b70 legacy=0x7f84180cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8414a00640 1 -- 172.21.15.146:0/1261536001 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628215976 0 0) 0x7f841806d140 con 0x7f84180c75a0 2024-09-16T01:51:40.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8416800640 1 -- 172.21.15.146:0/1261536001 >> v1:172.21.15.146:6789/0 conn(0x7f84180c75a0 legacy=0x7f84180cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.479+0000 7f8416800640 1 -- 172.21.15.146:0/1261536001 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841806d700 con 0x7f841815b340 2024-09-16T01:51:40.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.480+0000 7f234ca00640 1 Processor -- start 2024-09-16T01:51:40.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.480+0000 7f234ca00640 1 -- start start 2024-09-16T01:51:40.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.481+0000 7f8416800640 1 --2- 172.21.15.146:0/1261536001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418166170 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8410009d90 tx=0x7f841002a580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b9bc9b4b342587b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.481+0000 7f8414a00640 1 -- 172.21.15.146:0/1261536001 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410007740 con 0x7f841815b340 2024-09-16T01:51:40.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.481+0000 7f8414a00640 1 -- 172.21.15.146:0/1261536001 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84100078e0 con 0x7f841815b340 2024-09-16T01:51:40.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.481+0000 7f8414a00640 1 -- 172.21.15.146:0/1261536001 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410007cb0 con 0x7f841815b340 2024-09-16T01:51:40.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f234ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f23480c68f0 0x7f234815e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f234ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480ca270 0x7f234815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f234ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f234815eee0 con 0x7f23480cb3a0 2024-09-16T01:51:40.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f234ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f234815e640 con 0x7f23480c68f0 2024-09-16T01:51:40.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f234ca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f234815e920 con 0x7f23480ca270 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f2346800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f23480cb3a0 0x7f23480cc000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54382/0 (socket says 172.21.15.146:54382) 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f2346800640 1 -- 172.21.15.146:0/260013706 learned_addr learned my addr 172.21.15.146:0/260013706 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f2345e00640 1 --2- 172.21.15.146:0/260013706 >> v2:172.21.15.146:3300/0 conn(0x7f23480c68f0 0x7f234815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.483+0000 7f2345400640 1 --2- 172.21.15.146:0/260013706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480ca270 0x7f234815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f8417200640 1 -- 172.21.15.146:0/1261536001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 msgr2=0x7f8418166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f8417200640 1 --2- 172.21.15.146:0/1261536001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418166170 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8410009d90 tx=0x7f841002a580 comp rx=0 tx=0).stop 2024-09-16T01:51:40.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f8417200640 1 -- 172.21.15.146:0/1261536001 shutdown_connections 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f8417200640 1 --2- 172.21.15.146:0/1261536001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418166170 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.482+0000 7f8417200640 1 -- 172.21.15.146:0/1261536001 >> 172.21.15.146:0/1261536001 conn(0x7f84180ccc20 msgr2=0x7f84180cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.483+0000 7f2345e00640 1 -- 172.21.15.146:0/260013706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480ca270 msgr2=0x7f234815b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.483+0000 7f2345e00640 1 --2- 172.21.15.146:0/260013706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480ca270 0x7f234815b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.483+0000 7f2345e00640 1 -- 172.21.15.146:0/260013706 >> v1:172.21.15.146:6789/0 conn(0x7f23480cb3a0 legacy=0x7f23480cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.483+0000 7f2345e00640 1 -- 172.21.15.146:0/260013706 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234815ec00 con 0x7f23480c68f0 2024-09-16T01:51:40.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.484+0000 7f2345e00640 1 --2- 172.21.15.146:0/260013706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480c68f0 0x7f234815e000 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f23380097a0 tx=0x7f233802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e7df2a9e3a9d6c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.484+0000 7f8417200640 1 -- 172.21.15.146:0/1261536001 shutdown_connections 2024-09-16T01:51:40.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.484+0000 7f8417200640 1 -- 172.21.15.146:0/1261536001 wait complete. 2024-09-16T01:51:40.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.485+0000 7f2344a00640 1 -- 172.21.15.146:0/260013706 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338037070 con 0x7f23480c68f0 2024-09-16T01:51:40.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.485+0000 7f2344a00640 1 -- 172.21.15.146:0/260013706 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2338032440 con 0x7f23480c68f0 2024-09-16T01:51:40.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.485+0000 7f2344a00640 1 -- 172.21.15.146:0/260013706 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233802ae60 con 0x7f23480c68f0 2024-09-16T01:51:40.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 -- 172.21.15.146:0/260013706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480c68f0 msgr2=0x7f234815e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 --2- 172.21.15.146:0/260013706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480c68f0 0x7f234815e000 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f23380097a0 tx=0x7f233802a110 comp rx=0 tx=0).stop 2024-09-16T01:51:40.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f8417200640 1 Processor -- start 2024-09-16T01:51:40.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 -- 172.21.15.146:0/260013706 shutdown_connections 2024-09-16T01:51:40.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 --2- 172.21.15.146:0/260013706 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480ca270 0x7f234815b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 --2- 172.21.15.146:0/260013706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480c68f0 0x7f234815e000 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 -- 172.21.15.146:0/260013706 >> 172.21.15.146:0/260013706 conn(0x7f234806cb70 msgr2=0x7f234806e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:40.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f8417200640 1 -- start start 2024-09-16T01:51:40.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.486+0000 7f234ca00640 1 -- 172.21.15.146:0/260013706 shutdown_connections 2024-09-16T01:51:40.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.487+0000 7f234ca00640 1 -- 172.21.15.146:0/260013706 wait complete. 2024-09-16T01:51:40.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8417200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84180c6b70 0x7f8418159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8417200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84180c75a0 0x7f84181f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8417200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8417200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f841815b7c0 con 0x7f84180c75a0 2024-09-16T01:51:40.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8417200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f84180cb050 con 0x7f841815b340 2024-09-16T01:51:40.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f234ca00640 1 Processor -- start 2024-09-16T01:51:40.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f234ca00640 1 -- start start 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8417200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f84180cbc80 con 0x7f84180c6b70 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8416800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8416800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60296/0 (socket says 172.21.15.146:60296) 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.488+0000 7f8416800640 1 -- 172.21.15.146:0/4109668139 learned_addr learned my addr 172.21.15.146:0/4109668139 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f8416800640 1 -- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84180c6b70 msgr2=0x7f8418159dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f8416800640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84180c6b70 0x7f8418159dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f8416800640 1 -- 172.21.15.146:0/4109668139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84180c75a0 msgr2=0x7f84181f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f8416800640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84180c75a0 0x7f84181f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f8416800640 1 -- 172.21.15.146:0/4109668139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841815ba00 con 0x7f841815b340 2024-09-16T01:51:40.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.489+0000 7f8416800640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418156ea0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8410009d90 tx=0x7f8410004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410040070 con 0x7f841815b340 2024-09-16T01:51:40.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8410031c80 con 0x7f841815b340 2024-09-16T01:51:40.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841003c620 con 0x7f841815b340 2024-09-16T01:51:40.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f234ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 0x7f23481f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f234ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480ca270 0x7f2348154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f2345e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 0x7f23481f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.490+0000 7f2345e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 0x7f23481f56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44038/0 (socket says 172.21.15.146:44038) 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f2345400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480ca270 0x7f2348154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f2345400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480ca270 0x7f2348154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55024/0 (socket says 172.21.15.146:55024) 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f234ca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23480cb3a0 0x7f2348155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f2345e00640 1 -- 172.21.15.146:0/3236789576 learned_addr learned my addr 172.21.15.146:0/3236789576 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23480cbad0 con 0x7f23480ca270 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23480cb3a0 0x7f2348155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f234815b970 con 0x7f23480cb3a0 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f23480c7770 con 0x7f23480c68f0 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345e00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23480cb3a0 msgr2=0x7f2348155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23480cb3a0 0x7f2348155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345e00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480ca270 msgr2=0x7f2348154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480ca270 0x7f2348154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345e00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23481554a0 con 0x7f23480c68f0 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23480ca270 0x7f2348154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8418159610 con 0x7f841815b340 2024-09-16T01:51:40.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.491+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841815a430 con 0x7f841815b340 2024-09-16T01:51:40.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.492+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 0x7f23481f56a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23380098c0 tx=0x7f23380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.493+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338037050 con 0x7f23480c68f0 2024-09-16T01:51:40.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.493+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f841806cd20 con 0x7f841815b340 2024-09-16T01:51:40.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.493+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2338043070 con 0x7f23480c68f0 2024-09-16T01:51:40.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.493+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338030b70 con 0x7f23480c68f0 2024-09-16T01:51:40.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.493+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2348072aa0 con 0x7f23480c68f0 2024-09-16T01:51:40.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.493+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2348155710 con 0x7f23480c68f0 2024-09-16T01:51:40.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.495+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f23480ca9c0 con 0x7f23480c68f0 2024-09-16T01:51:40.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.496+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8410039040 con 0x7f841815b340 2024-09-16T01:51:40.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.496+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2338030d10 con 0x7f23480c68f0 2024-09-16T01:51:40.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.497+0000 7f840e000640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83f806c300 0x7f83f806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.497+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5799+0+0 (secure 0 0 0) 0x7f84100acc20 con 0x7f841815b340 2024-09-16T01:51:40.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.498+0000 7f8415e00640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83f806c300 0x7f83f806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.498+0000 7f8415e00640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83f806c300 0x7f83f806e7c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f84080015c0 tx=0x7f8408007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.502+0000 7f233e000640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f232806c300 0x7f232806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.502+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f841007b280 con 0x7f841815b340 2024-09-16T01:51:40.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.502+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f232806c300 0x7f232806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.503+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(77..77 src has 1..77) ==== 5799+0+0 (secure 0 0 0) 0x7f23380ac3d0 con 0x7f23480c68f0 2024-09-16T01:51:40.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.503+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v77) ==== 54+0+16 (secure 0 0 0) 0x7f23480ca9c0 con 0x7f23480c68f0 2024-09-16T01:51:40.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.504+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23480c4100 0x7f23481571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.504+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23480c5650 con 0x7f23480c4100 2024-09-16T01:51:40.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.504+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23480c4100 0x7f23481571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.505+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f232806c300 0x7f232806e7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f234815a770 tx=0x7f2334009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.505+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23480c4100 0x7f23481571f0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.507+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23480c5650 con 0x7f23480c4100 2024-09-16T01:51:40.514 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:40.512+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:40.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.546+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f234815e000 con 0x7f23480c4100 2024-09-16T01:51:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.548+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f234815e000 con 0x7f23480c4100 2024-09-16T01:51:40.551 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.549+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23480c44e0 0x7f23481f8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.549+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f23480c85b0 con 0x7f23480c44e0 2024-09-16T01:51:40.552 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:40.552 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:40.552 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:40.552 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:40.552 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:40.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.551+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23480c44e0 0x7f23481f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.552+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23480c44e0 0x7f23481f8190 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.553+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23480c85b0 con 0x7f23480c44e0 2024-09-16T01:51:40.565 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:40.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.583+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2348157c40 con 0x7f23480c44e0 2024-09-16T01:51:40.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.587+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2348157c40 con 0x7f23480c44e0 2024-09-16T01:51:40.589 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:40.589 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:40.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.587+0000 7f2333400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23481f8be0 0x7f2348062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.587+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f23481f8fe0 con 0x7f23481f8be0 2024-09-16T01:51:40.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.599+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23481f8be0 0x7f2348062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.615 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:40.613+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:40.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.615+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23481f8be0 0x7f2348062630 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.616+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f233401b040 con 0x7f23481f8be0 2024-09-16T01:51:40.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.637+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f23480c3440 con 0x7f23481f8be0 2024-09-16T01:51:40.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.640+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23480c3440 con 0x7f23481f8be0 2024-09-16T01:51:40.644 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:40.645 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:40.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.641+0000 7f2333400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2348063080 0x7f234815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.641+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f23480c3440 con 0x7f2348063080 2024-09-16T01:51:40.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.642+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2348063080 0x7f234815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.643+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2348063080 0x7f234815bbf0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.649+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23480c3440 con 0x7f2348063080 2024-09-16T01:51:40.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.679+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f23480c3440 con 0x7f2348063080 2024-09-16T01:51:40.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.684+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f23480c3440 con 0x7f2348063080 2024-09-16T01:51:40.686 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:40.686 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.686+0000 7f2333400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f234815c640 0x7f2348209510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.686+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f234815cac0 con 0x7f234815c640 2024-09-16T01:51:40.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.688+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f234815c640 0x7f2348209510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.691+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f234815c640 0x7f2348209510 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.695+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f234815cac0 con 0x7f234815c640 2024-09-16T01:51:40.715 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:40.713+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:40.732 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:19 3.1f 3:f8cab2eb:::10000000002.00000010:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 50.331051232, 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075358271000000004, 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.733 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 19 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398080+0000", 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.398086+0000", 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2149999999999998e-06 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438214+0000", 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040127599999999999 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438231+0000", 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7028e-05 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438238+0000", 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8249999999999999e-06 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438293+0000", 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5007999999999999e-05 2024-09-16T01:51:40.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438402+0000", 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000109379 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.438491+0000", 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8774000000000001e-05 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.442442+0000", 2024-09-16T01:51:40.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039511320000000004 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473344+0000", 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030902466999999999 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473394+0000", 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9264000000000001e-05 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.473438+0000", 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4579000000000001e-05 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:24 3.0 3:028eacff:::10000000002.00000015:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 50.287169837999997, 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.073041954000000006, 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.739 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 24 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441961+0000", 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.441963+0000", 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7379999999999999e-06 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508113+0000", 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066149523000000002 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508127+0000", 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4678000000000001e-05 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508135+0000", 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4959999999999999e-06 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508200+0000", 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5062999999999997e-05 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508374+0000", 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000174322 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.508512+0000", 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013740099999999999 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514816+0000", 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0063044199999999998 2024-09-16T01:51:40.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514914+0000", 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7590000000000006e-05 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.514951+0000", 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7348999999999998e-05 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.515003+0000", 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2373999999999997e-05 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 50.205931077000002, 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.745 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:51:40.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:122 3.0 3:00eacacc:::10000000002.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.359217831000002, 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068671870999999995, 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.751 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 122 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369913+0000", 2024-09-16T01:51:40.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.369915+0000", 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5990000000000001e-06 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434232+0000", 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064317038000000007 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434250+0000", 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8028000000000001e-05 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434258+0000", 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6720000000000004e-06 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434311+0000", 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2914999999999997e-05 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434411+0000", 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100045 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.434501+0000", 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494000000000005e-05 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.437130+0000", 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026284059999999998 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438521+0000", 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0013915290000000001 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438553+0000", 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1992999999999997e-05 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.438585+0000", 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2152e-05 2024-09-16T01:51:40.756 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:545 3.38 3:1d8b64ca:::100000001f7.0000001e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.101476097999999, 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075800996999999995, 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.757 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 545 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627655+0000", 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.627656+0000", 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.176e-06 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655590+0000", 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027933824999999999 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655601+0000", 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1270999999999999e-05 2024-09-16T01:51:40.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655608+0000", 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.55e-06 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655659+0000", 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0946999999999999e-05 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655756+0000", 2024-09-16T01:51:40.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7448000000000006e-05 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.655857+0000", 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100863 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.659273+0000", 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034161180000000001 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703393+0000", 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044119393 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703427+0000", 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4063000000000002e-05 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.703456+0000", 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9343e-05 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.762 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:546 3.38 3:1d0ca575:::100000001f7.0000001f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.068211688000002, 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068250515999999997, 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.763 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 546 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660920+0000", 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.660921+0000", 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.539e-06 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709606+0000", 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048684573000000002 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709617+0000", 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1447999999999999e-05 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709622+0000", 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3000000000000001e-06 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709691+0000", 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8879000000000002e-05 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709788+0000", 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6768999999999995e-05 2024-09-16T01:51:40.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.709893+0000", 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010466799999999999 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.715292+0000", 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005399165 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729109+0000", 2024-09-16T01:51:40.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013817352 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729143+0000", 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4059e-05 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.729170+0000", 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6764e-05 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:574 3.27 3:e46e0c7f:::100000001f7.0000003b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.724475131999998, 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084267634999999994, 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 574 2024-09-16T01:51:40.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:51:40.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004659+0000", 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5129999999999999e-06 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032641+0000", 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027982080999999999 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032656+0000", 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5699999999999999e-05 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032666+0000", 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5599999999999999e-06 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032719+0000", 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3066999999999999e-05 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032827+0000", 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010841900000000001 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032929+0000", 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101128 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.036882+0000", 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039538949999999998 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088843+0000", 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051960682000000001 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088878+0000", 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.485e-05 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088924+0000", 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5739999999999999e-05 2024-09-16T01:51:40.774 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:590 3.29 3:966924c4:::100000001f7.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.612964107, 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074607536000000002, 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.775 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 590 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116167+0000", 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.116170+0000", 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162459+0000", 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046289624000000001 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162473+0000", 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3776e-05 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162481+0000", 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2889999999999998e-06 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162559+0000", 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7101000000000006e-05 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162675+0000", 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116933 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.162764+0000", 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8343000000000004e-05 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.167505+0000", 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047412519999999996 2024-09-16T01:51:40.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190706+0000", 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023201366000000001 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190742+0000", 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5757999999999999e-05 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.190775+0000", 2024-09-16T01:51:40.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2484999999999998e-05 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:625 3.3d 3:bee2a002:::100000001f7.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.288487676999999, 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067884265999999999, 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.781 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 625 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440644+0000", 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.440646+0000", 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0710000000000002e-06 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502844+0000", 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062197927 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502866+0000", 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2580999999999999e-05 2024-09-16T01:51:40.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502877+0000", 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0823e-05 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.502939+0000", 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2360000000000006e-05 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.503095+0000", 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015614099999999999 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.503243+0000", 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014767400000000001 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.507276+0000", 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040332340000000001 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508460+0000", 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011837740000000001 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508496+0000", 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5540999999999997e-05 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.508528+0000", 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2140000000000001e-05 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.786 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:670 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.848434953000002, 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066678565999999995, 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 670 2024-09-16T01:51:40.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880696+0000", 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.880698+0000", 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2890000000000001e-06 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903674+0000", 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022976465000000001 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903699+0000", 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4675e-05 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903712+0000", 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2935000000000001e-05 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903788+0000", 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6676999999999998e-05 2024-09-16T01:51:40.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.903928+0000", 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013916300000000001 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.904044+0000", 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011618200000000001 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.907487+0000", 2024-09-16T01:51:40.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034430839999999999 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947289+0000", 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039801925000000002 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947330+0000", 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.121e-05 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.947375+0000", 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4960999999999999e-05 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:712 3.1e 3:7933c15e:::100000001f7.000000c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.443216141000001, 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067308615000000002, 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.793 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 712 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285915+0000", 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.285917+0000", 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0370000000000001e-06 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330533+0000", 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044615783999999999 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330550+0000", 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6878999999999999e-05 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330558+0000", 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7000000000000008e-06 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330621+0000", 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3100000000000002e-05 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330773+0000", 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015215199999999999 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.330932+0000", 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015931100000000001 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.335841+0000", 2024-09-16T01:51:40.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004909348 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353141+0000", 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017299833000000001 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353182+0000", 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0849000000000001e-05 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.353224+0000", 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1622000000000002e-05 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:872 3.18 3:19643448:::100000001f7.00000065:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.579486486, 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.069066814000000004, 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.799 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 872 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149645+0000", 2024-09-16T01:51:40.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.149650+0000", 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8899999999999998e-06 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209584+0000", 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059934556 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209621+0000", 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6378999999999998e-05 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209629+0000", 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6410000000000008e-06 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209715+0000", 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6013999999999993e-05 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209807+0000", 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2020000000000003e-05 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.209966+0000", 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158889 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.216152+0000", 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006185467 2024-09-16T01:51:40.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218614+0000", 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024625630000000001 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218669+0000", 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5104e-05 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.218712+0000", 2024-09-16T01:51:40.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2290999999999999e-05 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1033 3.29 3:94178d8a:::100000001f8.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 2.5382640969999999, 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086462834000000002, 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.805 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1033 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190869+0000", 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3099999999999999e-06 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224163+0000", 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.033293469999999999 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224185+0000", 2024-09-16T01:51:40.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1987000000000001e-05 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224197+0000", 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.222e-05 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224896+0000", 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00069902199999999999 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.225064+0000", 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016828399999999999 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.225208+0000", 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014386800000000001 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.231378+0000", 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006169796 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277260+0000", 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045882091999999999 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277295+0000", 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4505999999999998e-05 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277330+0000", 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5278999999999999e-05 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.810 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1063 3.31 3:8fb8ba0c:::100000001f8.00000024:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.566900+0000", 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 2.16223163, 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071814635000000002, 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1063 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.566900+0000", 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.566900+0000", 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.566901+0000", 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.691e-06 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.617790+0000", 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050888744 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.617807+0000", 2024-09-16T01:51:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7070000000000001e-05 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.617815+0000", 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4059999999999999e-06 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.617892+0000", 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7107000000000006e-05 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.618008+0000", 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011644199999999999 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.618111+0000", 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102979 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.622780+0000", 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046689460000000002 2024-09-16T01:51:40.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.638645+0000", 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015865382000000001 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.638681+0000", 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5286000000000001e-05 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.638714+0000", 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3581999999999998e-05 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1133 3.38 3:1cae2fc4:::100000001f8.0000006a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.360979+0000", 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.3681520039999999, 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083133998000000001, 2024-09-16T01:51:40.817 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1133 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.360979+0000", 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.360979+0000", 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.360980+0000", 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2309999999999999e-06 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404652+0000", 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.043671158000000002 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404670+0000", 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8031999999999999e-05 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404681+0000", 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1781e-05 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404749+0000", 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7257999999999999e-05 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404891+0000", 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014246899999999999 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.405017+0000", 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000126174 2024-09-16T01:51:40.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.408883+0000", 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038658730000000001 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.444000+0000", 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035117122000000001 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.444060+0000", 2024-09-16T01:51:40.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9781000000000002e-05 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.444113+0000", 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3118999999999997e-05 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1137 3.29 3:9673bc10:::100000001f8.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.3242700680000001, 2024-09-16T01:51:40.823 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084134489000000007, 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1137 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:51:40.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404862+0000", 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0270000000000001e-06 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458066+0000", 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053203817 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458080+0000", 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3536999999999999e-05 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458085+0000", 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8130000000000001e-06 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458167+0000", 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1797000000000003e-05 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458306+0000", 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013863699999999999 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458472+0000", 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016593700000000001 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.462073+0000", 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003601131 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488893+0000", 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026819912000000001 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488949+0000", 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5889e-05 2024-09-16T01:51:40.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488996+0000", 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6992e-05 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1174 3.31 3:8e8a532d:::100000001f8.00000093:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.805466+0000", 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.92366516799999998, 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.069711801000000004, 2024-09-16T01:51:40.829 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1174 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.805466+0000", 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.805466+0000", 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.805468+0000", 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7290000000000001e-06 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.849896+0000", 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044428648000000001 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.849922+0000", 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.525e-05 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.849934+0000", 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2429000000000001e-05 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.850012+0000", 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7849000000000006e-05 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.850147+0000", 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013533800000000001 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.850270+0000", 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012238900000000001 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.854791+0000", 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045208100000000001 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.875091+0000", 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0203003 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.875135+0000", 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4635999999999998e-05 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.875178+0000", 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2422999999999997e-05 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1181 3.38 3:1e6b0f01:::100000001f8.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.878999+0000", 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.85013182300000001, 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.080968997000000001, 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.835 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1181 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.878999+0000", 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.878999+0000", 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.879001+0000", 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2619999999999999e-06 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926095+0000", 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047094233999999999 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926110+0000", 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5279999999999999e-05 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926120+0000", 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0052e-05 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926184+0000", 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3660999999999999e-05 2024-09-16T01:51:40.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926325+0000", 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014081400000000001 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926480+0000", 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000154857 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.930130+0000", 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036503460000000001 2024-09-16T01:51:40.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.959902+0000", 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029772526000000001 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.959938+0000", 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6003000000000001e-05 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.959968+0000", 2024-09-16T01:51:40.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9961999999999999e-05 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1204 3.1e 3:7a97cce9:::100000001f8.000000b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.135683+0000", 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.59344798600000004, 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.080493544, 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.841 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1204 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.135683+0000", 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.135683+0000", 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.135685+0000", 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.305e-06 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194149+0000", 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058464147000000001 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194163+0000", 2024-09-16T01:51:40.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3910999999999999e-05 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194170+0000", 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0910000000000004e-06 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194245+0000", 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5118999999999997e-05 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194387+0000", 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014192799999999999 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194503+0000", 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011599600000000001 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.198531+0000", 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040281029999999999 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.216096+0000", 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017565537999999999 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.216136+0000", 2024-09-16T01:51:40.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9527999999999998e-05 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.216177+0000", 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0877999999999997e-05 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1207 3.17 3:ea5aa73c:::100000001f8.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.194188+0000", 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.53494278100000003, 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067499686000000003, 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:40.849 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1207 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194188+0000", 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194188+0000", 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:40.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194189+0000", 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.239966+0000", 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.04577672 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.239983+0000", 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6294999999999999e-05 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.239990+0000", 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8800000000000008e-06 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:40.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.240050+0000", 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9932999999999997e-05 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.240159+0000", 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010877700000000001 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.240261+0000", 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101909 2024-09-16T01:51:40.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.243261+0000", 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029996760000000002 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.261586+0000", 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.018325049 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.261643+0000", 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7219000000000002e-05 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.261688+0000", 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5207999999999998e-05 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:40.856 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:40.856 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:40.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.737+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f234815cac0 con 0x7f234815c640 2024-09-16T01:51:40.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.740+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f234815cac0 con 0x7f234815c640 2024-09-16T01:51:40.858 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:40.859 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:40.859 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:40.855+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:40.859 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:40.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.744+0000 7f2333400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f2348209f60 0x7f234820c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.744+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f234820f120 con 0x7f2348209f60 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.745+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f2348209f60 0x7f234820c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.745+0000 7f2345400640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f2348209f60 0x7f234820c350 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.749+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f234820f120 con 0x7f2348209f60 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.821+0000 7f2333400640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f234820c910 con 0x7f2348209f60 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.823+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f234820c910 con 0x7f2348209f60 2024-09-16T01:51:40.860 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:40.861 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:40.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.824+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f231c001650 0x7f231c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.824+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f231c006bb0 con 0x7f231c001650 2024-09-16T01:51:40.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.825+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f231c001650 0x7f231c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.834+0000 7f2346800640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f231c001650 0x7f231c003b10 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.846+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f231c006bb0 con 0x7f231c001650 2024-09-16T01:51:40.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.881+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fdfd80cf0d0 con 0x7fdfd80d65a0 2024-09-16T01:51:40.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.906+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f231c0058d0 con 0x7f231c001650 2024-09-16T01:51:40.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.909+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f231c0058d0 con 0x7f231c001650 2024-09-16T01:51:40.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.909+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f84180ca280 con 0x7f841815b340 2024-09-16T01:51:40.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.911+0000 7f840e000640 1 -- 172.21.15.146:0/4109668139 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v77) ==== 81+0+34868 (secure 0 0 0) 0x7f841007ab80 con 0x7f841815b340 2024-09-16T01:51:40.913 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:51:40.914 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:40.914 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 77, 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "created": "2024-09-16T01:48:34.399638+0000", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "modified": "2024-09-16T01:51:40.006327+0000", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "last_up_change": "2024-09-16T01:48:45.748664+0000", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "last_in_change": "2024-09-16T01:51:17.905443+0000", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_num": 5799936, 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_set": [ 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "pglog_hardlimit", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "purged_snapdirs", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_deletes", 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "sortbitwise" 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:40.915 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_version": 6, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_snapshot": "", 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_max": 3, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd": 8, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "require_min_compat_client": "luminous", 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "min_compat_client": "luminous", 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "require_osd_release": "squid", 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_crimson": false, 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "pools": [ 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.916 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 1, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": ".mgr", 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:46.570355+0000", 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:51:40.917 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 1, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 1, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 1, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 1, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 1, 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:51:40.918 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "18", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:51:40.919 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:51:40.920 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:51:40.921 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_max": 32, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 1 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": {} 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.922 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 6.9000000953674316, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 6.9000000953674316, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.28999999165534973, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 2, 2024-09-16T01:51:40.923 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_metadata", 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:56.008811+0000", 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:51:40.924 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 32, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 32, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 32, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 32, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 32, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:51:40.925 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "21", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:51:40.926 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:51:40.927 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:51:40.928 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_bias": 4, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 16, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_priority": 5 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": "cephfs" 2024-09-16T01:51:40.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.9700000286102295, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.9700000286102295, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2.25, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2.25, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 3, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_data", 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:57.207121+0000", 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:51:40.931 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 64, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 64, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 64, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 64, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 64, 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:51:40.932 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "22", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:51:40.933 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:51:40.934 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:51:40.935 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 64, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "target_size_ratio": 0.89999997615814209 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: "data": "cephfs" 2024-09-16T01:51:40.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.4299999475479126, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.4299999475479126, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 1.1399999856948853, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 1.6299999952316284, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 1.6299999952316284, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "osds": [ 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 72, 2024-09-16T01:51:40.938 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6824", 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6825", 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.939 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6826", 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6827", 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.940 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6830", 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6831", 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.941 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6828", 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6829", 2024-09-16T01:51:40.942 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6825/61440064", 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6827/61440064", 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6831/61440064", 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6829/61440064", 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.943 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0.93414306640625, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6808", 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6809", 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.945 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6810", 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6811", 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.946 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6814", 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6815", 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.947 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6812", 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6813", 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.948 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6809/2375201615", 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6811/2375201615", 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6815/2375201615", 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6813/2375201615", 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.949 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.950 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6800", 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6801", 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.951 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6802", 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6803", 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6806", 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6807", 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6804", 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6805", 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.954 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6801/1818492127", 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6803/1818492127", 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6807/1818492127", 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6805/1818492127", 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.955 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.956 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6816", 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6817", 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6818", 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6819", 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6822", 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6823", 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6820", 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6821", 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6817/1633804394", 2024-09-16T01:51:40.960 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6819/1633804394", 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1633804394", 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6821/1633804394", 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0.8634185791015625, 2024-09-16T01:51:40.961 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 13, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6800", 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6801", 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6802", 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6803", 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6806", 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6807", 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.965 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6804", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6805", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6801/154390265", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6803/154390265", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6807/154390265", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6805/154390265", 2024-09-16T01:51:40.966 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.967 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6824", 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.968 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6825", 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6826", 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6827", 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.970 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6830", 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6831", 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6828", 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6829", 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6825/356357279", 2024-09-16T01:51:40.972 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6827/356357279", 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6831/356357279", 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6829/356357279", 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.973 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6808", 2024-09-16T01:51:40.974 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6809", 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6810", 2024-09-16T01:51:40.975 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6811", 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.976 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6814", 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6815", 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.977 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6812", 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6813", 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6809/3661001615", 2024-09-16T01:51:40.978 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6811/3661001615", 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6815/3661001615", 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6813/3661001615", 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:51:40.979 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 0, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 35, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6816", 2024-09-16T01:51:40.980 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6817", 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6818", 2024-09-16T01:51:40.981 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6819", 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6822", 2024-09-16T01:51:40.982 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6823", 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6820", 2024-09-16T01:51:40.983 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6821", 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6817/1279166573", 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6819/1279166573", 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6823/1279166573", 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6821/1279166573", 2024-09-16T01:51:40.984 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_xinfo": [ 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.985 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.135308+0000", 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.290750+0000", 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.712744+0000", 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.987 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.988678+0000", 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.988 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.198114+0000", 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.085332+0000", 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.394321+0000", 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:51:40.990 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.236983+0000", 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap": [], 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_items": [ 2024-09-16T01:51:40.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.2c", 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 0, 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 7 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "pgid": "3.3b", 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "mappings": [ 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:40.992 INFO:teuthology.orchestra.run.smithi146.stdout: "from": 0, 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "to": 7 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_primaries": [], 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_temp": [], 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_temp": [], 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklist": {}, 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "range_blocklist": {}, 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profiles": { 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "default": { 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-failure-domain": "osd", 2024-09-16T01:51:40.993 INFO:teuthology.orchestra.run.smithi146.stdout: "k": "2", 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "m": "1", 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin": "jerasure", 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "technique": "reed_sol_van" 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps_queue": [], 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "new_removed_snaps": [], 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "new_purged_snaps": [], 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_node_flags": {}, 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "device_class_flags": {}, 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode": { 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_enabled": false, 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_bucket_count": 0, 2024-09-16T01:51:40.994 INFO:teuthology.orchestra.run.smithi146.stdout: "degraded_stretch_mode": 0, 2024-09-16T01:51:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "recovering_stretch_mode": 0, 2024-09-16T01:51:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_bucket": 0 2024-09-16T01:51:40.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:40.995 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:40.996 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:40.996 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:40.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.913+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f231c004810 0x7f231c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:40.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.913+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f231c009f20 con 0x7f231c004810 2024-09-16T01:51:40.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.914+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f231c004810 0x7f231c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:40.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.914+0000 7f2345e00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f231c004810 0x7f231c004cc0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:40.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.916+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f231c009f20 con 0x7f231c004810 2024-09-16T01:51:40.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.921+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83f806c300 msgr2=0x7f83f806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.921+0000 7f8417200640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83f806c300 0x7f83f806e7c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f84080015c0 tx=0x7f8408007930 comp rx=0 tx=0).stop 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.921+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 msgr2=0x7f8418156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.921+0000 7f8417200640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418156ea0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8410009d90 tx=0x7f8410004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.923+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 shutdown_connections 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.923+0000 7f8417200640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83f806c300 0x7f83f806e7c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.923+0000 7f8417200640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f841815b340 0x7f8418156ea0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.923+0000 7f8417200640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84180c75a0 0x7f84181f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.923+0000 7f8417200640 1 --2- 172.21.15.146:0/4109668139 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84180c6b70 0x7f8418159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:40.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.923+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 >> 172.21.15.146:0/4109668139 conn(0x7f84180ccc20 msgr2=0x7f84180cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:40.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.926+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 shutdown_connections 2024-09-16T01:51:40.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.926+0000 7f8417200640 1 -- 172.21.15.146:0/4109668139 wait complete. 2024-09-16T01:51:40.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.952+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f231c008d50 con 0x7f231c004810 2024-09-16T01:51:40.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.954+0000 7f233e000640 1 -- 172.21.15.146:0/3236789576 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f231c008d50 con 0x7f231c004810 2024-09-16T01:51:40.999 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:41.001 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:40.994+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:41.001 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:41.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23480c4100 msgr2=0x7f23481571f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f23480c4100 0x7f23481571f0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23480c44e0 msgr2=0x7f23481f8190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f23480c44e0 0x7f23481f8190 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23481f8be0 msgr2=0x7f2348062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f23481f8be0 0x7f2348062630 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2348063080 msgr2=0x7f234815bbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f2348063080 0x7f234815bbf0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f234815c640 msgr2=0x7f2348209510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f234815c640 0x7f2348209510 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f2348209f60 msgr2=0x7f234820c350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f2348209f60 0x7f234820c350 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f231c001650 msgr2=0x7f231c003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f231c001650 0x7f231c003b10 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f231c004810 msgr2=0x7f231c004cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f231c004810 0x7f231c004cc0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f232806c300 msgr2=0x7f232806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f232806c300 0x7f232806e7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f234815a770 tx=0x7f2334009190 comp rx=0 tx=0).stop 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f2346800640 1 -- 172.21.15.146:0/3236789576 reap_dead start 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f2346800640 1 -- 172.21.15.146:0/3236789576 reap_dead start 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f2346800640 1 -- 172.21.15.146:0/3236789576 reap_dead start 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f2346800640 1 -- 172.21.15.146:0/3236789576 reap_dead start 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 msgr2=0x7f23481f56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f2346800640 1 -- 172.21.15.146:0/3236789576 reap_dead start 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 0x7f23481f56a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23380098c0 tx=0x7f23380046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 shutdown_connections 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 --2- 172.21.15.146:0/3236789576 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23480c68f0 0x7f23481f56a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.955+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 >> 172.21.15.146:0/3236789576 conn(0x7f234806cb70 msgr2=0x7f23480cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:41.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.956+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 shutdown_connections 2024-09-16T01:51:41.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:40.956+0000 7f234ca00640 1 -- 172.21.15.146:0/3236789576 wait complete. 2024-09-16T01:51:41.005 INFO:tasks.thrashosds.thrasher:j is {'epoch': 77, 'fsid': '4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d', 'created': '2024-09-16T01:48:34.399638+0000', 'modified': '2024-09-16T01:51:40.006327+0000', 'last_up_change': '2024-09-16T01:48:45.748664+0000', 'last_in_change': '2024-09-16T01:51:17.905443+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:48:46.570355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:48:56.008811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:48:57.207121+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4299999475479126, 'score_stable': 1.4299999475479126, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e7758ddf-75ff-43c7-abae-3647b170f7c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 61440064}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 61440064}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 61440064}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 61440064}]}, 'public_addr': '172.21.15.146:6825/61440064', 'cluster_addr': '172.21.15.146:6827/61440064', 'heartbeat_back_addr': '172.21.15.146:6831/61440064', 'heartbeat_front_addr': '172.21.15.146:6829/61440064', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a065e77e-26f0-4245-abe8-e36c2fb4436b', 'up': 1, 'in': 1, 'weight': 0.93414306640625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 2375201615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 2375201615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 2375201615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 2375201615}]}, 'public_addr': '172.21.15.146:6809/2375201615', 'cluster_addr': '172.21.15.146:6811/2375201615', 'heartbeat_back_addr': '172.21.15.146:6815/2375201615', 'heartbeat_front_addr': '172.21.15.146:6813/2375201615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fefb6d8f-6799-4344-a977-85ab983ec0cb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 1818492127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 1818492127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 1818492127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 1818492127}]}, 'public_addr': '172.21.15.146:6801/1818492127', 'cluster_addr': '172.21.15.146:6803/1818492127', 'heartbeat_back_addr': '172.21.15.146:6807/1818492127', 'heartbeat_front_addr': '172.21.15.146:6805/1818492127', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9d1a2f96-efcb-4a73-b5db-ec5e2a63122a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 1633804394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 1633804394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1633804394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 1633804394}]}, 'public_addr': '172.21.15.146:6817/1633804394', 'cluster_addr': '172.21.15.146:6819/1633804394', 'heartbeat_back_addr': '172.21.15.146:6823/1633804394', 'heartbeat_front_addr': '172.21.15.146:6821/1633804394', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c', 'up': 1, 'in': 1, 'weight': 0.8634185791015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6800', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6801', 'nonce': 154390265}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6802', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6803', 'nonce': 154390265}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6806', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6807', 'nonce': 154390265}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6804', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6805', 'nonce': 154390265}]}, 'public_addr': '172.21.15.161:6801/154390265', 'cluster_addr': '172.21.15.161:6803/154390265', 'heartbeat_back_addr': '172.21.15.161:6807/154390265', 'heartbeat_front_addr': '172.21.15.161:6805/154390265', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '11562dd5-d537-4d88-a282-e8687caebad6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6824', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6825', 'nonce': 356357279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6826', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6827', 'nonce': 356357279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6830', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6831', 'nonce': 356357279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6828', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6829', 'nonce': 356357279}]}, 'public_addr': '172.21.15.161:6825/356357279', 'cluster_addr': '172.21.15.161:6827/356357279', 'heartbeat_back_addr': '172.21.15.161:6831/356357279', 'heartbeat_front_addr': '172.21.15.161:6829/356357279', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b31adf01-6768-49cf-a232-127b9d0d2dc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6808', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6809', 'nonce': 3661001615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6810', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6811', 'nonce': 3661001615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6814', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6815', 'nonce': 3661001615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6812', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6813', 'nonce': 3661001615}]}, 'public_addr': '172.21.15.161:6809/3661001615', 'cluster_addr': '172.21.15.161:6811/3661001615', 'heartbeat_back_addr': '172.21.15.161:6815/3661001615', 'heartbeat_front_addr': '172.21.15.161:6813/3661001615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '67c95c28-b600-430c-9474-a902cf46a73b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6816', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6817', 'nonce': 1279166573}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6818', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6819', 'nonce': 1279166573}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6822', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6823', 'nonce': 1279166573}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6820', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6821', 'nonce': 1279166573}]}, 'public_addr': '172.21.15.161:6817/1279166573', 'cluster_addr': '172.21.15.161:6819/1279166573', 'heartbeat_back_addr': '172.21.15.161:6823/1279166573', 'heartbeat_front_addr': '172.21.15.161:6821/1279166573', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.135308+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.290750+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.712744+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.988678+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.198114+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.085332+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.394321+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.236983+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.2c', 'mappings': [{'from': 0, 'to': 7}]}, {'pgid': '3.3b', 'mappings': [{'from': 0, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T01:51:41.005 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.3b 2024-09-16T01:51:41.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.3b 2024-09-16T01:51:41.022 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:41.022 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:41.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.025+0000 7fdfd6a00640 1 -- 172.21.15.146:0/1394447596 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v78) ==== 87+0+0 (secure 0 0 0) 0x7fdfe00d4660 con 0x7fdfd80d65a0 2024-09-16T01:51:41.029 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:51:41.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:41.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfc006c330 msgr2=0x7fdfc006e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfc006c330 0x7fdfc006e7f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0006fd0 tx=0x7fdfd0008040 comp rx=0 tx=0).stop 2024-09-16T01:51:41.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfd80d65a0 msgr2=0x7fdfd8044da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfd80d65a0 0x7fdfd8044da0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0051790 tx=0x7fdfe0069890 comp rx=0 tx=0).stop 2024-09-16T01:51:41.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 shutdown_connections 2024-09-16T01:51:41.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfc006c330 0x7fdfc006e7f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfd80d65a0 0x7fdfd8044da0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8050900 0x7fdfd8048c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 --2- 172.21.15.146:0/1394447596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfd804b960 0x7fdfd8049c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.042+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 >> 172.21.15.146:0/1394447596 conn(0x7fdfd804fc50 msgr2=0x7fdfd80da020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:41.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.043+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 shutdown_connections 2024-09-16T01:51:41.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.043+0000 7fdfdfe00640 1 -- 172.21.15.146:0/1394447596 wait complete. 2024-09-16T01:51:41.132 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:41.127+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:41.198 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:41.196+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:41.222 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:41.222 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:41.233 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:41.302 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:41.297+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:41.392 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:41.393 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:41.400 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:41.398+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:41.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:41.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.410+0000 7fe314400640 1 Processor -- start 2024-09-16T01:51:41.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.411+0000 7fe314400640 1 -- start start 2024-09-16T01:51:41.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.412+0000 7fe314400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe30c0c7f30 0x7fe30c0c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.412+0000 7fe314400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe30c0c7a60 0x7fe30c06dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.412+0000 7fe314400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe30c155110 con 0x7fe30c154d30 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.412+0000 7fe314400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe30c0c6bb0 con 0x7fe30c0c7f30 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.412+0000 7fe314400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe30c073140 con 0x7fe30c0c7a60 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe311a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe30c154d30 0x7fe30c0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:54392/0 (socket says 172.21.15.146:54392) 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe311a00640 1 -- 172.21.15.146:0/1947835436 learned_addr learned my addr 172.21.15.146:0/1947835436 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:41.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe312400640 1 --2- 172.21.15.146:0/1947835436 >> v2:172.21.15.161:3300/0 conn(0x7fe30c0c7a60 0x7fe30c06dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe311000640 1 --2- 172.21.15.146:0/1947835436 >> v2:172.21.15.146:3300/0 conn(0x7fe30c0c7f30 0x7fe30c0c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe312400640 1 -- 172.21.15.146:0/1947835436 >> v2:172.21.15.146:3300/0 conn(0x7fe30c0c7f30 msgr2=0x7fe30c0c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe312400640 1 --2- 172.21.15.146:0/1947835436 >> v2:172.21.15.146:3300/0 conn(0x7fe30c0c7f30 0x7fe30c0c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe312400640 1 -- 172.21.15.146:0/1947835436 >> v1:172.21.15.146:6789/0 conn(0x7fe30c154d30 legacy=0x7fe30c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.413+0000 7fe312400640 1 -- 172.21.15.146:0/1947835436 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c0c6e00 con 0x7fe30c0c7a60 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.415+0000 7fe312400640 1 --2- 172.21.15.146:0/1947835436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c0c7a60 0x7fe30c06dae0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe30c06cc90 tx=0x7fe30402a630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d295c43302b4383d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.415+0000 7fe30be00640 1 -- 172.21.15.146:0/1947835436 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304007740 con 0x7fe30c0c7a60 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.415+0000 7fe30be00640 1 -- 172.21.15.146:0/1947835436 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3040078e0 con 0x7fe30c0c7a60 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.415+0000 7fe30be00640 1 -- 172.21.15.146:0/1947835436 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304007740 con 0x7fe30c0c7a60 2024-09-16T01:51:41.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 -- 172.21.15.146:0/1947835436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c0c7a60 msgr2=0x7fe30c06dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 --2- 172.21.15.146:0/1947835436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c0c7a60 0x7fe30c06dae0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe30c06cc90 tx=0x7fe30402a630 comp rx=0 tx=0).stop 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 -- 172.21.15.146:0/1947835436 shutdown_connections 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 --2- 172.21.15.146:0/1947835436 >> v2:172.21.15.146:3300/0 conn(0x7fe30c0c7f30 0x7fe30c0c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 --2- 172.21.15.146:0/1947835436 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c0c7a60 0x7fe30c06dae0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 -- 172.21.15.146:0/1947835436 >> 172.21.15.146:0/1947835436 conn(0x7fe30c06f3c0 msgr2=0x7fe30c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 -- 172.21.15.146:0/1947835436 shutdown_connections 2024-09-16T01:51:41.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.416+0000 7fe314400640 1 -- 172.21.15.146:0/1947835436 wait complete. 2024-09-16T01:51:41.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.418+0000 7fe314400640 1 Processor -- start 2024-09-16T01:51:41.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.418+0000 7fe314400640 1 -- start start 2024-09-16T01:51:41.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.419+0000 7fe314400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe30c0c7f30 0x7fe30c1428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:41.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.420+0000 7fe314400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 0x7fe30c146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:41.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.420+0000 7fe314400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe30c140a20 0x7fe30c145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:41.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.420+0000 7fe314400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c072e00 con 0x7fe30c0c7f30 2024-09-16T01:51:41.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.420+0000 7fe314400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c072920 con 0x7fe30c154d30 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.420+0000 7fe314400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c0718c0 con 0x7fe30c140a20 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.421+0000 7fe311000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 0x7fe30c146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.421+0000 7fe311000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 0x7fe30c146950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60318/0 (socket says 172.21.15.146:60318) 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.421+0000 7fe311000640 1 -- 172.21.15.146:0/1482778157 learned_addr learned my addr 172.21.15.146:0/1482778157 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe311000640 1 -- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe30c140a20 msgr2=0x7fe30c145570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe311000640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe30c140a20 0x7fe30c145570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe311000640 1 -- 172.21.15.146:0/1482778157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe30c0c7f30 msgr2=0x7fe30c1428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe311000640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe30c0c7f30 0x7fe30c1428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe311000640 1 -- 172.21.15.146:0/1482778157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c15b8e0 con 0x7fe30c154d30 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe311000640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 0x7fe30c146950 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800e9b0 tx=0x7fe2f800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8017070 con 0x7fe30c154d30 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f800c9a0 con 0x7fe30c154d30 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.422+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800cd90 con 0x7fe30c154d30 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.423+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c143e90 con 0x7fe30c154d30 2024-09-16T01:51:41.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.423+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c1e61a0 con 0x7fe30c154d30 2024-09-16T01:51:41.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.427+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe30c144ae0 con 0x7fe30c154d30 2024-09-16T01:51:41.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.427+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe2f801a070 con 0x7fe30c154d30 2024-09-16T01:51:41.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.429+0000 7fe309600640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe2ec06c380 0x7fe2ec06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:41.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.429+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5799+0+0 (secure 0 0 0) 0x7fe2f808b530 con 0x7fe30c154d30 2024-09-16T01:51:41.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.430+0000 7fe311a00640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe2ec06c380 0x7fe2ec06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:41.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.430+0000 7fe311a00640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe2ec06c380 0x7fe2ec06e840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe300007fd0 tx=0x7fe3000079a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:41.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.434+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe2f8059b90 con 0x7fe30c154d30 2024-09-16T01:51:41.502 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:41.500+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:41.566 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:41.566 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:41.575 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:41.610 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:41.604+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:41.701 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:41.701 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:41.703 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:41.701+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:41.711 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:41.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:41.799+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "3.3b"} v 0) -- 0x7fe30c0c5f00 con 0x7fe30c154d30 2024-09-16T01:51:41.804 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:41.802+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:41.872 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:41.872 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:41.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:41.922 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:41.920+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:42.008 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:42.005+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:42.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.057+0000 7fe309600640 1 -- 172.21.15.146:0/1482778157 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3b"}]=0 clear 3.3b pg_upmap_items mapping v79) ==== 118+0+0 (secure 0 0 0) 0x7fe2f8059490 con 0x7fe30c154d30 2024-09-16T01:51:42.063 INFO:teuthology.orchestra.run.smithi146.stderr:clear 3.3b pg_upmap_items mapping 2024-09-16T01:51:42.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe2ec06c380 msgr2=0x7fe2ec06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:42.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe2ec06c380 0x7fe2ec06e840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe300007fd0 tx=0x7fe3000079a0 comp rx=0 tx=0).stop 2024-09-16T01:51:42.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 msgr2=0x7fe30c146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:42.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 0x7fe30c146950 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800e9b0 tx=0x7fe2f800ee70 comp rx=0 tx=0).stop 2024-09-16T01:51:42.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 shutdown_connections 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe2ec06c380 0x7fe2ec06e840 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe300007fd0 tx=0x7fe3000079a0 comp rx=0 tx=0).stop 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe30c140a20 0x7fe30c145570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe30c154d30 0x7fe30c146950 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 --2- 172.21.15.146:0/1482778157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe30c0c7f30 0x7fe30c1428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.069+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 >> 172.21.15.146:0/1482778157 conn(0x7fe30c06f3c0 msgr2=0x7fe30c0c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.070+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 shutdown_connections 2024-09-16T01:51:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:42.070+0000 7fe314400640 1 -- 172.21.15.146:0/1482778157 wait complete. 2024-09-16T01:51:42.108 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:42.106+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:42.209 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:42.207+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:42.236 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:42.237 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:42.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:42.309 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:42.307+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:42.410 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:42.408+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:42.511 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:42.509+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:42.527 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:42.528 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:42.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:42.658 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:42.652+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:42.715 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:42.711+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:42.813 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:42.812+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:42.877 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:42.877 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:42.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:42.932 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:42.930+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:43.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:43.015 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:43.012+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:43.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:51:43.123 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:43.122+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:43.205 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:43.205 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:43.216 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:43.214+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:43.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:43.318 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:43.316+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:43.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.366+0000 7fdfd1000640 1 Processor -- start 2024-09-16T01:51:43.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.367+0000 7fdfd1000640 1 -- start start 2024-09-16T01:51:43.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.369+0000 7fdfd1000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdfcc0c13f0 0x7fdfcc02d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.369+0000 7fdfd1000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.369+0000 7fdfd1000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfcc0c9c50 con 0x7fdfcc0c9870 2024-09-16T01:51:43.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.369+0000 7fdfd1000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdfcc1510f0 con 0x7fdfcc0c13f0 2024-09-16T01:51:43.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.369+0000 7fdfd1000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdfcc0c8970 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.370+0000 7fdfc2a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdfcc0c13f0 0x7fdfcc02d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.370+0000 7fdfcb400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdfcc0c9870 0x7fdfcc157290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:55144/0 (socket says 172.21.15.146:55144) 2024-09-16T01:51:43.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.370+0000 7fdfcb400640 1 -- 172.21.15.146:0/3084834597 learned_addr learned my addr 172.21.15.146:0/3084834597 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:43.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.371+0000 7fdfc2a00640 1 -- 172.21.15.146:0/3084834597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 msgr2=0x7fdfcc0cd8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:51:43.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.371+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/3084834597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc0cd8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.371+0000 7fdfc2a00640 1 -- 172.21.15.146:0/3084834597 >> v1:172.21.15.146:6789/0 conn(0x7fdfcc0c9870 legacy=0x7fdfcc157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.372+0000 7fdfc2a00640 1 -- 172.21.15.146:0/3084834597 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc156ad0 con 0x7fdfcc0c13f0 2024-09-16T01:51:43.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.373+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/3084834597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc02d530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009a00 tx=0x7fdfb402b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797c8c3159bf9fab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.376+0000 7fdfcaa00640 1 -- 172.21.15.146:0/3084834597 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4037070 con 0x7fdfcc0c13f0 2024-09-16T01:51:43.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.376+0000 7fdfcaa00640 1 -- 172.21.15.146:0/3084834597 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb4032600 con 0x7fdfcc0c13f0 2024-09-16T01:51:43.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.376+0000 7fdfcaa00640 1 -- 172.21.15.146:0/3084834597 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40329d0 con 0x7fdfcc0c13f0 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.378+0000 7fdfd1000640 1 -- 172.21.15.146:0/3084834597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 msgr2=0x7fdfcc02d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.378+0000 7fdfd1000640 1 --2- 172.21.15.146:0/3084834597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc02d530 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009a00 tx=0x7fdfb402b5b0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.379+0000 7fdfd1000640 1 -- 172.21.15.146:0/3084834597 shutdown_connections 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.379+0000 7fdfd1000640 1 --2- 172.21.15.146:0/3084834597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc0cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.379+0000 7fdfd1000640 1 --2- 172.21.15.146:0/3084834597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc02d530 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.379+0000 7fdfd1000640 1 -- 172.21.15.146:0/3084834597 >> 172.21.15.146:0/3084834597 conn(0x7fdfcc073570 msgr2=0x7fdfcc073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:43.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.379+0000 7fdfd1000640 1 -- 172.21.15.146:0/3084834597 shutdown_connections 2024-09-16T01:51:43.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.379+0000 7fdfd1000640 1 -- 172.21.15.146:0/3084834597 wait complete. 2024-09-16T01:51:43.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.382+0000 7fdfd1000640 1 Processor -- start 2024-09-16T01:51:43.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.382+0000 7fdfd1000640 1 -- start start 2024-09-16T01:51:43.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfd1000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfd1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc1e0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfd1000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfcc0c9870 0x7fdfcc1e1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfd1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc156d70 con 0x7fdfcc0c13f0 2024-09-16T01:51:43.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfd1000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc0c8690 con 0x7fdfcc0c9870 2024-09-16T01:51:43.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfd1000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdfcc0c8b80 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfcb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfcb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39526/0 (socket says 172.21.15.146:39526) 2024-09-16T01:51:43.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.384+0000 7fdfcb400640 1 -- 172.21.15.146:0/1202376365 learned_addr learned my addr 172.21.15.146:0/1202376365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:43.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc1e0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfcc0c9870 0x7fdfcc1e1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfcb400640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfcc0c9870 msgr2=0x7fdfcc1e1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfcc0c9870 0x7fdfcc1e1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfcb400640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 msgr2=0x7fdfcc1e0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc1e0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.385+0000 7fdfcb400640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc1dff70 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.386+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfcc0c13f0 0x7fdfcc1e0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.386+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc151510 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00caa0 tx=0x7fdfbc00cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.387+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc017070 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.387+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfbc00ac00 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.387+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfcc1e0230 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.388+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc012400 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.389+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfcc1e07f0 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.391+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdfbc01a070 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.391+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdfcc073120 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.392+0000 7fdfc3e00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfa806c2b0 0x7fdfa806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.393+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfa806c2b0 0x7fdfa806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.393+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(80..80 src has 1..80) ==== 5770+0+0 (secure 0 0 0) 0x7fdfbc089450 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.393+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v80) ==== 54+0+16 (secure 0 0 0) 0x7fdfcc0c9c50 con 0x7fdfcc02cdf0 2024-09-16T01:51:43.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.394+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfa806c2b0 0x7fdfa806e770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4034000 tx=0x7fdfb4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.394+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fdfcc153750 0x7fdfcc149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.394+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc02c7f0 con 0x7fdfcc153750 2024-09-16T01:51:43.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.394+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fdfcc153750 0x7fdfcc149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.395+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fdfcc153750 0x7fdfcc149650 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.399+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc02c7f0 con 0x7fdfcc153750 2024-09-16T01:51:43.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.414+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc06daf0 con 0x7fdfcc153750 2024-09-16T01:51:43.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.416+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc06daf0 con 0x7fdfcc153750 2024-09-16T01:51:43.418 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.416+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fdfcc1e1ed0 0x7fdfcc1e22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.416+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc1e27f0 con 0x7fdfcc1e1ed0 2024-09-16T01:51:43.419 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:43.417+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:43.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.417+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fdfcc1e1ed0 0x7fdfcc1e22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.417+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fdfcc1e1ed0 0x7fdfcc1e22b0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.420+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc1e27f0 con 0x7fdfcc1e1ed0 2024-09-16T01:51:43.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.439+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc06d530 con 0x7fdfcc1e1ed0 2024-09-16T01:51:43.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.441+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc06d530 con 0x7fdfcc1e1ed0 2024-09-16T01:51:43.444 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:43.444 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:43.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.442+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fdfcc1487e0 0x7fdfcc148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.442+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc149100 con 0x7fdfcc1487e0 2024-09-16T01:51:43.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.443+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fdfcc1487e0 0x7fdfcc148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.445+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fdfcc1487e0 0x7fdfcc148bc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.461+0000 7f2f1a600640 1 Processor -- start 2024-09-16T01:51:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.461+0000 7f2f1a600640 1 -- start start 2024-09-16T01:51:43.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.463+0000 7f2f1a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f1406fee0 0x7f2f141509a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.464+0000 7f2f1a600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f14150ee0 con 0x7f2f140c70e0 2024-09-16T01:51:43.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.464+0000 7f2f1a600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f14071ce0 con 0x7f2f14072dd0 2024-09-16T01:51:43.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.464+0000 7f2f1a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f2f14157580 con 0x7f2f1406fee0 2024-09-16T01:51:43.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.465+0000 7f2f13e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2f14072dd0 0x7f2f1406da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:39940/0 (socket says 172.21.15.146:39940) 2024-09-16T01:51:43.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.465+0000 7f2f13e00640 1 -- 172.21.15.146:0/3557027060 learned_addr learned my addr 172.21.15.146:0/3557027060 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:43.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.465+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840380998 0 0) 0x7f2f14071ce0 con 0x7f2f14072dd0 2024-09-16T01:51:43.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.465+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ef8003520 con 0x7f2f14072dd0 2024-09-16T01:51:43.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.475+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc149100 con 0x7fdfcc1487e0 2024-09-16T01:51:43.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1185890969 0 0) 0x7f2ef8003520 con 0x7f2f14072dd0 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 >> v2:172.21.15.146:3300/0 conn(0x7f2f1406fee0 msgr2=0x7f2f141509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 --2- 172.21.15.146:0/3557027060 >> v2:172.21.15.146:3300/0 conn(0x7f2f1406fee0 0x7f2f141509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 >> v1:172.21.15.146:6789/0 conn(0x7f2f140c70e0 legacy=0x7f2f14072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f14156c50 con 0x7f2f14072dd0 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f2f0c0019f0 con 0x7f2f14072dd0 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 >> v1:172.21.15.161:6789/0 conn(0x7f2f14072dd0 legacy=0x7f2f1406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ef80039b0 0x7f2ef8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef8004800 0x7f2ef8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80052e0 0x7f2ef800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2f14157580 con 0x7f2ef80052e0 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef800c1e0 con 0x7f2ef8004800 2024-09-16T01:51:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.468+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2ef800c3e0 con 0x7f2ef80039b0 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.469+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80052e0 0x7f2ef800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.470+0000 7f2f13e00640 1 -- 172.21.15.146:0/3557027060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ef80039b0 msgr2=0x7f2ef8003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.475+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2ef80039b0 0x7f2ef8003db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.475+0000 7f2f13e00640 1 -- 172.21.15.146:0/3557027060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef8004800 msgr2=0x7f2ef8004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.475+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef8004800 0x7f2ef8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.475+0000 7f2f13e00640 1 -- 172.21.15.146:0/3557027060 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f14156c50 con 0x7f2ef80052e0 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.477+0000 7f2f13e00640 1 -- 172.21.15.146:0/3557027060 reap_dead start 2024-09-16T01:51:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.477+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80052e0 0x7f2ef800b9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c010d00 tx=0x7f2ef8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d65c9711582b5b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.478+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c010900 con 0x7f2ef80052e0 2024-09-16T01:51:43.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.478+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0c010270 con 0x7f2ef80052e0 2024-09-16T01:51:43.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.479+0000 7f2f1a600640 1 -- 172.21.15.146:0/3557027060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80052e0 msgr2=0x7f2ef800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.480+0000 7f2f12000640 1 -- 172.21.15.146:0/3557027060 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c010900 con 0x7f2ef80052e0 2024-09-16T01:51:43.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.480+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80052e0 0x7f2ef800b9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c010d00 tx=0x7f2ef8005830 comp rx=0 tx=0).stop 2024-09-16T01:51:43.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.481+0000 7f2f1a600640 1 -- 172.21.15.146:0/3557027060 shutdown_connections 2024-09-16T01:51:43.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.481+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3557027060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80052e0 0x7f2ef800b9c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.481+0000 7f2f1a600640 1 -- 172.21.15.146:0/3557027060 >> 172.21.15.146:0/3557027060 conn(0x7f2f1406f3c0 msgr2=0x7f2f1406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:43.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.485+0000 7f2f1a600640 1 -- 172.21.15.146:0/3557027060 shutdown_connections 2024-09-16T01:51:43.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.485+0000 7f2f1a600640 1 -- 172.21.15.146:0/3557027060 wait complete. 2024-09-16T01:51:43.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.504+0000 7f2f1a600640 1 Processor -- start 2024-09-16T01:51:43.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.510+0000 7f2f1a600640 1 -- start start 2024-09-16T01:51:43.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.513+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc149100 con 0x7fdfcc1487e0 2024-09-16T01:51:43.520 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:43.517+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:43.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.517+0000 7f2f1a600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 0x7f2f141efe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.518+0000 7f2f1a600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2f14142500 0x7f2f141ec7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc149100 con 0x7fdfcc1487e0 2024-09-16T01:51:43.522 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:43.523 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:43.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fdfcc062b40 0x7fdfcc062fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc149100 con 0x7fdfcc062b40 2024-09-16T01:51:43.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7f2f1a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f1414ac90 0x7f2f1414b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7f2f1a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2f140c9a90 con 0x7f2f1414ac90 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7f2f1a600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f14157210 con 0x7f2ef80052e0 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7f2f1a600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2f140718c0 con 0x7f2f14142500 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.518+0000 7f2f13400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 0x7f2f141efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7f2f13400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 0x7f2f141efe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54548/0 (socket says 172.21.15.146:54548) 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.520+0000 7f2f13400640 1 -- 172.21.15.146:0/3786708263 learned_addr learned my addr 172.21.15.146:0/3786708263 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:43.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.521+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f1414ac90 0x7f2f1414b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.523+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fdfcc062b40 0x7fdfcc062fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.523+0000 7f2f13e00640 1 -- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2f14142500 msgr2=0x7f2f141ec7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.523+0000 7f2f12a00640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2f14142500 0x7f2f141ec7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.523+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2f14142500 0x7f2f141ec7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.523+0000 7f2f13e00640 1 -- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 msgr2=0x7f2f141efe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.523+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 0x7f2f141efe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.524+0000 7f2f13e00640 1 -- 172.21.15.146:0/3786708263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f14158520 con 0x7f2f1414ac90 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.524+0000 7f2f13400640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 0x7f2f141efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:43.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.524+0000 7f2f12a00640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2f14142500 0x7f2f141ec7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:43.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.524+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fdfcc062b40 0x7fdfcc062fa0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.524+0000 7f2f13e00640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f1414ac90 0x7f2f1414b110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2f140739f0 tx=0x7f2f0c001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.525+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c04a070 con 0x7f2f1414ac90 2024-09-16T01:51:43.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.526+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc149100 con 0x7fdfcc062b40 2024-09-16T01:51:43.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.527+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f0c03b430 con 0x7f2f1414ac90 2024-09-16T01:51:43.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.527+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c040040 con 0x7f2f1414ac90 2024-09-16T01:51:43.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.532+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f140cc640 con 0x7f2f1414ac90 2024-09-16T01:51:43.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.532+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f140cc8e0 con 0x7f2f1414ac90 2024-09-16T01:51:43.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.544+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f140c9da0 con 0x7f2f1414ac90 2024-09-16T01:51:43.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.545+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2f0c03b800 con 0x7f2f1414ac90 2024-09-16T01:51:43.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.548+0000 7f2f0b400640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2ef406c2b0 0x7f2ef406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.551+0000 7f2f13400640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2ef406c2b0 0x7f2ef406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.552+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5770+0+0 (secure 0 0 0) 0x7f2f0c0b3650 con 0x7f2f1414ac90 2024-09-16T01:51:43.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.554+0000 7f2f13400640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2ef406c2b0 0x7f2ef406e770 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009e90 tx=0x7f2f040091c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.555+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc149100 con 0x7fdfcc062b40 2024-09-16T01:51:43.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.559+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc149100 con 0x7fdfcc062b40 2024-09-16T01:51:43.561 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:43.562 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:43.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.559+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fdfcc0cbce0 0x7fdfcc0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.560+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc0cc680 con 0x7fdfcc0cbce0 2024-09-16T01:51:43.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.560+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fdfcc0cbce0 0x7fdfcc0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.560+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fdfcc0cbce0 0x7fdfcc0cc140 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.561+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc0cc680 con 0x7fdfcc0cbce0 2024-09-16T01:51:43.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.563+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2f0c081690 con 0x7f2f1414ac90 2024-09-16T01:51:43.596 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:43.596 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:43.596 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:43.597 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:43.597 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:43.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.610+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc0cc680 con 0x7fdfcc0cbce0 2024-09-16T01:51:43.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.616+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc0cc680 con 0x7fdfcc0cbce0 2024-09-16T01:51:43.618 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:43.618 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:43.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.616+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fdfcc06c120 0x7fdfcc06c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.616+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc06cac0 con 0x7fdfcc06c120 2024-09-16T01:51:43.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:43.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.617+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fdfcc06c120 0x7fdfcc06c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.618+0000 7fdfc2a00640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fdfcc06c120 0x7fdfcc06c580 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.619+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc06cac0 con 0x7fdfcc06c120 2024-09-16T01:51:43.622 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:43.620+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:43.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.647+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc06cac0 con 0x7fdfcc06c120 2024-09-16T01:51:43.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.675+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc06cac0 con 0x7fdfcc06c120 2024-09-16T01:51:43.681 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:43.681 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:43.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.679+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fdfcc2030f0 0x7fdfcc2054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.679+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc208330 con 0x7fdfcc2030f0 2024-09-16T01:51:43.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.681+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fdfcc2030f0 0x7fdfcc2054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.682+0000 7fdfd2600640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fdfcc2030f0 0x7fdfcc2054e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.683+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc208330 con 0x7fdfcc2030f0 2024-09-16T01:51:43.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.716+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc205aa0 con 0x7fdfcc2030f0 2024-09-16T01:51:43.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.719+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc205aa0 con 0x7fdfcc2030f0 2024-09-16T01:51:43.722 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:43.722 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:43.720+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:43.722 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:43.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.721+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fdfcc2061c0 0x7fdfcc206640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:43.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.721+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdfcc20b6d0 con 0x7fdfcc2061c0 2024-09-16T01:51:43.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.721+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fdfcc2061c0 0x7fdfcc206640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:43.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.722+0000 7fdfcb400640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fdfcc2061c0 0x7fdfcc206640 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:43.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.723+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfcc20b6d0 con 0x7fdfcc2061c0 2024-09-16T01:51:43.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.751+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdfcc206c90 con 0x7fdfcc2061c0 2024-09-16T01:51:43.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.752+0000 7fdfc3e00640 1 -- 172.21.15.146:0/1202376365 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdfcc206c90 con 0x7fdfcc2061c0 2024-09-16T01:51:43.755 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:43.756 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:43.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fdfcc153750 msgr2=0x7fdfcc149650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fdfcc153750 0x7fdfcc149650 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fdfcc1e1ed0 msgr2=0x7fdfcc1e22b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fdfcc1e1ed0 0x7fdfcc1e22b0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fdfcc1487e0 msgr2=0x7fdfcc148bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fdfcc1487e0 0x7fdfcc148bc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fdfcc062b40 msgr2=0x7fdfcc062fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fdfcc062b40 0x7fdfcc062fa0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fdfcc0cbce0 msgr2=0x7fdfcc0cc140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fdfcc0cbce0 0x7fdfcc0cc140 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fdfcc06c120 msgr2=0x7fdfcc06c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fdfcc06c120 0x7fdfcc06c580 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fdfcc2030f0 msgr2=0x7fdfcc2054e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fdfcc2030f0 0x7fdfcc2054e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fdfcc2061c0 msgr2=0x7fdfcc206640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fdfcc2061c0 0x7fdfcc206640 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd2600640 1 -- 172.21.15.146:0/1202376365 reap_dead start 2024-09-16T01:51:43.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd2600640 1 -- 172.21.15.146:0/1202376365 reap_dead start 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfa806c2b0 msgr2=0x7fdfa806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfa806c2b0 0x7fdfa806e770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4034000 tx=0x7fdfb4008030 comp rx=0 tx=0).stop 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 msgr2=0x7fdfcc151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.753+0000 7fdfd1000640 1 --2- 172.21.15.146:0/1202376365 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfcc02cdf0 0x7fdfcc151510 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00caa0 tx=0x7fdfbc00cf60 comp rx=0 tx=0).stop 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd2600640 1 -- 172.21.15.146:0/1202376365 reap_dead start 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd2600640 1 -- 172.21.15.146:0/1202376365 reap_dead start 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd2600640 1 -- 172.21.15.146:0/1202376365 reap_dead start 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 shutdown_connections 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 >> 172.21.15.146:0/1202376365 conn(0x7fdfcc073570 msgr2=0x7fdfcc071260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 shutdown_connections 2024-09-16T01:51:43.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.754+0000 7fdfd1000640 1 -- 172.21.15.146:0/1202376365 wait complete. 2024-09-16T01:51:43.827 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:43.825+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:43.926 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:43.924+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:43.998+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f2f14072070 con 0x7f2f1414ac90 2024-09-16T01:51:44.002 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:44.002 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:44.002 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:44.002 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:44.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 176.230965521, 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:51:44.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:44.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:7 3.2a 3:557f550e:::10000000002.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e54)", 2024-09-16T01:51:44.009 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.740625186999999, 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081303988999999993, 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 7 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258694+0000", 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.258696+0000", 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7320000000000001e-06 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305572+0000", 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046875737000000001 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305609+0000", 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7704999999999999e-05 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.305623+0000", 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3536000000000001e-05 2024-09-16T01:51:44.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310196+0000", 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045734269999999997 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310344+0000", 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147181 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.310523+0000", 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017908400000000001 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.327195+0000", 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016672724999999999 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339893+0000", 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012697559000000001 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339953+0000", 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0084999999999998e-05 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.339998+0000", 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5218e-05 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:441 3.1c 3:3913eadd:::10000000002.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:44.015 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.390637452999997, 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072104032999999998, 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 441 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608682+0000", 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.608684+0000", 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4959999999999999e-06 2024-09-16T01:51:44.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650015+0000", 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041330193000000001 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650045+0000", 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0335e-05 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650057+0000", 2024-09-16T01:51:44.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650158+0000", 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100837 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650266+0000", 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107975 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.019 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.650426+0000", 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016051800000000001 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.663669+0000", 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013242594 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680672+0000", 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017003041999999999 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680737+0000", 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4750000000000002e-05 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.680786+0000", 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9217e-05 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.613513181000002, 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:51:44.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:51:44.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:51:44.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:524 3.19 3:9adb8721:::100000001f7.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.520450569000001, 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092555282000000003, 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 524 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478871+0000", 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0700000000000001e-06 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538563+0000", 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059691821999999999 2024-09-16T01:51:44.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538594+0000", 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1285000000000003e-05 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538606+0000", 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1688999999999999e-05 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538706+0000", 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100752 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538872+0000", 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016575699999999999 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.539058+0000", 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018612500000000001 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.547102+0000", 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080436529999999996 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571308+0000", 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024205946999999998 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571375+0000", 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6623000000000002e-05 2024-09-16T01:51:44.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571424+0000", 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9558999999999998e-05 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:541 3.1c 3:387f027e:::100000001f7.0000001a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.378499347000002, 2024-09-16T01:51:44.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084290084000000001, 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 541 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620823+0000", 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8169999999999999e-06 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670392+0000", 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049568677999999998 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670432+0000", 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0312999999999998e-05 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670448+0000", 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5781e-05 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670540+0000", 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1920000000000001e-05 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670705+0000", 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016490299999999999 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670881+0000", 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017675899999999999 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.683357+0000", 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012476179 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.704990+0000", 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021632195 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.705053+0000", 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3291000000000001e-05 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.705110+0000", 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7247999999999998e-05 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:557 3.10 3:0b5e6b00:::100000001f7.0000002a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.792602+0000", 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.206717136000002, 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.069233480999999999, 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 557 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.792602+0000", 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.792602+0000", 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.792605+0000", 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0750000000000002e-06 2024-09-16T01:51:44.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.816871+0000", 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024265641000000001 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.816911+0000", 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0240000000000001e-05 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.816930+0000", 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8689e-05 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.817116+0000", 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185918 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.817302+0000", 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018637300000000001 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.817444+0000", 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000141865 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.830032+0000", 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012587891 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.861681+0000", 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031649259999999999 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.861755+0000", 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4040000000000003e-05 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.861836+0000", 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0488999999999995e-05 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:681 3.10 3:0ad5cbe1:::100000001f7.000000a6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.924809+0000", 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.074510858, 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070284214999999997, 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 681 2024-09-16T01:51:44.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.924809+0000", 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.924809+0000", 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.924811+0000", 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981398+0000", 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056587232000000001 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981424+0000", 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5729e-05 2024-09-16T01:51:44.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981436+0000", 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1799e-05 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981563+0000", 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012704200000000001 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.981726+0000", 2024-09-16T01:51:44.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016336599999999999 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.982383+0000", 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000656709 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.992308+0000", 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0099254040000000005 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.994970+0000", 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00266161 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.995039+0000", 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9146999999999994e-05 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.995093+0000", 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3820999999999999e-05 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.053 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.821496232000001, 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.054 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:51:44.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:51:44.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.059 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.785963609, 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:51:44.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:51:44.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:819 3.37 3:ef985689:::100000001f7.00000030:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.622420+0000", 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.376899542, 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066382658999999997, 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.066 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 819 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.622420+0000", 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.622420+0000", 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.622422+0000", 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1679999999999998e-06 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673256+0000", 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050833761999999998 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673314+0000", 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7805e-05 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673327+0000", 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3526000000000001e-05 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673407+0000", 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9488999999999997e-05 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673508+0000", 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101817 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.673673+0000", 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016442200000000001 2024-09-16T01:51:44.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.682588+0000", 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089146429999999999 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.688676+0000", 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060883780000000002 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.688749+0000", 2024-09-16T01:51:44.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3073999999999999e-05 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.688803+0000", 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3575000000000001e-05 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:834 3.2a 3:57cc18b9:::100000001f7.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.742928+0000", 2024-09-16T01:51:44.072 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.256391708999999, 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075641033999999996, 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 834 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.742928+0000", 2024-09-16T01:51:44.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.742928+0000", 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.742930+0000", 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6280000000000001e-06 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.074 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781555+0000", 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038624533000000003 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781568+0000", 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2988e-05 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781610+0000", 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2323000000000001e-05 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781709+0000", 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8333999999999997e-05 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.781826+0000", 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117321 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.782003+0000", 2024-09-16T01:51:44.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017726800000000001 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.817329+0000", 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035325803000000003 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.818457+0000", 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011279490000000001 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.818516+0000", 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8743999999999999e-05 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.818569+0000", 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3143000000000003e-05 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1134 3.21 3:87440b43:::100000001f8.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:44.078 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.375674+0000", 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.6236452200000002, 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.069400464999999995, 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1134 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.375674+0000", 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.375674+0000", 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.375676+0000", 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.176e-06 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.418271+0000", 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042594745000000003 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.081 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.418299+0000", 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8215999999999999e-05 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.418316+0000", 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6867e-05 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.418798+0000", 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00048198000000000002 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.418978+0000", 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000179829 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.419160+0000", 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018192800000000001 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.442851+0000", 2024-09-16T01:51:44.083 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023690834000000001 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.444955+0000", 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021039779999999998 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.445021+0000", 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6558999999999998e-05 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.445075+0000", 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3353000000000002e-05 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1147 3.19 3:9848b992:::100000001f8.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.511489+0000", 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4878303639999997, 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084795828000000004, 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.085 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1147 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.511489+0000", 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.511489+0000", 2024-09-16T01:51:44.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.511491+0000", 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3630000000000001e-06 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.560928+0000", 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049436969999999997 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.560961+0000", 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2478000000000003e-05 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.560972+0000", 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0634e-05 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.561207+0000", 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023557499999999999 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.561352+0000", 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144974 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.561505+0000", 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152437 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.583038+0000", 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021533845999999999 2024-09-16T01:51:44.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.596082+0000", 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013043974 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.596220+0000", 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000137947 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.596285+0000", 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4629999999999996e-05 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1193 3.37 3:ef2b5fb8:::100000001f8.000000a6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.002921+0000", 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9963988449999999, 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077504510999999998, 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.091 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1193 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.002921+0000", 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.002921+0000", 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.002923+0000", 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7719999999999999e-06 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.062515+0000", 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059591465000000003 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.062650+0000", 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000134855 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.062668+0000", 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7978e-05 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.062987+0000", 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00031971300000000001 2024-09-16T01:51:44.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.063164+0000", 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017672599999999999 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.063336+0000", 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172208 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.071176+0000", 2024-09-16T01:51:44.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0078396790000000004 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.080308+0000", 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091317929999999992 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.080374+0000", 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5919000000000004e-05 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.080425+0000", 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1403000000000002e-05 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1211 3.1c 3:3941380e:::100000001f8.000000b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.157062+0000", 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.84225773, 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079436243000000004, 2024-09-16T01:51:44.097 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1211 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.157062+0000", 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.157062+0000", 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.157065+0000", 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8090000000000001e-06 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.185690+0000", 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028625454000000002 2024-09-16T01:51:44.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.185715+0000", 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.497e-05 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.185724+0000", 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2059999999999996e-06 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.185792+0000", 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8058000000000005e-05 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.185899+0000", 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107159 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.185992+0000", 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2777999999999998e-05 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.214394+0000", 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028402138 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236390+0000", 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021995423 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236454+0000", 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3868000000000006e-05 2024-09-16T01:51:44.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236498+0000", 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4379999999999999e-05 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1224 3.1a 3:5b959896:::100000001f8.000000c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.377085+0000", 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.6222340979999998, 2024-09-16T01:51:44.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067600409, 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1224 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.377085+0000", 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.377085+0000", 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.377088+0000", 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5119999999999998e-06 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.415976+0000", 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038888132999999998 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.416125+0000", 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014943399999999999 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.416204+0000", 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8157000000000004e-05 2024-09-16T01:51:44.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.417042+0000", 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083827499999999998 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.419540+0000", 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0024983990000000001 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:44.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.421151+0000", 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001610853 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.441900+0000", 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020749057000000001 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.444555+0000", 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002654605 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.444633+0000", 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.8256000000000004e-05 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.444686+0000", 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2728000000000001e-05 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1461 3.1c 3:39a1ca69:::100000001f8.000001b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:44.109 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.58862281400000005, 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105254967, 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1461 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:51:44.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410699+0000", 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.232e-06 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476134+0000", 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065435190000000004 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476173+0000", 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8878999999999998e-05 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476186+0000", 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3016e-05 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476284+0000", 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7583000000000004e-05 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476440+0000", 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015621799999999999 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476629+0000", 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018937399999999999 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.505615+0000", 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028986212000000001 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515833+0000", 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010217944 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515901+0000", 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7250999999999997e-05 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515952+0000", 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1068e-05 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1471 3.21 3:84a15e9e:::100000001f8.000001bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.520659+0000", 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.47866025699999998, 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071211036000000005, 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1471 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.520659+0000", 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.520659+0000", 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.520662+0000", 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.48e-06 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.564853+0000", 2024-09-16T01:51:44.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044190898999999999 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.564885+0000", 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2511000000000001e-05 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.564899+0000", 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.382e-05 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.565000+0000", 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010158900000000001 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.565169+0000", 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016851299999999999 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.565333+0000", 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163542 2024-09-16T01:51:44.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.578309+0000", 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012976059 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.591789+0000", 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013480199 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.591836+0000", 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7379999999999997e-05 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.591870+0000", 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4044000000000001e-05 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:51:44.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.419874672, 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:44.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:51:44.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:44.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:44.127 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:44.129 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.065+0000 7f2f0b400640 1 -- 172.21.15.146:0/3786708263 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v81) ==== 97+0+0 (secure 0 0 0) 0x7f2f0c085990 con 0x7f2f1414ac90 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2ef406c2b0 msgr2=0x7f2ef406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2ef406c2b0 0x7f2ef406e770 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009e90 tx=0x7f2f040091c0 comp rx=0 tx=0).stop 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f1414ac90 msgr2=0x7f2f1414b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f1414ac90 0x7f2f1414b110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2f140739f0 tx=0x7f2f0c001cf0 comp rx=0 tx=0).stop 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 shutdown_connections 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2ef406c2b0 0x7f2ef406e770 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f1414ac90 0x7f2f1414b110 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2f14142500 0x7f2f141ec7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:44.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 --2- 172.21.15.146:0/3786708263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef80052e0 0x7f2f141efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:44.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 >> 172.21.15.146:0/3786708263 conn(0x7f2f1406f3c0 msgr2=0x7f2f1406f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:44.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 shutdown_connections 2024-09-16T01:51:44.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:44.070+0000 7f2f1a600640 1 -- 172.21.15.146:0/3786708263 wait complete. 2024-09-16T01:51:44.132 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:44.127+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:44.229 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:44.228+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:44.335 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:44.333+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:44.431 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:44.429+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:44.519 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:44.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:44.519 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:44.519 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:44.533 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:44.531+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:44.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:44.633 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:44.631+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:44.734 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:44.732+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:44.835 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:44.833+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:44.875 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:44.876 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:44.876 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:44.876 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:44.876 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:44.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:44.944 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:44.937+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:45.037 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:45.035+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:45.138 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:45.136+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:87 3.3a 3:5da5f84e:::10000000002.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:45.171 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.146586124000002, 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074094462, 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 87 2024-09-16T01:51:45.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020380+0000", 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.020383+0000", 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4820000000000001e-06 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042859+0000", 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022475964000000001 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042890+0000", 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1655000000000001e-05 2024-09-16T01:51:45.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.042904+0000", 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4022999999999999e-05 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043003+0000", 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8924000000000004e-05 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043158+0000", 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015449699999999999 2024-09-16T01:51:45.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.043296+0000", 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000138708 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.060494+0000", 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017197288000000002 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094350+0000", 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033856167999999999 2024-09-16T01:51:45.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094415+0000", 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4982999999999998e-05 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.094475+0000", 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9769999999999999e-05 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.177 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:437 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.580131827999999, 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067623057, 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 437 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:45.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586834+0000", 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.586836+0000", 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3960000000000001e-06 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626664+0000", 2024-09-16T01:51:45.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.039827763000000002 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626906+0000", 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000242155 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.626942+0000", 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6282000000000003e-05 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627180+0000", 2024-09-16T01:51:45.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000238273 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627280+0000", 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100127 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.627422+0000", 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014201900000000001 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.646355+0000", 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018932462000000001 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654340+0000", 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079853360000000009 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654407+0000", 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7005e-05 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.654457+0000", 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0238999999999998e-05 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:457 3.26 3:64ff7f1d:::10000000002.000000c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.392305241999999, 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075061167999999998, 2024-09-16T01:51:45.183 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 457 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774661+0000", 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.774664+0000", 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.587e-06 2024-09-16T01:51:45.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804118+0000", 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029454668999999999 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804142+0000", 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3910999999999999e-05 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804151+0000", 2024-09-16T01:51:45.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8619999999999992e-06 2024-09-16T01:51:45.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804222+0000", 2024-09-16T01:51:45.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0643999999999997e-05 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804324+0000", 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102132 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.804467+0000", 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014311099999999999 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.841516+0000", 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037049058000000003 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849624+0000", 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081077600000000003 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849681+0000", 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7068999999999999e-05 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.849722+0000", 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1365000000000001e-05 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.504642938, 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:51:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:51:45.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:51:45.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:634 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.652067732999999, 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068283101999999998, 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.197 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 634 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514898+0000", 2024-09-16T01:51:45.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.514901+0000", 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5299999999999999e-06 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549207+0000", 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034306171000000003 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549224+0000", 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6976000000000001e-05 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549235+0000", 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0682999999999999e-05 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549568+0000", 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00033294599999999998 2024-09-16T01:51:45.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549714+0000", 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014617499999999999 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.549858+0000", 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144356 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.566205+0000", 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01634683 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583078+0000", 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016872402000000002 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583135+0000", 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7286e-05 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:15.583182+0000", 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6746999999999998e-05 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:696 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.039842397000001, 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071884475000000003, 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.203 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 696 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127124+0000", 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.127126+0000", 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9369999999999998e-06 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159227+0000", 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032101044000000002 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159250+0000", 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3332999999999998e-05 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159260+0000", 2024-09-16T01:51:45.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6900000000000004e-06 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159343+0000", 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3373000000000003e-05 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159484+0000", 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014100399999999999 2024-09-16T01:51:45.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.159646+0000", 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162044 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.186104+0000", 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026457794999999999 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198896+0000", 2024-09-16T01:51:45.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012791749 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.198959+0000", 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3387999999999997e-05 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:16.199008+0000", 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9118e-05 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:785 3.11 e69/20 3:8b60708a:::100000001f7.0000000e:head v 69'38, mlcod=65'35)", 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:45.210 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.912101997000001, 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082167670999999998, 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254864+0000", 2024-09-16T01:51:45.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254865+0000", 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9799999999999999e-07 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286626+0000", 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031761664000000002 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286647+0000", 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1268e-05 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286659+0000", 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1276e-05 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286743+0000", 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4488999999999997e-05 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.286772+0000", 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9082999999999999e-05 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.336965+0000", 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050192716999999998 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337032+0000", 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6876000000000001e-05 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1031 3.e 3:73e84e43:::100000001f8.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.9950273940000001, 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.093093621000000001, 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1031 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.215 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171941+0000", 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2230000000000001e-06 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182392+0000", 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010451225999999999 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182418+0000", 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5561000000000001e-05 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182431+0000", 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3012e-05 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186260+0000", 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038291340000000001 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186410+0000", 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014975200000000001 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186556+0000", 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146587 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.210170+0000", 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023614109000000001 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.264915+0000", 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054744835999999998 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.264984+0000", 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8753999999999999e-05 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.265032+0000", 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8427000000000002e-05 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.221 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.8757836709999998, 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:51:45.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:51:45.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:51:45.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:51:45.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:51:45.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:51:45.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:51:45.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:51:45.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:51:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.846622816, 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:51:45.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:51:45.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:51:45.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.238 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.8142712479999998, 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:51:45.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7983239820000003, 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:51:45.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:51:45.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1127 3.36 e76/20 3:6e2a8853:::100000001f8.00000064:head v 76'45, mlcod=76'43)", 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.380105+0000", 2024-09-16T01:51:45.251 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7868616880000001, 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082221984999999997, 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.380105+0000", 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.380105+0000", 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.380106+0000", 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.373e-06 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.381306+0000", 2024-09-16T01:51:45.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012000629999999999 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386611+0000", 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0053045799999999997 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386631+0000", 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0604000000000001e-05 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386699+0000", 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7756000000000007e-05 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386722+0000", 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2603e-05 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462293+0000", 2024-09-16T01:51:45.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.075571743999999996 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462327+0000", 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3262000000000001e-05 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1132 3.5 3:a0fd78c8:::100000001f8.00000069:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7757892850000001, 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095513232000000003, 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1132 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391179+0000", 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.66e-06 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408341+0000", 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017162686999999999 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408396+0000", 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5034000000000002e-05 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408412+0000", 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5685e-05 2024-09-16T01:51:45.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408710+0000", 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029794899999999999 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408870+0000", 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000160168 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.409033+0000", 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016284700000000001 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480660+0000", 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071627535000000006 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486552+0000", 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0058913079999999996 2024-09-16T01:51:45.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486634+0000", 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2051000000000004e-05 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486690+0000", 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6307999999999997e-05 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.262 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1161 2.14 2:2e2fa760:::200.00000001:head [write 167677~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e76)", 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.408489+0000", 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.7584773419999999, 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.066457844000000002, 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408489+0000", 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408489+0000", 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408491+0000", 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9810000000000002e-06 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408496+0000", 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6169999999999998e-06 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408506+0000", 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8220000000000002e-06 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408513+0000", 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7789999999999998e-06 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.410753+0000", 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022403449999999999 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.410846+0000", 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2962999999999997e-05 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.410983+0000", 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013657 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451987+0000", 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041004314 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.474838+0000", 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022850342999999999 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.474894+0000", 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6335999999999997e-05 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.474947+0000", 2024-09-16T01:51:45.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2774e-05 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1154 3.16 3:69b0082d:::100000001f8.0000007f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.595750+0000", 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5712159139999997, 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077456938000000003, 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.269 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1154 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.595750+0000", 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.595750+0000", 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.595753+0000", 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626218+0000", 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030465401 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626259+0000", 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0729000000000003e-05 2024-09-16T01:51:45.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626274+0000", 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4701999999999999e-05 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626379+0000", 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010513700000000001 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626549+0000", 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016985599999999999 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626728+0000", 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017889199999999999 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.650020+0000", 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023292262000000001 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.673095+0000", 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023074879 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.673158+0000", 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3017999999999999e-05 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.673207+0000", 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9395999999999999e-05 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.274 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1155 3.e 3:7121bfba:::100000001f8.00000080:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.626316+0000", 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5406500579999998, 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070178986999999998, 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1155 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626316+0000", 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626316+0000", 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.626318+0000", 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9700000000000002e-06 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.681984+0000", 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055665474999999999 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.682004+0000", 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0098e-05 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.682017+0000", 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3713999999999999e-05 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.682098+0000", 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0309999999999995e-05 2024-09-16T01:51:45.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.682256+0000", 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157878 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.682419+0000", 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016359299999999999 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.696241+0000", 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013822039 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.696377+0000", 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013594599999999999 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.696445+0000", 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7329000000000005e-05 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.696495+0000", 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0634999999999999e-05 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1202 3.e 3:71b02bc6:::100000001f8.000000af:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.099706+0000", 2024-09-16T01:51:45.281 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0672603069999997, 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064163865, 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1202 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.099706+0000", 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.099706+0000", 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.099708+0000", 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2170000000000001e-06 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.144025+0000", 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044317144000000003 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.144064+0000", 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8279999999999999e-05 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.144078+0000", 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4338000000000001e-05 2024-09-16T01:51:45.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.144175+0000", 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6917999999999996e-05 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.144334+0000", 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015918200000000001 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:45.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.144486+0000", 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015216599999999999 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.152648+0000", 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0081614879999999997 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.163744+0000", 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011096007 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.163817+0000", 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3506999999999999e-05 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.163870+0000", 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2617999999999997e-05 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.287 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1531 3.16 3:6865cba7:::100000001f8.000001f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:44.281546+0000", 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.88541973900000004, 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068965182999999999, 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1531 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.281546+0000", 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.281546+0000", 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.281548+0000", 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.815e-06 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.322810+0000", 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041261704000000003 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.322937+0000", 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012707300000000001 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.322996+0000", 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8748000000000001e-05 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.323178+0000", 2024-09-16T01:51:45.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000181788 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.323358+0000", 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180633 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.323519+0000", 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016034 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.346936+0000", 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023417506000000001 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.350397+0000", 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034610439999999999 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.350460+0000", 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3178999999999999e-05 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.350512+0000", 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1353000000000001e-05 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1579 3.3c 3:3e9b6e6d:::100000001f8.00000228:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:44.809755+0000", 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.35721164399999999, 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081668402000000001, 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1579 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.809755+0000", 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.809755+0000", 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.809757+0000", 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-06 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.863386+0000", 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053629033 2024-09-16T01:51:45.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.863412+0000", 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5086e-05 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.863421+0000", 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8989999999999996e-06 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.863506+0000", 2024-09-16T01:51:45.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5013999999999996e-05 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.863685+0000", 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017872 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.863822+0000", 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000136398 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.887274+0000", 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023452845 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.891315+0000", 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004040653 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.891374+0000", 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8470000000000001e-05 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.891423+0000", 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9400000000000001e-05 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.300 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:45.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:45.302 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:45.300+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:45.402 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:45.401+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:45.503 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:45.501+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:45.597 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:45.597 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:45.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.597 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1636 3.11 3:8b5a2b4a:::100000001f8.00000261:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:45.546645+0000", 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.049574228999999997, 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049603398999999999, 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1636 2024-09-16T01:51:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.546645+0000", 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.546645+0000", 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.546647+0000", 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3609999999999999e-06 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.572802+0000", 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026155156999999998 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.572951+0000", 2024-09-16T01:51:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148577 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.573050+0000", 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9474000000000003e-05 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.574015+0000", 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096483300000000001 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.574574+0000", 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055861699999999997 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.574883+0000", 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030932900000000001 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.595747+0000", 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020863577000000001 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:51:45.603 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:45.604 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:45.603+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:45.620 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:45.714 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:45.711+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:45.782 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:45.806 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:45.805+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:45.907 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:45.905+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:45.992 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:45.993 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:45.993 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:45.993 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:45.993 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:46.008 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:46.006+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:46.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:46.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.077+0000 7f67e0400640 1 Processor -- start 2024-09-16T01:51:46.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.078+0000 7f67e0400640 1 -- start start 2024-09-16T01:51:46.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.083+0000 7f67e0400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 0x7f67d80c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.086+0000 7f67dda00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f67d806c700 0x7f67d8072a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:39944/0 (socket says 172.21.15.146:39944) 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.086+0000 7f67dd000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 0x7f67d80c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.086+0000 7f67dd000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 0x7f67d80c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39548/0 (socket says 172.21.15.146:39548) 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.086+0000 7f67e0400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f67d8071c70 0x7f67d806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67e0400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d8151080 con 0x7f67d806c700 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67e0400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f67d81575b0 con 0x7f67d8071c70 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67e0400640 1 -- 172.21.15.146:0/2646695522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f67d8157890 con 0x7f67d8073140 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67dda00640 1 -- 172.21.15.146:0/2646695522 learned_addr learned my addr 172.21.15.146:0/2646695522 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:46.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67de400640 1 --2- 172.21.15.146:0/2646695522 >> v2:172.21.15.146:3300/0 conn(0x7f67d8071c70 0x7f67d806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67dd000640 1 -- 172.21.15.146:0/2646695522 >> v2:172.21.15.146:3300/0 conn(0x7f67d8071c70 msgr2=0x7f67d806ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:46.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67dd000640 1 --2- 172.21.15.146:0/2646695522 >> v2:172.21.15.146:3300/0 conn(0x7f67d8071c70 0x7f67d806ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67dd000640 1 -- 172.21.15.146:0/2646695522 >> v1:172.21.15.161:6789/0 conn(0x7f67d806c700 legacy=0x7f67d8072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.087+0000 7f67dd000640 1 -- 172.21.15.146:0/2646695522 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d806fad0 con 0x7f67d8073140 2024-09-16T01:51:46.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.089+0000 7f67dd000640 1 --2- 172.21.15.146:0/2646695522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 0x7f67d80c85a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009d90 tx=0x7f67cc02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b12473c65bcacdc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.089+0000 7f67d7e00640 1 -- 172.21.15.146:0/2646695522 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc007740 con 0x7f67d8073140 2024-09-16T01:51:46.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.089+0000 7f67d7e00640 1 -- 172.21.15.146:0/2646695522 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc0078e0 con 0x7f67d8073140 2024-09-16T01:51:46.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.089+0000 7f67d7e00640 1 -- 172.21.15.146:0/2646695522 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc007cb0 con 0x7f67d8073140 2024-09-16T01:51:46.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.092+0000 7f67e0400640 1 -- 172.21.15.146:0/2646695522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 msgr2=0x7f67d80c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.092+0000 7f67e0400640 1 --2- 172.21.15.146:0/2646695522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 0x7f67d80c85a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009d90 tx=0x7f67cc02a560 comp rx=0 tx=0).stop 2024-09-16T01:51:46.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.093+0000 7f67e0400640 1 -- 172.21.15.146:0/2646695522 shutdown_connections 2024-09-16T01:51:46.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.093+0000 7f67e0400640 1 --2- 172.21.15.146:0/2646695522 >> v2:172.21.15.146:3300/0 conn(0x7f67d8071c70 0x7f67d806ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.093+0000 7f67e0400640 1 --2- 172.21.15.146:0/2646695522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d8073140 0x7f67d80c85a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.093+0000 7f67e0400640 1 -- 172.21.15.146:0/2646695522 >> 172.21.15.146:0/2646695522 conn(0x7f67d806bc10 msgr2=0x7f67d806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:46.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.095+0000 7f67e0400640 1 -- 172.21.15.146:0/2646695522 shutdown_connections 2024-09-16T01:51:46.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.096+0000 7f67e0400640 1 -- 172.21.15.146:0/2646695522 wait complete. 2024-09-16T01:51:46.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.103+0000 7f67e0400640 1 Processor -- start 2024-09-16T01:51:46.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.104+0000 7f67e0400640 1 -- start start 2024-09-16T01:51:46.107 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:46.107+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:46.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.108+0000 7f67e0400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d806c700 0x7f67d8155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.108+0000 7f67e0400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 0x7f67d81f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67e0400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d8073140 0x7f67d8138d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67dd000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 0x7f67d81f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67dd000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 0x7f67d81f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54562/0 (socket says 172.21.15.146:54562) 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67e0400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f67d806f3a0 con 0x7f67d8073140 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67dd000640 1 -- 172.21.15.146:0/732942854 learned_addr learned my addr 172.21.15.146:0/732942854 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67d806f220 con 0x7f67d8071c70 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.109+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f67d806f520 con 0x7f67d806c700 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.110+0000 7f67dd000640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d806c700 msgr2=0x7f67d8155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.110+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67d806c700 0x7f67d8155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.110+0000 7f67dd000640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d8073140 msgr2=0x7f67d8138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.110+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d8073140 0x7f67d8138d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.110+0000 7f67dd000640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d8149b30 con 0x7f67d8071c70 2024-09-16T01:51:46.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.110+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 0x7f67d81f2430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67cc02ab50 tx=0x7f67cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.111+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc031810 con 0x7f67d8071c70 2024-09-16T01:51:46.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.111+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc0319b0 con 0x7f67d8071c70 2024-09-16T01:51:46.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.111+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc031d80 con 0x7f67d8071c70 2024-09-16T01:51:46.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.111+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d8149dd0 con 0x7f67d8071c70 2024-09-16T01:51:46.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.111+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d814a310 con 0x7f67d8071c70 2024-09-16T01:51:46.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.113+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f67d806ece0 con 0x7f67d8071c70 2024-09-16T01:51:46.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.114+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f67cc04b3f0 con 0x7f67d8071c70 2024-09-16T01:51:46.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.117+0000 7f67d5600640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67bc06c3b0 0x7f67bc06e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.118+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5770+0+0 (secure 0 0 0) 0x7f67cc0b4e60 con 0x7f67d8071c70 2024-09-16T01:51:46.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.118+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v81) ==== 54+0+16 (secure 0 0 0) 0x7f67cc046070 con 0x7f67d8071c70 2024-09-16T01:51:46.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.119+0000 7f67c6a00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f67d8148140 0x7f67d81392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.119+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67d8149620 con 0x7f67d8148140 2024-09-16T01:51:46.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.120+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67bc06c3b0 0x7f67bc06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.121+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67bc06c3b0 0x7f67bc06e870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000f60 tx=0x7f67c80079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.121+0000 7f67de400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f67d8148140 0x7f67d81392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.123+0000 7f67de400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f67d8148140 0x7f67d81392b0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.124+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67d000a970 con 0x7f67d8148140 2024-09-16T01:51:46.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:51:46.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.153+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67d8165460 con 0x7f67d8148140 2024-09-16T01:51:46.168 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:46.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.165+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67d8165460 con 0x7f67d8148140 2024-09-16T01:51:46.168 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:46.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.166+0000 7f67c6a00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67d8148f90 0x7f67d814c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.167+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f67d806bff0 con 0x7f67d8148f90 2024-09-16T01:51:46.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.168+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67d8148f90 0x7f67d814c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.169+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67d8148f90 0x7f67d814c1a0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.172+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67d806bff0 con 0x7f67d8148f90 2024-09-16T01:51:46.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.193+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67d81f2020 con 0x7f67d8148f90 2024-09-16T01:51:46.208 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:46.208 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:46.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.204+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67d81f2020 con 0x7f67d8148f90 2024-09-16T01:51:46.209 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:46.207+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:46.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.216+0000 7f67c6a00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f67d8062630 0x7f67d8062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.216+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f67d814cfb0 con 0x7f67d8062630 2024-09-16T01:51:46.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.222+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f67d8062630 0x7f67d8062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.224+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f67d8062630 0x7f67d8062a30 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.226+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67d814cfb0 con 0x7f67d8062630 2024-09-16T01:51:46.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.259+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67d80631c0 con 0x7f67d8062630 2024-09-16T01:51:46.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.261+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67d80631c0 con 0x7f67d8062630 2024-09-16T01:51:46.263 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:46.264 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:46.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.261+0000 7f67c6a00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f67d81548e0 0x7f67d8154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.262+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f67d8072db0 con 0x7f67d81548e0 2024-09-16T01:51:46.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.266+0000 7f67de400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f67d81548e0 0x7f67d8154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.267+0000 7f67de400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f67d81548e0 0x7f67d8154d20 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.269+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67d8072db0 con 0x7f67d81548e0 2024-09-16T01:51:46.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.298+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67d8155530 con 0x7f67d81548e0 2024-09-16T01:51:46.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.304+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67d8155530 con 0x7f67d81548e0 2024-09-16T01:51:46.307 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:46.307 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:46.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.305+0000 7f67c6a00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f67d81ff640 0x7f67d8201a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.305+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f67d80c8140 con 0x7f67d81ff640 2024-09-16T01:51:46.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.306+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f67d81ff640 0x7f67d8201a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.306+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f67d81ff640 0x7f67d8201a30 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.309 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:46.307+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:46.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.307+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67c8007390 con 0x7f67d81ff640 2024-09-16T01:51:46.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.332+0000 7f67c6a00640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67d8139850 con 0x7f67d81ff640 2024-09-16T01:51:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.335+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67d8139850 con 0x7f67d81ff640 2024-09-16T01:51:46.337 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.335+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f67b0001650 0x7f67b0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.335+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f67b0006bb0 con 0x7f67b0001650 2024-09-16T01:51:46.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.336+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f67b0001650 0x7f67b0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.348+0000 7f67dd000640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f67b0001650 0x7f67b0003b10 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.349+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67b0006bb0 con 0x7f67b0001650 2024-09-16T01:51:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.379+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67b00058d0 con 0x7f67b0001650 2024-09-16T01:51:46.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.381+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67b00058d0 con 0x7f67b0001650 2024-09-16T01:51:46.383 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.381+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f67b00048a0 0x7f67b0004d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.381+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f67b0006690 con 0x7f67b00048a0 2024-09-16T01:51:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.382+0000 7f67de400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f67b00048a0 0x7f67b0004d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.382+0000 7f67de400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f67b00048a0 0x7f67b0004d00 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.383+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67b0006690 con 0x7f67b00048a0 2024-09-16T01:51:46.388 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:46.388 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:46.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.402+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67b00065e0 con 0x7f67b00048a0 2024-09-16T01:51:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.404+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67b00065e0 con 0x7f67b00048a0 2024-09-16T01:51:46.406 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.404+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f67b0005890 0x7f67b0005d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.404+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f67b0006330 con 0x7f67b0005890 2024-09-16T01:51:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.405+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f67b0005890 0x7f67b0005d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.405+0000 7f67dda00640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f67b0005890 0x7f67b0005d40 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.406+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67b0006330 con 0x7f67b0005890 2024-09-16T01:51:46.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.428+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67b000ada0 con 0x7f67b0005890 2024-09-16T01:51:46.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.433+0000 7f67d5600640 1 -- 172.21.15.146:0/732942854 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67b000ada0 con 0x7f67b0005890 2024-09-16T01:51:46.439 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:46.439 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:46.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f67d8148140 msgr2=0x7f67d81392b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f67d8148140 0x7f67d81392b0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67d8148f90 msgr2=0x7f67d814c1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f67d8148f90 0x7f67d814c1a0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f67d8062630 msgr2=0x7f67d8062a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f67d8062630 0x7f67d8062a30 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f67d81548e0 msgr2=0x7f67d8154d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f67d81548e0 0x7f67d8154d20 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f67d81ff640 msgr2=0x7f67d8201a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f67d81ff640 0x7f67d8201a30 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f67b0001650 msgr2=0x7f67b0003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f67b0001650 0x7f67b0003b10 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f67b00048a0 msgr2=0x7f67b0004d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f67b00048a0 0x7f67b0004d00 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f67b0005890 msgr2=0x7f67b0005d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.437+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f67b0005890 0x7f67b0005d40 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67de400640 1 -- 172.21.15.146:0/732942854 reap_dead start 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67de400640 1 -- 172.21.15.146:0/732942854 reap_dead start 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67de400640 1 -- 172.21.15.146:0/732942854 reap_dead start 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67de400640 1 -- 172.21.15.146:0/732942854 reap_dead start 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67de400640 1 -- 172.21.15.146:0/732942854 reap_dead start 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67bc06c3b0 msgr2=0x7f67bc06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67bc06c3b0 0x7f67bc06e870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000f60 tx=0x7f67c80079e0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 msgr2=0x7f67d81f2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.438+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 0x7f67d81f2430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67cc02ab50 tx=0x7f67cc004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.439+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 shutdown_connections 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.439+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f67bc06c3b0 0x7f67bc06e870 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000f60 tx=0x7f67c80079e0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.439+0000 7f67e0400640 1 --2- 172.21.15.146:0/732942854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d8071c70 0x7f67d81f2430 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67cc02ab50 tx=0x7f67cc004a40 comp rx=0 tx=0).stop 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.439+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 >> 172.21.15.146:0/732942854 conn(0x7f67d806bc10 msgr2=0x7f67d80c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.439+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 shutdown_connections 2024-09-16T01:51:46.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.439+0000 7f67e0400640 1 -- 172.21.15.146:0/732942854 wait complete. 2024-09-16T01:51:46.450 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:46.449+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:46.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.450+0000 7ff072600640 1 Processor -- start 2024-09-16T01:51:46.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.451+0000 7ff072600640 1 -- start start 2024-09-16T01:51:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff072600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff072600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06c15b1d0 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff072600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06c15b4b0 con 0x7ff06c0cc0c0 2024-09-16T01:51:46.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff072600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff06c0cbd00 con 0x7ff06c0cb2f0 2024-09-16T01:51:46.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff06aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff06b400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff06c0cc0c0 0x7ff06c15e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:39956/0 (socket says 172.21.15.146:39956) 2024-09-16T01:51:46.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.453+0000 7ff06b400640 1 -- 172.21.15.146:0/3702357455 learned_addr learned my addr 172.21.15.146:0/3702357455 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:46.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.454+0000 7ff06a000640 1 -- 172.21.15.146:0/3702357455 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1478934462 0 0) 0x7ff06c15b4b0 con 0x7ff06c0cc0c0 2024-09-16T01:51:46.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.454+0000 7ff06a000640 1 -- 172.21.15.146:0/3702357455 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff050003520 con 0x7ff06c0cc0c0 2024-09-16T01:51:46.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.454+0000 7ff06aa00640 1 -- 172.21.15.146:0/3702357455 >> v1:172.21.15.161:6789/0 conn(0x7ff06c0cc0c0 legacy=0x7ff06c15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.454+0000 7ff06aa00640 1 -- 172.21.15.146:0/3702357455 >> v1:172.21.15.146:6789/0 conn(0x7ff06c0cc4a0 legacy=0x7ff06c165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.454+0000 7ff06aa00640 1 -- 172.21.15.146:0/3702357455 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c15b790 con 0x7ff06c0cb2f0 2024-09-16T01:51:46.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.455+0000 7ff06aa00640 1 --2- 172.21.15.146:0/3702357455 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15aab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff0580097d0 tx=0x7ff05802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51b5f4083a7a3e5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.456+0000 7ff06a000640 1 -- 172.21.15.146:0/3702357455 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058037070 con 0x7ff06c0cb2f0 2024-09-16T01:51:46.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.456+0000 7ff06a000640 1 -- 172.21.15.146:0/3702357455 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff058032440 con 0x7ff06c0cb2f0 2024-09-16T01:51:46.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.456+0000 7ff06a000640 1 -- 172.21.15.146:0/3702357455 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058032810 con 0x7ff06c0cb2f0 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 -- 172.21.15.146:0/3702357455 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 msgr2=0x7ff06c15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 --2- 172.21.15.146:0/3702357455 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15aab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff0580097d0 tx=0x7ff05802a210 comp rx=0 tx=0).stop 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 -- 172.21.15.146:0/3702357455 shutdown_connections 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 --2- 172.21.15.146:0/3702357455 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15aab0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 -- 172.21.15.146:0/3702357455 >> 172.21.15.146:0/3702357455 conn(0x7ff06c06d560 msgr2=0x7ff06c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 -- 172.21.15.146:0/3702357455 shutdown_connections 2024-09-16T01:51:46.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.457+0000 7ff072600640 1 -- 172.21.15.146:0/3702357455 wait complete. 2024-09-16T01:51:46.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.460+0000 7ff072600640 1 Processor -- start 2024-09-16T01:51:46.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.460+0000 7ff072600640 1 -- start start 2024-09-16T01:51:46.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.461+0000 7ff072600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.461+0000 7ff072600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 0x7ff06c156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.461+0000 7ff072600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff06c0cc4a0 0x7ff06c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.461+0000 7ff072600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c15e540 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.461+0000 7ff072600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c15ba70 con 0x7ff06c0cc0c0 2024-09-16T01:51:46.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.461+0000 7ff072600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff06c0cc8b0 con 0x7ff06c0cb2f0 2024-09-16T01:51:46.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06b400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06aa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 0x7ff06c156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff06c0cc4a0 0x7ff06c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06b400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39574/0 (socket says 172.21.15.146:39574) 2024-09-16T01:51:46.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06aa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 0x7ff06c156370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54574/0 (socket says 172.21.15.146:54574) 2024-09-16T01:51:46.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06b400640 1 -- 172.21.15.146:0/3005708893 learned_addr learned my addr 172.21.15.146:0/3005708893 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:46.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 -- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 msgr2=0x7ff06c15a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 -- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 msgr2=0x7ff06c156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:46.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 0x7ff06c156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:46.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 -- 172.21.15.146:0/3005708893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c0cf330 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06aa00640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 0x7ff06c156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06b400640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.462+0000 7ff06be00640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff06c0cc4a0 0x7ff06c1568b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff06400c730 tx=0x7ff06400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.463+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff064016020 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.463+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06c0c80b0 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.463+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff064006d50 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.463+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff064003bb0 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.463+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06c0c8670 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.464+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06c0c9120 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.466+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff0640050d0 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.467+0000 7ff063400640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff04c06c2b0 0x7ff04c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:46.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.468+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5770+0+0 (secure 0 0 0) 0x7ff06408aed0 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.468+0000 7ff06b400640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff04c06c2b0 0x7ff04c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:46.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.468+0000 7ff06b400640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff04c06c2b0 0x7ff04c06e770 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0011e0 tx=0x7ff05c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:46.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.472+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff064020020 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.512 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:46.511+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:46.528 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:46.528 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:46.538 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:46.614 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:46.612+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:46.697 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:46.698 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:46.708 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:46.753 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:46.752+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:46.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:46.798+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7ff06c0c5560 con 0x7ff06c0cc4a0 2024-09-16T01:51:46.815 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:46.813+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:46.833 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:46.833 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:46.844 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:46.920 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:46.918+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:46.991 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:46.992 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.001 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:47.017 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:47.015+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:47.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.105+0000 7ff063400640 1 -- 172.21.15.146:0/3005708893 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v82) ==== 91+0+0 (secure 0 0 0) 0x7ff064058f20 con 0x7ff06c0cc4a0 2024-09-16T01:51:47.108 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:47.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:47.109 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:47.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-09-16T01:51:47.112 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:51:47.120 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.120 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.115+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff04c06c2b0 msgr2=0x7ff04c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:47.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.115+0000 7ff072600640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff04c06c2b0 0x7ff04c06e770 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0011e0 tx=0x7ff05c009040 comp rx=0 tx=0).stop 2024-09-16T01:51:47.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.115+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff06c0cc4a0 msgr2=0x7ff06c1568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.115+0000 7ff072600640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff06c0cc4a0 0x7ff06c1568b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff06400c730 tx=0x7ff06400cbf0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.115+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 shutdown_connections 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff04c06c2b0 0x7ff04c06e770 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff06c0cc4a0 0x7ff06c1568b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff06c0cc0c0 0x7ff06c156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 --2- 172.21.15.146:0/3005708893 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff06c0cb2f0 0x7ff06c15a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 >> 172.21.15.146:0/3005708893 conn(0x7ff06c06d560 msgr2=0x7ff06c0cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 shutdown_connections 2024-09-16T01:51:47.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.116+0000 7ff072600640 1 -- 172.21.15.146:0/3005708893 wait complete. 2024-09-16T01:51:47.122 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:47.120+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:47.131 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:47.222 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:47.219+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:47.252 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.252 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.263 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:47.320 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:47.318+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:47.379 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.379 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.390 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:47.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.419+0000 7f0aae800640 1 Processor -- start 2024-09-16T01:51:47.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.420+0000 7f0aae800640 1 -- start start 2024-09-16T01:51:47.424 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:47.421+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:47.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.422+0000 7f0aae800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa8150c10 0x7f0aa8072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:47.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.422+0000 7f0aae800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0aa80c70e0 0x7f0aa80727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:47.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.422+0000 7f0aae800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aa81569d0 con 0x7f0aa8072dd0 2024-09-16T01:51:47.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.422+0000 7f0aae800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa8157580 con 0x7f0aa80c70e0 2024-09-16T01:51:47.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.422+0000 7f0aae800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa8071ce0 con 0x7f0aa8150c10 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa8150c10 0x7f0aa8072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa8150c10 0x7f0aa8072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39582/0 (socket says 172.21.15.146:39582) 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa7400640 1 -- 172.21.15.146:0/796181879 learned_addr learned my addr 172.21.15.146:0/796181879 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa6a00640 1 --2- 172.21.15.146:0/796181879 >> v2:172.21.15.161:3300/0 conn(0x7f0aa80c70e0 0x7f0aa80727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa6a00640 1 -- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa8150c10 msgr2=0x7f0aa8072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa6a00640 1 --2- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa8150c10 0x7f0aa8072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa6a00640 1 -- 172.21.15.146:0/796181879 >> v1:172.21.15.146:6789/0 conn(0x7f0aa8072dd0 legacy=0x7f0aa806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:47.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.423+0000 7f0aa6a00640 1 -- 172.21.15.146:0/796181879 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa8156c50 con 0x7f0aa80c70e0 2024-09-16T01:51:47.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.425+0000 7f0aa6a00640 1 --2- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa80c70e0 0x7f0aa80727d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8157700 tx=0x7f0a9402a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c49d69bb62a9e3c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:47.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.427+0000 7f0aa6000640 1 -- 172.21.15.146:0/796181879 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94037070 con 0x7f0aa80c70e0 2024-09-16T01:51:47.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.427+0000 7f0aa6000640 1 -- 172.21.15.146:0/796181879 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9402ae50 con 0x7f0aa80c70e0 2024-09-16T01:51:47.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.427+0000 7f0aa6000640 1 -- 172.21.15.146:0/796181879 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94032660 con 0x7f0aa80c70e0 2024-09-16T01:51:47.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.428+0000 7f0aae800640 1 -- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa80c70e0 msgr2=0x7f0aa80727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:47.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.428+0000 7f0aae800640 1 --2- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa80c70e0 0x7f0aa80727d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8157700 tx=0x7f0a9402a140 comp rx=0 tx=0).stop 2024-09-16T01:51:47.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.428+0000 7f0aae800640 1 -- 172.21.15.146:0/796181879 shutdown_connections 2024-09-16T01:51:47.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.428+0000 7f0aae800640 1 --2- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa80c70e0 0x7f0aa80727d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.428+0000 7f0aae800640 1 --2- 172.21.15.146:0/796181879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa8150c10 0x7f0aa8072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.428+0000 7f0aae800640 1 -- 172.21.15.146:0/796181879 >> 172.21.15.146:0/796181879 conn(0x7f0aa806f3c0 msgr2=0x7f0aa806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:47.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.430+0000 7f0aae800640 1 -- 172.21.15.146:0/796181879 shutdown_connections 2024-09-16T01:51:47.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.436+0000 7f0aae800640 1 -- 172.21.15.146:0/796181879 wait complete. 2024-09-16T01:51:47.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.439+0000 7f0aae800640 1 Processor -- start 2024-09-16T01:51:47.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.440+0000 7f0aae800640 1 -- start start 2024-09-16T01:51:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.441+0000 7f0aae800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8072dd0 0x7f0aa8200f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.441+0000 7f0aae800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 0x7f0aa814f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.441+0000 7f0aae800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aa8150c10 0x7f0aa814f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.441+0000 7f0aae800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa80c9a90 con 0x7f0aa8150c10 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.441+0000 7f0aae800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa8157210 con 0x7f0aa8072dd0 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.441+0000 7f0aae800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa80718c0 con 0x7f0aa80c70e0 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.442+0000 7f0aa6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 0x7f0aa814f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.442+0000 7f0aa6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 0x7f0aa814f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39592/0 (socket says 172.21.15.146:39592) 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.442+0000 7f0aa6a00640 1 -- 172.21.15.146:0/1351126809 learned_addr learned my addr 172.21.15.146:0/1351126809 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.443+0000 7f0aa7400640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8072dd0 0x7f0aa8200f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.443+0000 7f0aa7e00640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aa8150c10 0x7f0aa814f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:47.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.446+0000 7f0aa6a00640 1 -- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8072dd0 msgr2=0x7f0aa8200f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:47.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.446+0000 7f0aa6a00640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8072dd0 0x7f0aa8200f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.446+0000 7f0aa6a00640 1 -- 172.21.15.146:0/1351126809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aa8150c10 msgr2=0x7f0aa814f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:47.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.447+0000 7f0aa6a00640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aa8150c10 0x7f0aa814f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:47.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.447+0000 7f0aa6a00640 1 -- 172.21.15.146:0/1351126809 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa80c7c80 con 0x7f0aa80c70e0 2024-09-16T01:51:47.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.447+0000 7f0aa7e00640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aa8150c10 0x7f0aa814f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:47.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.447+0000 7f0aa7400640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8072dd0 0x7f0aa8200f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:51:47.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.448+0000 7f0aa6a00640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 0x7f0aa814f490 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402a730 tx=0x7f0a940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:47.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.449+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94041020 con 0x7f0aa80c70e0 2024-09-16T01:51:47.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.450+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9403d670 con 0x7f0aa80c70e0 2024-09-16T01:51:47.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.450+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94041020 con 0x7f0aa80c70e0 2024-09-16T01:51:47.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.450+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa814a2e0 con 0x7f0aa80c70e0 2024-09-16T01:51:47.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.451+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa80c59e0 con 0x7f0aa80c70e0 2024-09-16T01:51:47.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.452+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aa80c8760 con 0x7f0aa80c70e0 2024-09-16T01:51:47.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.454+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0a9402ad10 con 0x7f0aa80c70e0 2024-09-16T01:51:47.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.455+0000 7f0a9f400640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a8806c4b0 0x7f0a8806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:47.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.455+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(82..82 src has 1..82) ==== 5770+0+0 (secure 0 0 0) 0x7f0a940ab6e0 con 0x7f0aa80c70e0 2024-09-16T01:51:47.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.456+0000 7f0aa7400640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a8806c4b0 0x7f0a8806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:47.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.457+0000 7f0aa7400640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a8806c4b0 0x7f0a8806e970 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800acc0 tx=0x7f0a9800d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:47.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.458+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0aa80c8760 con 0x7f0aa80c70e0 2024-09-16T01:51:47.521 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:47.519+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:47.523 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.523 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.534 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:47.623 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:47.621+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:47.657 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.658 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.667 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:47.723 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:47.722+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:47.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:47.745+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd out", "ids": ["4"]} v 0) -- 0x7f0aa806fee0 con 0x7f0aa80c70e0 2024-09-16T01:51:47.749 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:51:47.746+0000 7f49bee00640 -1 mon.a@0(leader).osd e82 definitely_dead 0 2024-09-16T01:51:47.783 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.784 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.795 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:47.826 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:47.824+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:47.925 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:47.924+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:47.955 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:47.955 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:47.964 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:48.026 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:48.024+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:48.124 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:48.125 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:48.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.122+0000 7f0a9f400640 1 -- 172.21.15.146:0/1351126809 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["4"]}]=0 marked out osd.4. v83) ==== 87+0+0 (secure 0 0 0) 0x7f0aa80c8760 con 0x7f0aa80c70e0 2024-09-16T01:51:48.126 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:48.125+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:48.127 INFO:teuthology.orchestra.run.smithi146.stderr:marked out osd.4. 2024-09-16T01:51:48.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:48.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.136+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a8806c4b0 msgr2=0x7f0a8806e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:48.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.136+0000 7f0aae800640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a8806c4b0 0x7f0a8806e970 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800acc0 tx=0x7f0a9800d040 comp rx=0 tx=0).stop 2024-09-16T01:51:48.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.136+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 msgr2=0x7f0aa814f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:48.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.136+0000 7f0aae800640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 0x7f0aa814f490 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402a730 tx=0x7f0a940046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.139+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 shutdown_connections 2024-09-16T01:51:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.139+0000 7f0aae800640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0a8806c4b0 0x7f0a8806e970 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.139+0000 7f0aae800640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aa8150c10 0x7f0aa814f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.139+0000 7f0aae800640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0aa80c70e0 0x7f0aa814f490 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.139+0000 7f0aae800640 1 --2- 172.21.15.146:0/1351126809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0aa8072dd0 0x7f0aa8200f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.139+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 >> 172.21.15.146:0/1351126809 conn(0x7f0aa806f3c0 msgr2=0x7f0aa80c7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:48.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.145+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 shutdown_connections 2024-09-16T01:51:48.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.145+0000 7f0aae800640 1 -- 172.21.15.146:0/1351126809 wait complete. 2024-09-16T01:51:48.229 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:48.227+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:48.328 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:48.326+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:48.429 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:48.428+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:48.434 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:48.434 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:48.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:48.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:48.530 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:48.529+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:48.632 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:48.630+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:48.736 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:48.734+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.803+0000 7feb42c00640 1 Processor -- start 2024-09-16T01:51:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.803+0000 7feb42c00640 1 -- start start 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb42c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feb3c06f9f0 0x7feb3c06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb42c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06fe20 0x7feb3c0c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb42c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3c0c7020 con 0x7feb3c0cce70 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb42c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7feb3c0c7300 con 0x7feb3c06f9f0 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb42c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7feb3c0c75e0 con 0x7feb3c06fe20 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb40c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06fe20 0x7feb3c0c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb3be00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7feb3c0cce70 0x7feb3c1577a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:39964/0 (socket says 172.21.15.146:39964) 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb3be00640 1 -- 172.21.15.146:0/988999449 learned_addr learned my addr 172.21.15.146:0/988999449 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.805+0000 7feb40c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06fe20 0x7feb3c0c4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39596/0 (socket says 172.21.15.146:39596) 2024-09-16T01:51:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.806+0000 7feb3b400640 1 --2- 172.21.15.146:0/988999449 >> v2:172.21.15.161:3300/0 conn(0x7feb3c06f9f0 0x7feb3c06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.806+0000 7feb3b400640 1 -- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06fe20 msgr2=0x7feb3c0c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:48.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.806+0000 7feb3b400640 1 --2- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06fe20 0x7feb3c0c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.806+0000 7feb3b400640 1 -- 172.21.15.146:0/988999449 >> v1:172.21.15.161:6789/0 conn(0x7feb3c0cce70 legacy=0x7feb3c1577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:48.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.806+0000 7feb3b400640 1 -- 172.21.15.146:0/988999449 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c0c78c0 con 0x7feb3c06f9f0 2024-09-16T01:51:48.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.807+0000 7feb3b400640 1 --2- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c06f9f0 0x7feb3c06eef0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0097a0 tx=0x7feb2c02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=539d09c607668199 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.807+0000 7feb3aa00640 1 -- 172.21.15.146:0/988999449 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c037070 con 0x7feb3c06f9f0 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb3aa00640 1 -- 172.21.15.146:0/988999449 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb2c032440 con 0x7feb3c06f9f0 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb3aa00640 1 -- 172.21.15.146:0/988999449 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c032810 con 0x7feb3c06f9f0 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb42c00640 1 -- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c06f9f0 msgr2=0x7feb3c06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb42c00640 1 --2- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c06f9f0 0x7feb3c06eef0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0097a0 tx=0x7feb2c02a210 comp rx=0 tx=0).stop 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb42c00640 1 -- 172.21.15.146:0/988999449 shutdown_connections 2024-09-16T01:51:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb42c00640 1 --2- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06fe20 0x7feb3c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb42c00640 1 --2- 172.21.15.146:0/988999449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c06f9f0 0x7feb3c06eef0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.808+0000 7feb42c00640 1 -- 172.21.15.146:0/988999449 >> 172.21.15.146:0/988999449 conn(0x7feb3c0cc380 msgr2=0x7feb3c0cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.809+0000 7feb42c00640 1 -- 172.21.15.146:0/988999449 shutdown_connections 2024-09-16T01:51:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.809+0000 7feb42c00640 1 -- 172.21.15.146:0/988999449 wait complete. 2024-09-16T01:51:48.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.810+0000 7feb42c00640 1 Processor -- start 2024-09-16T01:51:48.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.810+0000 7feb42c00640 1 -- start start 2024-09-16T01:51:48.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.811+0000 7feb42c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06f9f0 0x7feb3c0b9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.811+0000 7feb42c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb3c0cce70 0x7feb3c14d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb42c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 0x7feb3c1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb42c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c0c6d00 con 0x7feb3c0cce70 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb42c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c0c7e80 con 0x7feb3c14f1a0 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb42c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7feb3c06f430 con 0x7feb3c06f9f0 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 0x7feb3c1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 0x7feb3c1513c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54622/0 (socket says 172.21.15.146:54622) 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 learned_addr learned my addr 172.21.15.146:0/3151429317 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb3b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb3c0cce70 0x7feb3c14d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06f9f0 0x7feb3c0b9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06f9f0 msgr2=0x7feb3c0b9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feb3c06f9f0 0x7feb3c0b9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb3c0cce70 msgr2=0x7feb3c14d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb3c0cce70 0x7feb3c14d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:48.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.812+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c152690 con 0x7feb3c14f1a0 2024-09-16T01:51:48.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.813+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 0x7feb3c1513c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feb2800e930 tx=0x7feb2800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.813+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28017070 con 0x7feb3c14f1a0 2024-09-16T01:51:48.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.813+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb2800c980 con 0x7feb3c14f1a0 2024-09-16T01:51:48.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.813+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3c157190 con 0x7feb3c14f1a0 2024-09-16T01:51:48.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.813+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2800cd70 con 0x7feb3c14f1a0 2024-09-16T01:51:48.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.813+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3c1f16a0 con 0x7feb3c14f1a0 2024-09-16T01:51:48.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.814+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7feb3c14f860 con 0x7feb3c14f1a0 2024-09-16T01:51:48.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.815+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7feb2801a070 con 0x7feb3c14f1a0 2024-09-16T01:51:48.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.816+0000 7feb33e00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feb1c06c2e0 0x7feb1c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.816+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5770+0+0 (secure 0 0 0) 0x7feb2808b730 con 0x7feb3c14f1a0 2024-09-16T01:51:48.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.816+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feb1c06c2e0 0x7feb1c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.816+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v83) ==== 54+0+16 (secure 0 0 0) 0x7feb3c14f860 con 0x7feb3c14f1a0 2024-09-16T01:51:48.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.817+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feb3c06fe20 0x7feb3c14d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.817+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb3c06eef0 con 0x7feb3c06fe20 2024-09-16T01:51:48.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.817+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feb1c06c2e0 0x7feb1c06e7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feb340012e0 tx=0x7feb34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.817+0000 7feb3b400640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feb3c06fe20 0x7feb3c14d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.818+0000 7feb3b400640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feb3c06fe20 0x7feb3c14d0b0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.819+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c06eef0 con 0x7feb3c06fe20 2024-09-16T01:51:48.833 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:48.831+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:48.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.836+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c157cc0 con 0x7feb3c06fe20 2024-09-16T01:51:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.839+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c157cc0 con 0x7feb3c06fe20 2024-09-16T01:51:48.842 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.839+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feb3c1f18e0 0x7feb3c146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.840+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7feb3c073730 con 0x7feb3c1f18e0 2024-09-16T01:51:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.840+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feb3c1f18e0 0x7feb3c146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.841+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feb3c1f18e0 0x7feb3c146770 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.842+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c073730 con 0x7feb3c1f18e0 2024-09-16T01:51:48.847 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:48.847 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:48.847 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:48.847 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:48.847 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:48.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.858+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c0cc9b0 con 0x7feb3c1f18e0 2024-09-16T01:51:48.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.861+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c0cc9b0 con 0x7feb3c1f18e0 2024-09-16T01:51:48.863 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:48.864 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:48.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.862+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feb3c1f40f0 0x7feb3c1f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.862+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7feb3c1471c0 con 0x7feb3c1f40f0 2024-09-16T01:51:48.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.870+0000 7feb3b400640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feb3c1f40f0 0x7feb3c1f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.872+0000 7feb3b400640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feb3c1f40f0 0x7feb3c1f44f0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:48.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.874+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c1471c0 con 0x7feb3c1f40f0 2024-09-16T01:51:48.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.922+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c1f4c80 con 0x7feb3c1f40f0 2024-09-16T01:51:48.933 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:48.932+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:48.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.933+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c1f4c80 con 0x7feb3c1f40f0 2024-09-16T01:51:48.936 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:48.936 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:48.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.933+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feb3c062cb0 0x7feb3c063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.933+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7feb3c0635d0 con 0x7feb3c062cb0 2024-09-16T01:51:48.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.934+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feb3c062cb0 0x7feb3c063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.935+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feb3c062cb0 0x7feb3c063090 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.942+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c0635d0 con 0x7feb3c062cb0 2024-09-16T01:51:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.966+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c14b1b0 con 0x7feb3c062cb0 2024-09-16T01:51:48.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.976+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c14b1b0 con 0x7feb3c062cb0 2024-09-16T01:51:48.979 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.977+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feb3c14b8d0 0x7feb3c02c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.977+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7feb3c14bcf0 con 0x7feb3c14b8d0 2024-09-16T01:51:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.977+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feb3c14b8d0 0x7feb3c02c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.978+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feb3c14b8d0 0x7feb3c02c640 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:48.979+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c14bcf0 con 0x7feb3c14b8d0 2024-09-16T01:51:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.007+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c14bcf0 con 0x7feb3c14b8d0 2024-09-16T01:51:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.010+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c14bcf0 con 0x7feb3c14b8d0 2024-09-16T01:51:49.012 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:49.013 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:49.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.011+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feb3c02d090 0x7feb3c02d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.011+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7feb3c02dab0 con 0x7feb3c02d090 2024-09-16T01:51:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.013+0000 7feb3b400640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feb3c02d090 0x7feb3c02d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.014+0000 7feb3b400640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feb3c02d090 0x7feb3c02d4f0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:49.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.015+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c02dab0 con 0x7feb3c02d090 2024-09-16T01:51:49.035 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:49.033+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:49.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.044+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c000950 con 0x7feb3c02d090 2024-09-16T01:51:49.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.046+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c000950 con 0x7feb3c02d090 2024-09-16T01:51:49.048 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:49.049 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:49.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.047+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feb3c207990 0x7feb3c209d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.047+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7feb3c20cd10 con 0x7feb3c207990 2024-09-16T01:51:49.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.050+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feb3c207990 0x7feb3c209d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.054+0000 7feb40c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feb3c207990 0x7feb3c209d80 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:49.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.055+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c20cd10 con 0x7feb3c207990 2024-09-16T01:51:49.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.083+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c20cce0 con 0x7feb3c207990 2024-09-16T01:51:49.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.085+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c20cce0 con 0x7feb3c207990 2024-09-16T01:51:49.088 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.086+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feb3c20bf60 0x7feb3c20c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.086+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7feb3c2100e0 con 0x7feb3c20bf60 2024-09-16T01:51:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.086+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feb3c20bf60 0x7feb3c20c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.087+0000 7feb3be00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feb3c20bf60 0x7feb3c20c3a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:49.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.088+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb3c2100e0 con 0x7feb3c20bf60 2024-09-16T01:51:49.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.121+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feb3c20a310 con 0x7feb3c20bf60 2024-09-16T01:51:49.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.123+0000 7feb33e00640 1 -- 172.21.15.146:0/3151429317 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feb3c20a310 con 0x7feb3c20bf60 2024-09-16T01:51:49.125 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:49.126 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:49.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feb3c06fe20 msgr2=0x7feb3c14d0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feb3c06fe20 0x7feb3c14d0b0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feb3c1f18e0 msgr2=0x7feb3c146770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feb3c1f18e0 0x7feb3c146770 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feb3c1f40f0 msgr2=0x7feb3c1f44f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feb3c1f40f0 0x7feb3c1f44f0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feb3c062cb0 msgr2=0x7feb3c063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feb3c062cb0 0x7feb3c063090 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feb3c14b8d0 msgr2=0x7feb3c02c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feb3c14b8d0 0x7feb3c02c640 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feb3c02d090 msgr2=0x7feb3c02d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feb3c02d090 0x7feb3c02d4f0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feb3c207990 msgr2=0x7feb3c209d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feb3c207990 0x7feb3c209d80 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feb3c20bf60 msgr2=0x7feb3c20c3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.125+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feb3c20bf60 0x7feb3c20c3a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 reap_dead start 2024-09-16T01:51:49.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 reap_dead start 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 reap_dead start 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 reap_dead start 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb40c00640 1 -- 172.21.15.146:0/3151429317 reap_dead start 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feb1c06c2e0 msgr2=0x7feb1c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feb1c06c2e0 0x7feb1c06e7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feb340012e0 tx=0x7feb34009040 comp rx=0 tx=0).stop 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 msgr2=0x7feb3c1513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.126+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 0x7feb3c1513c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feb2800e930 tx=0x7feb2800edf0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.127+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 shutdown_connections 2024-09-16T01:51:49.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.127+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feb1c06c2e0 0x7feb1c06e7a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.127+0000 7feb42c00640 1 --2- 172.21.15.146:0/3151429317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb3c14f1a0 0x7feb3c1513c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.127+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 >> 172.21.15.146:0/3151429317 conn(0x7feb3c0cc380 msgr2=0x7feb3c0c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:49.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.127+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 shutdown_connections 2024-09-16T01:51:49.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.127+0000 7feb42c00640 1 -- 172.21.15.146:0/3151429317 wait complete. 2024-09-16T01:51:49.139 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:49.137+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:49.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:51:49.238 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:49.236+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:40 3.f 3:f045ad74:::10000000002.00000025:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:50.565654+0000", 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.676526850999998, 2024-09-16T01:51:49.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.068719732000000006, 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 40 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.565654+0000", 2024-09-16T01:51:49.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.565654+0000", 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.565656+0000", 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.586666+0000", 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021009762000000001 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.586720+0000", 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4871999999999999e-05 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.586771+0000", 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0507000000000003e-05 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.586942+0000", 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017098999999999999 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.587081+0000", 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013884100000000001 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.587218+0000", 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013681899999999999 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.605774+0000", 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018556848000000001 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634258+0000", 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028483623 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634318+0000", 2024-09-16T01:51:49.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9651000000000002e-05 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:50.634373+0000", 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5739000000000003e-05 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:85 3.36 3:6fe880e3:::10000000002.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:51.017084+0000", 2024-09-16T01:51:49.253 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.225096545, 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.078597749999999994, 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 85 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.017084+0000", 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.017084+0000", 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.017085+0000", 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.356e-06 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.040713+0000", 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023627763 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.040738+0000", 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5256e-05 2024-09-16T01:51:49.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.040750+0000", 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1651000000000001e-05 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.040823+0000", 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2432e-05 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.040916+0000", 2024-09-16T01:51:49.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3218000000000001e-05 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.041004+0000", 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7805000000000004e-05 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.063374+0000", 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022370891 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095553+0000", 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032178352 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095633+0000", 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0425000000000003e-05 2024-09-16T01:51:49.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:51.095682+0000", 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8600999999999998e-05 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:478 3.c 3:32521e74:::10000000002.000000db:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 49.217091644, 2024-09-16T01:51:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.88174018200000004, 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 478 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:51:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025090+0000", 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2699999999999998e-07 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047858+0000", 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022768360000000001 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047892+0000", 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3855e-05 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047918+0000", 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5595000000000001e-05 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048026+0000", 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108242 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048035+0000", 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3940000000000002e-06 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828462+0000", 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.78042635999999999 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831445+0000", 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029837000000000002 2024-09-16T01:51:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882498+0000", 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051052510000000002 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882597+0000", 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8874000000000002e-05 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882746+0000", 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149457 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.903108+0000", 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020361516 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906632+0000", 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035239960000000002 2024-09-16T01:51:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906720+0000", 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7952000000000001e-05 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906829+0000", 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109444 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:527 3.c 3:31390f1a:::100000001f7.0000000c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.514558+0000", 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.727622373999999, 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.078147525999999995, 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 527 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.514558+0000", 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.514558+0000", 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.514560+0000", 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.471e-06 2024-09-16T01:51:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.536951+0000", 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022391547000000001 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537000+0000", 2024-09-16T01:51:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8770999999999998e-05 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537025+0000", 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4488e-05 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537181+0000", 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015645700000000001 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537297+0000", 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116339 2024-09-16T01:51:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.537411+0000", 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113552 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.550698+0000", 2024-09-16T01:51:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013287192 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.592566+0000", 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041868181999999997 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.592649+0000", 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3041000000000007e-05 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.592706+0000", 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6486000000000001e-05 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:771 3.36 3:6fe38dc2:::100000001f7.00000000:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.174066+0000", 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.068114212000001, 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070459834999999998, 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 771 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.174066+0000", 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.174066+0000", 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.174069+0000", 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9519999999999999e-06 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.185330+0000", 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011260177999999999 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.185366+0000", 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6597999999999998e-05 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.185442+0000", 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5737999999999999e-05 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.186872+0000", 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00143059 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.187125+0000", 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025282899999999999 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.187422+0000", 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029690800000000002 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.230778+0000", 2024-09-16T01:51:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043355994000000002 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.244416+0000", 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013637639 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.244479+0000", 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3068e-05 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.244526+0000", 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7341e-05 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:775 3.15 3:a92727ba:::100000001f7.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.185487+0000", 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.056693762999998, 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073889921999999997, 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 775 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.185487+0000", 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.185487+0000", 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.185489+0000", 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9199999999999998e-06 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.209275+0000", 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023786333999999999 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212399+0000", 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003123841 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212421+0000", 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2340999999999998e-05 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212677+0000", 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025534700000000001 2024-09-16T01:51:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212790+0000", 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113157 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212961+0000", 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017079899999999999 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.254639+0000", 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041678585999999997 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.259227+0000", 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0045875630000000002 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.259314+0000", 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6792000000000005e-05 2024-09-16T01:51:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.259377+0000", 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3242000000000002e-05 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:777 3.2c 3:36199bb1:::100000001f7.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.029715732, 2024-09-16T01:51:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106547593, 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 777 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212467+0000", 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1399999999999998e-06 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221218+0000", 2024-09-16T01:51:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087507980000000006 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221238+0000", 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9729999999999999e-05 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221249+0000", 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1462e-05 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222669+0000", 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001420017 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222797+0000", 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127572 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222974+0000", 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017753699999999999 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.258753+0000", 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035779068999999997 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318898+0000", 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060144465000000001 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318963+0000", 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5098999999999995e-05 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.319012+0000", 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9703999999999997e-05 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:782 3.2c 3:36568ae2:::100000001f7.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.245221+0000", 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.996959853, 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083119175000000003, 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 782 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.245221+0000", 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.245221+0000", 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.245222+0000", 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.035e-06 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.267460+0000", 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022237784 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.267489+0000", 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9439999999999999e-05 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.267503+0000", 2024-09-16T01:51:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4083000000000001e-05 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.270741+0000", 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032376919999999999 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.270842+0000", 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101576 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.271025+0000", 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000182547 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.294540+0000", 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023515404 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.328215+0000", 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033674295999999999 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.328290+0000", 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5722000000000004e-05 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.328340+0000", 2024-09-16T01:51:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9595999999999997e-05 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:837 3.15 3:abde3564:::100000001f7.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.811708+0000", 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.430472364, 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072304048999999995, 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 837 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.811708+0000", 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.811708+0000", 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.811711+0000", 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.43e-06 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.869910+0000", 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058199031999999998 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.869926+0000", 2024-09-16T01:51:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6597000000000001e-05 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.869934+0000", 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0180000000000001e-06 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.870011+0000", 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6553999999999999e-05 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.870148+0000", 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013739899999999999 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.870317+0000", 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000168901 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.875327+0000", 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050094379999999997 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.883939+0000", 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0086127029999999993 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.883979+0000", 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9520000000000001e-05 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.884012+0000", 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3457000000000002e-05 2024-09-16T01:51:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1127 3.36 3:6e2a8853:::100000001f8.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.9379873419999996, 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15931991200000001, 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1127 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304196+0000", 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.526e-06 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354927+0000", 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050730866999999999 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355561+0000", 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063468300000000004 2024-09-16T01:51:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355571+0000", 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6199999999999994e-06 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355665+0000", 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3875000000000005e-05 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355819+0000", 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015463900000000001 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355964+0000", 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144241 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.393080+0000", 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037116647000000003 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463398+0000", 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070317222999999998 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463456+0000", 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8749999999999998e-05 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463513+0000", 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6841e-05 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1192 3.36 3:6f9f445e:::100000001f8.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.981387+0000", 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2607939500000001, 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.081977499999999995, 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1192 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.981387+0000", 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.981387+0000", 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.981388+0000", 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7799999999999999e-06 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.051545+0000", 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070156394999999996 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.051574+0000", 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9626999999999999e-05 2024-09-16T01:51:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.051616+0000", 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1182e-05 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.051739+0000", 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012315899999999999 2024-09-16T01:51:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.051900+0000", 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016118299999999999 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.052066+0000", 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165947 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.063257+0000", 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011191548000000001 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.063270+0000", 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2067e-05 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.063321+0000", 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1127e-05 2024-09-16T01:51:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.063364+0000", 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3485000000000001e-05 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1218 3.3f 3:ffd44cf4:::100000001f8.000000bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.9618063009999993, 2024-09-16T01:51:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.121928352, 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1218 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280376+0000", 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8980000000000001e-06 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360384+0000", 2024-09-16T01:51:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080007935000000002 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360415+0000", 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1007999999999999e-05 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360429+0000", 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4233000000000001e-05 2024-09-16T01:51:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360518+0000", 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8465000000000001e-05 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360685+0000", 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016704299999999999 2024-09-16T01:51:49.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:49.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360842+0000", 2024-09-16T01:51:49.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157496 2024-09-16T01:51:49.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.379376+0000", 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018533701999999999 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402192+0000", 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022815932000000001 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402254+0000", 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2378000000000005e-05 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402303+0000", 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8262e-05 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1460 3.2a 3:574d3868:::100000001f8.000001b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.8558134669999999, 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114279699, 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1460 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386369+0000", 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1969999999999999e-06 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447453+0000", 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061083239999999997 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447479+0000", 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5941000000000001e-05 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447490+0000", 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1967000000000001e-05 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447627+0000", 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013671700000000001 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447784+0000", 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015693 2024-09-16T01:51:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447948+0000", 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016412100000000001 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.467094+0000", 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019145590000000001 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.497455+0000", 2024-09-16T01:51:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030361091 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.499539+0000", 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020837780000000001 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.500647+0000", 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108127 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1474 3.39 3:9e943776:::100000001f8.000001bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6605000199999997, 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.116146155, 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1474 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581683+0000", 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6740000000000001e-06 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639340+0000", 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057656583999999997 2024-09-16T01:51:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639448+0000", 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107707 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639499+0000", 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1391999999999999e-05 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648061+0000", 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085624870000000006 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648374+0000", 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000312307 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648681+0000", 2024-09-16T01:51:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030683700000000002 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.693784+0000", 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045103740000000003 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697710+0000", 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039253650000000001 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697780+0000", 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0134000000000003e-05 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697827+0000", 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6928000000000002e-05 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1480 3.2a 3:55ceaa15:::100000001f8.000001c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.639647+0000", 2024-09-16T01:51:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.6025334249999998, 2024-09-16T01:51:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067731026999999999, 2024-09-16T01:51:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1480 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639647+0000", 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639647+0000", 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639649+0000", 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4810000000000001e-06 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.660369+0000", 2024-09-16T01:51:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020720431000000001 2024-09-16T01:51:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.660389+0000", 2024-09-16T01:51:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9874999999999999e-05 2024-09-16T01:51:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.660400+0000", 2024-09-16T01:51:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.092e-05 2024-09-16T01:51:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.660496+0000", 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6158999999999999e-05 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.660693+0000", 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019704999999999999 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.660856+0000", 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016307100000000001 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.703630+0000", 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042773698999999998 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.707253+0000", 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0036230949999999998 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.707320+0000", 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6682999999999998e-05 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.707378+0000", 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8563000000000002e-05 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1503 3.2a 3:54fb97d1:::100000001f8.000001dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.890714+0000", 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.3514667420000004, 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076412117000000002, 2024-09-16T01:51:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1503 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.890714+0000", 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.890714+0000", 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.890716+0000", 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.368e-06 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.947359+0000", 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056642672999999998 2024-09-16T01:51:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.947373+0000", 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3842000000000001e-05 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.947380+0000", 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.627e-06 2024-09-16T01:51:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.947470+0000", 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0037000000000005e-05 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.947649+0000", 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178889 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.947821+0000", 2024-09-16T01:51:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017131899999999999 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.965991+0000", 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018170154000000001 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.967013+0000", 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001022413 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.967077+0000", 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3672999999999998e-05 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.967126+0000", 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9122000000000002e-05 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1525 3.f 3:f35be96e:::100000001f8.000001f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:44.213629+0000", 2024-09-16T01:51:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0285514960000004, 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.069802838000000006, 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1525 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.213629+0000", 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.213629+0000", 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.213632+0000", 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1099999999999999e-06 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.239087+0000", 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025454843000000001 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.239119+0000", 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2231999999999999e-05 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.239133+0000", 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3256e-05 2024-09-16T01:51:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.239230+0000", 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7002000000000004e-05 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.239393+0000", 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016327599999999999 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.239559+0000", 2024-09-16T01:51:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016619599999999999 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.252739+0000", 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013180018999999999 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.283325+0000", 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030585919999999999 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.283386+0000", 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.084e-05 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.283432+0000", 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6143999999999997e-05 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1558 3.2a 3:572f240d:::100000001f8.00000213:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:44.551475+0000", 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.6907057830000003, 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079255174999999997, 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1558 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.551475+0000", 2024-09-16T01:51:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.551475+0000", 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.551476+0000", 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.519e-06 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.595726+0000", 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044249860000000002 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.595750+0000", 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3567999999999999e-05 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.595761+0000", 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1209999999999999e-05 2024-09-16T01:51:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.595862+0000", 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101519 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.596012+0000", 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149698 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.596174+0000", 2024-09-16T01:51:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161702 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.625562+0000", 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029388153 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.630607+0000", 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050451419999999999 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.630679+0000", 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1851e-05 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.630730+0000", 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0952999999999998e-05 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1576 3.2a 3:54489c98:::100000001f8.00000225:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:44.770216+0000", 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.4719643109999998, 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072846157999999994, 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1576 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.770216+0000", 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.770216+0000", 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.770218+0000", 2024-09-16T01:51:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6729999999999999e-06 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.811733+0000", 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041515294000000001 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.811765+0000", 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1341999999999999e-05 2024-09-16T01:51:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.811777+0000", 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2442e-05 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.813102+0000", 2024-09-16T01:51:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0013249169999999999 2024-09-16T01:51:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.813278+0000", 2024-09-16T01:51:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017616600000000001 2024-09-16T01:51:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.813449+0000", 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000170848 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.827677+0000", 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01422753 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.842944+0000", 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015267698999999999 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.843011+0000", 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7238999999999998e-05 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:44.843062+0000", 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1007999999999997e-05 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1921 3.35 3:aefe9194:::100000001f8.0000037e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.857822+0000", 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.38435829199999999, 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.086432075999999997, 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1921 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.857822+0000", 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.857822+0000", 2024-09-16T01:51:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.857825+0000", 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9890000000000001e-06 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.879958+0000", 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022132723 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.879982+0000", 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3952e-05 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.879995+0000", 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3519999999999999e-05 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880085+0000", 2024-09-16T01:51:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9326999999999994e-05 2024-09-16T01:51:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880252+0000", 2024-09-16T01:51:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016694999999999999 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880418+0000", 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016624700000000001 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.891303+0000", 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010885137 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.944151+0000", 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052848322000000003 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.944208+0000", 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6360999999999998e-05 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.944254+0000", 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6548000000000002e-05 2024-09-16T01:51:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:49.387 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:49.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:49.388 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:49.375+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:49.478 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:49.477+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:49.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.522+0000 7f55aba00640 1 Processor -- start 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.523+0000 7f55aba00640 1 -- start start 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.525+0000 7f55aba00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55a4168200 0x7f55a40c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.525+0000 7f55aba00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55a40c7430 0x7f55a40c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.525+0000 7f55aba00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a416ad60 con 0x7f55a4164d70 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.525+0000 7f55aba00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f55a416b040 con 0x7f55a40c7430 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.525+0000 7f55aba00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f55a416b320 con 0x7f55a4168200 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.530+0000 7f55a3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55a4168200 0x7f55a40c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.530+0000 7f55a3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55a4168200 0x7f55a40c8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54624/0 (socket says 172.21.15.146:54624) 2024-09-16T01:51:49.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.530+0000 7f55a3e00640 1 -- 172.21.15.146:0/1509270855 learned_addr learned my addr 172.21.15.146:0/1509270855 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.538+0000 7f55a3e00640 1 -- 172.21.15.146:0/1509270855 >> v2:172.21.15.146:3300/0 conn(0x7f55a40c7430 msgr2=0x7f55a40c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.531+0000 7f55a3400640 1 -- 172.21.15.146:0/1509270855 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277330046 0 0) 0x7f55a416ad60 con 0x7f55a4164d70 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.534+0000 7f55a9a00640 1 --2- 172.21.15.146:0/1509270855 >> v2:172.21.15.146:3300/0 conn(0x7f55a40c7430 0x7f55a40c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.539+0000 7f55a3e00640 1 --2- 172.21.15.146:0/1509270855 >> v2:172.21.15.146:3300/0 conn(0x7f55a40c7430 0x7f55a40c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.539+0000 7f55a3e00640 1 -- 172.21.15.146:0/1509270855 >> v1:172.21.15.161:6789/0 conn(0x7f55a4164d70 legacy=0x7f55a4167af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.539+0000 7f55a3e00640 1 -- 172.21.15.146:0/1509270855 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a416b600 con 0x7f55a4168200 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.540+0000 7f55a9a00640 1 --2- 172.21.15.146:0/1509270855 >> v2:172.21.15.146:3300/0 conn(0x7f55a40c7430 0x7f55a40c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:49.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.541+0000 7f55a3e00640 1 --2- 172.21.15.146:0/1509270855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a4168200 0x7f55a40c8ad0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f55a40c6870 tx=0x7f559002a520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=807b5a6542ec1f87 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:49.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.542+0000 7f55a3400640 1 -- 172.21.15.146:0/1509270855 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5590036070 con 0x7f55a4168200 2024-09-16T01:51:49.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.542+0000 7f55a3400640 1 -- 172.21.15.146:0/1509270855 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f559000ae50 con 0x7f55a4168200 2024-09-16T01:51:49.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.542+0000 7f55a3400640 1 -- 172.21.15.146:0/1509270855 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559000a760 con 0x7f55a4168200 2024-09-16T01:51:49.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.544+0000 7f55aba00640 1 -- 172.21.15.146:0/1509270855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a4168200 msgr2=0x7f55a40c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.544+0000 7f55aba00640 1 --2- 172.21.15.146:0/1509270855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a4168200 0x7f55a40c8ad0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f55a40c6870 tx=0x7f559002a520 comp rx=0 tx=0).stop 2024-09-16T01:51:49.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.545+0000 7f55aba00640 1 -- 172.21.15.146:0/1509270855 shutdown_connections 2024-09-16T01:51:49.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.545+0000 7f55aba00640 1 --2- 172.21.15.146:0/1509270855 >> v2:172.21.15.146:3300/0 conn(0x7f55a40c7430 0x7f55a40c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.545+0000 7f55aba00640 1 --2- 172.21.15.146:0/1509270855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a4168200 0x7f55a40c8ad0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.545+0000 7f55aba00640 1 -- 172.21.15.146:0/1509270855 >> 172.21.15.146:0/1509270855 conn(0x7f55a406f3c0 msgr2=0x7f55a406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:49.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.545+0000 7f55aba00640 1 -- 172.21.15.146:0/1509270855 shutdown_connections 2024-09-16T01:51:49.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.545+0000 7f55aba00640 1 -- 172.21.15.146:0/1509270855 wait complete. 2024-09-16T01:51:49.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.551+0000 7f55aba00640 1 Processor -- start 2024-09-16T01:51:49.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.551+0000 7f55aba00640 1 -- start start 2024-09-16T01:51:49.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.553+0000 7f55aba00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a40c7430 0x7f55a40cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.553+0000 7f55aba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 0x7f55a40c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.554+0000 7f55aba00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55a4168200 0x7f55a4163690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.554+0000 7f55a3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 0x7f55a40c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.554+0000 7f55a3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 0x7f55a40c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49694/0 (socket says 172.21.15.146:49694) 2024-09-16T01:51:49.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55a9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a40c7430 0x7f55a40cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55a9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a40c7430 0x7f55a40cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54634/0 (socket says 172.21.15.146:54634) 2024-09-16T01:51:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55aba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f55a40c6bb0 con 0x7f55a4164d70 2024-09-16T01:51:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55aba00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f55a40c3970 con 0x7f55a40c7430 2024-09-16T01:51:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55aba00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f55a40718c0 con 0x7f55a4168200 2024-09-16T01:51:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55a3e00640 1 -- 172.21.15.146:0/725352213 learned_addr learned my addr 172.21.15.146:0/725352213 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:49.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.555+0000 7f55a9a00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55a4168200 0x7f55a4163690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.556+0000 7f55a9a00640 1 -- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a40c7430 msgr2=0x7f55a40cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.556+0000 7f55a9a00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a40c7430 0x7f55a40cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.556+0000 7f55a9a00640 1 -- 172.21.15.146:0/725352213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 msgr2=0x7f55a40c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:49.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.557+0000 7f55a9a00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 0x7f55a40c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:49.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.557+0000 7f55a9a00640 1 -- 172.21.15.146:0/725352213 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a41fe8a0 con 0x7f55a4168200 2024-09-16T01:51:49.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.557+0000 7f55a3e00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 0x7f55a40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:49.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.558+0000 7f55a9a00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55a4168200 0x7f55a4163690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f559c014410 tx=0x7f559c0148d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:49.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.558+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c01e020 con 0x7f55a4168200 2024-09-16T01:51:49.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.558+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f559c01a850 con 0x7f55a4168200 2024-09-16T01:51:49.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.558+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c01ac40 con 0x7f55a4168200 2024-09-16T01:51:49.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.560+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a41fda40 con 0x7f55a4168200 2024-09-16T01:51:49.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.560+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a41fe000 con 0x7f55a4168200 2024-09-16T01:51:49.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.562+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a40500d0 con 0x7f55a4168200 2024-09-16T01:51:49.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.567+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f559c021020 con 0x7f55a4168200 2024-09-16T01:51:49.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.569+0000 7f55a0c00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f558406c2b0 0x7f558406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.569+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(84..84 src has 1..84) ==== 5795+0+0 (secure 0 0 0) 0x7f559c094470 con 0x7f55a4168200 2024-09-16T01:51:49.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.570+0000 7f55a9000640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f558406c2b0 0x7f558406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:49.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.571+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f55a40500d0 con 0x7f55a4168200 2024-09-16T01:51:49.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:49.573+0000 7f55a9000640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f558406c2b0 0x7f558406e770 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5598000f60 tx=0x7f5598009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:49.579 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:49.578+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:49.686 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:49.683+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:49.791 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:49.789+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:49.850 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:49.850 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:49.850 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:49.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:49.881 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:49.880+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:49.982 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:49.981+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:50.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.071+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f55a4165150 con 0x7f55a4168200 2024-09-16T01:51:50.083 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:50.083+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:50.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.158+0000 7f55a0c00640 1 -- 172.21.15.146:0/725352213 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v85) ==== 101+0+0 (secure 0 0 0) 0x7f559c028020 con 0x7f55a4168200 2024-09-16T01:51:50.161 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:51:50.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.169+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f558406c2b0 msgr2=0x7f558406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:50.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.169+0000 7f55aba00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f558406c2b0 0x7f558406e770 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5598000f60 tx=0x7f5598009040 comp rx=0 tx=0).stop 2024-09-16T01:51:50.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55a4168200 msgr2=0x7f55a4163690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55a4168200 0x7f55a4163690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f559c014410 tx=0x7f559c0148d0 comp rx=0 tx=0).stop 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 shutdown_connections 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f558406c2b0 0x7f558406e770 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55a4168200 0x7f55a4163690 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55a4164d70 0x7f55a40c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 --2- 172.21.15.146:0/725352213 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55a40c7430 0x7f55a40cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 >> 172.21.15.146:0/725352213 conn(0x7f55a406f3c0 msgr2=0x7f55a4162040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 shutdown_connections 2024-09-16T01:51:50.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:50.170+0000 7f55aba00640 1 -- 172.21.15.146:0/725352213 wait complete. 2024-09-16T01:51:50.184 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:50.183+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:50.274 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:50.274 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:50.275 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:50.275 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:50.287 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:50.283+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:50.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:50.386 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:50.384+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:50.486 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:50.485+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:50.591 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:50.588+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:50.688 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:50.687+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:51:50.698 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 182.92424304100001, 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:51:50.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:51:50.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:50.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:51:50.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:51:50.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:51:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:51:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:51:50.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.306790700999997, 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:51:50.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:51:50.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:51:50.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:50.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:524 3.19 3:9adb8721:::100000001f7.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.213728089, 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092555282000000003, 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.718 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 524 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478869+0000", 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.478871+0000", 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0700000000000001e-06 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538563+0000", 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059691821999999999 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538594+0000", 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1285000000000003e-05 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538606+0000", 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1688999999999999e-05 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538706+0000", 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100752 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.538872+0000", 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016575699999999999 2024-09-16T01:51:50.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.539058+0000", 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018612500000000001 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.547102+0000", 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080436529999999996 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571308+0000", 2024-09-16T01:51:50.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024205946999999998 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571375+0000", 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6623000000000002e-05 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.571424+0000", 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9558999999999998e-05 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:541 3.1c 3:387f027e:::100000001f7.0000001a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.071776866999997, 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084290084000000001, 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 541 2024-09-16T01:51:50.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620820+0000", 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.620823+0000", 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8169999999999999e-06 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670392+0000", 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049568677999999998 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670432+0000", 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0312999999999998e-05 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670448+0000", 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5781e-05 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670540+0000", 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1920000000000001e-05 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670705+0000", 2024-09-16T01:51:50.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016490299999999999 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.670881+0000", 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017675899999999999 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.683357+0000", 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012476179 2024-09-16T01:51:50.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.704990+0000", 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021632195 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.705053+0000", 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3291000000000001e-05 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.705110+0000", 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7247999999999998e-05 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:50.730 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.514773752, 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.731 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:51:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:51:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:51:50.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:51:50.737 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.479241128999998, 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.738 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:51:50.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:51:50.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1147 3.19 3:9848b992:::100000001f8.00000078:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.511489+0000", 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.181107883999999, 2024-09-16T01:51:50.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084795828000000004, 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1147 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.511489+0000", 2024-09-16T01:51:50.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.511489+0000", 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.511491+0000", 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3630000000000001e-06 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.560928+0000", 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049436969999999997 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.560961+0000", 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2478000000000003e-05 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.560972+0000", 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0634e-05 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.561207+0000", 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023557499999999999 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.561352+0000", 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144974 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.561505+0000", 2024-09-16T01:51:50.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000152437 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.583038+0000", 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021533845999999999 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.596082+0000", 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013043974 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.596220+0000", 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000137947 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.596285+0000", 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4629999999999996e-05 2024-09-16T01:51:50.750 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1461 3.1c 3:39a1ca69:::100000001f8.000001b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.2819003340000004, 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105254967, 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.751 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1461 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410699+0000", 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.232e-06 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476134+0000", 2024-09-16T01:51:50.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065435190000000004 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476173+0000", 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8878999999999998e-05 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476186+0000", 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3016e-05 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476284+0000", 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7583000000000004e-05 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476440+0000", 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015621799999999999 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476629+0000", 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018937399999999999 2024-09-16T01:51:50.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.505615+0000", 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028986212000000001 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515833+0000", 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010217944 2024-09-16T01:51:50.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515901+0000", 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7250999999999997e-05 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515952+0000", 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1068e-05 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.1131521920000003, 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.758 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:51:50.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:51:50.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1492 3.37 3:ec778d4e:::100000001f8.000001d1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.799860+0000", 2024-09-16T01:51:50.764 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.8927367730000002, 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083667876000000002, 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1492 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.799860+0000", 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.799860+0000", 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.799862+0000", 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0619999999999999e-06 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.823326+0000", 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023463984 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.823372+0000", 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5697e-05 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.823387+0000", 2024-09-16T01:51:50.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4608e-05 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.823800+0000", 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00041323500000000002 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.824141+0000", 2024-09-16T01:51:50.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034073799999999999 2024-09-16T01:51:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.824603+0000", 2024-09-16T01:51:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000462433 2024-09-16T01:51:50.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.845223+0000", 2024-09-16T01:51:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020620013 2024-09-16T01:51:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.883429+0000", 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038205910000000003 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.883488+0000", 2024-09-16T01:51:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8826000000000002e-05 2024-09-16T01:51:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.883528+0000", 2024-09-16T01:51:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0370000000000001e-05 2024-09-16T01:51:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1634 3.19 3:9a92d45b:::100000001f8.0000025f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:45.506649+0000", 2024-09-16T01:51:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.1859481120000002, 2024-09-16T01:51:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084347031000000003, 2024-09-16T01:51:50.774 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1634 2024-09-16T01:51:50.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.506649+0000", 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.506649+0000", 2024-09-16T01:51:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.506651+0000", 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3159999999999999e-06 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.521246+0000", 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014594968999999999 2024-09-16T01:51:50.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.521279+0000", 2024-09-16T01:51:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3009000000000002e-05 2024-09-16T01:51:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.521293+0000", 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4083000000000001e-05 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.521508+0000", 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000214792 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.521680+0000", 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017178199999999999 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.521840+0000", 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015971299999999999 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.563873+0000", 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042033145000000001 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.590849+0000", 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026976472000000001 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.590907+0000", 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7626e-05 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.590996+0000", 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9123999999999996e-05 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.5489406539999999, 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:51:50.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:51:50.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.487153352, 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:51:50.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:51:50.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:51:50.794 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1931 3.39 e84/83 3:9d369e22:::100000001f8.00000388:head v 84'56, mlcod=0'0)", 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4645336959999999, 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110959082, 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228064+0000", 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5499999999999997e-07 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229687+0000", 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00162302 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229706+0000", 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8459999999999999e-05 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229719+0000", 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2986000000000001e-05 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229824+0000", 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105163 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229850+0000", 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6621000000000001e-05 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.338768+0000", 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108917369 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339022+0000", 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025460800000000003 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.799 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1932 3.39 e84/83 3:9da8c7a0:::100000001f8.00000389:head v 84'57, mlcod=0'0)", 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4575517549999999, 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10422508900000001, 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235046+0000", 2024-09-16T01:51:50.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0589999999999999e-06 2024-09-16T01:51:50.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245188+0000", 2024-09-16T01:51:50.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010141739 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245228+0000", 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9755999999999997e-05 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245244+0000", 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6475000000000001e-05 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.249380+0000", 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041353170000000003 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.249407+0000", 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7586e-05 2024-09-16T01:51:50.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339027+0000", 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089619521999999993 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339270+0000", 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024363499999999999 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1938 3.39 e84/83 3:9faf1682:::100000001f8.0000038f:head v 84'58, mlcod=0'0)", 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.44730694, 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.101947118, 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245292+0000", 2024-09-16T01:51:50.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5030000000000001e-06 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253345+0000", 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080530840000000003 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253363+0000", 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8281999999999999e-05 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253379+0000", 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6252999999999998e-05 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254278+0000", 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089897700000000002 2024-09-16T01:51:50.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254304+0000", 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5559e-05 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347185+0000", 2024-09-16T01:51:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092881693000000001 2024-09-16T01:51:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347237+0000", 2024-09-16T01:51:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1767000000000001e-05 2024-09-16T01:51:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1947 3.2c 3:3557c0c2:::100000001f8.00000398:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.258704+0000", 2024-09-16T01:51:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.433893372, 2024-09-16T01:51:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.088759880999999999, 2024-09-16T01:51:50.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1947 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.258704+0000", 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.258704+0000", 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.258705+0000", 2024-09-16T01:51:50.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.745e-06 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.266690+0000", 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0079851220000000007 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.266719+0000", 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8493999999999999e-05 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.266731+0000", 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1790999999999999e-05 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.267734+0000", 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001002775 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.267878+0000", 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143993 2024-09-16T01:51:50.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268045+0000", 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016779800000000001 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.323782+0000", 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055736815000000002 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347384+0000", 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023602049 2024-09-16T01:51:50.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347430+0000", 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5346000000000003e-05 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347463+0000", 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3952999999999998e-05 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1951 3.a e84/20 3:53c75786:::100000001f8.0000039c:head v 84'32, mlcod=81'30)", 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.271209+0000", 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.4213884079999999, 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083797774000000005, 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.271209+0000", 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.271209+0000", 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.271209+0000", 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0499999999999999e-07 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.279721+0000", 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085118079999999992 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.279752+0000", 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0756000000000002e-05 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.279765+0000", 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.31e-05 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.279923+0000", 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158269 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.279951+0000", 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8138000000000001e-05 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.354911+0000", 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074959632999999998 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355006+0000", 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5665e-05 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1963 3.19 3:990e9bf2:::100000001f8.000003a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.450671+0000", 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.241926036, 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095157485999999999, 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.824 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1963 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.450671+0000", 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.450671+0000", 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.450673+0000", 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2280000000000001e-06 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491124+0000", 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040450429000000003 2024-09-16T01:51:50.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491156+0000", 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2715000000000001e-05 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491170+0000", 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3899e-05 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491383+0000", 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021269099999999999 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491542+0000", 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015885699999999999 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:50.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491709+0000", 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166825 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.541315+0000", 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049606712999999997 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.542828+0000", 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015124610000000001 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.542889+0000", 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1555000000000003e-05 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.545828+0000", 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029391130000000001 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.64537090600000002, 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:50.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:50.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:50.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:50.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:51:50.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:50.837 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:50.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:50.839 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:50.837+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:50.939 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:50.938+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:51.040 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:51.039+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:51.141 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:51.140+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:51.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:2085 3.29 e85/20 3:95bdb1e0:::100000001f8.00000422:head v 85'43, mlcod=85'41)", 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.198671+0000", 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.027259928999999999, 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027288494999999999, 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:51:51.227 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.198671+0000", 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.198671+0000", 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:51.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.198672+0000", 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2600000000000001e-07 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.202523+0000", 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038514410000000002 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.202550+0000", 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7242000000000001e-05 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.202562+0000", 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1620000000000001e-05 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.202698+0000", 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013639199999999999 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.202726+0000", 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7239000000000001e-05 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:51.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:51.231 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:51.231 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:51.231 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:51:51.231 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:51:51.231 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:51.245 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:51.243+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:51.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:51.366 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:51.364+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:51.443 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:51.442+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:51.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.510+0000 7f427e200640 1 Processor -- start 2024-09-16T01:51:51.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.510+0000 7f427e200640 1 -- start start 2024-09-16T01:51:51.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f427e200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42780c7a10 0x7f4278150d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f427e200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427806ed10 con 0x7f4278071a10 2024-09-16T01:51:51.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f427e200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f427806eff0 con 0x7f4278150930 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f427e200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f42780725c0 con 0x7f42780c7a10 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f4277e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42780c7a10 0x7f4278150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f4277e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42780c7a10 0x7f4278150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54640/0 (socket says 172.21.15.146:54640) 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.512+0000 7f4277e00640 1 -- 172.21.15.146:0/1921124747 learned_addr learned my addr 172.21.15.146:0/1921124747 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.513+0000 7f4277e00640 1 -- 172.21.15.146:0/1921124747 >> v1:172.21.15.161:6789/0 conn(0x7f4278150930 legacy=0x7f42780c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.513+0000 7f4277e00640 1 -- 172.21.15.146:0/1921124747 >> v1:172.21.15.146:6789/0 conn(0x7f4278071a10 legacy=0x7f4278073280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:51.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.513+0000 7f4277e00640 1 -- 172.21.15.146:0/1921124747 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427806f2d0 con 0x7f42780c7a10 2024-09-16T01:51:51.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.514+0000 7f4277e00640 1 --2- 172.21.15.146:0/1921124747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780c7a10 0x7f4278150d60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4268009870 tx=0x7f426802a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb804c79ba3c0b52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.515+0000 7f4276000640 1 -- 172.21.15.146:0/1921124747 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268037070 con 0x7f42780c7a10 2024-09-16T01:51:51.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.515+0000 7f4276000640 1 -- 172.21.15.146:0/1921124747 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4268032640 con 0x7f42780c7a10 2024-09-16T01:51:51.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.515+0000 7f4276000640 1 -- 172.21.15.146:0/1921124747 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268032a10 con 0x7f42780c7a10 2024-09-16T01:51:51.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.516+0000 7f427e200640 1 -- 172.21.15.146:0/1921124747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780c7a10 msgr2=0x7f4278150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.516+0000 7f427e200640 1 --2- 172.21.15.146:0/1921124747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780c7a10 0x7f4278150d60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4268009870 tx=0x7f426802a330 comp rx=0 tx=0).stop 2024-09-16T01:51:51.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.516+0000 7f427e200640 1 -- 172.21.15.146:0/1921124747 shutdown_connections 2024-09-16T01:51:51.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.516+0000 7f427e200640 1 --2- 172.21.15.146:0/1921124747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780c7a10 0x7f4278150d60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.516+0000 7f427e200640 1 -- 172.21.15.146:0/1921124747 >> 172.21.15.146:0/1921124747 conn(0x7f427806bc10 msgr2=0x7f427806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:51.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.517+0000 7f427e200640 1 -- 172.21.15.146:0/1921124747 shutdown_connections 2024-09-16T01:51:51.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.517+0000 7f427e200640 1 -- 172.21.15.146:0/1921124747 wait complete. 2024-09-16T01:51:51.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.520+0000 7f427e200640 1 Processor -- start 2024-09-16T01:51:51.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.520+0000 7f427e200640 1 -- start start 2024-09-16T01:51:51.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f427e200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 0x7f4278211b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f427e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42780c7a10 0x7f42782106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f427e200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4278150930 0x7f4278210c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f427e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4278159fe0 con 0x7f42780c7a10 2024-09-16T01:51:51.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f427e200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f427806f890 con 0x7f4278150930 2024-09-16T01:51:51.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f427e200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f427815a160 con 0x7f4278071a10 2024-09-16T01:51:51.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f4277400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 0x7f4278211b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f4277400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 0x7f4278211b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39626/0 (socket says 172.21.15.146:39626) 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.522+0000 7f4277400640 1 -- 172.21.15.146:0/214118865 learned_addr learned my addr 172.21.15.146:0/214118865 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42780c7a10 0x7f42782106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277400640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4278150930 msgr2=0x7f4278210c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4278150930 0x7f4278210c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4278150930 0x7f4278210c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277400640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42780c7a10 msgr2=0x7f42782106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42780c7a10 0x7f42782106c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277400640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4278212500 con 0x7f4278071a10 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4278150930 0x7f4278210c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42780c7a10 0x7f42782106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:51.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.523+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 0x7f4278211b60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f427000ba50 tx=0x7f427000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.524+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270019070 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.524+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427000cc20 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.524+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270014440 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.525+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42781493f0 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.525+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42780c32a0 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.529+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f42781587a0 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.530+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f427001c070 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.531+0000 7f426f400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425806c380 0x7f425806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.532+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(86..86 src has 1..86) ==== 5770+0+0 (secure 0 0 0) 0x7f427008b280 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.532+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v86) ==== 54+0+16 (secure 0 0 0) 0x7f42781587a0 con 0x7f4278071a10 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.532+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4278150310 0x7f427813be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.532+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42781491e0 con 0x7f4278150310 2024-09-16T01:51:51.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.538+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425806c380 0x7f425806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.539+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4278150310 0x7f427813be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.539+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425806c380 0x7f425806e840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f427806c5f0 tx=0x7f4264009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.545 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:51.543+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:51.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.543+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4278150310 0x7f427813be80 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.545+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42781491e0 con 0x7f4278150310 2024-09-16T01:51:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.582+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4278150d60 con 0x7f4278150310 2024-09-16T01:51:51.589 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:51.590 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:51.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.587+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4278150d60 con 0x7f4278150310 2024-09-16T01:51:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.594+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4278157300 0x7f4278142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.594+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f42782120a0 con 0x7f4278157300 2024-09-16T01:51:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.595+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4278157300 0x7f4278142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.597+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4278157300 0x7f4278142120 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.606+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42782120a0 con 0x7f4278157300 2024-09-16T01:51:51.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.631+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f42782122b0 con 0x7f4278157300 2024-09-16T01:51:51.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.634+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f42782122b0 con 0x7f4278157300 2024-09-16T01:51:51.636 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:51.637 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:51.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.634+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4278062630 0x7f4278062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.634+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4278147ed0 con 0x7f4278062630 2024-09-16T01:51:51.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.635+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4278062630 0x7f4278062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.635+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4278062630 0x7f4278062a30 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.637+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4278147ed0 con 0x7f4278062630 2024-09-16T01:51:51.645 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:51.643+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:51.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.660+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f42781483b0 con 0x7f4278062630 2024-09-16T01:51:51.662 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:51.662 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:51.662 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:51.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.677+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f42781483b0 con 0x7f4278062630 2024-09-16T01:51:51.680 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:51.680 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:51.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.679+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4278063330 0x7f4278147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.679+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f42781477a0 con 0x7f4278063330 2024-09-16T01:51:51.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.679+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4278063330 0x7f4278147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.680+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4278063330 0x7f4278147260 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:51.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.682+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42781477a0 con 0x7f4278063330 2024-09-16T01:51:51.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.702+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4278147cb0 con 0x7f4278063330 2024-09-16T01:51:51.715 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.706+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4278147cb0 con 0x7f4278063330 2024-09-16T01:51:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.706+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f427802ccc0 0x7f427802d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.706+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f427802d660 con 0x7f427802ccc0 2024-09-16T01:51:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.707+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f427802ccc0 0x7f427802d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.710+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f427802ccc0 0x7f427802d120 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.718+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f427802d660 con 0x7f427802ccc0 2024-09-16T01:51:51.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.739+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f427802d870 con 0x7f427802ccc0 2024-09-16T01:51:51.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.742+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f427802d870 con 0x7f427802ccc0 2024-09-16T01:51:51.744 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:51.744 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:51.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.742+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4278223390 0x7f4278225780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.742+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f427802db90 con 0x7f4278223390 2024-09-16T01:51:51.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.742+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4278223390 0x7f4278225780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.744+0000 7f4276a00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4278223390 0x7f4278225780 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.747 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:51.744+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:51.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.745+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42781483b0 con 0x7f4278223390 2024-09-16T01:51:51.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.768+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4278225cc0 con 0x7f4278223390 2024-09-16T01:51:51.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.770+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4278225cc0 con 0x7f4278223390 2024-09-16T01:51:51.773 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:51.773 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:51.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.771+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42782264f0 0x7f42782288e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.771+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f427822b6b0 con 0x7f42782264f0 2024-09-16T01:51:51.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.772+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42782264f0 0x7f42782288e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.776+0000 7f4277e00640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42782264f0 0x7f42782288e0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.777+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f427822b6b0 con 0x7f42782264f0 2024-09-16T01:51:51.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.792+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4278228e20 con 0x7f42782264f0 2024-09-16T01:51:51.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.794+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4278228e20 con 0x7f42782264f0 2024-09-16T01:51:51.797 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:51.797 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:51.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.795+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4278229540 0x7f42782299a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:51.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.795+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f427822e9f0 con 0x7f4278229540 2024-09-16T01:51:51.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.795+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4278229540 0x7f42782299a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:51.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.797+0000 7f4277400640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4278229540 0x7f42782299a0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:51.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.798+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f427822e9f0 con 0x7f4278229540 2024-09-16T01:51:51.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.827+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4278229ee0 con 0x7f4278229540 2024-09-16T01:51:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.831+0000 7f426f400640 1 -- 172.21.15.146:0/214118865 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4278229ee0 con 0x7f4278229540 2024-09-16T01:51:51.837 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:51.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4278150310 msgr2=0x7f427813be80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f4278150310 0x7f427813be80 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4278157300 msgr2=0x7f4278142120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4278157300 0x7f4278142120 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4278062630 msgr2=0x7f4278062a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f4278062630 0x7f4278062a30 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4278063330 msgr2=0x7f4278147260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f4278063330 0x7f4278147260 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f427802ccc0 msgr2=0x7f427802d120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f427802ccc0 0x7f427802d120 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4278223390 msgr2=0x7f4278225780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f4278223390 0x7f4278225780 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42782264f0 msgr2=0x7f42782288e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f42782264f0 0x7f42782288e0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4278229540 msgr2=0x7f42782299a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4278229540 0x7f42782299a0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f4277e00640 1 -- 172.21.15.146:0/214118865 reap_dead start 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f4277e00640 1 -- 172.21.15.146:0/214118865 reap_dead start 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f4277e00640 1 -- 172.21.15.146:0/214118865 reap_dead start 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f4277e00640 1 -- 172.21.15.146:0/214118865 reap_dead start 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.836+0000 7f4277e00640 1 -- 172.21.15.146:0/214118865 reap_dead start 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425806c380 msgr2=0x7f425806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425806c380 0x7f425806e840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f427806c5f0 tx=0x7f4264009040 comp rx=0 tx=0).stop 2024-09-16T01:51:51.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 msgr2=0x7f4278211b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 0x7f4278211b60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f427000ba50 tx=0x7f427000bf10 comp rx=0 tx=0).stop 2024-09-16T01:51:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 shutdown_connections 2024-09-16T01:51:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f425806c380 0x7f425806e840 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f427806c5f0 tx=0x7f4264009040 comp rx=0 tx=0).stop 2024-09-16T01:51:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 --2- 172.21.15.146:0/214118865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4278071a10 0x7f4278211b60 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f427000ba50 tx=0x7f427000bf10 comp rx=0 tx=0).stop 2024-09-16T01:51:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.837+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 >> 172.21.15.146:0/214118865 conn(0x7f427806bc10 msgr2=0x7f427815a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:51.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.838+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 shutdown_connections 2024-09-16T01:51:51.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:51.838+0000 7f427e200640 1 -- 172.21.15.146:0/214118865 wait complete. 2024-09-16T01:51:51.846 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:51.844+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:51.947 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:51.946+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:51.993 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:51.994 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:52.016 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:52.048 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:52.047+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:52.149 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:52.148+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:52.207 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:51:52.249 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:52.248+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:52.340 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:52.340 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:52.353 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:52.351+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:52.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:52.452 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:52.450+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:52.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.537+0000 7f3355600640 1 Processor -- start 2024-09-16T01:51:52.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.538+0000 7f3355600640 1 -- start start 2024-09-16T01:51:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.540+0000 7f3355600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33480d8bb0 0x7f334804ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.540+0000 7f3355600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f334804f870 0x7f3348005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.540+0000 7f3355600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3348050520 con 0x7f3348007340 2024-09-16T01:51:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.540+0000 7f3355600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3348050800 con 0x7f33480d8bb0 2024-09-16T01:51:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.540+0000 7f3355600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f334804f070 con 0x7f334804f870 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.541+0000 7f3354c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f334804f870 0x7f3348005d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.541+0000 7f334fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33480d8bb0 0x7f334804ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.541+0000 7f334f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3348007340 0x7f334800e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:55188/0 (socket says 172.21.15.146:55188) 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.541+0000 7f334f400640 1 -- 172.21.15.146:0/3097944662 learned_addr learned my addr 172.21.15.146:0/3097944662 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.542+0000 7f334fe00640 1 -- 172.21.15.146:0/3097944662 >> v2:172.21.15.161:3300/0 conn(0x7f334804f870 msgr2=0x7f3348005d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.542+0000 7f334fe00640 1 --2- 172.21.15.146:0/3097944662 >> v2:172.21.15.161:3300/0 conn(0x7f334804f870 0x7f3348005d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.542+0000 7f334fe00640 1 -- 172.21.15.146:0/3097944662 >> v1:172.21.15.146:6789/0 conn(0x7f3348007340 legacy=0x7f334800e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.543+0000 7f334fe00640 1 -- 172.21.15.146:0/3097944662 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334804f350 con 0x7f33480d8bb0 2024-09-16T01:51:52.554 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:52.552+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:52.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.552+0000 7f334fe00640 1 --2- 172.21.15.146:0/3097944662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804ea00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3348051f30 tx=0x7f334402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b679cfe114a0b3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:52.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.553+0000 7f334ea00640 1 -- 172.21.15.146:0/3097944662 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344037070 con 0x7f33480d8bb0 2024-09-16T01:51:52.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.553+0000 7f334ea00640 1 -- 172.21.15.146:0/3097944662 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3344032440 con 0x7f33480d8bb0 2024-09-16T01:51:52.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.553+0000 7f334ea00640 1 -- 172.21.15.146:0/3097944662 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344037070 con 0x7f33480d8bb0 2024-09-16T01:51:52.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.555+0000 7f3355600640 1 -- 172.21.15.146:0/3097944662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 msgr2=0x7f334804ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:52.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.555+0000 7f3355600640 1 --2- 172.21.15.146:0/3097944662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804ea00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3348051f30 tx=0x7f334402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:51:52.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.558+0000 7f3355600640 1 -- 172.21.15.146:0/3097944662 shutdown_connections 2024-09-16T01:51:52.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.558+0000 7f3355600640 1 --2- 172.21.15.146:0/3097944662 >> v2:172.21.15.161:3300/0 conn(0x7f334804f870 0x7f3348005d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:52.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.558+0000 7f3355600640 1 --2- 172.21.15.146:0/3097944662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804ea00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:52.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.558+0000 7f3355600640 1 -- 172.21.15.146:0/3097944662 >> 172.21.15.146:0/3097944662 conn(0x7f3348054470 msgr2=0x7f3348054880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:52.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.558+0000 7f3355600640 1 -- 172.21.15.146:0/3097944662 shutdown_connections 2024-09-16T01:51:52.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.558+0000 7f3355600640 1 -- 172.21.15.146:0/3097944662 wait complete. 2024-09-16T01:51:52.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.561+0000 7f3355600640 1 Processor -- start 2024-09-16T01:51:52.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.561+0000 7f3355600640 1 -- start start 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.563+0000 7f3355600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 0x7f33480d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.563+0000 7f3355600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f334804f870 0x7f33480d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.563+0000 7f3355600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.563+0000 7f334fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 0x7f33480d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.563+0000 7f334fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 0x7f33480d3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54668/0 (socket says 172.21.15.146:54668) 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f334f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f334804f870 0x7f33480d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f3354c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f3354c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49734/0 (socket says 172.21.15.146:49734) 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f334f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f334804f870 0x7f33480d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39636/0 (socket says 172.21.15.146:39636) 2024-09-16T01:51:52.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f3355600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3348049340 con 0x7f33480d8bb0 2024-09-16T01:51:52.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f3355600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f33480491c0 con 0x7f3348007340 2024-09-16T01:51:52.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f3355600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f334800eae0 con 0x7f334804f870 2024-09-16T01:51:52.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f334fe00640 1 -- 172.21.15.146:0/2541882030 learned_addr learned my addr 172.21.15.146:0/2541882030 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:52.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f334fe00640 1 -- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f334804f870 msgr2=0x7f33480d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:52.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f334fe00640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f334804f870 0x7f33480d03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:52.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.564+0000 7f334fe00640 1 -- 172.21.15.146:0/2541882030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 msgr2=0x7f334804b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:52.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.565+0000 7f334fe00640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:52.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.565+0000 7f334fe00640 1 -- 172.21.15.146:0/2541882030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334817c1e0 con 0x7f3348007340 2024-09-16T01:51:52.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.565+0000 7f3354c00640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:52.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.566+0000 7f334fe00640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 0x7f33480d3d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3344035040 tx=0x7f33440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:52.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.567+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344037040 con 0x7f3348007340 2024-09-16T01:51:52.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.567+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33440325f0 con 0x7f3348007340 2024-09-16T01:51:52.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.567+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33440329c0 con 0x7f3348007340 2024-09-16T01:51:52.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.569+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334804d710 con 0x7f3348007340 2024-09-16T01:51:52.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.569+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f334804dcb0 con 0x7f3348007340 2024-09-16T01:51:52.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.573+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33480551d0 con 0x7f3348007340 2024-09-16T01:51:52.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.576+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3344044020 con 0x7f3348007340 2024-09-16T01:51:52.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.578+0000 7f3343e00640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f333006c380 0x7f333006e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:52.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.578+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5770+0+0 (secure 0 0 0) 0x7f33440ab970 con 0x7f3348007340 2024-09-16T01:51:52.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.580+0000 7f334f400640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f333006c380 0x7f333006e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:52.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.582+0000 7f334f400640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f333006c380 0x7f333006e840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33480d7f60 tx=0x7f333c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:52.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.583+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f334817c1e0 con 0x7f3348007340 2024-09-16T01:51:52.657 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:52.652+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:52.718 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:52.718 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:52.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:52.776 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:52.774+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:52.856 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:52.854+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:52.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:52.933+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f33480df570 con 0x7f3348007340 2024-09-16T01:51:52.963 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:52.961+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:53.057 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:53.056+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:53.122 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:53.123 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:53.139 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:53.182 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:53.180+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:53.188 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:51:53.188 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:51:53.189 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T01:51:53.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-16T01:51:53.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.230+0000 7f3343e00640 1 -- 172.21.15.146:0/2541882030 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v88) ==== 87+0+0 (secure 0 0 0) 0x7f3344079ac0 con 0x7f3348007340 2024-09-16T01:51:53.232 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:51:53.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.234+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f333006c380 msgr2=0x7f333006e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.234+0000 7f3355600640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f333006c380 0x7f333006e840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33480d7f60 tx=0x7f333c009040 comp rx=0 tx=0).stop 2024-09-16T01:51:53.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.234+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 msgr2=0x7f33480d3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.234+0000 7f3355600640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 0x7f33480d3d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3344035040 tx=0x7f33440046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.236+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 shutdown_connections 2024-09-16T01:51:53.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.236+0000 7f3355600640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f333006c380 0x7f333006e840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.236+0000 7f3355600640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33480d8bb0 0x7f334804b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.236+0000 7f3355600640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f334804f870 0x7f33480d03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.236+0000 7f3355600640 1 --2- 172.21.15.146:0/2541882030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3348007340 0x7f33480d3d40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.236+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 >> 172.21.15.146:0/2541882030 conn(0x7f3348054470 msgr2=0x7f3348005240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:53.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.241+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 shutdown_connections 2024-09-16T01:51:53.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.241+0000 7f3355600640 1 -- 172.21.15.146:0/2541882030 wait complete. 2024-09-16T01:51:53.261 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:53.258+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:53.266 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:53.267 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:53.276 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:53.360 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:53.359+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:53.402 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:53.402 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:53.411 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:53.495 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:53.494+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:53.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.498+0000 7fd055c00640 1 Processor -- start 2024-09-16T01:51:53.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.498+0000 7fd055c00640 1 -- start start 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.499+0000 7fd055c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0500c75a0 0x7fd05015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.499+0000 7fd055c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0500cc040 0x7fd0500cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.499+0000 7fd055c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd05006d140 con 0x7fd0500c6b70 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.499+0000 7fd055c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd05015b880 con 0x7fd0500cc040 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.499+0000 7fd055c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd0500cab20 con 0x7fd0500c75a0 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0500c75a0 0x7fd05015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0500c75a0 0x7fd05015b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50432/0 (socket says 172.21.15.146:50432) 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 -- 172.21.15.146:0/116371205 learned_addr learned my addr 172.21.15.146:0/116371205 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:53.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd055200640 1 --2- 172.21.15.146:0/116371205 >> v2:172.21.15.146:3300/0 conn(0x7fd0500cc040 0x7fd0500cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 -- 172.21.15.146:0/116371205 >> v2:172.21.15.146:3300/0 conn(0x7fd0500cc040 msgr2=0x7fd0500cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:53.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 --2- 172.21.15.146:0/116371205 >> v2:172.21.15.146:3300/0 conn(0x7fd0500cc040 0x7fd0500cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 -- 172.21.15.146:0/116371205 >> v1:172.21.15.146:6789/0 conn(0x7fd0500c6b70 legacy=0x7fd0500cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.500+0000 7fd047400640 1 -- 172.21.15.146:0/116371205 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05006d420 con 0x7fd0500c75a0 2024-09-16T01:51:53.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.501+0000 7fd047400640 1 --2- 172.21.15.146:0/116371205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500c75a0 0x7fd05015b340 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd0500c3ca0 tx=0x7fd03802a1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a17c824d64adc417 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.502+0000 7fd04f400640 1 -- 172.21.15.146:0/116371205 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038037070 con 0x7fd0500c75a0 2024-09-16T01:51:53.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.502+0000 7fd04f400640 1 -- 172.21.15.146:0/116371205 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038032440 con 0x7fd0500c75a0 2024-09-16T01:51:53.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.502+0000 7fd04f400640 1 -- 172.21.15.146:0/116371205 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038032810 con 0x7fd0500c75a0 2024-09-16T01:51:53.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.504+0000 7fd055c00640 1 -- 172.21.15.146:0/116371205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500c75a0 msgr2=0x7fd05015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.504+0000 7fd055c00640 1 --2- 172.21.15.146:0/116371205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500c75a0 0x7fd05015b340 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd0500c3ca0 tx=0x7fd03802a1f0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.505+0000 7fd055c00640 1 -- 172.21.15.146:0/116371205 shutdown_connections 2024-09-16T01:51:53.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.505+0000 7fd055c00640 1 --2- 172.21.15.146:0/116371205 >> v2:172.21.15.146:3300/0 conn(0x7fd0500cc040 0x7fd0500cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.505+0000 7fd055c00640 1 --2- 172.21.15.146:0/116371205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500c75a0 0x7fd05015b340 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.505+0000 7fd055c00640 1 -- 172.21.15.146:0/116371205 >> 172.21.15.146:0/116371205 conn(0x7fd0500ccc20 msgr2=0x7fd0500cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:53.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.505+0000 7fd055c00640 1 -- 172.21.15.146:0/116371205 shutdown_connections 2024-09-16T01:51:53.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.505+0000 7fd055c00640 1 -- 172.21.15.146:0/116371205 wait complete. 2024-09-16T01:51:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.507+0000 7fd055c00640 1 Processor -- start 2024-09-16T01:51:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.507+0000 7fd055c00640 1 -- start start 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0500c6b70 0x7fd05015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0500c75a0 0x7fd0500c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500cc040 0x7fd050157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500cb050 con 0x7fd0500c75a0 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500cad70 con 0x7fd0500cc040 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd0500cbc80 con 0x7fd0500c6b70 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500cc040 0x7fd050157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd047400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0500c75a0 0x7fd0500c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd04fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0500c6b70 0x7fd05015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500cc040 0x7fd050157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50438/0 (socket says 172.21.15.146:50438) 2024-09-16T01:51:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd055200640 1 -- 172.21.15.146:0/2774969875 learned_addr learned my addr 172.21.15.146:0/2774969875 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd047400640 1 -- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0500c6b70 msgr2=0x7fd05015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd047400640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd0500c6b70 0x7fd05015ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd047400640 1 -- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500cc040 msgr2=0x7fd050157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.509+0000 7fd047400640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500cc040 0x7fd050157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd047400640 1 -- 172.21.15.146:0/2774969875 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05015d9d0 con 0x7fd0500c75a0 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd055200640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0500cc040 0x7fd050157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd047400640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0500c75a0 0x7fd0500c5630 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd038031000 tx=0x7fd0380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038041030 con 0x7fd0500c75a0 2024-09-16T01:51:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038030500 con 0x7fd0500c75a0 2024-09-16T01:51:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038037040 con 0x7fd0500c75a0 2024-09-16T01:51:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.510+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05015dcb0 con 0x7fd0500c75a0 2024-09-16T01:51:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.511+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05015e270 con 0x7fd0500c75a0 2024-09-16T01:51:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.511+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd05006cd20 con 0x7fd0500c75a0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.514+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd038044070 con 0x7fd0500c75a0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.515+0000 7fd04cc00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd02c06c2b0 0x7fd02c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.515+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7fd0380ac500 con 0x7fd0500c75a0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.515+0000 7fd04fe00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd02c06c2b0 0x7fd02c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.515+0000 7fd04cc00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd02c074570 0x7fd02c0769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.515+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd038001d30 con 0x7fd02c074570 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.515+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fd0380ac8b0 con 0x7fd0500c75a0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.516+0000 7fd055200640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd02c074570 0x7fd02c0769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.516+0000 7fd04fe00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd02c06c2b0 0x7fd02c06e770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd0400016e0 tx=0x7fd040009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.516+0000 7fd055200640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd02c074570 0x7fd02c0769b0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.517+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd038001d30 con 0x7fd02c074570 2024-09-16T01:51:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.551+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fd05006da90 con 0x7fd02c074570 2024-09-16T01:51:53.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.552+0000 7fd04cc00640 1 -- 172.21.15.146:0/2774969875 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fd05006da90 con 0x7fd02c074570 2024-09-16T01:51:53.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd02c074570 msgr2=0x7fd02c0769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055c00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fd02c074570 0x7fd02c0769b0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd02c06c2b0 msgr2=0x7fd02c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055c00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd02c06c2b0 0x7fd02c06e770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd0400016e0 tx=0x7fd040009040 comp rx=0 tx=0).stop 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0500c75a0 msgr2=0x7fd0500c5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055c00640 1 --2- 172.21.15.146:0/2774969875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0500c75a0 0x7fd0500c5630 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd038031000 tx=0x7fd0380046c0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.554+0000 7fd055200640 1 -- 172.21.15.146:0/2774969875 reap_dead start 2024-09-16T01:51:53.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.555+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 shutdown_connections 2024-09-16T01:51:53.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.555+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 >> 172.21.15.146:0/2774969875 conn(0x7fd0500ccc20 msgr2=0x7fd0500c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:53.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.555+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 shutdown_connections 2024-09-16T01:51:53.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.555+0000 7fd055c00640 1 -- 172.21.15.146:0/2774969875 wait complete. 2024-09-16T01:51:53.562 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:53.561+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:53.573 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:53.573 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:53.583 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:51:53.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-16T01:51:53.668 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:53.667+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:53.744 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:53.744 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:53.754 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:51:53.795 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:53.794+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:53.865 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:53.864+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:53.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.898+0000 7f60ef200640 1 Processor -- start 2024-09-16T01:51:53.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.898+0000 7f60ef200640 1 -- start start 2024-09-16T01:51:53.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ef200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f60f00c6b70 0x7f60f015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ef200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f006d140 con 0x7f60f00cc040 2024-09-16T01:51:53.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ef200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f006d420 con 0x7f60f00c75a0 2024-09-16T01:51:53.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ef200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f60f015b880 con 0x7f60f00c6b70 2024-09-16T01:51:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ede00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f60f00c6b70 0x7f60f015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ede00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f60f00c6b70 0x7f60f015b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50442/0 (socket says 172.21.15.146:50442) 2024-09-16T01:51:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.900+0000 7f60ede00640 1 -- 172.21.15.146:0/4066018544 learned_addr learned my addr 172.21.15.146:0/4066018544 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.901+0000 7f60eca00640 1 -- 172.21.15.146:0/4066018544 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076537572 0 0) 0x7f60f006d420 con 0x7f60f00c75a0 2024-09-16T01:51:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.901+0000 7f60eca00640 1 -- 172.21.15.146:0/4066018544 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60d4003520 con 0x7f60f00c75a0 2024-09-16T01:51:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.901+0000 7f60ede00640 1 -- 172.21.15.146:0/4066018544 >> v1:172.21.15.161:6789/0 conn(0x7f60f00c75a0 legacy=0x7f60f00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.901+0000 7f60ede00640 1 -- 172.21.15.146:0/4066018544 >> v1:172.21.15.146:6789/0 conn(0x7f60f00cc040 legacy=0x7f60f0165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.901+0000 7f60ede00640 1 -- 172.21.15.146:0/4066018544 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f006d700 con 0x7f60f00c6b70 2024-09-16T01:51:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.903+0000 7f60ede00640 1 --2- 172.21.15.146:0/4066018544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c6b70 0x7f60f015b340 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009870 tx=0x7f60dc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49153c72fe152dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.903+0000 7f60eca00640 1 -- 172.21.15.146:0/4066018544 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc037070 con 0x7f60f00c6b70 2024-09-16T01:51:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.904+0000 7f60eca00640 1 -- 172.21.15.146:0/4066018544 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60dc032440 con 0x7f60f00c6b70 2024-09-16T01:51:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.904+0000 7f60eca00640 1 -- 172.21.15.146:0/4066018544 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc032810 con 0x7f60f00c6b70 2024-09-16T01:51:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.905+0000 7f60ef200640 1 -- 172.21.15.146:0/4066018544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c6b70 msgr2=0x7f60f015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.905+0000 7f60ef200640 1 --2- 172.21.15.146:0/4066018544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c6b70 0x7f60f015b340 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f60dc009870 tx=0x7f60dc02a230 comp rx=0 tx=0).stop 2024-09-16T01:51:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.905+0000 7f60ef200640 1 -- 172.21.15.146:0/4066018544 shutdown_connections 2024-09-16T01:51:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.905+0000 7f60ef200640 1 --2- 172.21.15.146:0/4066018544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c6b70 0x7f60f015b340 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.905+0000 7f60ef200640 1 -- 172.21.15.146:0/4066018544 >> 172.21.15.146:0/4066018544 conn(0x7f60f00ccc20 msgr2=0x7f60f00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:53.908 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:53.908 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:53.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.906+0000 7f60ef200640 1 -- 172.21.15.146:0/4066018544 shutdown_connections 2024-09-16T01:51:53.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.906+0000 7f60ef200640 1 -- 172.21.15.146:0/4066018544 wait complete. 2024-09-16T01:51:53.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.908+0000 7f60ef200640 1 Processor -- start 2024-09-16T01:51:53.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.909+0000 7f60ef200640 1 -- start start 2024-09-16T01:51:53.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.911+0000 7f60ef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f00c6b70 0x7f60f015ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.911+0000 7f60ef200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c75a0 0x7f60f0071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.911+0000 7f60ef200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60f00cc040 0x7f60f0072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ef200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f60f00cb050 con 0x7f60f00c6b70 2024-09-16T01:51:53.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ef200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f60f00cab20 con 0x7f60f00c75a0 2024-09-16T01:51:53.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ef200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f60f00cbc80 con 0x7f60f00cc040 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ede00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f00c6b70 0x7f60f015ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ede00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f00c6b70 0x7f60f015ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55448/0 (socket says 172.21.15.146:55448) 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ede00640 1 -- 172.21.15.146:0/4081109573 learned_addr learned my addr 172.21.15.146:0/4081109573 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.912+0000 7f60ee800640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60f00cc040 0x7f60f0072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.913+0000 7f60ee800640 1 -- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c75a0 msgr2=0x7f60f0071cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.913+0000 7f60ee800640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60f00c75a0 0x7f60f0071cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.913+0000 7f60ee800640 1 -- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f00c6b70 msgr2=0x7f60f015ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.913+0000 7f60ee800640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f00c6b70 0x7f60f015ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.913+0000 7f60ee800640 1 -- 172.21.15.146:0/4081109573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f01f5a30 con 0x7f60f00cc040 2024-09-16T01:51:53.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.914+0000 7f60ede00640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60f00c6b70 0x7f60f015ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:53.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.915+0000 7f60ee800640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60f00cc040 0x7f60f0072230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c440 tx=0x7f60e800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.916+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8017070 con 0x7f60f00cc040 2024-09-16T01:51:53.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.916+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e80035f0 con 0x7f60f00cc040 2024-09-16T01:51:53.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.916+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e80039e0 con 0x7f60f00cc040 2024-09-16T01:51:53.920 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:51:53.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.919+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f01f5750 con 0x7f60f00cc040 2024-09-16T01:51:53.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.919+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f006f2b0 con 0x7f60f00cc040 2024-09-16T01:51:53.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.920+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f60f006cd20 con 0x7f60f00cc040 2024-09-16T01:51:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.923+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f60e801a070 con 0x7f60f00cc040 2024-09-16T01:51:53.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.924+0000 7f60e6000640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f60d006c300 0x7f60d006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.924+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f60e808a9a0 con 0x7f60f00cc040 2024-09-16T01:51:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.924+0000 7f60e6000640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f60d0074640 0x7f60d0076a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.924+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60e8038e50 con 0x7f60d0074640 2024-09-16T01:51:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.924+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f60e808ad50 con 0x7f60f00cc040 2024-09-16T01:51:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.925+0000 7f60ed400640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f60d0074640 0x7f60d0076a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.925+0000 7f60ede00640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f60d006c300 0x7f60d006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.926+0000 7f60ed400640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f60d0074640 0x7f60d0076a80 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.927+0000 7f60ede00640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f60d006c300 0x7f60d006e7c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f60dc02a270 tx=0x7f60dc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:53.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.928+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60e8038e50 con 0x7f60d0074640 2024-09-16T01:51:53.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.948+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f60f006da90 con 0x7f60d0074640 2024-09-16T01:51:53.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.949+0000 7f60e6000640 1 -- 172.21.15.146:0/4081109573 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f60f006da90 con 0x7f60d0074640 2024-09-16T01:51:53.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.954+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f60d0074640 msgr2=0x7f60d0076a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.954+0000 7f60ef200640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f60d0074640 0x7f60d0076a80 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:53.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.956+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f60d006c300 msgr2=0x7f60d006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.956+0000 7f60ef200640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f60d006c300 0x7f60d006e7c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f60dc02a270 tx=0x7f60dc02e000 comp rx=0 tx=0).stop 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.956+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60f00cc040 msgr2=0x7f60f0072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.956+0000 7f60ef200640 1 --2- 172.21.15.146:0/4081109573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60f00cc040 0x7f60f0072230 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c440 tx=0x7f60e800c900 comp rx=0 tx=0).stop 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.957+0000 7f60ee800640 1 -- 172.21.15.146:0/4081109573 reap_dead start 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.957+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 shutdown_connections 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.957+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 >> 172.21.15.146:0/4081109573 conn(0x7f60f00ccc20 msgr2=0x7f60f00c8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.957+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 shutdown_connections 2024-09-16T01:51:53.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:53.958+0000 7f60ef200640 1 -- 172.21.15.146:0/4081109573 wait complete. 2024-09-16T01:51:53.967 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:53.965+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:53.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-16T01:51:54.067 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:54.066+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:54.072 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:54.072 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:54.081 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:51:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.148+0000 7f9223400640 1 Processor -- start 2024-09-16T01:51:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.149+0000 7f9223400640 1 -- start start 2024-09-16T01:51:54.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f9223400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f921c15b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f9223400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921c15e0b0 con 0x7f921c154c10 2024-09-16T01:51:54.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f9223400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921c15e390 con 0x7f921c0cca70 2024-09-16T01:51:54.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f9223400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f921c15de60 con 0x7f921c0cbe00 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f9220a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f921c154c10 0x7f921c073340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46522/0 (socket says 172.21.15.146:46522) 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f921be00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f921c0cca70 0x7f921c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:38938/0 (socket says 172.21.15.146:38938) 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.152+0000 7f9220a00640 1 -- 172.21.15.146:0/458558914 learned_addr learned my addr 172.21.15.146:0/458558914 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f9221400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f921c15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279581382 0 0) 0x7f921c15e0b0 con 0x7f921c154c10 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9204003520 con 0x7f921c154c10 2024-09-16T01:51:54.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098460351 0 0) 0x7f921c15e390 con 0x7f921c0cca70 2024-09-16T01:51:54.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f921c15e0b0 con 0x7f921c0cca70 2024-09-16T01:51:54.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1399851259 0 0) 0x7f9204003520 con 0x7f921c154c10 2024-09-16T01:51:54.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 msgr2=0x7f921c15b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:54.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f921c15b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 >> v1:172.21.15.161:6789/0 conn(0x7f921c0cca70 legacy=0x7f921c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.153+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c15e670 con 0x7f921c154c10 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f92100015a0 con 0x7f921c154c10 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 >> v1:172.21.15.146:6789/0 conn(0x7f921c154c10 legacy=0x7f921c073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204004bd0 0x7f9204004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f92040037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9204003ce0 0x7f9204004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f921c15de60 con 0x7f9204003ce0 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f920400ba70 con 0x7f9204004bd0 2024-09-16T01:51:54.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.154+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f920400bc20 con 0x7f921c0cbe00 2024-09-16T01:51:54.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.155+0000 7f921be00640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f92040037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.155+0000 7f9221400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9204003ce0 0x7f9204004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.155+0000 7f921be00640 1 -- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204004bd0 msgr2=0x7f9204004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:54.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.155+0000 7f921be00640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9204004bd0 0x7f9204004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.155+0000 7f921be00640 1 -- 172.21.15.146:0/458558914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9204003ce0 msgr2=0x7f9204004160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:54.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.155+0000 7f921be00640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9204003ce0 0x7f9204004160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.156+0000 7f921be00640 1 -- 172.21.15.146:0/458558914 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9204003520 con 0x7f921c0cbe00 2024-09-16T01:51:54.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.158+0000 7f921be00640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f92040037a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9208001900 tx=0x7f9204005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e9760ce4e89109 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.158+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9208001500 con 0x7f921c0cbe00 2024-09-16T01:51:54.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.158+0000 7f921b400640 1 -- 172.21.15.146:0/458558914 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920802e070 con 0x7f921c0cbe00 2024-09-16T01:51:54.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.159+0000 7f9223400640 1 -- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 msgr2=0x7f92040037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.159+0000 7f9223400640 1 --2- 172.21.15.146:0/458558914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c0cbe00 0x7f92040037a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9208001900 tx=0x7f9204005830 comp rx=0 tx=0).stop 2024-09-16T01:51:54.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.159+0000 7f9221400640 1 -- 172.21.15.146:0/458558914 reap_dead start 2024-09-16T01:51:54.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.160+0000 7f9223400640 1 -- 172.21.15.146:0/458558914 shutdown_connections 2024-09-16T01:51:54.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.160+0000 7f9223400640 1 -- 172.21.15.146:0/458558914 >> 172.21.15.146:0/458558914 conn(0x7f921c06ce80 msgr2=0x7f921c06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:54.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.161+0000 7f9223400640 1 -- 172.21.15.146:0/458558914 shutdown_connections 2024-09-16T01:51:54.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.161+0000 7f9223400640 1 -- 172.21.15.146:0/458558914 wait complete. 2024-09-16T01:51:54.167 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:54.166+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:54.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.166+0000 7f9223400640 1 Processor -- start 2024-09-16T01:51:54.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.166+0000 7f9223400640 1 -- start start 2024-09-16T01:51:54.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.168+0000 7f9223400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c15d370 0x7f921c1e4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.169+0000 7f9223400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f921c15d750 0x7f921c0c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.169+0000 7f921be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f921c15d750 0x7f921c0c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.169+0000 7f921be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f921c15d750 0x7f921c0c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55472/0 (socket says 172.21.15.146:55472) 2024-09-16T01:51:54.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.169+0000 7f921be00640 1 -- 172.21.15.146:0/3059611438 learned_addr learned my addr 172.21.15.146:0/3059611438 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:54.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.170+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921c1f8460 0x7f921c14dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.170+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c15d370 0x7f921c1e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.170+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921c1f8460 0x7f921c14dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.170+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f921c154ff0 con 0x7f921c15d750 2024-09-16T01:51:54.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.170+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f921c15ec30 con 0x7f921c1f8460 2024-09-16T01:51:54.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.170+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f921c073a50 con 0x7f921c15d370 2024-09-16T01:51:54.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.172+0000 7f921be00640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c15d370 msgr2=0x7f921c1e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.172+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c15d370 0x7f921c1e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.172+0000 7f921be00640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921c1f8460 msgr2=0x7f921c14dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.172+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921c1f8460 0x7f921c14dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.172+0000 7f921be00640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c0ca980 con 0x7f921c15d750 2024-09-16T01:51:54.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.173+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921c1f8460 0x7f921c14dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:54.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.173+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f921c15d750 0x7f921c0c9a70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f920802a780 tx=0x7f9208001300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.174+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921c15d370 0x7f921c1e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:54.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.174+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9208043070 con 0x7f921c15d750 2024-09-16T01:51:54.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.175+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921c073410 con 0x7f921c15d750 2024-09-16T01:51:54.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.175+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921c0ca6a0 con 0x7f921c15d750 2024-09-16T01:51:54.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.176+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9208033670 con 0x7f921c15d750 2024-09-16T01:51:54.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.177+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f921c0cafa0 con 0x7f921c15d750 2024-09-16T01:51:54.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.177+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9208033a20 con 0x7f921c15d750 2024-09-16T01:51:54.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.179+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9208033bc0 con 0x7f921c15d750 2024-09-16T01:51:54.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.180+0000 7f9218c00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91fc06c330 0x7f91fc06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.180+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f92080ab3a0 con 0x7f921c15d750 2024-09-16T01:51:54.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.181+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v88) ==== 54+0+16 (secure 0 0 0) 0x7f92080ab880 con 0x7f921c15d750 2024-09-16T01:51:54.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.181+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91fc06c330 0x7f91fc06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.181+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f921c1f88e0 0x7f921c0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.182+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f921c1f88e0 0x7f921c0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.181+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f921c0ce720 con 0x7f921c1f88e0 2024-09-16T01:51:54.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.182+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91fc06c330 0x7f91fc06e7f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9210001130 tx=0x7f9210009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.183+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f921c1f88e0 0x7f921c0cd710 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.184+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c0ce720 con 0x7f921c1f88e0 2024-09-16T01:51:54.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.213+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c0ce440 con 0x7f921c1f88e0 2024-09-16T01:51:54.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.215+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c0ce440 con 0x7f921c1f88e0 2024-09-16T01:51:54.217 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:54.218 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:54.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.216+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f921c0c78a0 0x7f921c0c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.216+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f921c075880 con 0x7f921c0c78a0 2024-09-16T01:51:54.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.218+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f921c0c78a0 0x7f921c0c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.219+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f921c0c78a0 0x7f921c0c7c80 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.219+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c075880 con 0x7f921c0c78a0 2024-09-16T01:51:54.234 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:54.234 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:54.244 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:51:54.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.248+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c1e51a0 con 0x7f921c0c78a0 2024-09-16T01:51:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.250+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c1e51a0 con 0x7f921c0c78a0 2024-09-16T01:51:54.252 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.251+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f921c0c8580 0x7f921c062b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.251+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f921c063080 con 0x7f921c0c8580 2024-09-16T01:51:54.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.252+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f921c0c8580 0x7f921c062b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.253+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f921c0c8580 0x7f921c062b40 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.254+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c063080 con 0x7f921c0c8580 2024-09-16T01:51:54.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.273+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c063560 con 0x7f921c0c8580 2024-09-16T01:51:54.285 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:54.284+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:54.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.287+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c063560 con 0x7f921c0c8580 2024-09-16T01:51:54.289 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:54.289 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:54.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.287+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f921c0cfad0 0x7f921c02c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.287+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f921c0cff10 con 0x7f921c0cfad0 2024-09-16T01:51:54.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.290+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f921c0cfad0 0x7f921c02c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.295+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f921c0cfad0 0x7f921c02c640 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.298+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c0cff10 con 0x7f921c0cfad0 2024-09-16T01:51:54.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.320+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c1e5350 con 0x7f921c0cfad0 2024-09-16T01:51:54.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.325+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c1e5350 con 0x7f921c0cfad0 2024-09-16T01:51:54.327 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:54.327 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:54.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.325+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f921c02d090 0x7f921c02d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.325+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f921c02da30 con 0x7f921c02d090 2024-09-16T01:51:54.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.326+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f921c02d090 0x7f921c02d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.326+0000 7f5047e00640 1 Processor -- start 2024-09-16T01:51:54.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.326+0000 7f5047e00640 1 -- start start 2024-09-16T01:51:54.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.326+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f921c02d090 0x7f921c02d4f0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.327+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c02da30 con 0x7f921c02d090 2024-09-16T01:51:54.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5047e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f504815aa70 0x7f50480cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5047e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f504815af00 con 0x7f50480c6290 2024-09-16T01:51:54.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5047e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f504815b1e0 con 0x7f50480c5eb0 2024-09-16T01:51:54.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5047e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f504806dbd0 con 0x7f504815aa70 2024-09-16T01:51:54.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5046a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f50480c5eb0 0x7f5048154910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:38944/0 (socket says 172.21.15.146:38944) 2024-09-16T01:51:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5046a00640 1 -- 172.21.15.146:0/2183128308 learned_addr learned my addr 172.21.15.146:0/2183128308 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5046000640 1 --2- 172.21.15.146:0/2183128308 >> v2:172.21.15.146:3300/0 conn(0x7f504815aa70 0x7f50480cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5045600640 1 -- 172.21.15.146:0/2183128308 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3218635905 0 0) 0x7f504815b1e0 con 0x7f50480c5eb0 2024-09-16T01:51:54.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.328+0000 7f5045600640 1 -- 172.21.15.146:0/2183128308 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502c003520 con 0x7f50480c5eb0 2024-09-16T01:51:54.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.329+0000 7f5046000640 1 -- 172.21.15.146:0/2183128308 >> v1:172.21.15.161:6789/0 conn(0x7f50480c5eb0 legacy=0x7f5048154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.329+0000 7f5046000640 1 -- 172.21.15.146:0/2183128308 >> v1:172.21.15.146:6789/0 conn(0x7f50480c6290 legacy=0x7f504806c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.329+0000 7f5046000640 1 -- 172.21.15.146:0/2183128308 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504815b4c0 con 0x7f504815aa70 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.330+0000 7f5046000640 1 --2- 172.21.15.146:0/2183128308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f504815aa70 0x7f50480cac30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5034009a00 tx=0x7f503402a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90333c2666997616 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.331+0000 7f5045600640 1 -- 172.21.15.146:0/2183128308 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034036070 con 0x7f504815aa70 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.331+0000 7f5045600640 1 -- 172.21.15.146:0/2183128308 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5034031600 con 0x7f504815aa70 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.331+0000 7f5045600640 1 -- 172.21.15.146:0/2183128308 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50340319d0 con 0x7f504815aa70 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 -- 172.21.15.146:0/2183128308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f504815aa70 msgr2=0x7f50480cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 --2- 172.21.15.146:0/2183128308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f504815aa70 0x7f50480cac30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5034009a00 tx=0x7f503402a540 comp rx=0 tx=0).stop 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 -- 172.21.15.146:0/2183128308 shutdown_connections 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 --2- 172.21.15.146:0/2183128308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f504815aa70 0x7f50480cac30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 -- 172.21.15.146:0/2183128308 >> 172.21.15.146:0/2183128308 conn(0x7f5048067b30 msgr2=0x7f5048067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 -- 172.21.15.146:0/2183128308 shutdown_connections 2024-09-16T01:51:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.332+0000 7f5047e00640 1 -- 172.21.15.146:0/2183128308 wait complete. 2024-09-16T01:51:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.334+0000 7f5047e00640 1 Processor -- start 2024-09-16T01:51:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.334+0000 7f5047e00640 1 -- start start 2024-09-16T01:51:54.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50480c5eb0 0x7f5048201680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50480c6290 0x7f50480c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50481fe3f0 0x7f504814ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5048155020 con 0x7f50480c5eb0 2024-09-16T01:51:54.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f504806c370 con 0x7f50480c6290 2024-09-16T01:51:54.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f50481551a0 con 0x7f50481fe3f0 2024-09-16T01:51:54.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50481fe3f0 0x7f504814ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50481fe3f0 0x7f504814ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:48858/0 (socket says 172.21.15.146:48858) 2024-09-16T01:51:54.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.335+0000 7f5047400640 1 -- 172.21.15.146:0/2251050550 learned_addr learned my addr 172.21.15.146:0/2251050550 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:54.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5046000640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50480c6290 0x7f50480c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5047400640 1 -- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50480c6290 msgr2=0x7f50480c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5047400640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50480c6290 0x7f50480c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5047400640 1 -- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50480c5eb0 msgr2=0x7f5048201680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5047400640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50480c5eb0 0x7f5048201680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5047400640 1 -- 172.21.15.146:0/2251050550 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50481506c0 con 0x7f50481fe3f0 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.336+0000 7f5047400640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50481fe3f0 0x7f504814ec80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f504000c730 tx=0x7f504000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.337+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040016020 con 0x7f50481fe3f0 2024-09-16T01:51:54.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.337+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5040006d50 con 0x7f50481fe3f0 2024-09-16T01:51:54.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.337+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040003bb0 con 0x7f50481fe3f0 2024-09-16T01:51:54.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.337+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50481509a0 con 0x7f50481fe3f0 2024-09-16T01:51:54.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.338+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50481fdbf0 con 0x7f50481fe3f0 2024-09-16T01:51:54.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.338+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f50481fdfd0 con 0x7f50481fe3f0 2024-09-16T01:51:54.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.345+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f504000a640 con 0x7f50481fe3f0 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.346+0000 7f503ea00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f502806c2b0 0x7f502806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.346+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f504008bff0 con 0x7f50481fe3f0 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.347+0000 7f503ea00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5028074570 0x7f50280769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.347+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f504000ae50 con 0x7f5028074570 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.347+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f504008c480 con 0x7f50481fe3f0 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.347+0000 7f5046a00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f502806c2b0 0x7f502806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.347+0000 7f5046000640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5028074570 0x7f50280769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.348+0000 7f5046000640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5028074570 0x7f50280769b0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.348+0000 7f5046a00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f502806c2b0 0x7f502806e770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5038000f60 tx=0x7f50380078f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.349+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f504000ae50 con 0x7f5028074570 2024-09-16T01:51:54.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.351+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c02da30 con 0x7f921c02d090 2024-09-16T01:51:54.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.352+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c02da30 con 0x7f921c02d090 2024-09-16T01:51:54.355 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:54.355 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:54.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.353+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f921c2094b0 0x7f921c20b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.353+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f921c20e670 con 0x7f921c2094b0 2024-09-16T01:51:54.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.353+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f921c2094b0 0x7f921c20b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.354+0000 7f921be00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f921c2094b0 0x7f921c20b8a0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.354+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c20e670 con 0x7f921c2094b0 2024-09-16T01:51:54.368 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:54.367+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:54.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.379+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f5048131910 con 0x7f5028074570 2024-09-16T01:51:54.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.381+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c20bde0 con 0x7f921c2094b0 2024-09-16T01:51:54.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.386+0000 7f503ea00640 1 -- 172.21.15.146:0/2251050550 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f5048131910 con 0x7f5028074570 2024-09-16T01:51:54.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.386+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c20bde0 con 0x7f921c2094b0 2024-09-16T01:51:54.389 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:54.390 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:54.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.388+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f921c20c500 0x7f921c20c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.388+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f921c2119e0 con 0x7f921c20c500 2024-09-16T01:51:54.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.388+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f921c20c500 0x7f921c20c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.388+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5028074570 msgr2=0x7f50280769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.388+0000 7f5047e00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5028074570 0x7f50280769b0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f502806c2b0 msgr2=0x7f502806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f502806c2b0 0x7f502806e770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5038000f60 tx=0x7f50380078f0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50481fe3f0 msgr2=0x7f504814ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 --2- 172.21.15.146:0/2251050550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50481fe3f0 0x7f504814ec80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f504000c730 tx=0x7f504000cbf0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f9221400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f921c20c500 0x7f921c20c940 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047400640 1 -- 172.21.15.146:0/2251050550 reap_dead start 2024-09-16T01:51:54.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 shutdown_connections 2024-09-16T01:51:54.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 >> 172.21.15.146:0/2251050550 conn(0x7f5048067b30 msgr2=0x7f504815a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:54.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 shutdown_connections 2024-09-16T01:51:54.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.389+0000 7f5047e00640 1 -- 172.21.15.146:0/2251050550 wait complete. 2024-09-16T01:51:54.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.390+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c2119e0 con 0x7f921c20c500 2024-09-16T01:51:54.409 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:54.409 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:54.420 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:51:54.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-16T01:51:54.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.425+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c20cf90 con 0x7f921c20c500 2024-09-16T01:51:54.436 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.428+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c20cf90 con 0x7f921c20c500 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.433+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f921c20d6b0 0x7f921c20dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.433+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f921c20e1f0 con 0x7f921c20d6b0 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.433+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f921c20d6b0 0x7f921c20dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.433+0000 7f9220a00640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f921c20d6b0 0x7f921c20dad0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.434+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f921c20e1f0 con 0x7f921c20d6b0 2024-09-16T01:51:54.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.464+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f921c20e1f0 con 0x7f921c20d6b0 2024-09-16T01:51:54.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9218c00640 1 -- 172.21.15.146:0/3059611438 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f921c20e1f0 con 0x7f921c20d6b0 2024-09-16T01:51:54.471 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:54.468+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:54.471 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:54.471 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f921c1f88e0 msgr2=0x7f921c0cd710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f921c1f88e0 0x7f921c0cd710 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f921c0c78a0 msgr2=0x7f921c0c7c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f921c0c78a0 0x7f921c0c7c80 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f921c0c8580 msgr2=0x7f921c062b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f921c0c8580 0x7f921c062b40 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f921c0cfad0 msgr2=0x7f921c02c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f921c0cfad0 0x7f921c02c640 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f921c02d090 msgr2=0x7f921c02d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f921c02d090 0x7f921c02d4f0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f921c2094b0 msgr2=0x7f921c20b8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f921c2094b0 0x7f921c20b8a0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f921c20c500 msgr2=0x7f921c20c940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f921c20c500 0x7f921c20c940 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f921c20d6b0 msgr2=0x7f921c20dad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.469+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f921c20d6b0 0x7f921c20dad0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9221400640 1 -- 172.21.15.146:0/3059611438 reap_dead start 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9221400640 1 -- 172.21.15.146:0/3059611438 reap_dead start 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91fc06c330 msgr2=0x7f91fc06e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91fc06c330 0x7f91fc06e7f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9210001130 tx=0x7f9210009040 comp rx=0 tx=0).stop 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f921c15d750 msgr2=0x7f921c0c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9223400640 1 --2- 172.21.15.146:0/3059611438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f921c15d750 0x7f921c0c9a70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f920802a780 tx=0x7f9208001300 comp rx=0 tx=0).stop 2024-09-16T01:51:54.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.470+0000 7f9221400640 1 -- 172.21.15.146:0/3059611438 reap_dead start 2024-09-16T01:51:54.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.471+0000 7f9221400640 1 -- 172.21.15.146:0/3059611438 reap_dead start 2024-09-16T01:51:54.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.471+0000 7f9221400640 1 -- 172.21.15.146:0/3059611438 reap_dead start 2024-09-16T01:51:54.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.471+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 shutdown_connections 2024-09-16T01:51:54.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.471+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 >> 172.21.15.146:0/3059611438 conn(0x7f921c06ce80 msgr2=0x7f921c0cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:54.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.471+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 shutdown_connections 2024-09-16T01:51:54.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.472+0000 7f9223400640 1 -- 172.21.15.146:0/3059611438 wait complete. 2024-09-16T01:51:54.547 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:54.548 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:54.548 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:54.548 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:54.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.549 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:38 3.24 3:2742a876:::10000000002.00000023:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:51:54.549 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.552269+0000", 2024-09-16T01:51:54.549 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 63.992338685999997, 2024-09-16T01:51:54.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.076686301999999998, 2024-09-16T01:51:54.549 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 38 2024-09-16T01:51:54.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.552269+0000", 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.552269+0000", 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.552270+0000", 2024-09-16T01:51:54.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.579e-06 2024-09-16T01:51:54.553 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.553 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.553 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.553 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593289+0000", 2024-09-16T01:51:54.553 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041018708000000001 2024-09-16T01:51:54.553 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593307+0000", 2024-09-16T01:51:54.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8382000000000001e-05 2024-09-16T01:51:54.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593316+0000", 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1880000000000002e-06 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593380+0000", 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4666999999999996e-05 2024-09-16T01:51:54.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593479+0000", 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8740999999999995e-05 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593568+0000", 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9142999999999997e-05 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.596725+0000", 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031567890000000001 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.628797+0000", 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032072554000000003 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.628879+0000", 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1046999999999998e-05 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.628955+0000", 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6503999999999998e-05 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:482 3.32 3:4c3c6424:::10000000002.000000df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.479905099, 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.78198721800000004, 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.558 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 482 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.559 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064704+0000", 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.578e-06 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102892+0000", 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038187684999999999 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102910+0000", 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8338999999999998e-05 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102918+0000", 2024-09-16T01:51:54.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.259e-06 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102991+0000", 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2638999999999994e-05 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.103003+0000", 2024-09-16T01:51:54.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2269999999999999e-05 2024-09-16T01:51:54.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.816160+0000", 2024-09-16T01:51:54.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.71315740100000002 2024-09-16T01:51:54.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:54.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.820892+0000", 2024-09-16T01:51:54.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047315539999999998 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835633+0000", 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014741232999999999 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835741+0000", 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107768 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835903+0000", 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016157399999999999 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.566 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846577+0000", 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010674197 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846586+0000", 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5149999999999995e-06 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846627+0000", 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0513000000000003e-05 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846690+0000", 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2693000000000004e-05 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:487 3.33 3:ce8011cb:::10000000002.000000e4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:51:54.568 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.439472465000001, 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.80604562300000004, 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 487 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105136+0000", 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1429999999999999e-06 2024-09-16T01:51:54.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131251+0000", 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026114741 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131264+0000", 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3325e-05 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131270+0000", 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1399999999999997e-06 2024-09-16T01:51:54.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131326+0000", 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5643000000000002e-05 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131334+0000", 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4020000000000002e-06 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.572 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.832106+0000", 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.700771794 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.833244+0000", 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011381679999999999 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890416+0000", 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057171588000000002 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890476+0000", 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0118999999999998e-05 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890571+0000", 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5011000000000002e-05 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.895250+0000", 2024-09-16T01:51:54.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046789140000000002 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911079+0000", 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015829296999999999 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911119+0000", 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9489000000000001e-05 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911181+0000", 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1848999999999997e-05 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:506 3.32 3:4c9ed383:::10000000002.000000f7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.261157472999997, 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.56375856099999999, 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.576 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 506 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283453+0000", 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9679999999999998e-06 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324488+0000", 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041035220999999997 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324503+0000", 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5167000000000001e-05 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324512+0000", 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2019999999999993e-06 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324603+0000", 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494999999999993e-05 2024-09-16T01:51:54.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324612+0000", 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0769999999999997e-06 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.821133+0000", 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.496521196 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.824031+0000", 2024-09-16T01:51:54.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028972809999999998 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841705+0000", 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017674083 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841807+0000", 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102268 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:54.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841968+0000", 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016139600000000001 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846962+0000", 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049941990000000004 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847135+0000", 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017284200000000001 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.582 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847161+0000", 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5193000000000001e-05 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847208+0000", 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7973000000000003e-05 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:538 3.33 3:cf13e063:::100000001f7.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:54.583 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.964124507999998, 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.109075775, 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 538 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.584 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580485+0000", 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1449999999999998e-06 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637145+0000", 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056659689999999999 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637165+0000", 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0092e-05 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637173+0000", 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0660000000000004e-06 2024-09-16T01:51:54.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637263+0000", 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0059e-05 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637365+0000", 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101922 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637467+0000", 2024-09-16T01:51:54.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101853 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.640930+0000", 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003463617 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689487+0000", 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048556903999999998 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689524+0000", 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6776999999999997e-05 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689559+0000", 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4650000000000002e-05 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:556 3.24 3:25dc7ea1:::100000001f7.00000029:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.796679+0000", 2024-09-16T01:51:54.589 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 39.747928436999999, 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.090422115999999997, 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 556 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.590 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.796679+0000", 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.796679+0000", 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.796681+0000", 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.792e-06 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.591 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.849943+0000", 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053262217000000001 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.849959+0000", 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5882000000000001e-05 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.849967+0000", 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0039999999999999e-06 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.850023+0000", 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5939000000000001e-05 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.850131+0000", 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108514 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.850222+0000", 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0735000000000005e-05 2024-09-16T01:51:54.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.855014+0000", 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047923649999999998 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.887020+0000", 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032005515999999998 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.887058+0000", 2024-09-16T01:51:54.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7659e-05 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.887101+0000", 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3492999999999998e-05 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:704 3.1 3:80d7e53a:::100000001f7.000000bd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.213670+0000", 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.330937792999997, 2024-09-16T01:51:54.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075342113000000002, 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 704 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.213670+0000", 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.213670+0000", 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.213671+0000", 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.358e-06 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.264547+0000", 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050875919999999998 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.264571+0000", 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4416000000000001e-05 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.264586+0000", 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4728e-05 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.264690+0000", 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010440799999999999 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.264799+0000", 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108344 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.264913+0000", 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114282 2024-09-16T01:51:54.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.269177+0000", 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042638260000000001 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.288937+0000", 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.019760427000000001 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.288974+0000", 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6616999999999999e-05 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.289012+0000", 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7787000000000003e-05 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:726 3.1 3:83186ff1:::100000001f7.000000d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:51:54.601 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.427655+0000", 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.116952711000003, 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083880706999999999, 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 726 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.427655+0000", 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.427655+0000", 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.427657+0000", 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.967e-06 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457541+0000", 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029884035 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457566+0000", 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4902e-05 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457578+0000", 2024-09-16T01:51:54.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2041e-05 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457642+0000", 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3866000000000001e-05 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457750+0000", 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010874099999999999 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:54.605 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457861+0000", 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011107699999999999 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.461779+0000", 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039180930000000001 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.511458+0000", 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049678587000000003 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.511494+0000", 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6241000000000001e-05 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.511535+0000", 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1156999999999999e-05 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1150 3.1 3:813e966a:::100000001f8.0000007b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:54.607 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.532675+0000", 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.011932570000001, 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078304481999999995, 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1150 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.532675+0000", 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.532675+0000", 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.532677+0000", 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1229999999999998e-06 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578519+0000", 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045842540000000001 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578539+0000", 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9128e-05 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578546+0000", 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.588e-06 2024-09-16T01:51:54.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578616+0000", 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9775000000000002e-05 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578728+0000", 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011250400000000001 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:54.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578819+0000", 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0882000000000002e-05 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.582868+0000", 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040486439999999997 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.610818+0000", 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027949544 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.610908+0000", 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0007999999999996e-05 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.610979+0000", 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1746000000000001e-05 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.613 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1239 3.33 3:cd5a60ff:::100000001f8.000000d4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 13.947228814000001, 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.093172247999999999, 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1239 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597381+0000", 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9139999999999998e-06 2024-09-16T01:51:54.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647202+0000", 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049821439000000002 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647225+0000", 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3061000000000002e-05 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647234+0000", 2024-09-16T01:51:54.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5620000000000004e-06 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647368+0000", 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000134141 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647475+0000", 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106896 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647575+0000", 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100463 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.650879+0000", 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033037740000000002 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690472+0000", 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039593415999999999 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690523+0000", 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0657e-05 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690551+0000", 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7925000000000001e-05 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.619 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1254 3.32 3:4f15b646:::100000001f8.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.765675+0000", 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 13.778931957999999, 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075545762000000002, 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.620 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1254 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.765675+0000", 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.765675+0000", 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.765677+0000", 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.046e-06 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.835765+0000", 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.070087836000000001 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.835779+0000", 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3712e-05 2024-09-16T01:51:54.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.835789+0000", 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4890000000000001e-06 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.835839+0000", 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0266999999999998e-05 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.835950+0000", 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0001117 2024-09-16T01:51:54.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.836048+0000", 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7887999999999995e-05 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.840799+0000", 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047502270000000001 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.841155+0000", 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00035594499999999998 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.841190+0000", 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5537999999999998e-05 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.841221+0000", 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1114000000000001e-05 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.625 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1501 3.24 3:26bdc0b7:::100000001f8.000001da:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.668171236999999, 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.092713806999999995, 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.626 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1501 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876439+0000", 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4279999999999999e-06 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952215+0000", 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075776423999999995 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952227+0000", 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2218999999999999e-05 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952234+0000", 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.533e-06 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952285+0000", 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1088000000000003e-05 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952410+0000", 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012510700000000001 2024-09-16T01:51:54.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952516+0000", 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106437 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.957009+0000", 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044929569999999997 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969064+0000", 2024-09-16T01:51:54.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012055066 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969104+0000", 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9687000000000001e-05 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969150+0000", 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5861e-05 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1619 3.b 3:d196788b:::100000001f8.00000250:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.2132749080000007, 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.094691490000000003, 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1619 2024-09-16T01:51:54.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331334+0000", 2024-09-16T01:51:54.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4619999999999999e-06 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418738+0000", 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.087404198000000002 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418758+0000", 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9513000000000001e-05 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418768+0000", 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0291e-05 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418832+0000", 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3544e-05 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418975+0000", 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143361 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.419119+0000", 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014366699999999999 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425110+0000", 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059917099999999999 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425951+0000", 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084087400000000003 2024-09-16T01:51:54.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425990+0000", 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8868999999999997e-05 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.426024+0000", 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4001000000000002e-05 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1953 3.7 3:e1d5d0f0:::100000001f8.0000039e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.2956021379999996, 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.108386528, 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1953 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249006+0000", 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1179999999999999e-06 2024-09-16T01:51:54.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307322+0000", 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058315141000000001 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307341+0000", 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9011000000000001e-05 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307349+0000", 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307416+0000", 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7377000000000002e-05 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307582+0000", 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016564800000000001 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307687+0000", 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104881 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.314441+0000", 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006754726 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357290+0000", 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042848857999999997 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357330+0000", 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9477000000000002e-05 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357392+0000", 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2009999999999998e-05 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1968 3.12 3:4ba9d71c:::100000001f8.000003ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.0933541130000002, 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10484845199999999, 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1968 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451256+0000", 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:51:54.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483869+0000", 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032612532 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483888+0000", 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9029e-05 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483894+0000", 2024-09-16T01:51:54.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.674e-06 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483963+0000", 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8381999999999997e-05 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484065+0000", 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010199 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484168+0000", 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103727 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.489008+0000", 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048394880000000003 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556018+0000", 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067010583999999998 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556061+0000", 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3062999999999997e-05 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556102+0000", 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0246000000000001e-05 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1988 3.12 3:4924d375:::100000001f8.000003c1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.7811530089999996, 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098484843000000002, 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1988 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763456+0000", 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7290000000000001e-06 2024-09-16T01:51:54.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819882+0000", 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056425945999999998 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819899+0000", 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6441000000000001e-05 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819905+0000", 2024-09-16T01:51:54.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8750000000000002e-06 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819953+0000", 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7774e-05 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820053+0000", 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9702000000000002e-05 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:54.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820148+0000", 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5147000000000002e-05 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.823215+0000", 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030665369999999998 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861864+0000", 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038649944999999998 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861903+0000", 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8126000000000001e-05 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861939+0000", 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6621000000000001e-05 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.655 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2002 3.33 3:cf002303:::100000001f8.000003cf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.991569+0000", 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.5530380030000002, 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.091091670999999999, 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2002 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.991569+0000", 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.991569+0000", 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.991571+0000", 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2699999999999999e-06 2024-09-16T01:51:54.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024450+0000", 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032879494000000002 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024469+0000", 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9083999999999999e-05 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024478+0000", 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9099999999999994e-06 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024546+0000", 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8093e-05 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024658+0000", 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111702 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024766+0000", 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108164 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.030065+0000", 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0052992789999999996 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.082555+0000", 2024-09-16T01:51:54.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052489593000000001 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.082611+0000", 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6415000000000001e-05 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.082661+0000", 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9666999999999997e-05 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2033 3.33 3:cdd3c950:::100000001f8.000003ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.437077+0000", 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.1075304639999999, 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074810461999999994, 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2033 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.662 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.437077+0000", 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.437077+0000", 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.437078+0000", 2024-09-16T01:51:54.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.311e-06 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.484167+0000", 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047089187999999997 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.484184+0000", 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6595999999999998e-05 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.484195+0000", 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1168999999999999e-05 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.484266+0000", 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0758000000000003e-05 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.484417+0000", 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015096700000000001 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.484612+0000", 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00019536799999999999 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.489583+0000", 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049709960000000001 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.511785+0000", 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022201439999999999 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.511840+0000", 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5631000000000003e-05 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.511887+0000", 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7037999999999999e-05 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.667 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2037 3.12 3:486bd0c2:::100000001f8.000003f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.0575372989999998, 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124209681, 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2037 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487072+0000", 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0590000000000001e-06 2024-09-16T01:51:54.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523700+0000", 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036627383999999999 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523718+0000", 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8297000000000001e-05 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523725+0000", 2024-09-16T01:51:54.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523780+0000", 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4598000000000001e-05 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523909+0000", 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012891700000000001 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:54.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.524030+0000", 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000120874 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.527427+0000", 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033977389999999999 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611203+0000", 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083775383999999994 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611243+0000", 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114000000000003e-05 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611280+0000", 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6965999999999998e-05 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2090 3.1 3:83370771:::100000001f8.00000427:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:51:54.673 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 3.3122448269999998, 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.097529115999999999, 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2090 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:54.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232365+0000", 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.945e-06 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264573+0000", 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032208891000000003 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264593+0000", 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9488999999999999e-05 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264601+0000", 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2050000000000002e-06 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264714+0000", 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011313499999999999 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264867+0000", 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000152805 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.265004+0000", 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013696499999999999 2024-09-16T01:51:54.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.268362+0000", 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033575039999999999 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329806+0000", 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061444812000000001 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329852+0000", 2024-09-16T01:51:54.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5367000000000001e-05 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329892+0000", 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9997999999999999e-05 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:54.679 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:54.681 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:51:54.681 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:54.679+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:54.784 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:54.782+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:54.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.794+0000 7f6564400640 1 Processor -- start 2024-09-16T01:51:54.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.795+0000 7f6564400640 1 -- start start 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.800+0000 7f6564400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f655c0c9650 0x7f655c071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.800+0000 7f6564400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c072410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f655c0c9650 0x7f655c071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6564400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c06dbd0 con 0x7f655c150c10 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6564400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f655c1569d0 con 0x7f655c0c9650 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6564400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f655c06deb0 con 0x7f655c0718c0 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f655c0c9650 0x7f655c071ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50462/0 (socket says 172.21.15.146:50462) 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 -- 172.21.15.146:0/3726587371 learned_addr learned my addr 172.21.15.146:0/3726587371 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6562400640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 -- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 msgr2=0x7f655c072410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:54.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c072410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 -- 172.21.15.146:0/3726587371 >> v1:172.21.15.146:6789/0 conn(0x7f655c150c10 legacy=0x7f655c072d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.801+0000 7f6561000640 1 -- 172.21.15.146:0/3726587371 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c06e190 con 0x7f655c0c9650 2024-09-16T01:51:54.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.802+0000 7f6562400640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c072410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:54.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.804+0000 7f6561000640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c0c9650 0x7f655c071ed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6550009870 tx=0x7f655002a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ffe097e5e4c3530 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.804+0000 7f655be00640 1 -- 172.21.15.146:0/3726587371 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550037070 con 0x7f655c0c9650 2024-09-16T01:51:54.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.804+0000 7f655be00640 1 -- 172.21.15.146:0/3726587371 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655002ae50 con 0x7f655c0c9650 2024-09-16T01:51:54.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.804+0000 7f655be00640 1 -- 172.21.15.146:0/3726587371 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550032860 con 0x7f655c0c9650 2024-09-16T01:51:54.807 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:54.808 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:54.808 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:51:54.808 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:54.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.805+0000 7f6564400640 1 -- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c0c9650 msgr2=0x7f655c071ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.805+0000 7f6564400640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c0c9650 0x7f655c071ed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6550009870 tx=0x7f655002a300 comp rx=0 tx=0).stop 2024-09-16T01:51:54.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.806+0000 7f6564400640 1 -- 172.21.15.146:0/3726587371 shutdown_connections 2024-09-16T01:51:54.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.806+0000 7f6564400640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c072410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.806+0000 7f6564400640 1 --2- 172.21.15.146:0/3726587371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c0c9650 0x7f655c071ed0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.806+0000 7f6564400640 1 -- 172.21.15.146:0/3726587371 >> 172.21.15.146:0/3726587371 conn(0x7f655c06f3c0 msgr2=0x7f655c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:54.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.807+0000 7f6564400640 1 -- 172.21.15.146:0/3726587371 shutdown_connections 2024-09-16T01:51:54.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.807+0000 7f6564400640 1 -- 172.21.15.146:0/3726587371 wait complete. 2024-09-16T01:51:54.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.810+0000 7f6564400640 1 Processor -- start 2024-09-16T01:51:54.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.810+0000 7f6564400640 1 -- start start 2024-09-16T01:51:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.811+0000 7f6564400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c1f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.811+0000 7f6564400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f655c0c9650 0x7f655c14eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.811+0000 7f6564400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c150c10 0x7f655c0bd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.811+0000 7f6564400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f655c072a10 con 0x7f655c0c9650 2024-09-16T01:51:54.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.811+0000 7f6564400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f655c06e750 con 0x7f655c150c10 2024-09-16T01:51:54.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.811+0000 7f6564400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f655c156e60 con 0x7f655c0718c0 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c150c10 0x7f655c0bd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c150c10 0x7f655c0bd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50472/0 (socket says 172.21.15.146:50472) 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 -- 172.21.15.146:0/3904806017 learned_addr learned my addr 172.21.15.146:0/3904806017 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 -- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 msgr2=0x7f655c1f16c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f655c0718c0 0x7f655c1f16c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 -- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f655c0c9650 msgr2=0x7f655c14eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f655c0c9650 0x7f655c14eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.812+0000 7f6562400640 1 -- 172.21.15.146:0/3904806017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c06fd60 con 0x7f655c150c10 2024-09-16T01:51:54.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.813+0000 7f6562400640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c150c10 0x7f655c0bd980 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f655400cef0 tx=0x7f655400a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.813+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655401a070 con 0x7f655c150c10 2024-09-16T01:51:54.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.813+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655400d860 con 0x7f655c150c10 2024-09-16T01:51:54.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.813+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655400dc50 con 0x7f655c150c10 2024-09-16T01:51:54.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.814+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c1543d0 con 0x7f655c150c10 2024-09-16T01:51:54.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.814+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c154d70 con 0x7f655c150c10 2024-09-16T01:51:54.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.817+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6538001030 con 0x7f655c150c10 2024-09-16T01:51:54.821 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:51:54.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.819+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f655401d070 con 0x7f655c150c10 2024-09-16T01:51:54.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.821+0000 7f6559600640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f653c06c330 0x7f653c06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.822+0000 7f6561a00640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f653c06c330 0x7f653c06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.823+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f655408be10 con 0x7f655c150c10 2024-09-16T01:51:54.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.823+0000 7f6561a00640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f653c06c330 0x7f653c06e7f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6548001440 tx=0x7f65480079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.823+0000 7f6559600640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f653c0745f0 0x7f653c076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:54.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.823+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f653c077100 con 0x7f653c0745f0 2024-09-16T01:51:54.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.824+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f6554055e60 con 0x7f655c150c10 2024-09-16T01:51:54.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.824+0000 7f6561000640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f653c0745f0 0x7f653c076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:54.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.829+0000 7f6561000640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f653c0745f0 0x7f653c076a30 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:54.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.836+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f653c077100 con 0x7f653c0745f0 2024-09-16T01:51:54.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.864+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f6538002dd0 con 0x7f653c0745f0 2024-09-16T01:51:54.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.864+0000 7f6559600640 1 -- 172.21.15.146:0/3904806017 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f6538002dd0 con 0x7f653c0745f0 2024-09-16T01:51:54.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.865+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f653c0745f0 msgr2=0x7f653c076a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.865+0000 7f6564400640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f653c0745f0 0x7f653c076a30 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.865+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f653c06c330 msgr2=0x7f653c06e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.865+0000 7f6564400640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f653c06c330 0x7f653c06e7f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6548001440 tx=0x7f65480079d0 comp rx=0 tx=0).stop 2024-09-16T01:51:54.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c150c10 msgr2=0x7f655c0bd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:54.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6564400640 1 --2- 172.21.15.146:0/3904806017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f655c150c10 0x7f655c0bd980 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f655400cef0 tx=0x7f655400a710 comp rx=0 tx=0).stop 2024-09-16T01:51:54.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6562400640 1 -- 172.21.15.146:0/3904806017 reap_dead start 2024-09-16T01:51:54.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 shutdown_connections 2024-09-16T01:51:54.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 >> 172.21.15.146:0/3904806017 conn(0x7f655c06f3c0 msgr2=0x7f655c0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:54.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 shutdown_connections 2024-09-16T01:51:54.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:54.866+0000 7f6564400640 1 -- 172.21.15.146:0/3904806017 wait complete. 2024-09-16T01:51:54.883 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:54.882+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:54.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-16T01:51:54.944 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:54.944 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:54.944 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:54.944 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:54.944 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:54.957 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:51:54.984 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:54.982+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:55.084 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:55.083+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 166.25006580300001, 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.113 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:51:55.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:51:55.118 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 166.24996729599999, 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:51:55.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:51:55.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.124 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:17 3.21 e55/41 3:86f42aac:::10000000002.0000000e:head v 55'1, mlcod=0'0)", 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 64.733673370000005, 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011486753000000001, 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376928+0000", 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0950000000000001e-06 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385558+0000", 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086297260000000008 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385576+0000", 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.806e-05 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385587+0000", 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1165999999999999e-05 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385660+0000", 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2346000000000002e-05 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385679+0000", 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9038999999999998e-05 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388352+0000", 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026728210000000001 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388414+0000", 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2500000000000001e-05 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:20 3.21 e55/41 3:85380590:::10000000002.00000011:head v 55'2, mlcod=0'0)", 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 64.660031235999995, 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011038759, 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.129 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.130 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.130 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450571+0000", 2024-09-16T01:51:55.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.111e-06 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458973+0000", 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0084022200000000002 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458993+0000", 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9945e-05 2024-09-16T01:51:55.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459003+0000", 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9860000000000006e-06 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459070+0000", 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7600000000000003e-05 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459092+0000", 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1860000000000001e-05 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.461569+0000", 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024764679999999999 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.461608+0000", 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9569e-05 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.134 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:62 3.c e55/35 3:31d8a57c:::10000000002.0000003b:head v 55'1, mlcod=0'0)", 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 64.271489254000002, 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011246087, 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839113+0000", 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.167e-06 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847457+0000", 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0083445559999999995 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847465+0000", 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7719999999999994e-06 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847469+0000", 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0820000000000001e-06 2024-09-16T01:51:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847509+0000", 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847524+0000", 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.850313+0000", 2024-09-16T01:51:55.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027888370000000002 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.850358+0000", 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4409000000000001e-05 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:772 3.21 e68/67 3:848ebb0b:::100000001f7.00000001:head v 68'21, mlcod=65'19)", 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 34.839205585000002, 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025350340999999998, 2024-09-16T01:51:55.139 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271396+0000", 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.144e-06 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293800+0000", 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022404103000000002 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293826+0000", 2024-09-16T01:51:55.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5809999999999999e-05 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293845+0000", 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8394e-05 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293958+0000", 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011299899999999999 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293987+0000", 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8952999999999999e-05 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.296701+0000", 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027147180000000001 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.296745+0000", 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.422e-05 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.143 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1085 3.c e76/67 3:33b8095f:::100000001f8.0000003a:head v 76'25, mlcod=74'23)", 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.234625012999999, 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013510061, 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.144 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:51:55.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8799999999999999e-07 2024-09-16T01:51:55.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885660+0000", 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0096844449999999999 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885680+0000", 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9292000000000001e-05 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885689+0000", 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7920000000000004e-06 2024-09-16T01:51:55.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885767+0000", 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7769999999999996e-05 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885790+0000", 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2697999999999999e-05 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.889443+0000", 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036530579999999998 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.889486+0000", 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2818000000000002e-05 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1088 3.2e e76/67 3:75be0f36:::100000001f8.0000003d:head v 76'22, mlcod=74'20)", 2024-09-16T01:51:55.151 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.194474524, 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010871123999999999, 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6399999999999998e-07 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923588+0000", 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.007461479 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923612+0000", 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3785999999999999e-05 2024-09-16T01:51:55.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923625+0000", 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3166999999999999e-05 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923698+0000", 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3603000000000007e-05 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923725+0000", 2024-09-16T01:51:55.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6622999999999999e-05 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.926947+0000", 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003222267 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.926997+0000", 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9935000000000002e-05 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.156 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1134 3.21 e76/67 3:87440b43:::100000001f8.0000006b:head v 76'37, mlcod=74'35)", 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.685334273, 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016188002, 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:51:55.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425267+0000", 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0200000000000001e-07 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438526+0000", 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013259675 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438543+0000", 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7163000000000002e-05 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438555+0000", 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1928e-05 2024-09-16T01:51:55.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438633+0000", 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7935999999999994e-05 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438653+0000", 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9695999999999999e-05 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.441400+0000", 2024-09-16T01:51:55.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027465649999999999 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.441454+0000", 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4836999999999997e-05 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1511 3.8 e81/67 3:131e2389:::100000001f8.000001e4:head v 81'52, mlcod=79'50)", 2024-09-16T01:51:55.161 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.074176+0000", 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.036424528, 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010211288000000001, 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.074176+0000", 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.074176+0000", 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.074176+0000", 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7499999999999999e-07 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.075440+0000", 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0012640310000000001 2024-09-16T01:51:55.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.075460+0000", 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9729e-05 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.075475+0000", 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4484000000000001e-05 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.075552+0000", 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6934999999999994e-05 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.075571+0000", 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9916e-05 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.084325+0000", 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0087531499999999995 2024-09-16T01:51:55.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.084387+0000", 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2867999999999995e-05 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1547 3.21 e81/67 3:86681322:::100000001f8.00000208:head v 81'45, mlcod=80'43)", 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475134+0000", 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.635466427000001, 2024-09-16T01:51:55.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.009723275, 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475134+0000", 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475134+0000", 2024-09-16T01:51:55.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475136+0000", 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4699999999999999e-06 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.481753+0000", 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0066170949999999999 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.481767+0000", 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3833e-05 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.481777+0000", 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0539e-05 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.481851+0000", 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3349999999999994e-05 2024-09-16T01:51:55.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.481873+0000", 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2622000000000001e-05 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.484803+0000", 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002929957 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.484858+0000", 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4409e-05 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1660 3.2e e81/67 3:766865b4:::100000001f8.00000279:head v 81'29, mlcod=79'27)", 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.789056+0000", 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.3215448609999996, 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010627068999999999, 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.171 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.789056+0000", 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.789056+0000", 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.789057+0000", 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0559999999999999e-06 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.796238+0000", 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0071809919999999998 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.796260+0000", 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2059999999999999e-05 2024-09-16T01:51:55.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.796268+0000", 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8800000000000008e-06 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.796336+0000", 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7960999999999995e-05 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.796362+0000", 2024-09-16T01:51:55.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5987e-05 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.799621+0000", 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0032589160000000002 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.799683+0000", 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2217000000000004e-05 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.175 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1678 3.21 e81/67 3:8795a5a4:::100000001f8.0000028b:head v 81'47, mlcod=81'45)", 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.1022190569999992, 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011676199, 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008383+0000", 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6500000000000008e-07 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015309+0000", 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0069268530000000002 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015320+0000", 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0733999999999999e-05 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015326+0000", 2024-09-16T01:51:55.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3279999999999996e-06 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015402+0000", 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6369e-05 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015426+0000", 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3966000000000001e-05 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.020011+0000", 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045850730000000003 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.020058+0000", 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6910999999999999e-05 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1691 3.c e81/67 3:320b9a73:::100000001f8.00000298:head v 81'34, mlcod=81'32)", 2024-09-16T01:51:55.180 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:46.158698+0000", 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 8.9519025330000002, 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010336523, 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.158698+0000", 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.158698+0000", 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.158699+0000", 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2240000000000001e-06 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.166140+0000", 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074405900000000004 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.166156+0000", 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5837000000000001e-05 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.166167+0000", 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0842999999999999e-05 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.166233+0000", 2024-09-16T01:51:55.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6647000000000001e-05 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.166255+0000", 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999999e-05 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.168992+0000", 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027363280000000001 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.169035+0000", 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2864000000000002e-05 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1927 3.8 e84/83 3:13b44a54:::100000001f8.00000384:head v 84'56, mlcod=0'0)", 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.9398047089999997, 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011956664000000001, 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.185 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.85e-07 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179466+0000", 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086694140000000003 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179476+0000", 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0341e-05 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179482+0000", 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.353e-06 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179532+0000", 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9956000000000001e-05 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179556+0000", 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3456e-05 2024-09-16T01:51:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.182704+0000", 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031482120000000001 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.182753+0000", 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8746999999999999e-05 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1947 3.2c e84/77 3:3557c0c2:::100000001f8.00000398:head v 84'41, mlcod=81'39)", 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.8229738170000003, 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016887692999999999, 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287629+0000", 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300794+0000", 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013164334999999999 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300803+0000", 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2520000000000005e-06 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300809+0000", 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2790000000000004e-06 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300867+0000", 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7485000000000003e-05 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300890+0000", 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.372e-05 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.304472+0000", 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035818379999999999 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.304515+0000", 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2311999999999997e-05 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2030 3.c e85/67 3:3025d64f:::100000001f8.000003eb:head v 85'41, mlcod=84'39)", 2024-09-16T01:51:55.194 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.437175+0000", 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.6734251990000004, 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010168748, 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.437175+0000", 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.437175+0000", 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.437176+0000", 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5600000000000005e-07 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.444480+0000", 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.007303744 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.444491+0000", 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1243999999999999e-05 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.444498+0000", 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2509999999999997e-06 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.444548+0000", 2024-09-16T01:51:55.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9302000000000003e-05 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.444563+0000", 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5150000000000001e-05 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.447306+0000", 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002742679 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.447344+0000", 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8621999999999997e-05 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2059 3.6 e85/83 3:6235bc65:::100000001f8.00000408:head v 85'44, mlcod=60'4)", 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.2783979089999997, 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010790019, 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.199 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836745+0000", 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045413040000000003 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836759+0000", 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4260999999999999e-05 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836765+0000", 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3249999999999996e-06 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836846+0000", 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0672000000000003e-05 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836866+0000", 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0489e-05 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.842932+0000", 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060658760000000004 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.842993+0000", 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0557000000000003e-05 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.203 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2105 3.21 e86/67 3:84901bec:::100000001f8.00000436:head v 86'54, mlcod=83'52)", 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 3.6152981510000002, 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016103329, 2024-09-16T01:51:55.204 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6400000000000001e-07 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507953+0000", 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012649781000000001 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507969+0000", 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507977+0000", 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8250000000000005e-06 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.508038+0000", 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0900000000000003e-05 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.508061+0000", 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2453999999999999e-05 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.511370+0000", 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003309181 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.511406+0000", 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6056999999999997e-05 2024-09-16T01:51:55.208 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2381 3.34 e88/83 3:2c202722:::100000001f8.0000054a:head v 88'56, mlcod=87'54)", 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.34366743900000002, 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012640500000000001, 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766935+0000", 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.606e-06 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774938+0000", 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080026449999999992 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774955+0000", 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7736000000000001e-05 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774965+0000", 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4930000000000004e-06 2024-09-16T01:51:55.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.775023+0000", 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8697999999999999e-05 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.775047+0000", 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3889000000000001e-05 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.779505+0000", 2024-09-16T01:51:55.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044578710000000004 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.779574+0000", 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8561999999999998e-05 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:51:55.213 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:55.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:51:55.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.156+0000 7fc9cca00640 1 Processor -- start 2024-09-16T01:51:55.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.156+0000 7fc9cca00640 1 -- start start 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9cca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c80c75a0 0x7fc9c815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9cca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9c806d140 con 0x7fc9c80cc040 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9cca00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9c806d420 con 0x7fc9c80c6b70 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9cca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c815b880 con 0x7fc9c80c75a0 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9c6800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc9c80c6b70 0x7fc9c80cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:38956/0 (socket says 172.21.15.146:38956) 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9c6800640 1 -- 172.21.15.146:0/1641922954 learned_addr learned my addr 172.21.15.146:0/1641922954 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.157+0000 7fc9c5e00640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c80c75a0 0x7fc9c815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747828066 0 0) 0x7fc9c806d420 con 0x7fc9c80c6b70 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9ac003520 con 0x7fc9c80c6b70 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 514922920 0 0) 0x7fc9ac003520 con 0x7fc9c80c6b70 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c80c75a0 msgr2=0x7fc9c815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c80c75a0 0x7fc9c815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 >> v1:172.21.15.146:6789/0 conn(0x7fc9c80cc040 legacy=0x7fc9c8165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c806d700 con 0x7fc9c80c6b70 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fc9b8001670 con 0x7fc9c80c6b70 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 >> v1:172.21.15.161:6789/0 conn(0x7fc9c80c6b70 legacy=0x7fc9c80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9ac0039b0 0x7fc9ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9ac004800 0x7fc9ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ac0051c0 0x7fc9ac0141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c815b880 con 0x7fc9ac004800 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac014900 con 0x7fc9ac0051c0 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.158+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9ac014b00 con 0x7fc9ac0039b0 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.159+0000 7fc9c6800640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9ac0039b0 0x7fc9ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.159+0000 7fc9c6800640 1 -- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ac0051c0 msgr2=0x7fc9ac0141d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.159+0000 7fc9c6800640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9ac0051c0 0x7fc9ac0141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.159+0000 7fc9c6800640 1 -- 172.21.15.146:0/1641922954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9ac004800 msgr2=0x7fc9ac004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.159+0000 7fc9c6800640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9ac004800 0x7fc9ac004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.159+0000 7fc9c6800640 1 -- 172.21.15.146:0/1641922954 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c806d700 con 0x7fc9ac0039b0 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.160+0000 7fc9c6800640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9ac0039b0 0x7fc9ac003dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800c240 tx=0x7fc9ac005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bb8c4b5901321a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.160+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8007d30 con 0x7fc9ac0039b0 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.160+0000 7fc9c5400640 1 -- 172.21.15.146:0/1641922954 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b8005ad0 con 0x7fc9ac0039b0 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.161+0000 7fc9cca00640 1 -- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9ac0039b0 msgr2=0x7fc9ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.161+0000 7fc9cca00640 1 --2- 172.21.15.146:0/1641922954 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9ac0039b0 0x7fc9ac003dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800c240 tx=0x7fc9ac005830 comp rx=0 tx=0).stop 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.161+0000 7fc9c7200640 1 -- 172.21.15.146:0/1641922954 reap_dead start 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.161+0000 7fc9cca00640 1 -- 172.21.15.146:0/1641922954 shutdown_connections 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.161+0000 7fc9cca00640 1 -- 172.21.15.146:0/1641922954 >> 172.21.15.146:0/1641922954 conn(0x7fc9c80ccc20 msgr2=0x7fc9c80cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.163+0000 7fc9cca00640 1 -- 172.21.15.146:0/1641922954 shutdown_connections 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.163+0000 7fc9cca00640 1 -- 172.21.15.146:0/1641922954 wait complete. 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.166+0000 7fc9cca00640 1 Processor -- start 2024-09-16T01:51:55.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.166+0000 7fc9cca00640 1 -- start start 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.167+0000 7fc9cca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8153c60 0x7fc9c815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.167+0000 7fc9cca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815eaf0 0x7fc9c815c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.167+0000 7fc9cca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c815ba20 0x7fc9c8150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.167+0000 7fc9cca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c80cb050 con 0x7fc9c8153c60 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.167+0000 7fc9cca00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c80cab20 con 0x7fc9c815ba20 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.167+0000 7fc9cca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c80cbc80 con 0x7fc9c815eaf0 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.168+0000 7fc9c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8153c60 0x7fc9c815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.168+0000 7fc9c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815eaf0 0x7fc9c815c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.168+0000 7fc9c5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815eaf0 0x7fc9c815c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:48898/0 (socket says 172.21.15.146:48898) 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.168+0000 7fc9c5e00640 1 -- 172.21.15.146:0/258852316 learned_addr learned my addr 172.21.15.146:0/258852316 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.168+0000 7fc9c7200640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c815ba20 0x7fc9c8150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c5e00640 1 -- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c815ba20 msgr2=0x7fc9c8150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c5e00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9c815ba20 0x7fc9c8150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c5e00640 1 -- 172.21.15.146:0/258852316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8153c60 msgr2=0x7fc9c815c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c5e00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8153c60 0x7fc9c815c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c5e00640 1 -- 172.21.15.146:0/258852316 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c81ec400 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c6800640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9c8153c60 0x7fc9c815c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.169+0000 7fc9c5e00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815eaf0 0x7fc9c815c900 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400d4d0 tx=0x7fc9b400d990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.170+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40125f0 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.170+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9b4019070 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.170+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40125f0 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.170+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c81ec670 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.171+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c81ecc30 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.173+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9a0001030 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.174+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc9b4012790 con 0x7fc9c815eaf0 2024-09-16T01:51:55.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.175+0000 7fc9bea00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9a806c4b0 0x7fc9a806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.176+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7fc9b4057020 con 0x7fc9c815eaf0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.176+0000 7fc9bea00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9a80752d0 0x7fc9a8077770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.176+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9a8077e40 con 0x7fc9a80752d0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.176+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fc9b40559b0 con 0x7fc9c815eaf0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.176+0000 7fc9c6800640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9a806c4b0 0x7fc9a806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.176+0000 7fc9c7200640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9a80752d0 0x7fc9a8077770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.178+0000 7fc9c7200640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9a80752d0 0x7fc9a8077770 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.178+0000 7fc9c6800640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9a806c4b0 0x7fc9a806e970 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8033000 tx=0x7fc9b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.179+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9a8077e40 con 0x7fc9a80752d0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.205+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fc9a0002dd0 con 0x7fc9a80752d0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.206+0000 7fc9bea00640 1 -- 172.21.15.146:0/258852316 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fc9a0002dd0 con 0x7fc9a80752d0 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.207+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9a80752d0 msgr2=0x7fc9a8077770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.207+0000 7fc9cca00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fc9a80752d0 0x7fc9a8077770 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9a806c4b0 msgr2=0x7fc9a806e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9cca00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc9a806c4b0 0x7fc9a806e970 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8033000 tx=0x7fc9b802e000 comp rx=0 tx=0).stop 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815eaf0 msgr2=0x7fc9c815c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9cca00640 1 --2- 172.21.15.146:0/258852316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc9c815eaf0 0x7fc9c815c900 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400d4d0 tx=0x7fc9b400d990 comp rx=0 tx=0).stop 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9c7200640 1 -- 172.21.15.146:0/258852316 reap_dead start 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 shutdown_connections 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.209+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 >> 172.21.15.146:0/258852316 conn(0x7fc9c80ccc20 msgr2=0x7fc9c815a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.210+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 shutdown_connections 2024-09-16T01:51:55.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.210+0000 7fc9cca00640 1 -- 172.21.15.146:0/258852316 wait complete. 2024-09-16T01:51:55.227 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:55.213+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:55.247 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-16T01:51:55.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:51:55.353 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:55.351+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:55.417 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:55.416+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:55.518 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:55.517+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:55.579 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:55.579 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:55.580 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:55.580 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:55.599 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:51:55.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.600+0000 7f1230800640 1 Processor -- start 2024-09-16T01:51:55.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.600+0000 7f1230800640 1 -- start start 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.601+0000 7f1230800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12280cc0c0 0x7f122815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.601+0000 7f1230800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12280cc4a0 0x7f1228165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.601+0000 7f1230800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122815b1d0 con 0x7f12280cb2f0 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.601+0000 7f1230800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f12280cbd00 con 0x7f12280cc4a0 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.601+0000 7f1230800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f122815ba70 con 0x7f12280cc0c0 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.602+0000 7f122d400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f12280cb2f0 0x7f122815e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46548/0 (socket says 172.21.15.146:46548) 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.602+0000 7f122d400640 1 -- 172.21.15.146:0/1725972 learned_addr learned my addr 172.21.15.146:0/1725972 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2312654295 0 0) 0x7f122815b1d0 con 0x7f12280cb2f0 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1210003520 con 0x7f12280cb2f0 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3634746482 0 0) 0x7f1210003520 con 0x7f12280cb2f0 2024-09-16T01:51:55.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12280cc0c0 msgr2=0x7f122815aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:55.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12280cc0c0 0x7f122815aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 >> v2:172.21.15.146:3300/0 conn(0x7f12280cc4a0 msgr2=0x7f1228165430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:55.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 --2- 172.21.15.146:0/1725972 >> v2:172.21.15.146:3300/0 conn(0x7f12280cc4a0 0x7f1228165430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122815b4b0 con 0x7f12280cb2f0 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f121c001670 con 0x7f12280cb2f0 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 >> v1:172.21.15.146:6789/0 conn(0x7f12280cb2f0 legacy=0x7f122815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12100039b0 0x7f1210003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1210004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.603+0000 7f122ca00640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12100052e0 0x7f12100141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.604+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f12280cbd00 con 0x7f1210004800 2024-09-16T01:51:55.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.604+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f122815ba70 con 0x7f12100039b0 2024-09-16T01:51:55.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.604+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f12100149d0 con 0x7f12100052e0 2024-09-16T01:51:55.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.604+0000 7f122d400640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1210004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.605+0000 7f122e800640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12100052e0 0x7f12100141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.606+0000 7f122d400640 1 -- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12100052e0 msgr2=0x7f12100141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.606+0000 7f122d400640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12100052e0 0x7f12100141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.606+0000 7f122d400640 1 -- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12100039b0 msgr2=0x7f1210003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.606+0000 7f122d400640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12100039b0 0x7f1210003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.606+0000 7f122d400640 1 -- 172.21.15.146:0/1725972 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122815b4b0 con 0x7f1210004800 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.606+0000 7f122e800640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12100052e0 0x7f12100141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.608+0000 7f122d400640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1210004c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c340 tx=0x7f1210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949e7e44106fafa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.609+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c036070 con 0x7f1210004800 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.609+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121c005ad0 con 0x7f1210004800 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.609+0000 7f122ca00640 1 -- 172.21.15.146:0/1725972 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c007d90 con 0x7f1210004800 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.610+0000 7f1230800640 1 -- 172.21.15.146:0/1725972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 msgr2=0x7f1210004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.610+0000 7f1230800640 1 --2- 172.21.15.146:0/1725972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1210004c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c340 tx=0x7f1210005830 comp rx=0 tx=0).stop 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.610+0000 7f122e800640 1 -- 172.21.15.146:0/1725972 reap_dead start 2024-09-16T01:51:55.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.611+0000 7f1230800640 1 -- 172.21.15.146:0/1725972 shutdown_connections 2024-09-16T01:51:55.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.611+0000 7f1230800640 1 -- 172.21.15.146:0/1725972 >> 172.21.15.146:0/1725972 conn(0x7f122806d560 msgr2=0x7f122806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:55.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.611+0000 7f1230800640 1 -- 172.21.15.146:0/1725972 shutdown_connections 2024-09-16T01:51:55.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.611+0000 7f1230800640 1 -- 172.21.15.146:0/1725972 wait complete. 2024-09-16T01:51:55.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.614+0000 7f1230800640 1 Processor -- start 2024-09-16T01:51:55.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.614+0000 7f1230800640 1 -- start start 2024-09-16T01:51:55.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f1230800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1228159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f1230800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1228150900 0x7f1228159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f1230800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122815d790 0x7f12281efc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f1230800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f122815e1d0 con 0x7f1210004800 2024-09-16T01:51:55.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f1230800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f122815e540 con 0x7f1228150900 2024-09-16T01:51:55.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f1230800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f12280cc8b0 con 0x7f122815d790 2024-09-16T01:51:55.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f122d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1228150900 0x7f1228159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.616+0000 7f122de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1228159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.619 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:55.617+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:55.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1228150900 0x7f1228159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50504/0 (socket says 172.21.15.146:50504) 2024-09-16T01:51:55.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 -- 172.21.15.146:0/3305805452 learned_addr learned my addr 172.21.15.146:0/3305805452 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122e800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122815d790 0x7f12281efc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 -- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122815d790 msgr2=0x7f12281efc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122815d790 0x7f12281efc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 -- 172.21.15.146:0/3305805452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 msgr2=0x7f1228159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1210004800 0x7f1228159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 -- 172.21.15.146:0/3305805452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122814f7b0 con 0x7f1228150900 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122e800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122815d790 0x7f12281efc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:55.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.617+0000 7f122d400640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1228150900 0x7f1228159230 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c340 tx=0x7f121c001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.620+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c041020 con 0x7f1228150900 2024-09-16T01:51:55.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.620+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121c0072f0 con 0x7f1228150900 2024-09-16T01:51:55.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.620+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c036040 con 0x7f1228150900 2024-09-16T01:51:55.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.620+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122814f510 con 0x7f1228150900 2024-09-16T01:51:55.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.620+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1228158d10 con 0x7f1228150900 2024-09-16T01:51:55.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.623+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f121c044020 con 0x7f1228150900 2024-09-16T01:51:55.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.624+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f12280c5560 con 0x7f1228150900 2024-09-16T01:51:55.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.625+0000 7f1222000640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120c06c380 0x7f120c06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.626+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f121c0ab3b0 con 0x7f1228150900 2024-09-16T01:51:55.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.626+0000 7f1222000640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f120c0745e0 0x7f120c076a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.626+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f121c04aec0 con 0x7f120c0745e0 2024-09-16T01:51:55.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.626+0000 7f122de00640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120c06c380 0x7f120c06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.627+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f121c0b2050 con 0x7f1228150900 2024-09-16T01:51:55.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.627+0000 7f122e800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f120c0745e0 0x7f120c076a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.629+0000 7f122de00640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120c06c380 0x7f120c06e840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1218006fd0 tx=0x7f1218008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.629+0000 7f122e800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f120c0745e0 0x7f120c076a20 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.630+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f121c04aec0 con 0x7f120c0745e0 2024-09-16T01:51:55.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.679+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f12280d12e0 con 0x7f120c0745e0 2024-09-16T01:51:55.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.694+0000 7f1222000640 1 -- 172.21.15.146:0/3305805452 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f12280d12e0 con 0x7f120c0745e0 2024-09-16T01:51:55.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f120c0745e0 msgr2=0x7f120c076a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f120c0745e0 0x7f120c076a20 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120c06c380 msgr2=0x7f120c06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120c06c380 0x7f120c06e840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1218006fd0 tx=0x7f1218008040 comp rx=0 tx=0).stop 2024-09-16T01:51:55.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1228150900 msgr2=0x7f1228159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 --2- 172.21.15.146:0/3305805452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1228150900 0x7f1228159230 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f121c00c340 tx=0x7f121c001970 comp rx=0 tx=0).stop 2024-09-16T01:51:55.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f122e800640 1 -- 172.21.15.146:0/3305805452 reap_dead start 2024-09-16T01:51:55.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 shutdown_connections 2024-09-16T01:51:55.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.701+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 >> 172.21.15.146:0/3305805452 conn(0x7f122806d560 msgr2=0x7f12281539f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:55.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.705+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 shutdown_connections 2024-09-16T01:51:55.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.705+0000 7f1230800640 1 -- 172.21.15.146:0/3305805452 wait complete. 2024-09-16T01:51:55.719 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:55.718+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:55.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-16T01:51:55.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.764+0000 7f54c7600640 1 Processor -- start 2024-09-16T01:51:55.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.765+0000 7f54c7600640 1 -- start start 2024-09-16T01:51:55.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.767+0000 7f54c7600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 0x7f54b8045e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.767+0000 7f54c7600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54b80e3100 0x7f54b804aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.767+0000 7f54c7600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54b80e3570 con 0x7f54b804a6b0 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.767+0000 7f54c7600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f54b8045980 con 0x7f54b80e3100 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.767+0000 7f54c7600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f54b80dd1a0 con 0x7f54b80dcdc0 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.772+0000 7f54c4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 0x7f54b8045e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.772+0000 7f54c4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 0x7f54b8045e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:48926/0 (socket says 172.21.15.146:48926) 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.772+0000 7f54c4c00640 1 -- 172.21.15.146:0/1807069830 learned_addr learned my addr 172.21.15.146:0/1807069830 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.772+0000 7f54c5600640 1 --2- 172.21.15.146:0/1807069830 >> v2:172.21.15.146:3300/0 conn(0x7f54b80e3100 0x7f54b804aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.773+0000 7f54c4c00640 1 -- 172.21.15.146:0/1807069830 >> v2:172.21.15.146:3300/0 conn(0x7f54b80e3100 msgr2=0x7f54b804aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.776+0000 7f54c4c00640 1 --2- 172.21.15.146:0/1807069830 >> v2:172.21.15.146:3300/0 conn(0x7f54b80e3100 0x7f54b804aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.776+0000 7f54c4c00640 1 -- 172.21.15.146:0/1807069830 >> v1:172.21.15.161:6789/0 conn(0x7f54b804a6b0 legacy=0x7f54b80e29f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:55.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.776+0000 7f54c4c00640 1 -- 172.21.15.146:0/1807069830 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b80e3820 con 0x7f54b80dcdc0 2024-09-16T01:51:55.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.777+0000 7f54c5600640 1 --2- 172.21.15.146:0/1807069830 >> v2:172.21.15.146:3300/0 conn(0x7f54b80e3100 0x7f54b804aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:55.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.777+0000 7f54c4c00640 1 --2- 172.21.15.146:0/1807069830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 0x7f54b8045e10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009870 tx=0x7f54b402a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d283ccf84c0220a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.781+0000 7f54bf400640 1 -- 172.21.15.146:0/1807069830 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4037070 con 0x7f54b80dcdc0 2024-09-16T01:51:55.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.781+0000 7f54bf400640 1 -- 172.21.15.146:0/1807069830 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b4032440 con 0x7f54b80dcdc0 2024-09-16T01:51:55.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.781+0000 7f54bf400640 1 -- 172.21.15.146:0/1807069830 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4037070 con 0x7f54b80dcdc0 2024-09-16T01:51:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 -- 172.21.15.146:0/1807069830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 msgr2=0x7f54b8045e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 --2- 172.21.15.146:0/1807069830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 0x7f54b8045e10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009870 tx=0x7f54b402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 -- 172.21.15.146:0/1807069830 shutdown_connections 2024-09-16T01:51:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 --2- 172.21.15.146:0/1807069830 >> v2:172.21.15.146:3300/0 conn(0x7f54b80e3100 0x7f54b804aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 --2- 172.21.15.146:0/1807069830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80dcdc0 0x7f54b8045e10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 -- 172.21.15.146:0/1807069830 >> 172.21.15.146:0/1807069830 conn(0x7f54b804fc80 msgr2=0x7f54b8050090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:55.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 -- 172.21.15.146:0/1807069830 shutdown_connections 2024-09-16T01:51:55.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.782+0000 7f54c7600640 1 -- 172.21.15.146:0/1807069830 wait complete. 2024-09-16T01:51:55.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.789+0000 7f54c7600640 1 Processor -- start 2024-09-16T01:51:55.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.789+0000 7f54c7600640 1 -- start start 2024-09-16T01:51:55.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.790+0000 7f54c7600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b804a6b0 0x7f54b80e1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.794+0000 7f54c7600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80dcdc0 0x7f54b80deaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.795+0000 7f54c7600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 0x7f54b80defe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.798+0000 7f54c7600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8045330 con 0x7f54b80dcdc0 2024-09-16T01:51:55.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.798+0000 7f54c7600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8045050 con 0x7f54b804a6b0 2024-09-16T01:51:55.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.798+0000 7f54c7600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f54b8045610 con 0x7f54b80e3100 2024-09-16T01:51:55.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.798+0000 7f54bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80dcdc0 0x7f54b80deaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.799+0000 7f54c5600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 0x7f54b80defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.799+0000 7f54c5600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 0x7f54b80defe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:48942/0 (socket says 172.21.15.146:48942) 2024-09-16T01:51:55.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.799+0000 7f54c5600640 1 -- 172.21.15.146:0/1744413645 learned_addr learned my addr 172.21.15.146:0/1744413645 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:55.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54c5600640 1 -- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b804a6b0 msgr2=0x7f54b80e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54c5600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b804a6b0 0x7f54b80e1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54c5600640 1 -- 172.21.15.146:0/1744413645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80dcdc0 msgr2=0x7f54b80deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:55.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54c5600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80dcdc0 0x7f54b80deaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:55.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54c5600640 1 -- 172.21.15.146:0/1744413645 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b8178cd0 con 0x7f54b80e3100 2024-09-16T01:51:55.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54bfe00640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80dcdc0 0x7f54b80deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:55.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.803+0000 7f54c5600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 0x7f54b80defe0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f54c006a930 tx=0x7f54c0069890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.804+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c00728b0 con 0x7f54b80e3100 2024-09-16T01:51:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.804+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c0072a50 con 0x7f54b80e3100 2024-09-16T01:51:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.804+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0072e60 con 0x7f54b80e3100 2024-09-16T01:51:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.804+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b80e28d0 con 0x7f54b80e3100 2024-09-16T01:51:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.805+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b80dfc70 con 0x7f54b80e3100 2024-09-16T01:51:55.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.819+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f54c0095480 con 0x7f54b80e3100 2024-09-16T01:51:55.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.824+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c005290 con 0x7f54b80e3100 2024-09-16T01:51:55.830 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:55.824+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:55.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.827+0000 7f54bcc00640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f54a006c380 0x7f54a006e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:55.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.829+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f54c0107af0 con 0x7f54b80e3100 2024-09-16T01:51:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.831+0000 7f54c4c00640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f54a006c380 0x7f54a006e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:55.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.833+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f54c00d62d0 con 0x7f54b80e3100 2024-09-16T01:51:55.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:55.834+0000 7f54c4c00640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f54a006c380 0x7f54a006e840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f54b4034000 tx=0x7f54b402c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:55.926 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:55.924+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:56.007 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:56.008 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:56.008 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:51:56.008 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:51:56.008 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:56.022 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:56.021+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:56.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:51:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.071+0000 7f4a61600640 1 Processor -- start 2024-09-16T01:51:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.072+0000 7f4a61600640 1 -- start start 2024-09-16T01:51:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.073+0000 7f4a61600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4a5c0cb2f0 0x7f4a5c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.073+0000 7f4a61600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cc4a0 0x7f4a5c165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.073+0000 7f4a61600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c15b1d0 con 0x7f4a5c0cc0c0 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.073+0000 7f4a61600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4a5c0cbd00 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.073+0000 7f4a61600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5c15ba70 con 0x7f4a5c0cc4a0 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.074+0000 7f4a5a000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4a5c0cb2f0 0x7f4a5c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.074+0000 7f4a5a000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4a5c0cb2f0 0x7f4a5c15aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50534/0 (socket says 172.21.15.146:50534) 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.074+0000 7f4a5a000640 1 -- 172.21.15.146:0/3728558203 learned_addr learned my addr 172.21.15.146:0/3728558203 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.074+0000 7f4a5b400640 1 --2- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cc4a0 0x7f4a5c165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.075+0000 7f4a5a000640 1 -- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cc4a0 msgr2=0x7f4a5c165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:56.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.075+0000 7f4a5a000640 1 --2- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cc4a0 0x7f4a5c165430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.075+0000 7f4a5a000640 1 -- 172.21.15.146:0/3728558203 >> v1:172.21.15.146:6789/0 conn(0x7f4a5c0cc0c0 legacy=0x7f4a5c15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:56.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.076+0000 7f4a5a000640 1 -- 172.21.15.146:0/3728558203 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c15b4b0 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.078+0000 7f4a5a000640 1 --2- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15aab0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009870 tx=0x7f4a4402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bff7ac293ab3d299 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.078+0000 7f4a59600640 1 -- 172.21.15.146:0/3728558203 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44037070 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.078+0000 7f4a59600640 1 -- 172.21.15.146:0/3728558203 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a44032440 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.078+0000 7f4a59600640 1 -- 172.21.15.146:0/3728558203 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44032810 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.079+0000 7f4a61600640 1 -- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cb2f0 msgr2=0x7f4a5c15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.080+0000 7f4a61600640 1 --2- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15aab0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009870 tx=0x7f4a4402a250 comp rx=0 tx=0).stop 2024-09-16T01:51:56.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.080+0000 7f4a61600640 1 -- 172.21.15.146:0/3728558203 shutdown_connections 2024-09-16T01:51:56.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.080+0000 7f4a61600640 1 --2- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cc4a0 0x7f4a5c165430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.080+0000 7f4a61600640 1 --2- 172.21.15.146:0/3728558203 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15aab0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.080+0000 7f4a61600640 1 -- 172.21.15.146:0/3728558203 >> 172.21.15.146:0/3728558203 conn(0x7f4a5c06d560 msgr2=0x7f4a5c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:56.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.080+0000 7f4a61600640 1 -- 172.21.15.146:0/3728558203 shutdown_connections 2024-09-16T01:51:56.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.081+0000 7f4a61600640 1 -- 172.21.15.146:0/3728558203 wait complete. 2024-09-16T01:51:56.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.083+0000 7f4a61600640 1 Processor -- start 2024-09-16T01:51:56.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.083+0000 7f4a61600640 1 -- start start 2024-09-16T01:51:56.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.084+0000 7f4a61600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.084+0000 7f4a61600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5c0cc0c0 0x7f4a5c1f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.084+0000 7f4a61600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cc4a0 0x7f4a5c0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.084+0000 7f4a61600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c15e1d0 con 0x7f4a5c0cc0c0 2024-09-16T01:51:56.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.084+0000 7f4a61600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c15e540 con 0x7f4a5c0cc4a0 2024-09-16T01:51:56.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.084+0000 7f4a61600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5c0cc8f0 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:48962/0 (socket says 172.21.15.146:48962) 2024-09-16T01:51:56.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 -- 172.21.15.146:0/24721515 learned_addr learned my addr 172.21.15.146:0/24721515 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:56.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5a000640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5c0cc0c0 0x7f4a5c1f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5b400640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cc4a0 0x7f4a5c0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 -- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cc4a0 msgr2=0x7f4a5c0b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cc4a0 0x7f4a5c0b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 -- 172.21.15.146:0/24721515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5c0cc0c0 msgr2=0x7f4a5c1f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5c0cc0c0 0x7f4a5c1f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5aa00640 1 -- 172.21.15.146:0/24721515 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c1593b0 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5a000640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5c0cc0c0 0x7f4a5c1f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.085+0000 7f4a5b400640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a5c0cc4a0 0x7f4a5c0b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.086+0000 7f4a5aa00640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15a440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00c480 tx=0x7f4a4c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.086+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c0052d0 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.086+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a4c005470 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.086+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c012870 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.086+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5c158b10 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.086+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5c1590d0 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.087+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a5c0c5560 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.089+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4a4c012a10 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.091+0000 7f4a52a00640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3806c2b0 0x7f4a3806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.091+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f4a4c08b130 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.091+0000 7f4a52a00640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a38074570 0x7f4a380769b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.091+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a4c014e50 con 0x7f4a38074570 2024-09-16T01:51:56.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.091+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f4a4c092050 con 0x7f4a5c0cb2f0 2024-09-16T01:51:56.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.094+0000 7f4a5b400640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a38074570 0x7f4a380769b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.094+0000 7f4a5a000640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3806c2b0 0x7f4a3806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.095+0000 7f4a5a000640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3806c2b0 0x7f4a3806e770 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4a44034000 tx=0x7f4a4402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.099+0000 7f4a5b400640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a38074570 0x7f4a380769b0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.103+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a4c014e50 con 0x7f4a38074570 2024-09-16T01:51:56.128 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:56.124+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:56.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.145+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f4a5c0b5840 con 0x7f4a38074570 2024-09-16T01:51:56.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.146+0000 7f4a52a00640 1 -- 172.21.15.146:0/24721515 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f4a5c0b5840 con 0x7f4a38074570 2024-09-16T01:51:56.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.149+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a38074570 msgr2=0x7f4a380769b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.149+0000 7f4a61600640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f4a38074570 0x7f4a380769b0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3806c2b0 msgr2=0x7f4a3806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a61600640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4a3806c2b0 0x7f4a3806e770 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4a44034000 tx=0x7f4a4402e000 comp rx=0 tx=0).stop 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cb2f0 msgr2=0x7f4a5c15a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a61600640 1 --2- 172.21.15.146:0/24721515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4a5c0cb2f0 0x7f4a5c15a440 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c00c480 tx=0x7f4a4c00c940 comp rx=0 tx=0).stop 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a5b400640 1 -- 172.21.15.146:0/24721515 reap_dead start 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 shutdown_connections 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.150+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 >> 172.21.15.146:0/24721515 conn(0x7f4a5c06d560 msgr2=0x7f4a5c153d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.151+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 shutdown_connections 2024-09-16T01:51:56.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.151+0000 7f4a61600640 1 -- 172.21.15.146:0/24721515 wait complete. 2024-09-16T01:51:56.237 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:56.235+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:56.323 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:51:56.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.247+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f549c005580 con 0x7f54b80e3100 2024-09-16T01:51:56.324 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:56.323+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:56.333 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:51:56.423 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:51:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:51:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:51:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:478 3.c 3:32521e74:::10000000002.000000db:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:51:56.424 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.392686132999998, 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.88174018200000004, 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 478 2024-09-16T01:51:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025090+0000", 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2699999999999998e-07 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047858+0000", 2024-09-16T01:51:56.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022768360000000001 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047892+0000", 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3855e-05 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047918+0000", 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5595000000000001e-05 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048026+0000", 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108242 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048035+0000", 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3940000000000002e-06 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828462+0000", 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.78042635999999999 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831445+0000", 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029837000000000002 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882498+0000", 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051052510000000002 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882597+0000", 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8874000000000002e-05 2024-09-16T01:51:56.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882746+0000", 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149457 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.903108+0000", 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020361516 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906632+0000", 2024-09-16T01:51:56.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035239960000000002 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906720+0000", 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7952000000000001e-05 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906829+0000", 2024-09-16T01:51:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109444 2024-09-16T01:51:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.434 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:777 3.2c 3:36199bb1:::100000001f7.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.205310220999998, 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106547593, 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.435 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 777 2024-09-16T01:51:56.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212467+0000", 2024-09-16T01:51:56.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1399999999999998e-06 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221218+0000", 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087507980000000006 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221238+0000", 2024-09-16T01:51:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9729999999999999e-05 2024-09-16T01:51:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221249+0000", 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1462e-05 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222669+0000", 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001420017 2024-09-16T01:51:56.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222797+0000", 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127572 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222974+0000", 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017753699999999999 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.258753+0000", 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035779068999999997 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318898+0000", 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060144465000000001 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318963+0000", 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5098999999999995e-05 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.319012+0000", 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9703999999999997e-05 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1127 3.36 3:6e2a8853:::100000001f8.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.113581831000001, 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15931991200000001, 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.444 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1127 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304196+0000", 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.526e-06 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354927+0000", 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050730866999999999 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355561+0000", 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063468300000000004 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355571+0000", 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6199999999999994e-06 2024-09-16T01:51:56.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355665+0000", 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3875000000000005e-05 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355819+0000", 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015463900000000001 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:56.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355964+0000", 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144241 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.393080+0000", 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037116647000000003 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463398+0000", 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070317222999999998 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463456+0000", 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8749999999999998e-05 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463513+0000", 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6841e-05 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1218 3.3f 3:ffd44cf4:::100000001f8.000000bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:51:56.450 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.137400790000001, 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.121928352, 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1218 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:51:56.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280376+0000", 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8980000000000001e-06 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360384+0000", 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080007935000000002 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360415+0000", 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1007999999999999e-05 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360429+0000", 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4233000000000001e-05 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360518+0000", 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8465000000000001e-05 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360685+0000", 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016704299999999999 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360842+0000", 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157496 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.379376+0000", 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018533701999999999 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402192+0000", 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022815932000000001 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402254+0000", 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2378000000000005e-05 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402303+0000", 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8262e-05 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1460 3.2a 3:574d3868:::100000001f8.000001b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.031407956000001, 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114279699, 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.456 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1460 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386369+0000", 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1969999999999999e-06 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447453+0000", 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061083239999999997 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447479+0000", 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5941000000000001e-05 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447490+0000", 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1967000000000001e-05 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447627+0000", 2024-09-16T01:51:56.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013671700000000001 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447784+0000", 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015693 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447948+0000", 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016412100000000001 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.467094+0000", 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019145590000000001 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.497455+0000", 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030361091 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.499539+0000", 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020837780000000001 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.500647+0000", 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108127 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1474 3.39 3:9e943776:::100000001f8.000001bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.836094509, 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.116146155, 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.462 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1474 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581683+0000", 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6740000000000001e-06 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639340+0000", 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057656583999999997 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639448+0000", 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107707 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639499+0000", 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1391999999999999e-05 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648061+0000", 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085624870000000006 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648374+0000", 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000312307 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648681+0000", 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030683700000000002 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.693784+0000", 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045103740000000003 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697710+0000", 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039253650000000001 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697780+0000", 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0134000000000003e-05 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697827+0000", 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6928000000000002e-05 2024-09-16T01:51:56.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1921 3.35 3:aefe9194:::100000001f8.0000037e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.857822+0000", 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5599527809999998, 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.086432075999999997, 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1921 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.857822+0000", 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.857822+0000", 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.857825+0000", 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9890000000000001e-06 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.879958+0000", 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022132723 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.879982+0000", 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3952e-05 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.879995+0000", 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3519999999999999e-05 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880085+0000", 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9326999999999994e-05 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880252+0000", 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016694999999999999 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880418+0000", 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016624700000000001 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.891303+0000", 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010885137 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.944151+0000", 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052848322000000003 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.944208+0000", 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6360999999999998e-05 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.944254+0000", 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6548000000000002e-05 2024-09-16T01:51:56.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1925 3.39 3:9ea071c3:::100000001f8.00000382:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5377119009999998, 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40497902499999999, 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1925 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880065+0000", 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7120000000000001e-06 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905598+0000", 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025532966000000001 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905629+0000", 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0797999999999999e-05 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905642+0000", 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905790+0000", 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014842100000000001 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905804+0000", 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3781000000000001e-05 2024-09-16T01:51:56.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184322+0000", 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27851797299999997 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184507+0000", 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185203 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184707+0000", 2024-09-16T01:51:56.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199287 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274277+0000", 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089570446999999997 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.284842+0000", 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010564756999999999 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.284919+0000", 2024-09-16T01:51:56.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6735000000000003e-05 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.285042+0000", 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123626 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1928 3.1b 3:dae8eecd:::100000001f8.00000385:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5085385120000003, 2024-09-16T01:51:56.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44792130200000002, 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1928 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909238+0000", 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.804e-06 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937404+0000", 2024-09-16T01:51:56.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028165758999999999 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937432+0000", 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7718000000000001e-05 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937443+0000", 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0923e-05 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937583+0000", 2024-09-16T01:51:56.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014051800000000001 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937597+0000", 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.367e-05 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321633+0000", 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.38403599100000002 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321793+0000", 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015996000000000001 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321969+0000", 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017598599999999999 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340671+0000", 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018701613999999998 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.356957+0000", 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016286421999999998 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.357041+0000", 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3857999999999996e-05 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.357158+0000", 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011707899999999999 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.486 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1931 3.39 3:9d369e22:::100000001f8.00000388:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.4800056499999998, 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40258944099999999, 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1931 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937771+0000", 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2839999999999999e-06 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961925+0000", 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02415401 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961962+0000", 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6869000000000002e-05 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961975+0000", 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.361e-05 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.962078+0000", 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102604 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.962091+0000", 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3216e-05 2024-09-16T01:51:56.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.194767+0000", 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.23267554400000001 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.194914+0000", 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147501 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.195084+0000", 2024-09-16T01:51:56.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016995000000000001 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274589+0000", 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079504732999999994 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340176+0000", 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065587200999999998 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340241+0000", 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4921999999999999e-05 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340359+0000", 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117997 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1932 3.39 3:9da8c7a0:::100000001f8.00000389:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:51:56.493 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.4501149460000002, 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.37297202400000001, 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1932 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967663+0000", 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4660000000000002e-06 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990244+0000", 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022581615999999999 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990262+0000", 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7672999999999998e-05 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990271+0000", 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8410000000000006e-06 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990369+0000", 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7793000000000004e-05 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990377+0000", 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5040000000000002e-06 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217601+0000", 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22722361699999999 2024-09-16T01:51:56.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217755+0000", 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015434699999999999 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217927+0000", 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017202499999999999 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274595+0000", 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056667704999999999 2024-09-16T01:51:56.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340459+0000", 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065864332999999997 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340504+0000", 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5265000000000001e-05 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340632+0000", 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012783900000000001 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1938 3.39 3:9faf1682:::100000001f8.0000038f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3671278009999996, 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.297091671, 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.500 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1938 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050649+0000", 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6559999999999999e-06 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067265+0000", 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016615959 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067308+0000", 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3154000000000001e-05 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067333+0000", 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5364e-05 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067439+0000", 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106051 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067453+0000", 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3093e-05 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226329+0000", 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15887599599999999 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226492+0000", 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016327299999999999 2024-09-16T01:51:56.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226687+0000", 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019515299999999999 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.323930+0000", 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097242771000000006 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347547+0000", 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023617610000000001 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347630+0000", 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2270999999999999e-05 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347739+0000", 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010932 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1944 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.3019579629999996, 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.23495701299999999, 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1944 2024-09-16T01:51:56.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115819+0000", 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.26e-06 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189079+0000", 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073259493999999994 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189092+0000", 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2648e-05 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189102+0000", 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0377000000000001e-05 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191220+0000", 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021178830000000001 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191347+0000", 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127391 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191478+0000", 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013041699999999999 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.211598+0000", 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020119978 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350671+0000", 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.139073536 2024-09-16T01:51:56.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350725+0000", 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4144000000000002e-05 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350774+0000", 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8884999999999997e-05 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1949 3.2f e84/20 3:f7d79abc:::100000001f8.0000039a:head v 84'35, mlcod=81'33)", 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.189040834, 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107175486, 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228736+0000", 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.522e-06 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230216+0000", 2024-09-16T01:51:56.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001480592 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230232+0000", 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5648999999999999e-05 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230241+0000", 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7749999999999998e-06 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230314+0000", 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3069000000000001e-05 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230342+0000", 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.828e-05 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.335832+0000", 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10549030299999999 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.335910+0000", 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7296e-05 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2041 3.f 3:f3670283:::100000001f8.000003f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.561740+0000", 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.8560352199999999, 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087053971999999993, 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.517 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2041 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.561740+0000", 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.561740+0000", 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.561741+0000", 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5290000000000001e-06 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.603438+0000", 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.04169643 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.603452+0000", 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4579999999999999e-05 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.603461+0000", 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1650000000000006e-06 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.605721+0000", 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00226078 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.605884+0000", 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000162366 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.606046+0000", 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000161872 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.648710+0000", 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042664703999999998 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.648723+0000", 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3191e-05 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.648759+0000", 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5717999999999999e-05 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.648794+0000", 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4637000000000001e-05 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2088 3.3b 3:de25c16a:::100000001f8.00000425:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.191559571, 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.113643656, 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.523 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2088 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226217+0000", 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8899999999999999e-06 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268408+0000", 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042191008000000002 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268436+0000", 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7183999999999999e-05 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268447+0000", 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1233000000000001e-05 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268657+0000", 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020985900000000001 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.269211+0000", 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055385500000000002 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.269780+0000", 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00056933999999999999 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.333074+0000", 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063294230000000007 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339717+0000", 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066431490000000001 2024-09-16T01:51:56.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339798+0000", 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0920000000000005e-05 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339859+0000", 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0988e-05 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2102 3.f 3:f2131ecb:::100000001f8.00000433:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.368652+0000", 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0491226139999998, 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098315575000000002, 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2102 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.529 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.368652+0000", 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.368652+0000", 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.368655+0000", 2024-09-16T01:51:56.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1150000000000001e-06 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.418206+0000", 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049550961999999997 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.418240+0000", 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4436e-05 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.418253+0000", 2024-09-16T01:51:56.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3151e-05 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.418409+0000", 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015614900000000001 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.418571+0000", 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016202799999999999 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.418738+0000", 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016681500000000001 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.456966+0000", 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038227621000000003 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.466842+0000", 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0098758590000000007 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.466915+0000", 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3449999999999996e-05 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.466968+0000", 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2988999999999997e-05 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.534 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2344 3.2a 3:57f71980:::100000001f8.00000525:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:54.326829+0000", 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.090946593, 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089569637999999993, 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2344 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.326829+0000", 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.326829+0000", 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.326830+0000", 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7770000000000001e-06 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.373437+0000", 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046607124 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.373466+0000", 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8428999999999998e-05 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.373478+0000", 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.224e-05 2024-09-16T01:51:56.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.376649+0000", 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031709059999999998 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.376825+0000", 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001763 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.376991+0000", 2024-09-16T01:51:56.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016531400000000001 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.386277+0000", 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0092864539999999995 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.416294+0000", 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030016839 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.416354+0000", 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9805000000000001e-05 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:54.416398+0000", 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4450000000000003e-05 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2445 3.2a 3:566ba4fe:::100000001f8.0000058a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:51:56.540 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.98962478200000004, 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10738498, 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2445 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:51:56.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428152+0000", 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5209999999999999e-06 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455890+0000", 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027737906 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455910+0000", 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0023000000000001e-05 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455920+0000", 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0628999999999999e-05 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455993+0000", 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2358000000000001e-05 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.456334+0000", 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034162400000000001 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.456560+0000", 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022543699999999999 2024-09-16T01:51:56.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.489902+0000", 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033341844000000002 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535444+0000", 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045542250999999999 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535495+0000", 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0973000000000001e-05 2024-09-16T01:51:56.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535535+0000", 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0414000000000003e-05 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2490 3.3f 3:fed847bb:::100000001f8.000005b7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.935689+0000", 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48208607599999997, 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087730112999999998, 2024-09-16T01:51:56.546 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2490 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.935689+0000", 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.935689+0000", 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.935691+0000", 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1730000000000002e-06 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.981435+0000", 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045743542999999998 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.981470+0000", 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5315999999999999e-05 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.981484+0000", 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3903999999999999e-05 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.981574+0000", 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9789000000000004e-05 2024-09-16T01:51:56.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.981743+0000", 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016939200000000001 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.981899+0000", 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015629000000000001 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.011692+0000", 2024-09-16T01:51:56.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029792802 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.023306+0000", 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011614173 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.023367+0000", 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0338999999999999e-05 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.023419+0000", 2024-09-16T01:51:56.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2392000000000003e-05 2024-09-16T01:51:56.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.552 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.552 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:51:56.552 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:51:56.552 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:56.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:51:56.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:51:56.555 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:56.551+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:56.655 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:56.653+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:56.755 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:56.754+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:56.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.835+0000 7f949e600640 1 Processor -- start 2024-09-16T01:51:56.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.835+0000 7f949e600640 1 -- start start 2024-09-16T01:51:56.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.837+0000 7f949e600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9498152a10 0x7f94980c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.837+0000 7f949e600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9498070470 con 0x7f94980c95d0 2024-09-16T01:51:56.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.837+0000 7f949e600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9498070ae0 con 0x7f94980c5140 2024-09-16T01:51:56.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.837+0000 7f949e600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f94980c5540 con 0x7f9498152a10 2024-09-16T01:51:56.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.839+0000 7f9496a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f94980c5140 0x7f94980ce260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:38962/0 (socket says 172.21.15.146:38962) 2024-09-16T01:51:56.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.839+0000 7f9496a00640 1 -- 172.21.15.146:0/3839007879 learned_addr learned my addr 172.21.15.146:0/3839007879 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:51:56.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.840+0000 7f9497400640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9498152a10 0x7f94980c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.840+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 991129830 0 0) 0x7f9498070ae0 con 0x7f94980c5140 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.840+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f947c003520 con 0x7f94980c5140 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.840+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2533035751 0 0) 0x7f9498070470 con 0x7f94980c95d0 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.840+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9498070ae0 con 0x7f94980c95d0 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 187775972 0 0) 0x7f947c003520 con 0x7f94980c5140 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9498152a10 msgr2=0x7f94980c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9498152a10 0x7f94980c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 >> v1:172.21.15.146:6789/0 conn(0x7f94980c95d0 legacy=0x7f94981630b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9498070dc0 con 0x7f94980c5140 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f9484001670 con 0x7f94980c5140 2024-09-16T01:51:56.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 >> v1:172.21.15.161:6789/0 conn(0x7f94980c5140 legacy=0x7f94980ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c003b40 0x7f947c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c004800 0x7f947c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f947c0051c0 0x7f947c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f94980c5540 con 0x7f947c004800 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f947c00bfb0 con 0x7f947c0051c0 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f947c00c180 con 0x7f947c003b40 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9497e00640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f947c0051c0 0x7f947c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.841+0000 7f9496a00640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c004800 0x7f947c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.842+0000 7f9497e00640 1 -- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c003b40 msgr2=0x7f947c003f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.842+0000 7f9497e00640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c003b40 0x7f947c003f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.842+0000 7f9497e00640 1 -- 172.21.15.146:0/3839007879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c004800 msgr2=0x7f947c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.842+0000 7f9497e00640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c004800 0x7f947c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.842+0000 7f9497e00640 1 -- 172.21.15.146:0/3839007879 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9498070dc0 con 0x7f947c0051c0 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.843+0000 7f9497e00640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f947c0051c0 0x7f947c00b9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9490001fe0 tx=0x7f947c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4f75e9864ec2866 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.843+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9490001c00 con 0x7f947c0051c0 2024-09-16T01:51:56.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.843+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9490006d50 con 0x7f947c0051c0 2024-09-16T01:51:56.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.844+0000 7f9496000640 1 -- 172.21.15.146:0/3839007879 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9490001c00 con 0x7f947c0051c0 2024-09-16T01:51:56.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.844+0000 7f949e600640 1 -- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f947c0051c0 msgr2=0x7f947c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.845+0000 7f949e600640 1 --2- 172.21.15.146:0/3839007879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f947c0051c0 0x7f947c00b9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9490001fe0 tx=0x7f947c005830 comp rx=0 tx=0).stop 2024-09-16T01:51:56.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.845+0000 7f9497e00640 1 -- 172.21.15.146:0/3839007879 reap_dead start 2024-09-16T01:51:56.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.845+0000 7f949e600640 1 -- 172.21.15.146:0/3839007879 shutdown_connections 2024-09-16T01:51:56.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.845+0000 7f949e600640 1 -- 172.21.15.146:0/3839007879 >> 172.21.15.146:0/3839007879 conn(0x7f949806c2a0 msgr2=0x7f9498067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:56.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.847+0000 7f949e600640 1 -- 172.21.15.146:0/3839007879 shutdown_connections 2024-09-16T01:51:56.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.850+0000 7f949e600640 1 -- 172.21.15.146:0/3839007879 wait complete. 2024-09-16T01:51:56.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.852+0000 7f949e600640 1 Processor -- start 2024-09-16T01:51:56.855 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:56.855+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:56.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.854+0000 7f949e600640 1 -- start start 2024-09-16T01:51:56.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.858+0000 7f949e600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c0051c0 0x7f949806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.859+0000 7f949e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f949814ed10 0x7f949814f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.859+0000 7f949e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 0x7f9498200c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.860+0000 7f9497e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 0x7f9498200c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.860+0000 7f9497e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 0x7f9498200c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55582/0 (socket says 172.21.15.146:55582) 2024-09-16T01:51:56.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.860+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 learned_addr learned my addr 172.21.15.146:0/68352005 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:56.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.861+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9498070130 con 0x7f94982007a0 2024-09-16T01:51:56.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.861+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c0051c0 0x7f949806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.861+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f94980ca4f0 con 0x7f949814ed10 2024-09-16T01:51:56.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.861+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f94980c86f0 con 0x7f947c0051c0 2024-09-16T01:51:56.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.861+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f949814ed10 0x7f949814f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.862+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c0051c0 msgr2=0x7f949806dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.862+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f947c0051c0 0x7f949806dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.862+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f949814ed10 msgr2=0x7f949814f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:56.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.862+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f949814ed10 0x7f949814f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:56.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.862+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949814e250 con 0x7f94982007a0 2024-09-16T01:51:56.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.865+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f949814ed10 0x7f949814f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:51:56.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.868+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 0x7f9498200c20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f949002aa90 tx=0x7f9490001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.869+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949003e020 con 0x7f94982007a0 2024-09-16T01:51:56.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.869+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9490003e30 con 0x7f94982007a0 2024-09-16T01:51:56.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.872+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9498153270 con 0x7f94982007a0 2024-09-16T01:51:56.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.873+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949814e510 con 0x7f94982007a0 2024-09-16T01:51:56.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.875+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9490001c00 con 0x7f94982007a0 2024-09-16T01:51:56.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.877+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9470005270 con 0x7f94982007a0 2024-09-16T01:51:56.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.888+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9490041020 con 0x7f94982007a0 2024-09-16T01:51:56.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.895+0000 7f948f400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f947806c380 0x7f947806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.897+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f947806c380 0x7f947806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.897+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5770+0+0 (secure 0 0 0) 0x7f94900ab560 con 0x7f94982007a0 2024-09-16T01:51:56.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.901+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v88) ==== 54+0+16 (secure 0 0 0) 0x7f9490048020 con 0x7f94982007a0 2024-09-16T01:51:56.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.902+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f947806c380 0x7f947806e840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9488006fd0 tx=0x7f9488008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.904+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9470000fa0 0x7f94700045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.904+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9470004b20 con 0x7f9470000fa0 2024-09-16T01:51:56.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.911+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9470000fa0 0x7f94700045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.930+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9470000fa0 0x7f94700045e0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.931+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9470004b20 con 0x7f9470000fa0 2024-09-16T01:51:56.957 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:56.955+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:56.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.958+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9470005090 con 0x7f9470000fa0 2024-09-16T01:51:56.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.961+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9470005090 con 0x7f9470000fa0 2024-09-16T01:51:56.963 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:56.963 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:56.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.961+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9470001c70 0x7f9470002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:56.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.961+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f94700022b0 con 0x7f9470001c70 2024-09-16T01:51:56.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.961+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9470001c70 0x7f9470002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:56.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.962+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9470001c70 0x7f9470002990 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:56.966 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:51:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:51:56.966 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:51:56.966 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:51:56.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.967+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94700022b0 con 0x7f9470001c70 2024-09-16T01:51:56.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:51:57.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:56.999+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9470003200 con 0x7f9470001c70 2024-09-16T01:51:57.006 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:57.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.002+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9470003200 con 0x7f9470001c70 2024-09-16T01:51:57.007 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:57.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.004+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9470009cd0 0x7f947000c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:57.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.004+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f947000ee90 con 0x7f9470009cd0 2024-09-16T01:51:57.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.004+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9470009cd0 0x7f947000c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:57.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.005+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9470009cd0 0x7f947000c0c0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:57.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.005+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f947000ee90 con 0x7f9470009cd0 2024-09-16T01:51:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.027+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f947000c850 con 0x7f9470009cd0 2024-09-16T01:51:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.029+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f947000c850 con 0x7f9470009cd0 2024-09-16T01:51:57.032 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.031+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f947000ea90 0x7f947000ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.031+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f94700128a0 con 0x7f947000ea90 2024-09-16T01:51:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.032+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f947000ea90 0x7f947000ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.032+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f947000ea90 0x7f947000ca00 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:57.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.033+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94700128a0 con 0x7f947000ea90 2024-09-16T01:51:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.050+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f947000d2c0 con 0x7f947000ea90 2024-09-16T01:51:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.052+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f947000d2c0 con 0x7f947000ea90 2024-09-16T01:51:57.054 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.052+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f947000d9e0 0x7f9470011050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.052+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f947000def0 con 0x7f947000d9e0 2024-09-16T01:51:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.053+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f947000d9e0 0x7f9470011050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.053+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f947000d9e0 0x7f9470011050 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.054+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f947000def0 con 0x7f947000d9e0 2024-09-16T01:51:57.058 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:57.056+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:57.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.080+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f947000de70 con 0x7f947000d9e0 2024-09-16T01:51:57.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.082+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f947000de70 con 0x7f947000d9e0 2024-09-16T01:51:57.084 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:57.085 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:57.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.083+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9470011b20 0x7f9470011f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:57.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.083+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9470019390 con 0x7f9470011b20 2024-09-16T01:51:57.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.083+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9470011b20 0x7f9470011f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:57.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.084+0000 7f9496a00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9470011b20 0x7f9470011f60 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:57.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.084+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9470019390 con 0x7f9470011b20 2024-09-16T01:51:57.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.104+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9470016b00 con 0x7f9470011b20 2024-09-16T01:51:57.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.107+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9470016b00 con 0x7f9470011b20 2024-09-16T01:51:57.109 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:57.110 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:57.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.107+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9470017180 0x7f9470019570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:57.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.107+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f947001c4f0 con 0x7f9470017180 2024-09-16T01:51:57.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.108+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9470017180 0x7f9470019570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:57.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.108+0000 7f9497e00640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9470017180 0x7f9470019570 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:57.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.109+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f947001c4f0 con 0x7f9470017180 2024-09-16T01:51:57.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.136+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9470019b00 con 0x7f9470017180 2024-09-16T01:51:57.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.139+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9470019b00 con 0x7f9470017180 2024-09-16T01:51:57.143 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:57.144 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:57.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.142+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f947001a220 0x7f947001a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:57.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.142+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f947001f8c0 con 0x7f947001a220 2024-09-16T01:51:57.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.143+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f947001a220 0x7f947001a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:57.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.146+0000 7f9497400640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f947001a220 0x7f947001a660 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:57.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.148+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f947001f8c0 con 0x7f947001a220 2024-09-16T01:51:57.159 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:57.158+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:57.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.182+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f947001e7c0 con 0x7f947001a220 2024-09-16T01:51:57.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.188+0000 7f948f400640 1 -- 172.21.15.146:0/68352005 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f947001e7c0 con 0x7f947001a220 2024-09-16T01:51:57.191 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:57.191 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:57.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9470000fa0 msgr2=0x7f94700045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9470000fa0 0x7f94700045e0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9470001c70 msgr2=0x7f9470002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9470001c70 0x7f9470002990 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9470009cd0 msgr2=0x7f947000c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9470009cd0 0x7f947000c0c0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f947000ea90 msgr2=0x7f947000ca00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f947000ea90 0x7f947000ca00 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f947000d9e0 msgr2=0x7f9470011050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f947000d9e0 0x7f9470011050 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9470011b20 msgr2=0x7f9470011f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9470011b20 0x7f9470011f60 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9470017180 msgr2=0x7f9470019570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9470017180 0x7f9470019570 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f947001a220 msgr2=0x7f947001a660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f947001a220 0x7f947001a660 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 reap_dead start 2024-09-16T01:51:57.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 reap_dead start 2024-09-16T01:51:57.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 reap_dead start 2024-09-16T01:51:57.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 reap_dead start 2024-09-16T01:51:57.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f9497e00640 1 -- 172.21.15.146:0/68352005 reap_dead start 2024-09-16T01:51:57.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f947806c380 msgr2=0x7f947806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f947806c380 0x7f947806e840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9488006fd0 tx=0x7f9488008040 comp rx=0 tx=0).stop 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 msgr2=0x7f9498200c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.189+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 0x7f9498200c20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f949002aa90 tx=0x7f9490001650 comp rx=0 tx=0).stop 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.190+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 shutdown_connections 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.190+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f947806c380 0x7f947806e840 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9488006fd0 tx=0x7f9488008040 comp rx=0 tx=0).stop 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.190+0000 7f949e600640 1 --2- 172.21.15.146:0/68352005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94982007a0 0x7f9498200c20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.190+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 >> 172.21.15.146:0/68352005 conn(0x7f949806c2a0 msgr2=0x7f9498067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.190+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 shutdown_connections 2024-09-16T01:51:57.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.192+0000 7f949e600640 1 -- 172.21.15.146:0/68352005 wait complete. 2024-09-16T01:51:57.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.245+0000 7f54bcc00640 1 -- 172.21.15.146:0/1744413645 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v89) ==== 97+0+0 (secure 0 0 0) 0x7f54c00d5bd0 con 0x7f54b80e3100 2024-09-16T01:51:57.247 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:51:57.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f54a006c380 msgr2=0x7f54a006e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f54a006c380 0x7f54a006e840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f54b4034000 tx=0x7f54b402c000 comp rx=0 tx=0).stop 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 msgr2=0x7f54b80defe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 0x7f54b80defe0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f54c006a930 tx=0x7f54c0069890 comp rx=0 tx=0).stop 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 shutdown_connections 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f54a006c380 0x7f54a006e840 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f54b4034000 tx=0x7f54b402c000 comp rx=0 tx=0).stop 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54b80e3100 0x7f54b80defe0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54b80dcdc0 0x7f54b80deaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 --2- 172.21.15.146:0/1744413645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54b804a6b0 0x7f54b80e1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:57.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.250+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 >> 172.21.15.146:0/1744413645 conn(0x7f54b804fc80 msgr2=0x7f54b804c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:57.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.253+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 shutdown_connections 2024-09-16T01:51:57.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:57.253+0000 7f54c7600640 1 -- 172.21.15.146:0/1744413645 wait complete. 2024-09-16T01:51:57.264 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:57.259+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:57.279 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:57.279 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:57.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:51:57.396 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:57.394+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:57.463 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:57.461+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:57.564 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:57.563+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:57.602 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:57.603 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:57.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:51:57.664 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:57.663+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:57.766 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:57.765+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:57.866 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:57.865+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:57.891 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:57.891 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:57.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:51:57.967 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:57.966+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:58.068 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:58.067+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:58.169 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:58.168+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:58.240 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:58.240 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:58.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:51:58.269 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:58.268+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:58.372 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:58.371+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:58.475 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:58.472+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:58.516 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:58.516 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:58.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:51:58.588 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:58.579+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:58.677 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:51:58.675+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:51:58.775 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:58.774+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:58.875 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:51:58.874+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:51:58.877 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:58.877 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:58.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:51:58.982 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:58.981+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:59.078 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:59.077+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:59.179 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:59.178+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:59.219 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:59.219 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:59.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:51:59.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:51:59.280 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:59.279+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:51:59.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:51:59.381 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:51:59.380+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:51:59.492 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:51:59.490+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:51:59.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.504+0000 7f44b6a00640 1 Processor -- start 2024-09-16T01:51:59.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.505+0000 7f44b6a00640 1 -- start start 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b6a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44b0154c10 0x7f44b015b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b6a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f44b00cca70 0x7f44b0073340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b00cafa0 0x7f44b0073880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b6a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f44b015de60 con 0x7f44b00cca70 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b6a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f44b015e0b0 con 0x7f44b0154c10 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b6a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f44b015e390 con 0x7f44b00cafa0 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44afe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44b0154c10 0x7f44b015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44afe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44b0154c10 0x7f44b015b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50568/0 (socket says 172.21.15.146:50568) 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44afe00640 1 -- 172.21.15.146:0/19125209 learned_addr learned my addr 172.21.15.146:0/19125209 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.507+0000 7f44b4a00640 1 --2- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b00cafa0 0x7f44b0073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.508+0000 7f44afe00640 1 -- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b00cafa0 msgr2=0x7f44b0073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.508+0000 7f44afe00640 1 --2- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b00cafa0 0x7f44b0073880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.508+0000 7f44afe00640 1 -- 172.21.15.146:0/19125209 >> v2:172.21.15.146:3300/0 conn(0x7f44b00cca70 msgr2=0x7f44b0073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.508+0000 7f44afe00640 1 --2- 172.21.15.146:0/19125209 >> v2:172.21.15.146:3300/0 conn(0x7f44b00cca70 0x7f44b0073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.508+0000 7f44afe00640 1 -- 172.21.15.146:0/19125209 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b015e670 con 0x7f44b0154c10 2024-09-16T01:51:59.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.509+0000 7f44afe00640 1 --2- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0154c10 0x7f44b015b370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f44b006df90 tx=0x7f44a002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b016bd22896501d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.512+0000 7f44aea00640 1 -- 172.21.15.146:0/19125209 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0037070 con 0x7f44b0154c10 2024-09-16T01:51:59.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.512+0000 7f44aea00640 1 -- 172.21.15.146:0/19125209 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a0032440 con 0x7f44b0154c10 2024-09-16T01:51:59.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.512+0000 7f44aea00640 1 -- 172.21.15.146:0/19125209 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0032810 con 0x7f44b0154c10 2024-09-16T01:51:59.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 -- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0154c10 msgr2=0x7f44b015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 --2- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0154c10 0x7f44b015b370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f44b006df90 tx=0x7f44a002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 -- 172.21.15.146:0/19125209 shutdown_connections 2024-09-16T01:51:59.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 --2- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b00cafa0 0x7f44b0073880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 --2- 172.21.15.146:0/19125209 >> v2:172.21.15.146:3300/0 conn(0x7f44b00cca70 0x7f44b0073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 --2- 172.21.15.146:0/19125209 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b0154c10 0x7f44b015b370 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.513+0000 7f44b6a00640 1 -- 172.21.15.146:0/19125209 >> 172.21.15.146:0/19125209 conn(0x7f44b006ce80 msgr2=0x7f44b006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:59.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.517+0000 7f44b6a00640 1 -- 172.21.15.146:0/19125209 shutdown_connections 2024-09-16T01:51:59.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.517+0000 7f44b6a00640 1 -- 172.21.15.146:0/19125209 wait complete. 2024-09-16T01:51:59.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.519+0000 7f44b6a00640 1 Processor -- start 2024-09-16T01:51:59.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.519+0000 7f44b6a00640 1 -- start start 2024-09-16T01:51:59.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.520+0000 7f44b6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b00cafa0 0x7f44b015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.520+0000 7f44b6a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b00cca70 0x7f44b0150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.520+0000 7f44b6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 0x7f44b00c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.520+0000 7f44b6a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f44b0154ff0 con 0x7f44b00cafa0 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.520+0000 7f44b6a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f44b015ec30 con 0x7f44b00cca70 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.520+0000 7f44b6a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f44b00cb6f0 con 0x7f44b0154c10 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 0x7f44b00c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44afe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b00cafa0 0x7f44b015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 0x7f44b00c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:49014/0 (socket says 172.21.15.146:49014) 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 learned_addr learned my addr 172.21.15.146:0/423659237 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b00cca70 msgr2=0x7f44b0150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44b00cca70 0x7f44b0150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b00cafa0 msgr2=0x7f44b015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b00cafa0 0x7f44b015ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.521+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b015c0e0 con 0x7f44b0154c10 2024-09-16T01:51:59.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.522+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 0x7f44b00c99a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44a8007e40 tx=0x7f44a800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.522+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8014620 con 0x7f44b0154c10 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.522+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a8019070 con 0x7f44b0154c10 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.522+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8014a20 con 0x7f44b0154c10 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.522+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b015c3c0 con 0x7f44b0154c10 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.522+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b00c91a0 con 0x7f44b0154c10 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.523+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f44b015b370 con 0x7f44b0154c10 2024-09-16T01:51:59.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.525+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f44a8014bc0 con 0x7f44b0154c10 2024-09-16T01:51:59.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.526+0000 7f44a7e00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f449006c2b0 0x7f449006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.526+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(89..89 src has 1..89) ==== 5770+0+0 (secure 0 0 0) 0x7f44a808bdb0 con 0x7f44b0154c10 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.527+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v89) ==== 54+0+16 (secure 0 0 0) 0x7f44a808e350 con 0x7f44b0154c10 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.527+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f44b00c93a0 0x7f44b00c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.527+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44b00b3510 con 0x7f44b00c93a0 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.528+0000 7f44af400640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f44b00c93a0 0x7f44b00c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.529+0000 7f44af400640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f44b00c93a0 0x7f44b00c9ee0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.531+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b00b3510 con 0x7f44b00c93a0 2024-09-16T01:51:59.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.542+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f449006c2b0 0x7f449006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.542+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f449006c2b0 0x7f449006e770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f44a0034000 tx=0x7f44a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.554+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b00c68c0 con 0x7f44b00c93a0 2024-09-16T01:51:59.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.559+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b00c68c0 con 0x7f44b00c93a0 2024-09-16T01:51:59.560 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:51:59.561 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:51:59.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.559+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f44b014e3f0 0x7f44b014e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.559+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f44b00c6300 con 0x7f44b014e3f0 2024-09-16T01:51:59.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.560+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f44b014e3f0 0x7f44b014e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.561+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f44b014e3f0 0x7f44b014e7d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.563+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b00c6300 con 0x7f44b014e3f0 2024-09-16T01:51:59.579 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:59.579 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:59.583 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:51:59.582+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:51:59.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.591+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b014ed10 con 0x7f44b014e3f0 2024-09-16T01:51:59.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.594+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b014ed10 con 0x7f44b014e3f0 2024-09-16T01:51:59.595 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:51:59.596 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:51:59.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.595+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f44b01f3f40 0x7f44b01f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.595+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f44b01f4860 con 0x7f44b01f3f40 2024-09-16T01:51:59.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.595+0000 7f44af400640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f44b01f3f40 0x7f44b01f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.596+0000 7f44af400640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f44b01f3f40 0x7f44b01f4320 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.600+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b01f4860 con 0x7f44b01f3f40 2024-09-16T01:51:59.607 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:51:59.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.637+0000 7f09bc400640 1 Processor -- start 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.637+0000 7f09bc400640 1 -- start start 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.638+0000 7f09bc400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b4072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.638+0000 7f09bc400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09b406e160 0x7f09b40727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.638+0000 7f09bc400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09b41569d0 con 0x7f09b40c70e0 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.638+0000 7f09bc400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f09b4157580 con 0x7f09b406e160 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.638+0000 7f09bc400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f09b4071ce0 con 0x7f09b4150c10 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.638+0000 7f09b9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b4072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.641+0000 7f09b9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b4072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:49030/0 (socket says 172.21.15.146:49030) 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.641+0000 7f09b9a00640 1 -- 172.21.15.146:0/794889221 learned_addr learned my addr 172.21.15.146:0/794889221 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.641+0000 7f09b9a00640 1 -- 172.21.15.146:0/794889221 >> v2:172.21.15.161:3300/0 conn(0x7f09b406e160 msgr2=0x7f09b40727d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:59.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.641+0000 7f09b9a00640 1 --2- 172.21.15.146:0/794889221 >> v2:172.21.15.161:3300/0 conn(0x7f09b406e160 0x7f09b40727d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.641+0000 7f09b9a00640 1 -- 172.21.15.146:0/794889221 >> v1:172.21.15.161:6789/0 conn(0x7f09b40c70e0 legacy=0x7f09b406da50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:51:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.641+0000 7f09b9a00640 1 -- 172.21.15.146:0/794889221 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b4156c50 con 0x7f09b4150c10 2024-09-16T01:51:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.643+0000 7f09b9a00640 1 --2- 172.21.15.146:0/794889221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b4072290 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009870 tx=0x7f09a402a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2800a83466442902 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.644+0000 7f09b3e00640 1 -- 172.21.15.146:0/794889221 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4037070 con 0x7f09b4150c10 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.644+0000 7f09b3e00640 1 -- 172.21.15.146:0/794889221 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a4032440 con 0x7f09b4150c10 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.644+0000 7f09b3e00640 1 -- 172.21.15.146:0/794889221 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4032810 con 0x7f09b4150c10 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.645+0000 7f09bc400640 1 -- 172.21.15.146:0/794889221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 msgr2=0x7f09b4072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.645+0000 7f09bc400640 1 --2- 172.21.15.146:0/794889221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b4072290 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009870 tx=0x7f09a402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.645+0000 7f09bc400640 1 -- 172.21.15.146:0/794889221 shutdown_connections 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.645+0000 7f09bc400640 1 --2- 172.21.15.146:0/794889221 >> v2:172.21.15.161:3300/0 conn(0x7f09b406e160 0x7f09b40727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.645+0000 7f09bc400640 1 --2- 172.21.15.146:0/794889221 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b4072290 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.645+0000 7f09bc400640 1 -- 172.21.15.146:0/794889221 >> 172.21.15.146:0/794889221 conn(0x7f09b406f3c0 msgr2=0x7f09b406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.646+0000 7f09bc400640 1 -- 172.21.15.146:0/794889221 shutdown_connections 2024-09-16T01:51:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.646+0000 7f09bc400640 1 -- 172.21.15.146:0/794889221 wait complete. 2024-09-16T01:51:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.648+0000 7f09bc400640 1 Processor -- start 2024-09-16T01:51:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.648+0000 7f09bc400640 1 -- start start 2024-09-16T01:51:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.649+0000 7f09bc400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 0x7f09b41e88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.649+0000 7f09b9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 0x7f09b41e88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.649+0000 7f09b9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 0x7f09b41e88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55614/0 (socket says 172.21.15.146:55614) 2024-09-16T01:51:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.649+0000 7f09b9a00640 1 -- 172.21.15.146:0/3503416486 learned_addr learned my addr 172.21.15.146:0/3503416486 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:51:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.649+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b40c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.650+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b41e90a0 0x7f09b41f85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.650+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f09b40c9a90 con 0x7f09b40c70e0 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.650+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09b4157210 con 0x7f09b41e90a0 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.650+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f09b40718c0 con 0x7f09b4150c10 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.650+0000 7f09b9000640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b40c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.650+0000 7f09ba400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b41e90a0 0x7f09b41f85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.651+0000 7f09b9a00640 1 -- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 msgr2=0x7f09b40c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.651+0000 7f09b9a00640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b40c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.651+0000 7f09b9a00640 1 -- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b41e90a0 msgr2=0x7f09b41f85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.651+0000 7f09b9a00640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b41e90a0 0x7f09b41f85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.651+0000 7f09b9a00640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b41e9f50 con 0x7f09b40c70e0 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.651+0000 7f09b9000640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b40c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:51:59.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.652+0000 7f09b9a00640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 0x7f09b41e88b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009990 tx=0x7f09a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.652+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4041070 con 0x7f09b40c70e0 2024-09-16T01:51:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.652+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a403c930 con 0x7f09b40c70e0 2024-09-16T01:51:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.652+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b41ea230 con 0x7f09b40c70e0 2024-09-16T01:51:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.652+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b41ea7f0 con 0x7f09b40c70e0 2024-09-16T01:51:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.652+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4037040 con 0x7f09b40c70e0 2024-09-16T01:51:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.654+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f098c005290 con 0x7f09b40c70e0 2024-09-16T01:51:59.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.658+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b0062900 con 0x7f44b01f3f40 2024-09-16T01:51:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.660+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f09a4044070 con 0x7f09b40c70e0 2024-09-16T01:51:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.662+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b0062900 con 0x7f44b01f3f40 2024-09-16T01:51:59.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.664+0000 7f09b1600640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099406c2e0 0x7f099406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.665+0000 7f09b9000640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099406c2e0 0x7f099406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.666+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5770+0+0 (secure 0 0 0) 0x7f09a40abfd0 con 0x7f09b40c70e0 2024-09-16T01:51:59.668 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:51:59.669 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:51:59.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.668+0000 7f09b9000640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099406c2e0 0x7f099406e7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f09b41e84d0 tx=0x7f09a00090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.668+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f44b00b3ff0 0x7f44b00b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.668+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f44b00633e0 con 0x7f44b00b3ff0 2024-09-16T01:51:59.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.671+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f09b41569d0 con 0x7f09b40c70e0 2024-09-16T01:51:59.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.671+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f44b00b3ff0 0x7f44b00b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.672+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f44b00b3ff0 0x7f44b00b4410 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.678+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b00633e0 con 0x7f44b00b3ff0 2024-09-16T01:51:59.683 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:59.682+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:59.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.713+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b00b4950 con 0x7f44b00b3ff0 2024-09-16T01:51:59.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.716+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b00b4950 con 0x7f44b00b3ff0 2024-09-16T01:51:59.718 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:51:59.718 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:51:59.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.717+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f44b002c640 0x7f44b002ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.717+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f44b002cfc0 con 0x7f44b002c640 2024-09-16T01:51:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.718+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f44b002c640 0x7f44b002ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.719 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:51:59.719 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:51:59.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.718+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f44b002c640 0x7f44b002ca80 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.729+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b002cfc0 con 0x7f44b002c640 2024-09-16T01:51:59.731 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:51:59.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.759+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b002cfc0 con 0x7f44b002c640 2024-09-16T01:51:59.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.761+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b002cfc0 con 0x7f44b002c640 2024-09-16T01:51:59.762 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:51:59.763 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:51:59.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.762+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f44b002d640 0x7f44b0206dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.762+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f44b002daa0 con 0x7f44b002d640 2024-09-16T01:51:59.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.763+0000 7f44af400640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f44b002d640 0x7f44b0206dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.765+0000 7f44af400640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f44b002d640 0x7f44b0206dc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.767+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b002daa0 con 0x7f44b002d640 2024-09-16T01:51:59.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.789+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b002daa0 con 0x7f44b002d640 2024-09-16T01:51:59.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.793+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b002daa0 con 0x7f44b002d640 2024-09-16T01:51:59.795 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:51:59.795 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:51:59.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.794+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f44b0207810 0x7f44b0209c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.794+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f44b020c9d0 con 0x7f44b0207810 2024-09-16T01:51:59.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.795+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f44b0207810 0x7f44b0209c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.795+0000 7f44b4a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f44b0207810 0x7f44b0209c00 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.796+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b020c9d0 con 0x7f44b0207810 2024-09-16T01:51:59.816 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:51:59.815+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:51:59.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.818+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b020a140 con 0x7f44b0207810 2024-09-16T01:51:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.820+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b020a140 con 0x7f44b0207810 2024-09-16T01:51:59.821 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:51:59.821 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:51:59.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.820+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f44b020a860 0x7f44b020aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:51:59.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.820+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f44b020fd40 con 0x7f44b020a860 2024-09-16T01:51:59.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.820+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f44b020a860 0x7f44b020aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:51:59.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.821+0000 7f44afe00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f44b020a860 0x7f44b020aca0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:51:59.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.822+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44b020fd40 con 0x7f44b020a860 2024-09-16T01:51:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.841+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f44b020b250 con 0x7f44b020a860 2024-09-16T01:51:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.843+0000 7f44a7e00640 1 -- 172.21.15.146:0/423659237 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f44b020b250 con 0x7f44b020a860 2024-09-16T01:51:59.844 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f44b00c93a0 msgr2=0x7f44b00c9ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f44b00c93a0 0x7f44b00c9ee0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f44b014e3f0 msgr2=0x7f44b014e7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f44b014e3f0 0x7f44b014e7d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f44b01f3f40 msgr2=0x7f44b01f4320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f44b01f3f40 0x7f44b01f4320 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f44b00b3ff0 msgr2=0x7f44b00b4410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f44b00b3ff0 0x7f44b00b4410 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f44b002c640 msgr2=0x7f44b002ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f44b002c640 0x7f44b002ca80 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f44b002d640 msgr2=0x7f44b0206dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f44b002d640 0x7f44b0206dc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f44b0207810 msgr2=0x7f44b0209c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f44b0207810 0x7f44b0209c00 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f44b020a860 msgr2=0x7f44b020aca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f44b020a860 0x7f44b020aca0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 reap_dead start 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.844+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 reap_dead start 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 reap_dead start 2024-09-16T01:51:59.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 reap_dead start 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f449006c2b0 msgr2=0x7f449006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f449006c2b0 0x7f449006e770 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f44a0034000 tx=0x7f44a002e000 comp rx=0 tx=0).stop 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 msgr2=0x7f44b00c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 0x7f44b00c99a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44a8007e40 tx=0x7f44a800e4b0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b4a00640 1 -- 172.21.15.146:0/423659237 reap_dead start 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 shutdown_connections 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 --2- 172.21.15.146:0/423659237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f44b0154c10 0x7f44b00c99a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 >> 172.21.15.146:0/423659237 conn(0x7f44b006ce80 msgr2=0x7f44b00c6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 shutdown_connections 2024-09-16T01:51:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:51:59.845+0000 7f44b6a00640 1 -- 172.21.15.146:0/423659237 wait complete. 2024-09-16T01:51:59.890 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:51:59.890 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:51:59.890 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:51:59.890 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:51:59.890 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:51:59.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:51:59.913 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:51:59.906+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:51:59.986 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:51:59.985+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:00.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.022+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f098c005580 con 0x7f09b40c70e0 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 69.202438659999999, 2024-09-16T01:52:00.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:00.062 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.064 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:52:00.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:00.067 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 68.290860347999995, 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.068 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:52:00.070 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.071 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:52:00.072 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 68.246600118000003, 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.074 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:52:00.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:52:00.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.079 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 68.174544776000005, 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.080 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:52:00.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 59.917538354999998, 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.086 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:52:00.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.089 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:52:00.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:52:00.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.093 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 59.882742055999998, 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.095 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:52:00.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:52:00.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:52:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 59.799293102999997, 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.101 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.102 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.106 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.599990316000003, 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.107 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:52:00.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:52:00.112 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.455113378, 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.113 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:52:00.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:52:00.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.414468954, 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.119 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:00.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:52:00.124 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.256526119999997, 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.125 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:52:00.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:52:00.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.131398859000001, 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.131 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:00.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.136 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.094598928000003, 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.137 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:52:00.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:52:00.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.142 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.931573286000003, 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:52:00.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:52:00.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.797835497000001, 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.149 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:52:00.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:52:00.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.154 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.103445383, 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:52:00.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:52:00.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.776470817, 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.161 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:52:00.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.635560677000001, 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.167 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:52:00.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:00.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:52:00.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.582757732999999, 2024-09-16T01:52:00.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:52:00.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.263499192999999, 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.178 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:52:00.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:52:00.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.184 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:00.186 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:00.186 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:00.185+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:00.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.268+0000 7f09b1600640 1 -- 172.21.15.146:0/3503416486 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v90) ==== 91+0+0 (secure 0 0 0) 0x7f09a407a130 con 0x7f09b40c70e0 2024-09-16T01:52:00.273 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:52:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.279+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099406c2e0 msgr2=0x7f099406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.279+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099406c2e0 0x7f099406e7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f09b41e84d0 tx=0x7f09a00090e0 comp rx=0 tx=0).stop 2024-09-16T01:52:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 msgr2=0x7f09b41e88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 0x7f09b41e88b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009990 tx=0x7f09a40046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 shutdown_connections 2024-09-16T01:52:00.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099406c2e0 0x7f099406e7a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:00.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09b41e90a0 0x7f09b41f85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:00.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09b4150c10 0x7f09b40c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:00.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 --2- 172.21.15.146:0/3503416486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09b40c70e0 0x7f09b41e88b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:00.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.282+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 >> 172.21.15.146:0/3503416486 conn(0x7f09b406f3c0 msgr2=0x7f09b406da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:00.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.283+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 shutdown_connections 2024-09-16T01:52:00.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:00.284+0000 7f09bc400640 1 -- 172.21.15.146:0/3503416486 wait complete. 2024-09-16T01:52:00.290 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:00.289+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:00.343 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:00.343 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:00.343 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:00.343 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:00.352 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:00.388 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:00.387+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:00.469 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:00.469 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:00.469 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:00.469 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:00.470 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:00.479 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:00.489 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:00.488+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:00.594 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:00.589+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:00.610 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:00.610 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:00.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:00.611 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:00.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.611 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:00.611 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:00.611 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 70.084059276999994, 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:00.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:52:00.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:52:00.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:574 3.27 3:e46e0c7f:::100000001f7.0000003b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 45.602603332000001, 2024-09-16T01:52:00.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084267634999999994, 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 574 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004659+0000", 2024-09-16T01:52:00.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5129999999999999e-06 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032641+0000", 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027982080999999999 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032656+0000", 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5699999999999999e-05 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032666+0000", 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5599999999999999e-06 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032719+0000", 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3066999999999999e-05 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032827+0000", 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010841900000000001 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032929+0000", 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101128 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.036882+0000", 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039538949999999998 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088843+0000", 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051960682000000001 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088878+0000", 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.485e-05 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088924+0000", 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5739999999999999e-05 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.624 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1033 3.29 3:94178d8a:::100000001f8.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.416392297000002, 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086462834000000002, 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1033 2024-09-16T01:52:00.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190869+0000", 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3099999999999999e-06 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224163+0000", 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.033293469999999999 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224185+0000", 2024-09-16T01:52:00.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1987000000000001e-05 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224197+0000", 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.222e-05 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224896+0000", 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00069902199999999999 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.225064+0000", 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016828399999999999 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.225208+0000", 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014386800000000001 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.231378+0000", 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006169796 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277260+0000", 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045882091999999999 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277295+0000", 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4505999999999998e-05 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277330+0000", 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5278999999999999e-05 2024-09-16T01:52:00.630 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1133 3.38 3:1cae2fc4:::100000001f8.0000006a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.360979+0000", 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.246280204000001, 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083133998000000001, 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.631 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1133 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.360979+0000", 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.360979+0000", 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.360980+0000", 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2309999999999999e-06 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404652+0000", 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.043671158000000002 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404670+0000", 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8031999999999999e-05 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404681+0000", 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1781e-05 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404749+0000", 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7257999999999999e-05 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404891+0000", 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014246899999999999 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.405017+0000", 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000126174 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.408883+0000", 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038658730000000001 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.444000+0000", 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035117122000000001 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.444060+0000", 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9781000000000002e-05 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.444113+0000", 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3118999999999997e-05 2024-09-16T01:52:00.636 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1137 3.29 3:9673bc10:::100000001f8.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.202398268, 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084134489000000007, 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.637 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1137 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:52:00.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404862+0000", 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0270000000000001e-06 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458066+0000", 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053203817 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458080+0000", 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3536999999999999e-05 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458085+0000", 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8130000000000001e-06 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458167+0000", 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1797000000000003e-05 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458306+0000", 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013863699999999999 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458472+0000", 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016593700000000001 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.462073+0000", 2024-09-16T01:52:00.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003601131 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488893+0000", 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026819912000000001 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488949+0000", 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5889e-05 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488996+0000", 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6992e-05 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1181 3.38 3:1e6b0f01:::100000001f8.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.878999+0000", 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.728260023000001, 2024-09-16T01:52:00.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.080968997000000001, 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1181 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.878999+0000", 2024-09-16T01:52:00.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.878999+0000", 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.879001+0000", 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2619999999999999e-06 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926095+0000", 2024-09-16T01:52:00.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.047094233999999999 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926110+0000", 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5279999999999999e-05 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926120+0000", 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0052e-05 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926184+0000", 2024-09-16T01:52:00.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3660999999999999e-05 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926325+0000", 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014081400000000001 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.926480+0000", 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000154857 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.930130+0000", 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036503460000000001 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.959902+0000", 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029772526000000001 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.959938+0000", 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6003000000000001e-05 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.959968+0000", 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9961999999999999e-05 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1204 3.1e 3:7a97cce9:::100000001f8.000000b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.135683+0000", 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.471576186, 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.080493544, 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.649 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1204 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.135683+0000", 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.135683+0000", 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.135685+0000", 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.305e-06 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194149+0000", 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058464147000000001 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194163+0000", 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3910999999999999e-05 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194170+0000", 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0910000000000004e-06 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194245+0000", 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5118999999999997e-05 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194387+0000", 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014192799999999999 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.194503+0000", 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011599600000000001 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.198531+0000", 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040281029999999999 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.216096+0000", 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017565537999999999 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.216136+0000", 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9527999999999998e-05 2024-09-16T01:52:00.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.216177+0000", 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0877999999999997e-05 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1246 3.31 3:8c5c37ef:::100000001f8.000000db:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.710465+0000", 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 19.896794512, 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084411384000000006, 2024-09-16T01:52:00.655 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1246 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.710465+0000", 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.710465+0000", 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.710468+0000", 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.729e-06 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738714+0000", 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028245936999999999 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738735+0000", 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1080000000000001e-05 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738746+0000", 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1751e-05 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738820+0000", 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3627999999999994e-05 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738976+0000", 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015629000000000001 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.739147+0000", 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017062499999999999 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.744247+0000", 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0051001279999999998 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.794797+0000", 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050549627 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.794842+0000", 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5664999999999997e-05 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.794876+0000", 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3924000000000002e-05 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1656 3.29 3:976aaeee:::100000001f8.00000275:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:00.661 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.729286+0000", 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.877973114, 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.085174584999999997, 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1656 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.662 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.729286+0000", 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.729286+0000", 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.729288+0000", 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1339999999999998e-06 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754379+0000", 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025090527000000001 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754397+0000", 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7966999999999999e-05 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754406+0000", 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3340000000000008e-06 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754469+0000", 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2571999999999997e-05 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754573+0000", 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103675 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754691+0000", 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118885 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.757993+0000", 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033016740000000001 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.814386+0000", 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056392721999999999 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.814425+0000", 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8741000000000001e-05 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.814461+0000", 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6353999999999998e-05 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1663 3.17 3:ea019aa2:::100000001f8.0000027c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.830568446999999, 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124529129, 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.668 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1663 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:00.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776694+0000", 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8320000000000002e-06 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866207+0000", 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089513588000000005 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866225+0000", 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7325000000000001e-05 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866232+0000", 2024-09-16T01:52:00.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6149999999999999e-06 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866286+0000", 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3863000000000003e-05 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866393+0000", 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106443 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866499+0000", 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010595299999999999 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.869501+0000", 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030023570000000002 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901146+0000", 2024-09-16T01:52:00.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031644671999999999 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901180+0000", 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4277999999999999e-05 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901220+0000", 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0203000000000001e-05 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1954 3.31 3:8d5ad0f7:::100000001f8.0000039f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.273995+0000", 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.333264194, 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083579280000000006, 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1954 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.675 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.273995+0000", 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.273995+0000", 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.273997+0000", 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3850000000000001e-06 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.331399+0000", 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057401985000000003 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.331412+0000", 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3118999999999999e-05 2024-09-16T01:52:00.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.331419+0000", 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2860000000000002e-06 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.331476+0000", 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7432000000000002e-05 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.331599+0000", 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012287999999999999 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.331744+0000", 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014420899999999999 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.334935+0000", 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003191747 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357511+0000", 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022575733000000001 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357544+0000", 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3379e-05 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357575+0000", 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.0125e-05 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2039 3.31 3:8f066b95:::100000001f8.000003f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:00.681 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.08496251, 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12858615900000001, 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2039 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522299+0000", 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0109999999999999e-06 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586678+0000", 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064378764000000005 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586695+0000", 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7159999999999998e-05 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586705+0000", 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9839999999999996e-06 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586760+0000", 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5099999999999998e-05 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586894+0000", 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013428999999999999 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.587008+0000", 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114178 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.590811+0000", 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038024529999999999 2024-09-16T01:52:00.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650793+0000", 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059982029999999999 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650843+0000", 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9876999999999997e-05 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650883+0000", 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0312000000000003e-05 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2052 3.29 3:96415e08:::100000001f8.00000401:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.683050+0000", 2024-09-16T01:52:00.688 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.9242098849999998, 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088613109999999995, 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2052 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.683050+0000", 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.683050+0000", 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.683051+0000", 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.46e-06 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730050+0000", 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046998637000000003 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730068+0000", 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8521000000000001e-05 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730078+0000", 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0282000000000001e-05 2024-09-16T01:52:00.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730147+0000", 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8736000000000001e-05 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730311+0000", 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016330899999999999 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730465+0000", 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015406299999999999 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.735811+0000", 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053466750000000004 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.771561+0000", 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035749360000000001 2024-09-16T01:52:00.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.771615+0000", 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3891999999999998e-05 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.771663+0000", 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8174999999999999e-05 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2092 3.1e 3:7aa4b9dd:::100000001f8.00000429:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.3433684560000003, 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10603082799999999, 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2092 2024-09-16T01:52:00.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263893+0000", 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2249999999999999e-06 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292346+0000", 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028453092999999999 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292359+0000", 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2306999999999999e-05 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292364+0000", 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292417+0000", 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2673000000000002e-05 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292522+0000", 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010544600000000001 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292624+0000", 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101473 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.295728+0000", 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031039010000000001 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369848+0000", 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074120309999999995 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369888+0000", 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9465000000000002e-05 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369922+0000", 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4193999999999998e-05 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2103 3.31 3:8c8a0fd7:::100000001f8.00000434:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.388257+0000", 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.2190027519999997, 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086636273, 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2103 2024-09-16T01:52:00.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.388257+0000", 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.388257+0000", 2024-09-16T01:52:00.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.388258+0000", 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4079999999999999e-06 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431717+0000", 2024-09-16T01:52:00.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.043458745 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431731+0000", 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3971000000000001e-05 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431738+0000", 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8460000000000002e-06 2024-09-16T01:52:00.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431797+0000", 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9125000000000001e-05 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431909+0000", 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112417 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.432018+0000", 2024-09-16T01:52:00.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108305 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.435049+0000", 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030311919999999998 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.474790+0000", 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039741548000000002 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.474855+0000", 2024-09-16T01:52:00.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4480000000000006e-05 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.474893+0000", 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8235999999999998e-05 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2489 3.27 3:e6c61997:::100000001f8.000005b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:55.921405+0000", 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.6858545950000003, 2024-09-16T01:52:00.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084631479999999995, 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2489 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.921405+0000", 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.921405+0000", 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.921406+0000", 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.638e-06 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.710 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973442+0000", 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052035855999999998 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973459+0000", 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6721000000000001e-05 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973484+0000", 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4807000000000002e-05 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973549+0000", 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5158000000000002e-05 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973702+0000", 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015326700000000001 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973876+0000", 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017350699999999999 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.977774+0000", 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038979319999999998 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.005962+0000", 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028188063999999999 2024-09-16T01:52:00.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.006000+0000", 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8355999999999997e-05 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.006036+0000", 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6174000000000003e-05 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2506 3.31 3:8dd8e9cf:::100000001f8.000005c7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.155701+0000", 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.4515588790000002, 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.085502210999999995, 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2506 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.715 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.155701+0000", 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.155701+0000", 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.716 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.155702+0000", 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.606e-06 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214365+0000", 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058663172999999999 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214381+0000", 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5695000000000001e-05 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214391+0000", 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8560000000000002e-06 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214481+0000", 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9932000000000006e-05 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214591+0000", 2024-09-16T01:52:00.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110648 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214689+0000", 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7826999999999997e-05 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.217696+0000", 2024-09-16T01:52:00.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003006908 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.241115+0000", 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023418575000000001 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.241161+0000", 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5768e-05 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.241203+0000", 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2222999999999999e-05 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2568 3.27 3:e5763115:::100000001f8.00000605:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.844862+0000", 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 3.7623972050000001, 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083917768000000004, 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.721 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2568 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.844862+0000", 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.722 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.844862+0000", 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.844865+0000", 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5100000000000001e-06 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.873298+0000", 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028433118 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.873316+0000", 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7856e-05 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.873323+0000", 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3259999999999998e-06 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.873380+0000", 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6535e-05 2024-09-16T01:52:00.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.873492+0000", 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112125 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.873585+0000", 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3467000000000005e-05 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.877432+0000", 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038467929999999998 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.928698+0000", 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051266363000000002 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.928745+0000", 2024-09-16T01:52:00.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6317999999999999e-05 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.928780+0000", 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5357e-05 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2860 3.1e 3:7942aff9:::100000001f8.00000729:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:59.900216+0000", 2024-09-16T01:52:00.727 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.70704335799999996, 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.081205720999999995, 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2860 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.900216+0000", 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.900216+0000", 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.900217+0000", 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.215e-06 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.928546+0000", 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028328669000000001 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.928560+0000", 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.448e-05 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.928568+0000", 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2339999999999997e-06 2024-09-16T01:52:00.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.928643+0000", 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5419000000000004e-05 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.928741+0000", 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7634999999999996e-05 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:00.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.928844+0000", 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103199 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.932627+0000", 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037833369999999999 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.981312+0000", 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048685000999999999 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.981375+0000", 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2594000000000004e-05 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.981422+0000", 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6937999999999997e-05 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2901 3.1f 3:fb8355dd:::100000001f8.00000752:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:00.356814+0000", 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.25044550900000001, 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089405411000000004, 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2901 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.734 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.356814+0000", 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.356814+0000", 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.356816+0000", 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.063e-06 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430011+0000", 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.073194542000000001 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430029+0000", 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8349999999999999e-05 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430037+0000", 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4440000000000008e-06 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430095+0000", 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7432999999999997e-05 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430198+0000", 2024-09-16T01:52:00.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010282599999999999 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430291+0000", 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3115999999999994e-05 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.433796+0000", 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035057759999999999 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:00.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.446128+0000", 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012331811 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.446172+0000", 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4150999999999998e-05 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.446219+0000", 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6898999999999999e-05 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.739 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.740 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.740 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:00.741 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:00.742 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:00.740+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:00.842 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:00.841+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:00.901 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:00.902 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:00.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.902 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2951 3.24 3:26922d50:::100000001f8.00000784:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:00.902 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:00.876255+0000", 2024-09-16T01:52:00.902 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.024324743999999999, 2024-09-16T01:52:00.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024351345999999999, 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:00.903 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2951 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.876255+0000", 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:00.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.876255+0000", 2024-09-16T01:52:00.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:00.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.876256+0000", 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.781e-06 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.899634+0000", 2024-09-16T01:52:00.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023377284000000002 2024-09-16T01:52:00.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:00.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.899646+0000", 2024-09-16T01:52:00.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2408e-05 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.899652+0000", 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1619999999999996e-06 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.899727+0000", 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4468999999999995e-05 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.899877+0000", 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015029299999999999 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:00.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.900027+0000", 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015003799999999999 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 1 2024-09-16T01:52:00.910 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:00.911 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:00.943 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:00.942+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:01.046 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:01.045+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:01.074 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:01.074 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:01.074 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:01.074 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:01.074 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:01.084 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:01.145 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:01.144+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:01.195 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:01.195 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:01.196 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.43369510715733894 2024-09-16T01:52:01.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 0.43369510715733894 2024-09-16T01:52:01.200 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:01.200 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:38 3.24 3:2742a876:::10000000002.00000023:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.552269+0000", 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 70.645906695999997, 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.076686301999999998, 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.201 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 38 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.552269+0000", 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.552269+0000", 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.552270+0000", 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.579e-06 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593289+0000", 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041018708000000001 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593307+0000", 2024-09-16T01:52:01.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8382000000000001e-05 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593316+0000", 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1880000000000002e-06 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593380+0000", 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4666999999999996e-05 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593479+0000", 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8740999999999995e-05 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.593568+0000", 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9142999999999997e-05 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.596725+0000", 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031567890000000001 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.628797+0000", 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032072554000000003 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.628879+0000", 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1046999999999998e-05 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.628955+0000", 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6503999999999998e-05 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.206 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:482 3.32 3:4c3c6424:::10000000002.000000df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 61.133473109000001, 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.78198721800000004, 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 482 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.207 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064704+0000", 2024-09-16T01:52:01.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.578e-06 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102892+0000", 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038187684999999999 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102910+0000", 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8338999999999998e-05 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102918+0000", 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.259e-06 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102991+0000", 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2638999999999994e-05 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.103003+0000", 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2269999999999999e-05 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.816160+0000", 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.71315740100000002 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.820892+0000", 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047315539999999998 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835633+0000", 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014741232999999999 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835741+0000", 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107768 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835903+0000", 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016157399999999999 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846577+0000", 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010674197 2024-09-16T01:52:01.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846586+0000", 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5149999999999995e-06 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846627+0000", 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0513000000000003e-05 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846690+0000", 2024-09-16T01:52:01.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2693000000000004e-05 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:487 3.33 3:ce8011cb:::10000000002.000000e4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 61.093040475000002, 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.80604562300000004, 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.214 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 487 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105136+0000", 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1429999999999999e-06 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131251+0000", 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026114741 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131264+0000", 2024-09-16T01:52:01.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3325e-05 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131270+0000", 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1399999999999997e-06 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131326+0000", 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5643000000000002e-05 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:01.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131334+0000", 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4020000000000002e-06 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.832106+0000", 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.700771794 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.833244+0000", 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011381679999999999 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890416+0000", 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057171588000000002 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890476+0000", 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0118999999999998e-05 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890571+0000", 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5011000000000002e-05 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.895250+0000", 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046789140000000002 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911079+0000", 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015829296999999999 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911119+0000", 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9489000000000001e-05 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.220 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911181+0000", 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1848999999999997e-05 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:506 3.32 3:4c9ed383:::10000000002.000000f7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:01.221 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 60.914725482999998, 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.56375856099999999, 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 506 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283453+0000", 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9679999999999998e-06 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324488+0000", 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041035220999999997 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324503+0000", 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5167000000000001e-05 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324512+0000", 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2019999999999993e-06 2024-09-16T01:52:01.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324603+0000", 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494999999999993e-05 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324612+0000", 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0769999999999997e-06 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.821133+0000", 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.496521196 2024-09-16T01:52:01.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.824031+0000", 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028972809999999998 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841705+0000", 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017674083 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841807+0000", 2024-09-16T01:52:01.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102268 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841968+0000", 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016139600000000001 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846962+0000", 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049941990000000004 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847135+0000", 2024-09-16T01:52:01.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017284200000000001 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847161+0000", 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5193000000000001e-05 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847208+0000", 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7973000000000003e-05 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.228 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:538 3.33 3:cf13e063:::100000001f7.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.617692517999998, 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.109075775, 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 538 2024-09-16T01:52:01.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580485+0000", 2024-09-16T01:52:01.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1449999999999998e-06 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637145+0000", 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056659689999999999 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637165+0000", 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0092e-05 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637173+0000", 2024-09-16T01:52:01.231 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0660000000000004e-06 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637263+0000", 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0059e-05 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637365+0000", 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101922 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637467+0000", 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101853 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.640930+0000", 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003463617 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689487+0000", 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048556903999999998 2024-09-16T01:52:01.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689524+0000", 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6776999999999997e-05 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689559+0000", 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4650000000000002e-05 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:556 3.24 3:25dc7ea1:::100000001f7.00000029:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.796679+0000", 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.401496447, 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.090422115999999997, 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 556 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.796679+0000", 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.796679+0000", 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.796681+0000", 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.792e-06 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.849943+0000", 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053262217000000001 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.849959+0000", 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5882000000000001e-05 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.849967+0000", 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0039999999999999e-06 2024-09-16T01:52:01.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.850023+0000", 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5939000000000001e-05 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.850131+0000", 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108514 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.850222+0000", 2024-09-16T01:52:01.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0735000000000005e-05 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.855014+0000", 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047923649999999998 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.887020+0000", 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032005515999999998 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.887058+0000", 2024-09-16T01:52:01.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7659e-05 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.887101+0000", 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3492999999999998e-05 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:726 3.1 3:83186ff1:::100000001f7.000000d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:16.427655+0000", 2024-09-16T01:52:01.240 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 44.770520720999997, 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083880706999999999, 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 726 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.427655+0000", 2024-09-16T01:52:01.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.427655+0000", 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.427657+0000", 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.967e-06 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457541+0000", 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029884035 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457566+0000", 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4902e-05 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457578+0000", 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2041e-05 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457642+0000", 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3866000000000001e-05 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457750+0000", 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010874099999999999 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.457861+0000", 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011107699999999999 2024-09-16T01:52:01.244 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.461779+0000", 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039180930000000001 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.511458+0000", 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049678587000000003 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.511494+0000", 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6241000000000001e-05 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.511535+0000", 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1156999999999999e-05 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1150 3.1 3:813e966a:::100000001f8.0000007b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:01.246 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.532675+0000", 2024-09-16T01:52:01.247 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.66550058, 2024-09-16T01:52:01.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078304481999999995, 2024-09-16T01:52:01.247 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.247 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.247 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.247 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1150 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.532675+0000", 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.532675+0000", 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.532677+0000", 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1229999999999998e-06 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578519+0000", 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045842540000000001 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578539+0000", 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9128e-05 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578546+0000", 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.588e-06 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578616+0000", 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9775000000000002e-05 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578728+0000", 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011250400000000001 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.578819+0000", 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0882000000000002e-05 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.582868+0000", 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0040486439999999997 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.610818+0000", 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027949544 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.610908+0000", 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0007999999999996e-05 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.610979+0000", 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1746000000000001e-05 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1239 3.33 3:cd5a60ff:::100000001f8.000000d4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.600796824, 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.093172247999999999, 2024-09-16T01:52:01.253 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1239 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597381+0000", 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9139999999999998e-06 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647202+0000", 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049821439000000002 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647225+0000", 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3061000000000002e-05 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647234+0000", 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5620000000000004e-06 2024-09-16T01:52:01.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647368+0000", 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000134141 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647475+0000", 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106896 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647575+0000", 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100463 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.650879+0000", 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033037740000000002 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690472+0000", 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039593415999999999 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690523+0000", 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0657e-05 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690551+0000", 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7925000000000001e-05 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.259 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1501 3.24 3:26bdc0b7:::100000001f8.000001da:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.321739247, 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.092713806999999995, 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1501 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876439+0000", 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4279999999999999e-06 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952215+0000", 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075776423999999995 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952227+0000", 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2218999999999999e-05 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952234+0000", 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.533e-06 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952285+0000", 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1088000000000003e-05 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952410+0000", 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012510700000000001 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952516+0000", 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106437 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.957009+0000", 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044929569999999997 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969064+0000", 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012055066 2024-09-16T01:52:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969104+0000", 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9687000000000001e-05 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969150+0000", 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5861e-05 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1619 3.b 3:d196788b:::100000001f8.00000250:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.866842918, 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.094691490000000003, 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1619 2024-09-16T01:52:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331334+0000", 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4619999999999999e-06 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418738+0000", 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.087404198000000002 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418758+0000", 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9513000000000001e-05 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418768+0000", 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0291e-05 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418832+0000", 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3544e-05 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418975+0000", 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143361 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.419119+0000", 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014366699999999999 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425110+0000", 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059917099999999999 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425951+0000", 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084087400000000003 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425990+0000", 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8868999999999997e-05 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.426024+0000", 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4001000000000002e-05 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1953 3.7 3:e1d5d0f0:::100000001f8.0000039e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.949170148, 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.108386528, 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1953 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249006+0000", 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1179999999999999e-06 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307322+0000", 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058315141000000001 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307341+0000", 2024-09-16T01:52:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9011000000000001e-05 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307349+0000", 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307416+0000", 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7377000000000002e-05 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307582+0000", 2024-09-16T01:52:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016564800000000001 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307687+0000", 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104881 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.314441+0000", 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006754726 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357290+0000", 2024-09-16T01:52:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042848857999999997 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357330+0000", 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9477000000000002e-05 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357392+0000", 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2009999999999998e-05 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1968 3.12 3:4ba9d71c:::100000001f8.000003ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.746922123000001, 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10484845199999999, 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1968 2024-09-16T01:52:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451256+0000", 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483869+0000", 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032612532 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483888+0000", 2024-09-16T01:52:01.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9029e-05 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483894+0000", 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.674e-06 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483963+0000", 2024-09-16T01:52:01.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8381999999999997e-05 2024-09-16T01:52:01.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484065+0000", 2024-09-16T01:52:01.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010199 2024-09-16T01:52:01.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484168+0000", 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103727 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.489008+0000", 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048394880000000003 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556018+0000", 2024-09-16T01:52:01.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067010583999999998 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556061+0000", 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3062999999999997e-05 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556102+0000", 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0246000000000001e-05 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.286 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1988 3.12 3:4924d375:::100000001f8.000003c1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.434721018999999, 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098484843000000002, 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.287 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1988 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763456+0000", 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7290000000000001e-06 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819882+0000", 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056425945999999998 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819899+0000", 2024-09-16T01:52:01.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6441000000000001e-05 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819905+0000", 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8750000000000002e-06 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819953+0000", 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7774e-05 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820053+0000", 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9702000000000002e-05 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820148+0000", 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5147000000000002e-05 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.823215+0000", 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030665369999999998 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861864+0000", 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038649944999999998 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861903+0000", 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8126000000000001e-05 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861939+0000", 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6621000000000001e-05 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2002 3.33 3:cf002303:::100000001f8.000003cf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.991569+0000", 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.206606013, 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.091091670999999999, 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.293 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2002 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.991569+0000", 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.991569+0000", 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.991571+0000", 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2699999999999999e-06 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024450+0000", 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032879494000000002 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024469+0000", 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9083999999999999e-05 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024478+0000", 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9099999999999994e-06 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024546+0000", 2024-09-16T01:52:01.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8093e-05 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024658+0000", 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111702 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.024766+0000", 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108164 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.030065+0000", 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0052992789999999996 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.082555+0000", 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052489593000000001 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.082611+0000", 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6415000000000001e-05 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.082661+0000", 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9666999999999997e-05 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2037 3.12 3:486bd0c2:::100000001f8.000003f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.711105309000001, 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124209681, 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.299 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2037 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487072+0000", 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0590000000000001e-06 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523700+0000", 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036627383999999999 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523718+0000", 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8297000000000001e-05 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523725+0000", 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523780+0000", 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4598000000000001e-05 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523909+0000", 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012891700000000001 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.524030+0000", 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000120874 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.527427+0000", 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033977389999999999 2024-09-16T01:52:01.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611203+0000", 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083775383999999994 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611243+0000", 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114000000000003e-05 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.305 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.305 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611280+0000", 2024-09-16T01:52:01.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6965999999999998e-05 2024-09-16T01:52:01.305 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.305 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.305 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2090 3.1 3:83370771:::100000001f8.00000427:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.9658128369999996, 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.097529115999999999, 2024-09-16T01:52:01.306 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2090 2024-09-16T01:52:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232365+0000", 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.945e-06 2024-09-16T01:52:01.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264573+0000", 2024-09-16T01:52:01.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032208891000000003 2024-09-16T01:52:01.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264593+0000", 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9488999999999999e-05 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264601+0000", 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2050000000000002e-06 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264714+0000", 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011313499999999999 2024-09-16T01:52:01.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264867+0000", 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000152805 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.265004+0000", 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013696499999999999 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.268362+0000", 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033575039999999999 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329806+0000", 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061444812000000001 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329852+0000", 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5367000000000001e-05 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329892+0000", 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9997999999999999e-05 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2480 3.1 3:8263755d:::100000001f8.000005ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:55.786552+0000", 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.4116232179999999, 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077110988000000005, 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.315 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2480 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.786552+0000", 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.786552+0000", 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.786554+0000", 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9230000000000001e-06 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.814723+0000", 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028168479 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.814740+0000", 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7615999999999999e-05 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.814749+0000", 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9109999999999999e-06 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.814840+0000", 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1262999999999997e-05 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.814999+0000", 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015850200000000001 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.815175+0000", 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017596500000000001 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.818597+0000", 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034216580000000002 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.863501+0000", 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044904080999999998 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.863591+0000", 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0218999999999998e-05 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.863663+0000", 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2371000000000002e-05 2024-09-16T01:52:01.320 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2565 3.33 3:cd383402:::100000001f8.00000602:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.774101+0000", 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.4240742690000001, 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.081753302, 2024-09-16T01:52:01.321 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2565 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.774101+0000", 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.774101+0000", 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.774103+0000", 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6339999999999999e-06 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.829194+0000", 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055090998000000002 2024-09-16T01:52:01.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.829208+0000", 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3888e-05 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.829215+0000", 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1919999999999999e-06 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.829275+0000", 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9930999999999999e-05 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.829418+0000", 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143172 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.829567+0000", 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014897200000000001 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.833266+0000", 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036994720000000001 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.855764+0000", 2024-09-16T01:52:01.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022497891999999999 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.855809+0000", 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4926000000000003e-05 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.855854+0000", 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5225000000000002e-05 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2583 3.12 3:494b1864:::100000001f8.00000614:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:01.330 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.942819+0000", 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.2553562029999998, 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077848758000000004, 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:01.331 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2583 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.942819+0000", 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.942819+0000", 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.942820+0000", 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:52:01.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.009157+0000", 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066336206999999994 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.009175+0000", 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8844000000000001e-05 2024-09-16T01:52:01.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.009183+0000", 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5320000000000001e-06 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.009257+0000", 2024-09-16T01:52:01.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4379000000000001e-05 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.009397+0000", 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013923900000000001 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.009526+0000", 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000129534 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.013192+0000", 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036663720000000002 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.020594+0000", 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0074017629999999996 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.020635+0000", 2024-09-16T01:52:01.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0799e-05 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.020668+0000", 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2963999999999998e-05 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.338 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:01.340 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:01.341 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:01.338+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:01.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.433+0000 7f8730c00640 1 Processor -- start 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.433+0000 7f8730c00640 1 -- start start 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.435+0000 7f8730c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f872c158fe0 0x7f872c0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.435+0000 7f8730c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f872c071d20 0x7f872c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.435+0000 7f8730c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f872c0725b0 con 0x7f872c06fad0 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.435+0000 7f8730c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f872c072890 con 0x7f872c071d20 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.435+0000 7f8730c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f872c0c8110 con 0x7f872c158fe0 2024-09-16T01:52:01.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.436+0000 7f8729e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f872c158fe0 0x7f872c0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:01.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.436+0000 7f8721400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f872c071d20 0x7f872c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:01.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.436+0000 7f8721400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f872c071d20 0x7f872c06f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55624/0 (socket says 172.21.15.146:55624) 2024-09-16T01:52:01.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.436+0000 7f8721400640 1 -- 172.21.15.146:0/3460363631 learned_addr learned my addr 172.21.15.146:0/3460363631 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:01.441 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:01.440+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:01.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.438+0000 7f8721400640 1 -- 172.21.15.146:0/3460363631 >> v2:172.21.15.161:3300/0 conn(0x7f872c158fe0 msgr2=0x7f872c0c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.438+0000 7f8721400640 1 --2- 172.21.15.146:0/3460363631 >> v2:172.21.15.161:3300/0 conn(0x7f872c158fe0 0x7f872c0c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.438+0000 7f8721400640 1 -- 172.21.15.146:0/3460363631 >> v1:172.21.15.146:6789/0 conn(0x7f872c06fad0 legacy=0x7f872c15eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.438+0000 7f8721400640 1 -- 172.21.15.146:0/3460363631 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c0c77e0 con 0x7f872c071d20 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.439+0000 7f8721400640 1 --2- 172.21.15.146:0/3460363631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c071d20 0x7f872c06f590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f872c06d830 tx=0x7f871402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba61ca496acae21e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.440+0000 7f8729400640 1 -- 172.21.15.146:0/3460363631 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714037070 con 0x7f872c071d20 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.440+0000 7f8729400640 1 -- 172.21.15.146:0/3460363631 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8714032440 con 0x7f872c071d20 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.440+0000 7f8729400640 1 -- 172.21.15.146:0/3460363631 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714032810 con 0x7f872c071d20 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.441+0000 7f8730c00640 1 -- 172.21.15.146:0/3460363631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c071d20 msgr2=0x7f872c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:01.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.441+0000 7f8730c00640 1 --2- 172.21.15.146:0/3460363631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c071d20 0x7f872c06f590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f872c06d830 tx=0x7f871402a230 comp rx=0 tx=0).stop 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.441+0000 7f8730c00640 1 -- 172.21.15.146:0/3460363631 shutdown_connections 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.441+0000 7f8730c00640 1 --2- 172.21.15.146:0/3460363631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c071d20 0x7f872c06f590 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.441+0000 7f8730c00640 1 --2- 172.21.15.146:0/3460363631 >> v2:172.21.15.161:3300/0 conn(0x7f872c158fe0 0x7f872c0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.441+0000 7f8730c00640 1 -- 172.21.15.146:0/3460363631 >> 172.21.15.146:0/3460363631 conn(0x7f872c06e670 msgr2=0x7f872c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.442+0000 7f8730c00640 1 -- 172.21.15.146:0/3460363631 shutdown_connections 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.442+0000 7f8730c00640 1 -- 172.21.15.146:0/3460363631 wait complete. 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.444+0000 7f8730c00640 1 Processor -- start 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.444+0000 7f8730c00640 1 -- start start 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8730c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c06fad0 0x7f872c160870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8730c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 0x7f872c160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:01.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8730c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f872c158fe0 0x7f872c161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8730c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f872c0c9be0 con 0x7f872c06fad0 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8730c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f872c0c7da0 con 0x7f872c071d20 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8730c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f872c1593c0 con 0x7f872c158fe0 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8729e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c06fad0 0x7f872c160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8721400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 0x7f872c160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8721400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 0x7f872c160db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:50604/0 (socket says 172.21.15.146:50604) 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8721400640 1 -- 172.21.15.146:0/2628616444 learned_addr learned my addr 172.21.15.146:0/2628616444 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f872a800640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f872c158fe0 0x7f872c161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f872a800640 1 -- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 msgr2=0x7f872c160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f872a800640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 0x7f872c160db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f872a800640 1 -- 172.21.15.146:0/2628616444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c06fad0 msgr2=0x7f872c160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f872a800640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c06fad0 0x7f872c160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f872a800640 1 -- 172.21.15.146:0/2628616444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c1433f0 con 0x7f872c158fe0 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.445+0000 7f8721400640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 0x7f872c160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.446+0000 7f872a800640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f872c158fe0 0x7f872c161ad0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f872400b440 tx=0x7f872400b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.446+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724015020 con 0x7f872c158fe0 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.446+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8724011830 con 0x7f872c158fe0 2024-09-16T01:52:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.446+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724011c20 con 0x7f872c158fe0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.446+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872c0cca60 con 0x7f872c158fe0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.446+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f872c0cbf30 con 0x7f872c158fe0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.447+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872c070410 con 0x7f872c158fe0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.448+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8724018020 con 0x7f872c158fe0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.449+0000 7f8722a00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f870806c2b0 0x7f870806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.449+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 5770+0+0 (secure 0 0 0) 0x7f872408b1e0 con 0x7f872c158fe0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.449+0000 7f8729e00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f870806c2b0 0x7f870806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.450+0000 7f8729e00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f870806c2b0 0x7f870806e770 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f871c011340 tx=0x7f871c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:01.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.454+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f872c070410 con 0x7f872c158fe0 2024-09-16T01:52:01.495 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:01.495 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:01.495 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:01.495 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:01.504 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:01.549 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:01.548+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:01.644 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:01.643+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:01.664 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:01.664 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:01.665 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:01.665 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:01.665 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:01.676 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:01.749 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:01.748+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:01.804 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:01.804 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:01.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 172.94171758300001, 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:52:01.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:52:01.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:52:01.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 172.94161907599999, 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.811 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:52:01.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:52:01.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:17 3.21 e55/41 3:86f42aac:::10000000002.0000000e:head v 55'1, mlcod=0'0)", 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:52:01.817 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 71.425325150000006, 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011486753000000001, 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376928+0000", 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0950000000000001e-06 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385558+0000", 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086297260000000008 2024-09-16T01:52:01.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385576+0000", 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.806e-05 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385587+0000", 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1165999999999999e-05 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385660+0000", 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2346000000000002e-05 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385679+0000", 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9038999999999998e-05 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388352+0000", 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026728210000000001 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388414+0000", 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2500000000000001e-05 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:20 3.21 e55/41 3:85380590:::10000000002.00000011:head v 55'2, mlcod=0'0)", 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 71.351683015999996, 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011038759, 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.822 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450571+0000", 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.111e-06 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458973+0000", 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0084022200000000002 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458993+0000", 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9945e-05 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459003+0000", 2024-09-16T01:52:01.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9860000000000006e-06 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459070+0000", 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7600000000000003e-05 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459092+0000", 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1860000000000001e-05 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.461569+0000", 2024-09-16T01:52:01.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024764679999999999 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.461608+0000", 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9569e-05 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:62 3.c e55/35 3:31d8a57c:::10000000002.0000003b:head v 55'1, mlcod=0'0)", 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 70.963141034000003, 2024-09-16T01:52:01.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011246087, 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839113+0000", 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.167e-06 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847457+0000", 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0083445559999999995 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847465+0000", 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7719999999999994e-06 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847469+0000", 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0820000000000001e-06 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847509+0000", 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847524+0000", 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.850313+0000", 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027888370000000002 2024-09-16T01:52:01.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.850358+0000", 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4409000000000001e-05 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:772 3.21 e68/67 3:848ebb0b:::100000001f7.00000001:head v 68'21, mlcod=65'19)", 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.530857365000003, 2024-09-16T01:52:01.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025350340999999998, 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271396+0000", 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.144e-06 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293800+0000", 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022404103000000002 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293826+0000", 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5809999999999999e-05 2024-09-16T01:52:01.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293845+0000", 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8394e-05 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293958+0000", 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011299899999999999 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293987+0000", 2024-09-16T01:52:01.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8952999999999999e-05 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.296701+0000", 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027147180000000001 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.296745+0000", 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.422e-05 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.836 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1085 3.c e76/67 3:33b8095f:::100000001f8.0000003a:head v 76'25, mlcod=74'23)", 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.926276793, 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013510061, 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:01.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8799999999999999e-07 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885660+0000", 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0096844449999999999 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885680+0000", 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9292000000000001e-05 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885689+0000", 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7920000000000004e-06 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885767+0000", 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7769999999999996e-05 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885790+0000", 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2697999999999999e-05 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.889443+0000", 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036530579999999998 2024-09-16T01:52:01.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.889486+0000", 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2818000000000002e-05 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1088 3.2e e76/67 3:75be0f36:::100000001f8.0000003d:head v 76'22, mlcod=74'20)", 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.886126304000001, 2024-09-16T01:52:01.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010871123999999999, 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.916126+0000", 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6399999999999998e-07 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923588+0000", 2024-09-16T01:52:01.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.007461479 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923612+0000", 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3785999999999999e-05 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923625+0000", 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3166999999999999e-05 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923698+0000", 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3603000000000007e-05 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.923725+0000", 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6622999999999999e-05 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.926947+0000", 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003222267 2024-09-16T01:52:01.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.926997+0000", 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9935000000000002e-05 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1134 3.21 e76/67 3:87440b43:::100000001f8.0000006b:head v 76'37, mlcod=74'35)", 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.376986053, 2024-09-16T01:52:01.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016188002, 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425267+0000", 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0200000000000001e-07 2024-09-16T01:52:01.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438526+0000", 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013259675 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438543+0000", 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7163000000000002e-05 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438555+0000", 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1928e-05 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438633+0000", 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7935999999999994e-05 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438653+0000", 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9695999999999999e-05 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.441400+0000", 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027465649999999999 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.441454+0000", 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4836999999999997e-05 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1678 3.21 e81/67 3:8795a5a4:::100000001f8.0000028b:head v 81'47, mlcod=81'45)", 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:52:01.851 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.793870837, 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011676199, 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008383+0000", 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6500000000000008e-07 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015309+0000", 2024-09-16T01:52:01.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0069268530000000002 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015320+0000", 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0733999999999999e-05 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015326+0000", 2024-09-16T01:52:01.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3279999999999996e-06 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015402+0000", 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6369e-05 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015426+0000", 2024-09-16T01:52:01.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3966000000000001e-05 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.020011+0000", 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045850730000000003 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.020058+0000", 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6910999999999999e-05 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.858 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1927 3.8 e84/83 3:13b44a54:::100000001f8.00000384:head v 84'56, mlcod=0'0)", 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 12.631456489, 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011956664000000001, 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.85e-07 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179466+0000", 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086694140000000003 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179476+0000", 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0341e-05 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179482+0000", 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.353e-06 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179532+0000", 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9956000000000001e-05 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179556+0000", 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3456e-05 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.182704+0000", 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031482120000000001 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.182753+0000", 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8746999999999999e-05 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1947 3.2c e84/77 3:3557c0c2:::100000001f8.00000398:head v 84'41, mlcod=81'39)", 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:52:01.863 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 12.514625597, 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016887692999999999, 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287629+0000", 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300794+0000", 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013164334999999999 2024-09-16T01:52:01.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300803+0000", 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2520000000000005e-06 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300809+0000", 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2790000000000004e-06 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300867+0000", 2024-09-16T01:52:01.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7485000000000003e-05 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300890+0000", 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.372e-05 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.304472+0000", 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035818379999999999 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.304515+0000", 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2311999999999997e-05 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2059 3.6 e85/83 3:6235bc65:::100000001f8.00000408:head v 85'44, mlcod=60'4)", 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.970049689, 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010790019, 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.868 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.832203+0000", 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3499999999999996e-07 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836745+0000", 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045413040000000003 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836759+0000", 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4260999999999999e-05 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836765+0000", 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3249999999999996e-06 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836846+0000", 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0672000000000003e-05 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.836866+0000", 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0489e-05 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.842932+0000", 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0060658760000000004 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.842993+0000", 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0557000000000003e-05 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.872 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2105 3.21 e86/67 3:84901bec:::100000001f8.00000436:head v 86'54, mlcod=83'52)", 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.306949931, 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016103329, 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6400000000000001e-07 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507953+0000", 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012649781000000001 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507969+0000", 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507977+0000", 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8250000000000005e-06 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.508038+0000", 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0900000000000003e-05 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.508061+0000", 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2453999999999999e-05 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.511370+0000", 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003309181 2024-09-16T01:52:01.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.511406+0000", 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6056999999999997e-05 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2381 3.34 e88/83 3:2c202722:::100000001f8.0000054a:head v 88'56, mlcod=87'54)", 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 7.0353192189999998, 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012640500000000001, 2024-09-16T01:52:01.877 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766935+0000", 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.606e-06 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774938+0000", 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080026449999999992 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774955+0000", 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7736000000000001e-05 2024-09-16T01:52:01.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774965+0000", 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4930000000000004e-06 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.775023+0000", 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8697999999999999e-05 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.775047+0000", 2024-09-16T01:52:01.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3889000000000001e-05 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.779505+0000", 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044578710000000004 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.779574+0000", 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8561999999999998e-05 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.881 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2520 3.2e e88/67 3:75f1801c:::100000001f8.000005d5:head v 88'45, mlcod=88'43)", 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.324696+0000", 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.4775563439999999, 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012355377000000001, 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.324696+0000", 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.324696+0000", 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.324697+0000", 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2500000000000004e-07 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333359+0000", 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086616520000000006 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333376+0000", 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7377999999999999e-05 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333385+0000", 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2229999999999996e-06 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333455+0000", 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9479000000000003e-05 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333480+0000", 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5079000000000002e-05 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.336995+0000", 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035154869999999999 2024-09-16T01:52:01.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.337052+0000", 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6153999999999998e-05 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2575 3.34 e88/83 3:2f1ca3fd:::100000001f8.0000060c:head v 88'61, mlcod=88'59)", 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.920492+0000", 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.8817609129999999, 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016208686999999999, 2024-09-16T01:52:01.886 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.920492+0000", 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.920492+0000", 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.920493+0000", 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.879999999999999e-07 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933099+0000", 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012606640000000001 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933114+0000", 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.438e-05 2024-09-16T01:52:01.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933120+0000", 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8990000000000004e-06 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933181+0000", 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0646999999999997e-05 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933205+0000", 2024-09-16T01:52:01.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.351e-05 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.936629+0000", 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034247420000000002 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.936700+0000", 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0881000000000001e-05 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2593 3.2c e88/77 3:35ced6af:::100000001f8.0000061e:head v 88'53, mlcod=88'51)", 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:57.150732+0000", 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.6515202670000004, 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010984147, 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.150732+0000", 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.150732+0000", 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.150734+0000", 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9539999999999998e-06 2024-09-16T01:52:01.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.156131+0000", 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053964670000000003 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.156143+0000", 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2349e-05 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.156149+0000", 2024-09-16T01:52:01.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0519999999999998e-06 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.156207+0000", 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8239999999999998e-05 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.156253+0000", 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5686999999999998e-05 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.161685+0000", 2024-09-16T01:52:01.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005431572 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:57.161716+0000", 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1826000000000003e-05 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2856 3.c e89/67 3:333b8194:::100000001f8.00000725:head v 89'51, mlcod=89'49)", 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:59.956289+0000", 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 1.8459632099999999, 2024-09-16T01:52:01.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012073321, 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.956289+0000", 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.956289+0000", 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.956291+0000", 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965386+0000", 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0090953500000000003 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965411+0000", 2024-09-16T01:52:01.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4992999999999999e-05 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965422+0000", 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1317e-05 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965488+0000", 2024-09-16T01:52:01.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5575999999999997e-05 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965507+0000", 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9029e-05 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.968322+0000", 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028148380000000001 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.968363+0000", 2024-09-16T01:52:01.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0729000000000003e-05 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:3010 3.2e e90/67 3:7418dfc9:::100000001f8.000007bf:head v 90'53, mlcod=90'51)", 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.20709545400000001, 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012532477, 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:01.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4700000000000002e-07 2024-09-16T01:52:01.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603459+0000", 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0083016960000000008 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603480+0000", 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0397000000000001e-05 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603493+0000", 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3293e-05 2024-09-16T01:52:01.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603570+0000", 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7446000000000004e-05 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603598+0000", 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7463000000000001e-05 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.607657+0000", 2024-09-16T01:52:01.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004059287 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.607689+0000", 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2447999999999998e-05 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:01.904 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:01.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:01.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:01.811+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 0.43369510715733894} v 0) -- 0x7f872c072140 con 0x7f872c158fe0 2024-09-16T01:52:01.906 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:01.905+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:01.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:02.007 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:02.006+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:02.108 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:02.107+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:02.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.158+0000 7f7ff2600640 1 Processor -- start 2024-09-16T01:52:02.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.158+0000 7f7ff2600640 1 -- start start 2024-09-16T01:52:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.159+0000 7f7ff2600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.159+0000 7f7ff2600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7fec0d0810 0x7f7fec15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.159+0000 7f7ff2600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fec0cb600 con 0x7f7fec154f40 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.159+0000 7f7ff2600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7fec0cbbc0 con 0x7f7fec0d0810 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.159+0000 7f7ff2600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7fec155320 con 0x7f7fec0c7770 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.159+0000 7f7feb400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7fec154f40 0x7f7fec15eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46572/0 (socket says 172.21.15.146:46572) 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7feb400640 1 -- 172.21.15.146:0/599375975 learned_addr learned my addr 172.21.15.146:0/599375975 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7febe00640 1 --2- 172.21.15.146:0/599375975 >> v2:172.21.15.161:3300/0 conn(0x7f7fec0d0810 0x7f7fec15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7feaa00640 1 --2- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7febe00640 1 -- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 msgr2=0x7f7fec0d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:02.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7febe00640 1 --2- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7febe00640 1 -- 172.21.15.146:0/599375975 >> v1:172.21.15.146:6789/0 conn(0x7f7fec154f40 legacy=0x7f7fec15eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.160+0000 7f7febe00640 1 -- 172.21.15.146:0/599375975 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fec0cb8e0 con 0x7f7fec0d0810 2024-09-16T01:52:02.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.161+0000 7f7febe00640 1 --2- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec15ab70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009d90 tx=0x7f7fe402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51e5362676f8b53b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.162+0000 7f7fea000640 1 -- 172.21.15.146:0/599375975 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4007740 con 0x7f7fec0d0810 2024-09-16T01:52:02.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.162+0000 7f7fea000640 1 -- 172.21.15.146:0/599375975 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe40078e0 con 0x7f7fec0d0810 2024-09-16T01:52:02.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.162+0000 7f7fea000640 1 -- 172.21.15.146:0/599375975 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4007cf0 con 0x7f7fec0d0810 2024-09-16T01:52:02.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 -- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 msgr2=0x7f7fec15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 --2- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec15ab70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009d90 tx=0x7f7fe402a680 comp rx=0 tx=0).stop 2024-09-16T01:52:02.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 -- 172.21.15.146:0/599375975 shutdown_connections 2024-09-16T01:52:02.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 --2- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 --2- 172.21.15.146:0/599375975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec15ab70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 -- 172.21.15.146:0/599375975 >> 172.21.15.146:0/599375975 conn(0x7f7fec06cb50 msgr2=0x7f7fec06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:02.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 -- 172.21.15.146:0/599375975 shutdown_connections 2024-09-16T01:52:02.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.163+0000 7f7ff2600640 1 -- 172.21.15.146:0/599375975 wait complete. 2024-09-16T01:52:02.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.165+0000 7f7ff2600640 1 Processor -- start 2024-09-16T01:52:02.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 -- start start 2024-09-16T01:52:02.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fec154f40 0x7f7fec1f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec0b6210 con 0x7f7fec154f40 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7fec0c7b90 con 0x7f7fec0d0810 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.166+0000 7f7ff2600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7fec15b0b0 con 0x7f7fec0c7770 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0c40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:49070/0 (socket says 172.21.15.146:49070) 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 -- 172.21.15.146:0/2040607379 learned_addr learned my addr 172.21.15.146:0/2040607379 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fec154f40 0x7f7fec1f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 msgr2=0x7f7fec0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fec154f40 msgr2=0x7f7fec1f48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fec154f40 0x7f7fec1f48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feb400640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fec15a540 con 0x7f7fec0c7770 2024-09-16T01:52:02.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.167+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fec0d0810 0x7f7fec0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.168+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0c40e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c3f0 tx=0x7f7fdc00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.168+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc017070 con 0x7f7fec0c7770 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.168+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fdc005ce0 con 0x7f7fec0c7770 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.168+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fec15a820 con 0x7f7fec0c7770 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.168+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0072f0 con 0x7f7fec0c7770 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.168+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fec1f40b0 con 0x7f7fec0c7770 2024-09-16T01:52:02.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.169+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7fec1f44e0 con 0x7f7fec0c7770 2024-09-16T01:52:02.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.170+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7fdc01a070 con 0x7f7fec0c7770 2024-09-16T01:52:02.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.171+0000 7f7fe3400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7fcc06c2b0 0x7f7fcc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.171+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 5770+0+0 (secure 0 0 0) 0x7f7fdc08a770 con 0x7f7fec0c7770 2024-09-16T01:52:02.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.171+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7fcc06c2b0 0x7f7fcc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.171+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v90) ==== 54+0+16 (secure 0 0 0) 0x7f7fdc090170 con 0x7f7fec0c7770 2024-09-16T01:52:02.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.172+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7fec159fe0 0x7f7fec14d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.172+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fec15b490 con 0x7f7fec159fe0 2024-09-16T01:52:02.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.172+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7fec159fe0 0x7f7fec14d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.172+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7fcc06c2b0 0x7f7fcc06e770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0b5eb0 tx=0x7f7fd8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.172+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7fec159fe0 0x7f7fec14d550 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.173+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec15b490 con 0x7f7fec159fe0 2024-09-16T01:52:02.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.188+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec0cbd70 con 0x7f7fec159fe0 2024-09-16T01:52:02.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.190+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec0cbd70 con 0x7f7fec159fe0 2024-09-16T01:52:02.191 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:02.191 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:02.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.190+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7fec1556f0 0x7f7fec1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.190+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7fec0cc8b0 con 0x7f7fec1556f0 2024-09-16T01:52:02.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.191+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7fec1556f0 0x7f7fec1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.191+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7fec1556f0 0x7f7fec1587d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.192+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec0cc8b0 con 0x7f7fec1556f0 2024-09-16T01:52:02.208 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:02.207+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:02.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.222+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec0c86b0 con 0x7f7fec1556f0 2024-09-16T01:52:02.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.225+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec0c86b0 con 0x7f7fec1556f0 2024-09-16T01:52:02.226 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:02.226 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:02.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.225+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7fec159220 0x7f7fec062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.225+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7fec159620 con 0x7f7fec159220 2024-09-16T01:52:02.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.225+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7fec159220 0x7f7fec062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.226+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7fec159220 0x7f7fec062630 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.227+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fd801b040 con 0x7f7fec159220 2024-09-16T01:52:02.234 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:02.234 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:02.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.256+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec0c83d0 con 0x7f7fec159220 2024-09-16T01:52:02.258 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:02.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.258+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec0c83d0 con 0x7f7fec159220 2024-09-16T01:52:02.260 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:02.260 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:02.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.259+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7fec063080 0x7f7fec150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.259+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7fec063500 con 0x7f7fec063080 2024-09-16T01:52:02.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.259+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7fec063080 0x7f7fec150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.260+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7fec063080 0x7f7fec150d60 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.261+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec063500 con 0x7f7fec063080 2024-09-16T01:52:02.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.285+0000 7f8722a00640 1 -- 172.21.15.146:0/2628616444 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 0.43369510715733894}]=0 reweighted osd.7 to 0.433695 (6f06) v91) ==== 135+0+0 (secure 0 0 0) 0x7f872401f020 con 0x7f872c158fe0 2024-09-16T01:52:02.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.290+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec063500 con 0x7f7fec063080 2024-09-16T01:52:02.291 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.7 to 0.433695 (6f06) 2024-09-16T01:52:02.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.293+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec063500 con 0x7f7fec063080 2024-09-16T01:52:02.294 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:02.294 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:02.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.293+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7fec02c640 0x7f7fec02cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.293+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7fec063500 con 0x7f7fec02c640 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.295+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f870806c2b0 msgr2=0x7f870806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.295+0000 7f8730c00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f870806c2b0 0x7f870806e770 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f871c011340 tx=0x7f871c00b040 comp rx=0 tx=0).stop 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.295+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f872c158fe0 msgr2=0x7f872c161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.295+0000 7f8730c00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f872c158fe0 0x7f872c161ad0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f872400b440 tx=0x7f872400b900 comp rx=0 tx=0).stop 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 shutdown_connections 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f870806c2b0 0x7f870806e770 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f872c158fe0 0x7f872c161ad0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f872c071d20 0x7f872c160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 --2- 172.21.15.146:0/2628616444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f872c06fad0 0x7f872c160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 >> 172.21.15.146:0/2628616444 conn(0x7f872c06e670 msgr2=0x7f872c157280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 shutdown_connections 2024-09-16T01:52:02.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.296+0000 7f8730c00640 1 -- 172.21.15.146:0/2628616444 wait complete. 2024-09-16T01:52:02.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.302+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7fec02c640 0x7f7fec02cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.303+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7fec02c640 0x7f7fec02cac0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.304+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec063500 con 0x7f7fec02c640 2024-09-16T01:52:02.311 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:02.310+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:02.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:52:02.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.335+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec063500 con 0x7f7fec02c640 2024-09-16T01:52:02.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.341+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec063500 con 0x7f7fec02c640 2024-09-16T01:52:02.342 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:02.343 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:02.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.341+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7fec02d510 0x7f7fec207400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.341+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7fec151c40 con 0x7f7fec02d510 2024-09-16T01:52:02.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.345+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7fec02d510 0x7f7fec207400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.348+0000 7f7feaa00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7fec02d510 0x7f7fec207400 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.349+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec151c40 con 0x7f7fec02d510 2024-09-16T01:52:02.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.375+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec02d970 con 0x7f7fec02d510 2024-09-16T01:52:02.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.382+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec02d970 con 0x7f7fec02d510 2024-09-16T01:52:02.384 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:02.384 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:02.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.383+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7fec207e50 0x7f7fec20a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.383+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7fec20d090 con 0x7f7fec207e50 2024-09-16T01:52:02.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.384+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7fec207e50 0x7f7fec20a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.387+0000 7f7febe00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7fec207e50 0x7f7fec20a240 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.388+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec20d090 con 0x7f7fec207e50 2024-09-16T01:52:02.411 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:02.410+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.442+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec20aa50 con 0x7f7fec207e50 2024-09-16T01:52:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.446+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec20aa50 con 0x7f7fec207e50 2024-09-16T01:52:02.447 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.446+0000 7f7ff2600640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7fec20b170 0x7f7fec20b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.447+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7fec210400 con 0x7f7fec20b170 2024-09-16T01:52:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.447+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7fec20b170 0x7f7fec20b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.448+0000 7f7feb400640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7fec20b170 0x7f7fec20b5b0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.448+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7fec210400 con 0x7f7fec20b170 2024-09-16T01:52:02.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.479+0000 7f7ff2600640 1 -- 172.21.15.146:0/2040607379 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7fec20f280 con 0x7f7fec20b170 2024-09-16T01:52:02.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.482+0000 7f7fe3400640 1 -- 172.21.15.146:0/2040607379 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7fec20f280 con 0x7f7fec20b170 2024-09-16T01:52:02.485 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:02.485 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:02.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.484+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7fec159fe0 msgr2=0x7f7fec14d550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.484+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f7fec159fe0 0x7f7fec14d550 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7fec1556f0 msgr2=0x7f7fec1587d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f7fec1556f0 0x7f7fec1587d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7fec159220 msgr2=0x7f7fec062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f7fec159220 0x7f7fec062630 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7fec063080 msgr2=0x7f7fec150d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f7fec063080 0x7f7fec150d60 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7fec02c640 msgr2=0x7f7fec02cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f7fec02c640 0x7f7fec02cac0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7fec02d510 msgr2=0x7f7fec207400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f7fec02d510 0x7f7fec207400 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7fec207e50 msgr2=0x7f7fec20a240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f7fec207e50 0x7f7fec20a240 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7fec20b170 msgr2=0x7f7fec20b5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f7fec20b170 0x7f7fec20b5b0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7febe00640 1 -- 172.21.15.146:0/2040607379 reap_dead start 2024-09-16T01:52:02.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7febe00640 1 -- 172.21.15.146:0/2040607379 reap_dead start 2024-09-16T01:52:02.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7febe00640 1 -- 172.21.15.146:0/2040607379 reap_dead start 2024-09-16T01:52:02.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7febe00640 1 -- 172.21.15.146:0/2040607379 reap_dead start 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.486+0000 7f7febe00640 1 -- 172.21.15.146:0/2040607379 reap_dead start 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.487+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7fcc06c2b0 msgr2=0x7f7fcc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.487+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7fcc06c2b0 0x7f7fcc06e770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0b5eb0 tx=0x7f7fd8009190 comp rx=0 tx=0).stop 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.487+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 msgr2=0x7f7fec0c40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.487+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0c40e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c3f0 tx=0x7f7fdc00c8b0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.488+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 shutdown_connections 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.488+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7fcc06c2b0 0x7f7fcc06e770 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.488+0000 7f7fe0c00640 1 --2- 172.21.15.146:0/2040607379 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fec0c7770 0x7f7fec0c40e0 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00c3f0 tx=0x7f7fdc00c8b0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.488+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 >> 172.21.15.146:0/2040607379 conn(0x7f7fec06cb50 msgr2=0x7f7fec153170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.488+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 shutdown_connections 2024-09-16T01:52:02.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.488+0000 7f7fe0c00640 1 -- 172.21.15.146:0/2040607379 wait complete. 2024-09-16T01:52:02.511 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:02.510+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:02.585 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:02.585 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:02.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:02.611 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:02.610+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:02.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.628+0000 7f6a89c00640 1 Processor -- start 2024-09-16T01:52:02.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.628+0000 7f6a89c00640 1 -- start start 2024-09-16T01:52:02.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.630+0000 7f6a89c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.630+0000 7f6a89c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a840c8b10 0x7f6a840c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.630+0000 7f6a89c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8415f430 con 0x7f6a84159320 2024-09-16T01:52:02.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.630+0000 7f6a89c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6a84162770 con 0x7f6a840c8b10 2024-09-16T01:52:02.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.630+0000 7f6a89c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8415f710 con 0x7f6a84158f40 2024-09-16T01:52:02.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.631+0000 7f6a83400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a840c8b10 0x7f6a840c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.632+0000 7f6a82000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6a84159320 0x7f6a84162c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:38968/0 (socket says 172.21.15.146:38968) 2024-09-16T01:52:02.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.632+0000 7f6a82000640 1 -- 172.21.15.146:0/358049005 learned_addr learned my addr 172.21.15.146:0/358049005 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:02.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.632+0000 7f6a82a00640 1 --2- 172.21.15.146:0/358049005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.633+0000 7f6a82a00640 1 -- 172.21.15.146:0/358049005 >> v2:172.21.15.161:3300/0 conn(0x7f6a840c8b10 msgr2=0x7f6a840c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:02.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.633+0000 7f6a82a00640 1 --2- 172.21.15.146:0/358049005 >> v2:172.21.15.161:3300/0 conn(0x7f6a840c8b10 0x7f6a840c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.633+0000 7f6a82a00640 1 -- 172.21.15.146:0/358049005 >> v1:172.21.15.161:6789/0 conn(0x7f6a84159320 legacy=0x7f6a84162c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.633+0000 7f6a82a00640 1 -- 172.21.15.146:0/358049005 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8415f9f0 con 0x7f6a84158f40 2024-09-16T01:52:02.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.634+0000 7f6a82a00640 1 --2- 172.21.15.146:0/358049005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8415ee10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6a740097d0 tx=0x7f6a7402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7660c461bef463 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.635+0000 7f6a81600640 1 -- 172.21.15.146:0/358049005 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74037070 con 0x7f6a84158f40 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.635+0000 7f6a81600640 1 -- 172.21.15.146:0/358049005 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a74032440 con 0x7f6a84158f40 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.635+0000 7f6a81600640 1 -- 172.21.15.146:0/358049005 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74032810 con 0x7f6a84158f40 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.635+0000 7f6a89c00640 1 -- 172.21.15.146:0/358049005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 msgr2=0x7f6a8415ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.635+0000 7f6a89c00640 1 --2- 172.21.15.146:0/358049005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8415ee10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6a740097d0 tx=0x7f6a7402a250 comp rx=0 tx=0).stop 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.638+0000 7f6a89c00640 1 -- 172.21.15.146:0/358049005 shutdown_connections 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.638+0000 7f6a89c00640 1 --2- 172.21.15.146:0/358049005 >> v2:172.21.15.161:3300/0 conn(0x7f6a840c8b10 0x7f6a840c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.638+0000 7f6a89c00640 1 --2- 172.21.15.146:0/358049005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8415ee10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.638+0000 7f6a89c00640 1 -- 172.21.15.146:0/358049005 >> 172.21.15.146:0/358049005 conn(0x7f6a8406f3c0 msgr2=0x7f6a8406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.639+0000 7f6a89c00640 1 -- 172.21.15.146:0/358049005 shutdown_connections 2024-09-16T01:52:02.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.639+0000 7f6a89c00640 1 -- 172.21.15.146:0/358049005 wait complete. 2024-09-16T01:52:02.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.642+0000 7f6a89c00640 1 Processor -- start 2024-09-16T01:52:02.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.642+0000 7f6a89c00640 1 -- start start 2024-09-16T01:52:02.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.643+0000 7f6a89c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a840c8b10 0x7f6a8415a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.644+0000 7f6a89c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8406fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.644+0000 7f6a89c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a84159320 0x7f6a842010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.644+0000 7f6a89c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a84163310 con 0x7f6a840c8b10 2024-09-16T01:52:02.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.644+0000 7f6a89c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a84162400 con 0x7f6a84159320 2024-09-16T01:52:02.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.644+0000 7f6a89c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a84072070 con 0x7f6a84158f40 2024-09-16T01:52:02.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.646+0000 7f6a82a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a840c8b10 0x7f6a8415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.646+0000 7f6a83400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a84159320 0x7f6a842010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.646+0000 7f6a82000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8406fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.646+0000 7f6a82000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8406fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:49094/0 (socket says 172.21.15.146:49094) 2024-09-16T01:52:02.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.646+0000 7f6a82000640 1 -- 172.21.15.146:0/587176626 learned_addr learned my addr 172.21.15.146:0/587176626 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:02.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.647+0000 7f6a82000640 1 -- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a84159320 msgr2=0x7f6a842010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.647+0000 7f6a82000640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a84159320 0x7f6a842010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.647+0000 7f6a82000640 1 -- 172.21.15.146:0/587176626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a840c8b10 msgr2=0x7f6a8415a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:02.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.647+0000 7f6a82000640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a840c8b10 0x7f6a8415a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:02.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.647+0000 7f6a82000640 1 -- 172.21.15.146:0/587176626 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a841fe3f0 con 0x7f6a84158f40 2024-09-16T01:52:02.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a82a00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a840c8b10 0x7f6a8415a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:02.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a82000640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8406fa10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c3f0 tx=0x7f6a7000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70017070 con 0x7f6a84158f40 2024-09-16T01:52:02.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a70005ce0 con 0x7f6a84158f40 2024-09-16T01:52:02.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a700072d0 con 0x7f6a84158f40 2024-09-16T01:52:02.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a840ca0e0 con 0x7f6a84158f40 2024-09-16T01:52:02.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.648+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a840ca380 con 0x7f6a84158f40 2024-09-16T01:52:02.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.650+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a84071930 con 0x7f6a84158f40 2024-09-16T01:52:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.651+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a7001a070 con 0x7f6a84158f40 2024-09-16T01:52:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.653+0000 7f6a7aa00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a6406c2b0 0x7f6a6406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.653+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(91..91 src has 1..91) ==== 5770+0+0 (secure 0 0 0) 0x7f6a7008a6a0 con 0x7f6a84158f40 2024-09-16T01:52:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.653+0000 7f6a82a00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a6406c2b0 0x7f6a6406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.654+0000 7f6a82a00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a6406c2b0 0x7f6a6406e770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a74034000 tx=0x7f6a7402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.657+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6a70058770 con 0x7f6a84158f40 2024-09-16T01:52:02.713 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:02.712+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:02.813 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:02.812+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:02.890 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:02.890 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:02.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:02.918 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:02.917+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:02.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:02.968+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f6a840c9b60 con 0x7f6a84158f40 2024-09-16T01:52:03.021 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:03.020+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:03.123 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:03.122+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:03.194 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:03.194 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:03.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:03.256 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:03.255+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:03.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.293+0000 7f6a7aa00640 1 -- 172.21.15.146:0/587176626 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v92) ==== 101+0+0 (secure 0 0 0) 0x7f6a7005ca70 con 0x7f6a84158f40 2024-09-16T01:52:03.297 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:52:03.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.298+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a6406c2b0 msgr2=0x7f6a6406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:03.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.298+0000 7f6a89c00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a6406c2b0 0x7f6a6406e770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a74034000 tx=0x7f6a7402e000 comp rx=0 tx=0).stop 2024-09-16T01:52:03.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.298+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 msgr2=0x7f6a8406fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:03.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.298+0000 7f6a89c00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8406fa10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000c3f0 tx=0x7f6a7000c8b0 comp rx=0 tx=0).stop 2024-09-16T01:52:03.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.300+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 shutdown_connections 2024-09-16T01:52:03.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.300+0000 7f6a89c00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a6406c2b0 0x7f6a6406e770 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:03.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.300+0000 7f6a89c00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a84159320 0x7f6a842010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:03.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.300+0000 7f6a89c00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a84158f40 0x7f6a8406fa10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:03.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.300+0000 7f6a89c00640 1 --2- 172.21.15.146:0/587176626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a840c8b10 0x7f6a8415a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:03.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.300+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 >> 172.21.15.146:0/587176626 conn(0x7f6a8406f3c0 msgr2=0x7f6a84157fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:03.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.302+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 shutdown_connections 2024-09-16T01:52:03.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:03.303+0000 7f6a89c00640 1 -- 172.21.15.146:0/587176626 wait complete. 2024-09-16T01:52:03.327 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:03.325+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:03.428 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:03.427+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:03.531 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:03.529+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:03.550 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:03.550 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:03.579 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:03.630 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:03.629+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:03.734 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:03.733+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:03.837 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:03.836+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:03.942 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:03.936+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:03.997 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:03.997 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:04.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:04.063 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:04.063+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:04.141 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:04.136+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:04.238 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:04.237+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:04.339 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:04.338+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:04.405 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:04.406 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:04.440 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:04.439+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:04.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:04.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:04.541 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:04.540+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:04.642 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:04.641+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:04.743 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:04.742+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:04.848 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:04.848 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:04.848 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:04.845+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.868+0000 7fa66a400640 1 Processor -- start 2024-09-16T01:52:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.869+0000 7fa66a400640 1 -- start start 2024-09-16T01:52:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa66a400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa6640c7c40 0x7fa6641676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa66a400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa66406f5f0 0x7fa6640c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa66a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa664072b20 con 0x7fa664164f90 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa66a400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa664165370 con 0x7fa6640c7c40 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa66a400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa664072e00 con 0x7fa66406f5f0 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa663400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa664164f90 0x7fa664072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46696/0 (socket says 172.21.15.146:46696) 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.870+0000 7fa663400640 1 -- 172.21.15.146:0/869013058 learned_addr learned my addr 172.21.15.146:0/869013058 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.875+0000 7fa663e00640 1 --2- 172.21.15.146:0/869013058 >> v2:172.21.15.161:3300/0 conn(0x7fa66406f5f0 0x7fa6640c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.878+0000 7fa662000640 1 -- 172.21.15.146:0/869013058 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405522572 0 0) 0x7fa664072b20 con 0x7fa664164f90 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.882+0000 7fa662000640 1 -- 172.21.15.146:0/869013058 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa648003500 con 0x7fa664164f90 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.876+0000 7fa662a00640 1 --2- 172.21.15.146:0/869013058 >> v2:172.21.15.146:3300/0 conn(0x7fa6640c7c40 0x7fa6641676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.883+0000 7fa662a00640 1 -- 172.21.15.146:0/869013058 >> v2:172.21.15.161:3300/0 conn(0x7fa66406f5f0 msgr2=0x7fa6640c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.883+0000 7fa662a00640 1 --2- 172.21.15.146:0/869013058 >> v2:172.21.15.161:3300/0 conn(0x7fa66406f5f0 0x7fa6640c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.883+0000 7fa662a00640 1 -- 172.21.15.146:0/869013058 >> v1:172.21.15.146:6789/0 conn(0x7fa664164f90 legacy=0x7fa664072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.883+0000 7fa662a00640 1 -- 172.21.15.146:0/869013058 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6640730e0 con 0x7fa6640c7c40 2024-09-16T01:52:04.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.890+0000 7fa662a00640 1 --2- 172.21.15.146:0/869013058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6640c7c40 0x7fa6641676b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa6640c9d50 tx=0x7fa65002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b3e79e3d8ae196 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.891+0000 7fa662000640 1 -- 172.21.15.146:0/869013058 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650037070 con 0x7fa6640c7c40 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.891+0000 7fa662000640 1 -- 172.21.15.146:0/869013058 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa650032440 con 0x7fa6640c7c40 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.891+0000 7fa662000640 1 -- 172.21.15.146:0/869013058 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650032810 con 0x7fa6640c7c40 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.892+0000 7fa66a400640 1 -- 172.21.15.146:0/869013058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6640c7c40 msgr2=0x7fa6641676b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.892+0000 7fa66a400640 1 --2- 172.21.15.146:0/869013058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6640c7c40 0x7fa6641676b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa6640c9d50 tx=0x7fa65002a230 comp rx=0 tx=0).stop 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.892+0000 7fa66a400640 1 -- 172.21.15.146:0/869013058 shutdown_connections 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.892+0000 7fa66a400640 1 --2- 172.21.15.146:0/869013058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6640c7c40 0x7fa6641676b0 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa6640c9d50 tx=0x7fa65002a230 comp rx=0 tx=0).stop 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.892+0000 7fa66a400640 1 --2- 172.21.15.146:0/869013058 >> v2:172.21.15.161:3300/0 conn(0x7fa66406f5f0 0x7fa6640c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.892+0000 7fa66a400640 1 -- 172.21.15.146:0/869013058 >> 172.21.15.146:0/869013058 conn(0x7fa66406bc10 msgr2=0x7fa66406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:04.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.898+0000 7fa66a400640 1 -- 172.21.15.146:0/869013058 shutdown_connections 2024-09-16T01:52:04.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.898+0000 7fa66a400640 1 -- 172.21.15.146:0/869013058 wait complete. 2024-09-16T01:52:04.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.921+0000 7fa66a400640 1 Processor -- start 2024-09-16T01:52:04.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.928+0000 7fa66a400640 1 -- start start 2024-09-16T01:52:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.937+0000 7fa66a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66406f5f0 0x7fa664169710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.940+0000 7fa66a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa664164f90 0x7fa66415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.940+0000 7fa663400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66406f5f0 0x7fa664169710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.940+0000 7fa66a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66415e9f0 0x7fa6640c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.940+0000 7fa66a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa66406f0d0 con 0x7fa66406f5f0 2024-09-16T01:52:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.940+0000 7fa66a400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa66406edf0 con 0x7fa66415e9f0 2024-09-16T01:52:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.940+0000 7fa66a400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa6640c3860 con 0x7fa664164f90 2024-09-16T01:52:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.941+0000 7fa663e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66415e9f0 0x7fa6640c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.941+0000 7fa663e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66415e9f0 0x7fa6640c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:35050/0 (socket says 172.21.15.146:35050) 2024-09-16T01:52:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.941+0000 7fa663e00640 1 -- 172.21.15.146:0/2465631568 learned_addr learned my addr 172.21.15.146:0/2465631568 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.942+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa664164f90 0x7fa66415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.946 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:04.944+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:04.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.944+0000 7fa662a00640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66415e9f0 msgr2=0x7fa6640c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:04.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.944+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa66415e9f0 0x7fa6640c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.944+0000 7fa662a00640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66406f5f0 msgr2=0x7fa664169710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.944+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66406f5f0 0x7fa664169710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.944+0000 7fa662a00640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa664168810 con 0x7fa664164f90 2024-09-16T01:52:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.944+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa66406f5f0 0x7fa664169710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.945+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa664164f90 0x7fa66415e4b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa650031000 tx=0x7fa6500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.947+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650032e20 con 0x7fa664164f90 2024-09-16T01:52:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.947+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65003fe50 con 0x7fa664164f90 2024-09-16T01:52:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.947+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650004250 con 0x7fa664164f90 2024-09-16T01:52:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.948+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66415efd0 con 0x7fa664164f90 2024-09-16T01:52:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.949+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641fd4f0 con 0x7fa664164f90 2024-09-16T01:52:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.960+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa63c005270 con 0x7fa664164f90 2024-09-16T01:52:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.960+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa650066630 con 0x7fa664164f90 2024-09-16T01:52:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.968+0000 7fa65b400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa64406c380 0x7fa64406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.968+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(92..92 src has 1..92) ==== 5770+0+0 (secure 0 0 0) 0x7fa6500b3530 con 0x7fa664164f90 2024-09-16T01:52:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.969+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa64406c380 0x7fa64406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.974+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa64406c380 0x7fa64406e840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa654007fd0 tx=0x7fa654009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:04.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.975+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v92) ==== 54+0+16 (secure 0 0 0) 0x7fa650044070 con 0x7fa664164f90 2024-09-16T01:52:04.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.977+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa63c000fa0 0x7fa63c0045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:04.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.977+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa63c005550 con 0x7fa63c000fa0 2024-09-16T01:52:04.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.979+0000 7fa663e00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa63c000fa0 0x7fa63c0045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.984+0000 7fa663e00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa63c000fa0 0x7fa63c0045e0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:04.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:04.986+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c005550 con 0x7fa63c000fa0 2024-09-16T01:52:05.046 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:05.045+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:05.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.079+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c004fe0 con 0x7fa63c000fa0 2024-09-16T01:52:05.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.088+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c004fe0 con 0x7fa63c000fa0 2024-09-16T01:52:05.090 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:05.090 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:05.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.092+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa63c001d00 0x7fa63c002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.092+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa63c00b390 con 0x7fa63c001d00 2024-09-16T01:52:05.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.093+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa63c001d00 0x7fa63c002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.098+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa63c001d00 0x7fa63c002990 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.110+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c00b390 con 0x7fa63c001d00 2024-09-16T01:52:05.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.144+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c0031a0 con 0x7fa63c001d00 2024-09-16T01:52:05.147 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:05.147+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:05.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.157+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c0031a0 con 0x7fa63c001d00 2024-09-16T01:52:05.159 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:05.160 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:05.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.159+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa63c009c80 0x7fa63c00a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.159+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa63c00a650 con 0x7fa63c009c80 2024-09-16T01:52:05.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.160+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa63c009c80 0x7fa63c00a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.161+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa63c009c80 0x7fa63c00a110 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.163+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c00a650 con 0x7fa63c009c80 2024-09-16T01:52:05.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.230+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c00aae0 con 0x7fa63c009c80 2024-09-16T01:52:05.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.240+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c00aae0 con 0x7fa63c009c80 2024-09-16T01:52:05.242 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.241+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa63c00da60 0x7fa63c00fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.241+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa63c00b190 con 0x7fa63c00da60 2024-09-16T01:52:05.247 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 50.581539126000003, 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:52:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1031 3.e 3:73e84e43:::100000001f8.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:52:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.071923582, 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.093093621000000001, 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1031 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171941+0000", 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2230000000000001e-06 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182392+0000", 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010451225999999999 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182418+0000", 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5561000000000001e-05 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182431+0000", 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3012e-05 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186260+0000", 2024-09-16T01:52:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038291340000000001 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186410+0000", 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014975200000000001 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186556+0000", 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146587 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.210170+0000", 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023614109000000001 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.264915+0000", 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054744835999999998 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.264984+0000", 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8753999999999999e-05 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.265032+0000", 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8427000000000002e-05 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.952679859, 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.923519003999999, 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.891167436, 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.277 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:05.246+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.241+0000 7fa663e00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa63c00da60 0x7fa63c00fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.245+0000 7fa663e00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa63c00da60 0x7fa63c00fe50 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.875220169999999, 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1132 3.5 3:a0fd78c8:::100000001f8.00000069:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.852685473000001, 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095513232000000003, 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1132 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391179+0000", 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.66e-06 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408341+0000", 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017162686999999999 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408396+0000", 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5034000000000002e-05 2024-09-16T01:52:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408412+0000", 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5685e-05 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408710+0000", 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029794899999999999 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408870+0000", 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000160168 2024-09-16T01:52:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.409033+0000", 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016284700000000001 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480660+0000", 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071627535000000006 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486552+0000", 2024-09-16T01:52:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0058913079999999996 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486634+0000", 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2051000000000004e-05 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486690+0000", 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6307999999999997e-05 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.328145005, 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.143215648999998, 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.140677840999999, 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1966 3.3c 3:3c832e37:::100000001f8.000003ab:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.7876431, 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10596815800000001, 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1966 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456221+0000", 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.125e-06 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520630+0000", 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064408699 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520660+0000", 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0122000000000001e-05 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520672+0000", 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2179e-05 2024-09-16T01:52:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520772+0000", 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.993e-05 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520970+0000", 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198097 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.521122+0000", 2024-09-16T01:52:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151065 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.535969+0000", 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014847075 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562086+0000", 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026117475000000001 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562144+0000", 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7806000000000002e-05 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562187+0000", 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3584999999999997e-05 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:05.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.531266608999999, 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:05.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:05.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2016 3.2b 3:d71f35f6:::100000001f8.000003dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.214685+0000", 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.029177559000001, 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087313960999999995, 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2016 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.214685+0000", 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.214685+0000", 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.214687+0000", 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4389999999999999e-06 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.252117+0000", 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037430120999999997 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.252137+0000", 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9358999999999999e-05 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.252148+0000", 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1618e-05 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.262932+0000", 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010783589999999999 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.263115+0000", 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018258 2024-09-16T01:52:05.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.263273+0000", 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157954 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.286664+0000", 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023391189 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.301826+0000", 2024-09-16T01:52:05.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015162604999999999 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.301891+0000", 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4797999999999999e-05 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.301999+0000", 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010770800000000001 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2096 3.16 3:68301556:::100000001f8.0000042d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.970728274000001, 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096328759999999999, 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2096 2024-09-16T01:52:05.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273136+0000", 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306295+0000", 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033158316 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306329+0000", 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4656999999999997e-05 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306353+0000", 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4244999999999999e-05 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306437+0000", 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3885000000000001e-05 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306614+0000", 2024-09-16T01:52:05.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000176656 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306762+0000", 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014849399999999999 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.368128+0000", 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061365140999999998 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369361+0000", 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001232934 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369413+0000", 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2123000000000002e-05 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369463+0000", 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0229000000000003e-05 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.337 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2449 3.2 3:4180d341:::100000001f8.0000058e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.446705+0000", 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7971577209999996, 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092182652000000004, 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.338 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2449 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.446705+0000", 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.446705+0000", 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.446707+0000", 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3659999999999999e-06 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.482929+0000", 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036221824 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485444+0000", 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025151800000000001 2024-09-16T01:52:05.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485459+0000", 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4813e-05 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485569+0000", 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011008199999999999 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.341 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485728+0000", 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159476 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485886+0000", 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157608 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.518770+0000", 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032883901 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.538767+0000", 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019997451999999999 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.538835+0000", 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7733999999999999e-05 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.538887+0000", 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2216000000000003e-05 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.343 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2464 3.3c 3:3d394cad:::100000001f8.0000059d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.652681+0000", 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5911813460000008, 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085579681000000005, 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.344 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2464 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.652681+0000", 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.652681+0000", 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.652683+0000", 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9910000000000001e-06 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.699320+0000", 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046636900000000002 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.699406+0000", 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6235000000000004e-05 2024-09-16T01:52:05.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.699446+0000", 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9857000000000001e-05 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.699752+0000", 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030638399999999999 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.700347+0000", 2024-09-16T01:52:05.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000594343 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.700993+0000", 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00064596200000000003 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.711245+0000", 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010252747 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.737712+0000", 2024-09-16T01:52:05.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026466867000000002 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.738021+0000", 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000308206 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.738261+0000", 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000240189 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2472 3.16 3:691ada3c:::100000001f8.000005a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.703799+0000", 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5400632840000004, 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090150464, 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2472 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.703799+0000", 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.703799+0000", 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.703801+0000", 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.193e-06 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749450+0000", 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045648376999999997 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749555+0000", 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010519200000000001 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749632+0000", 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6812999999999998e-05 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749835+0000", 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000203387 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749993+0000", 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015814299999999999 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.750149+0000", 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015568900000000001 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.763637+0000", 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013488081000000001 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.793839+0000", 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030202217999999999 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.793899+0000", 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0084999999999998e-05 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.355 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.793950+0000", 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0285999999999999e-05 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2475 3.26 3:65c4544b:::100000001f8.000005a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.749804+0000", 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.4940584660000003, 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.086982682000000006, 2024-09-16T01:52:05.356 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2475 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749804+0000", 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749804+0000", 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.749806+0000", 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6339999999999999e-06 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.819402+0000", 2024-09-16T01:52:05.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.069596100999999994 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.819431+0000", 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9470000000000001e-05 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.819442+0000", 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1154e-05 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.819669+0000", 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000226732 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.819823+0000", 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153883 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.819986+0000", 2024-09-16T01:52:05.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016337099999999999 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.835685+0000", 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015698457999999998 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.836517+0000", 2024-09-16T01:52:05.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00083223900000000003 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.836658+0000", 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014146300000000001 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.836787+0000", 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012817699999999999 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2989 3.e 3:715a0782:::100000001f8.000007aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.298775+0000", 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.9450870419999999, 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092600565999999995, 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2989 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.298775+0000", 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.298775+0000", 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.298778+0000", 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5900000000000002e-06 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342873+0000", 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044094792000000001 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342890+0000", 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7659000000000002e-05 2024-09-16T01:52:05.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342902+0000", 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1389999999999999e-05 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342982+0000", 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9728999999999995e-05 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.343135+0000", 2024-09-16T01:52:05.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015343999999999999 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.343278+0000", 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014258000000000001 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.386435+0000", 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043157790000000001 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.391251+0000", 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048159270000000002 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.391316+0000", 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4589000000000001e-05 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.391376+0000", 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0080000000000001e-05 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3056 3.16 3:68f2fc0a:::100000001f8.000007ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 3.2356354409999999, 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094053392, 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3056 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008229+0000", 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.756e-06 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062258+0000", 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054028822999999997 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062288+0000", 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0896999999999999e-05 2024-09-16T01:52:05.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062302+0000", 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3112000000000001e-05 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062499+0000", 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019759500000000001 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062685+0000", 2024-09-16T01:52:05.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186281 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062841+0000", 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155847 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.074322+0000", 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011480959000000001 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:05.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102173+0000", 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027851130000000002 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102236+0000", 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2749000000000005e-05 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102280+0000", 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4243000000000003e-05 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.375 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.375 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:05.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.247+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c00b190 con 0x7fa63c00da60 2024-09-16T01:52:05.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.292+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c0106a0 con 0x7fa63c00da60 2024-09-16T01:52:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.297+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c0106a0 con 0x7fa63c00da60 2024-09-16T01:52:05.377 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:05.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:52:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.298+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa63c012690 0x7fa63c0108b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.298+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa63c016420 con 0x7fa63c012690 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.299+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa63c012690 0x7fa63c0108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.300+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa63c012690 0x7fa63c0108b0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.301+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c016420 con 0x7fa63c012690 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.330+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c010e70 con 0x7fa63c012690 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.332+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c010e70 con 0x7fa63c012690 2024-09-16T01:52:05.379 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.333+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa63c011590 0x7fa63c0119f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.333+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa63c015ff0 con 0x7fa63c011590 2024-09-16T01:52:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.333+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa63c011590 0x7fa63c0119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.334+0000 7fa662a00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa63c011590 0x7fa63c0119f0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.335+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c015ff0 con 0x7fa63c011590 2024-09-16T01:52:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.357+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c015f70 con 0x7fa63c011590 2024-09-16T01:52:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.360+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c015f70 con 0x7fa63c011590 2024-09-16T01:52:05.381 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.360+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa63c0152b0 0x7fa63c015710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.360+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa63c015cd0 con 0x7fa63c0152b0 2024-09-16T01:52:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.361+0000 7fa663e00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa63c0152b0 0x7fa63c015710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.362+0000 7fa663e00640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa63c0152b0 0x7fa63c015710 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.363+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c015cd0 con 0x7fa63c0152b0 2024-09-16T01:52:05.383 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:05.378+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:05.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.388+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c015cd0 con 0x7fa63c0152b0 2024-09-16T01:52:05.394 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:05.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.390+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c015cd0 con 0x7fa63c0152b0 2024-09-16T01:52:05.394 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:05.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.392+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa63c01ab90 0x7fa63c01cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.392+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa63c01fe60 con 0x7fa63c01ab90 2024-09-16T01:52:05.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.393+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa63c01ab90 0x7fa63c01cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.394+0000 7fa663400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa63c01ab90 0x7fa63c01cf80 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.398+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c01fe60 con 0x7fa63c01ab90 2024-09-16T01:52:05.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.448+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa63c01fe60 con 0x7fa63c01ab90 2024-09-16T01:52:05.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.450+0000 7fa65b400640 1 -- 172.21.15.146:0/2465631568 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa63c01fe60 con 0x7fa63c01ab90 2024-09-16T01:52:05.452 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:05.452 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa63c000fa0 msgr2=0x7fa63c0045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa63c000fa0 0x7fa63c0045e0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa63c001d00 msgr2=0x7fa63c002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa63c001d00 0x7fa63c002990 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa63c009c80 msgr2=0x7fa63c00a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa63c009c80 0x7fa63c00a110 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa63c00da60 msgr2=0x7fa63c00fe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa63c00da60 0x7fa63c00fe50 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa63c012690 msgr2=0x7fa63c0108b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa63c012690 0x7fa63c0108b0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.451+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa63c011590 msgr2=0x7fa63c0119f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa63c011590 0x7fa63c0119f0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa63c0152b0 msgr2=0x7fa63c015710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa63c0152b0 0x7fa63c015710 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa63c01ab90 msgr2=0x7fa63c01cf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa63c01ab90 0x7fa63c01cf80 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa64406c380 msgr2=0x7fa64406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa64406c380 0x7fa64406e840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa654007fd0 tx=0x7fa654009040 comp rx=0 tx=0).stop 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa664164f90 msgr2=0x7fa66415e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.452+0000 7fa66a400640 1 --2- 172.21.15.146:0/2465631568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa664164f90 0x7fa66415e4b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa650031000 tx=0x7fa6500046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa663e00640 1 -- 172.21.15.146:0/2465631568 reap_dead start 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa663e00640 1 -- 172.21.15.146:0/2465631568 reap_dead start 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa663e00640 1 -- 172.21.15.146:0/2465631568 reap_dead start 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa663e00640 1 -- 172.21.15.146:0/2465631568 reap_dead start 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa663e00640 1 -- 172.21.15.146:0/2465631568 reap_dead start 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 shutdown_connections 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.453+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 >> 172.21.15.146:0/2465631568 conn(0x7fa66406bc10 msgr2=0x7fa6641641a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:05.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.455+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 shutdown_connections 2024-09-16T01:52:05.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.457+0000 7fa66a400640 1 -- 172.21.15.146:0/2465631568 wait complete. 2024-09-16T01:52:05.479 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:05.478+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:05.586 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:05.583+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:05.680 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:05.679+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:05.764 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:05.764 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:05.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3357 3.22 3:46260675:::100000001f8.0000091a:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.724667+0000", 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.038218249000000003, 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038257804999999999, 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3357 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:05.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.724667+0000", 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.724667+0000", 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.724668+0000", 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.759e-06 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.757096+0000", 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032427984 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.757124+0000", 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7277e-05 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.757134+0000", 2024-09-16T01:52:05.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0689e-05 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.757355+0000", 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022017599999999999 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.758289+0000", 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093412699999999998 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:05.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.759248+0000", 2024-09-16T01:52:05.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00095981400000000002 2024-09-16T01:52:05.769 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.769 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:05.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:05.772 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:05.772 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:52:05.772 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:05.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.784+0000 7f1c27200640 1 Processor -- start 2024-09-16T01:52:05.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.785+0000 7f1c27200640 1 -- start start 2024-09-16T01:52:05.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.788+0000 7f1c27200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c28157e10 0x7f1c280b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.788+0000 7f1c27200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c280b1200 0x7f1c2815a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.788+0000 7f1c27200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c2806cb30 con 0x7f1c2815ae60 2024-09-16T01:52:05.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.788+0000 7f1c27200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1c2815edf0 con 0x7f1c280b1200 2024-09-16T01:52:05.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.788+0000 7f1c27200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2815e960 con 0x7f1c28157e10 2024-09-16T01:52:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.789+0000 7f1c25400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c280b1200 0x7f1c2815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.789+0000 7f1c25400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c280b1200 0x7f1c2815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37406/0 (socket says 172.21.15.146:37406) 2024-09-16T01:52:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.789+0000 7f1c25400640 1 -- 172.21.15.146:0/2990000705 learned_addr learned my addr 172.21.15.146:0/2990000705 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.790+0000 7f1c25400640 1 -- 172.21.15.146:0/2990000705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c28157e10 msgr2=0x7f1c280b0c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.790+0000 7f1c25400640 1 --2- 172.21.15.146:0/2990000705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c28157e10 0x7f1c280b0c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.790+0000 7f1c25400640 1 -- 172.21.15.146:0/2990000705 >> v1:172.21.15.146:6789/0 conn(0x7f1c2815ae60 legacy=0x7f1c280c8900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:05.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.790+0000 7f1c25400640 1 -- 172.21.15.146:0/2990000705 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2815e030 con 0x7f1c280b1200 2024-09-16T01:52:05.796 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:05.789+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:05.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.794+0000 7f1c25400640 1 --2- 172.21.15.146:0/2990000705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c280b1200 0x7f1c2815a920 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009e50 tx=0x7f1c2002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58194ac768951522 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.795+0000 7f1c24a00640 1 -- 172.21.15.146:0/2990000705 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20036070 con 0x7f1c280b1200 2024-09-16T01:52:05.797 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:05.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.796+0000 7f1c24a00640 1 -- 172.21.15.146:0/2990000705 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c20031600 con 0x7f1c280b1200 2024-09-16T01:52:05.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.796+0000 7f1c24a00640 1 -- 172.21.15.146:0/2990000705 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200319d0 con 0x7f1c280b1200 2024-09-16T01:52:05.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.800+0000 7f1c27200640 1 -- 172.21.15.146:0/2990000705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c280b1200 msgr2=0x7f1c2815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.800+0000 7f1c27200640 1 --2- 172.21.15.146:0/2990000705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c280b1200 0x7f1c2815a920 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009e50 tx=0x7f1c2002a780 comp rx=0 tx=0).stop 2024-09-16T01:52:05.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.809+0000 7f1c27200640 1 -- 172.21.15.146:0/2990000705 shutdown_connections 2024-09-16T01:52:05.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.809+0000 7f1c27200640 1 --2- 172.21.15.146:0/2990000705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c280b1200 0x7f1c2815a920 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.809+0000 7f1c27200640 1 --2- 172.21.15.146:0/2990000705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c28157e10 0x7f1c280b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.809+0000 7f1c27200640 1 -- 172.21.15.146:0/2990000705 >> 172.21.15.146:0/2990000705 conn(0x7f1c28067950 msgr2=0x7f1c28067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:05.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.813+0000 7f1c27200640 1 -- 172.21.15.146:0/2990000705 shutdown_connections 2024-09-16T01:52:05.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.813+0000 7f1c27200640 1 -- 172.21.15.146:0/2990000705 wait complete. 2024-09-16T01:52:05.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.815+0000 7f1c27200640 1 Processor -- start 2024-09-16T01:52:05.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.816+0000 7f1c27200640 1 -- start start 2024-09-16T01:52:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.820+0000 7f1c27200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c280b1200 0x7f1c282051e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.820+0000 7f1c27200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c28157e10 0x7f1c28154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.821+0000 7f1c27200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 0x7f1c281548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.821+0000 7f1c26800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 0x7f1c281548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.822+0000 7f1c26800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 0x7f1c281548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37422/0 (socket says 172.21.15.146:37422) 2024-09-16T01:52:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.822+0000 7f1c26800640 1 -- 172.21.15.146:0/2094261155 learned_addr learned my addr 172.21.15.146:0/2094261155 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.822+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c280c84d0 con 0x7f1c2815ae60 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.822+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2815e5f0 con 0x7f1c28157e10 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.822+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c280c9010 con 0x7f1c280b1200 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.822+0000 7f1c25400640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c28157e10 0x7f1c28154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.823+0000 7f1c25e00640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c280b1200 0x7f1c282051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.823+0000 7f1c26800640 1 -- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c280b1200 msgr2=0x7f1c282051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.823+0000 7f1c26800640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c280b1200 0x7f1c282051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.823+0000 7f1c26800640 1 -- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c28157e10 msgr2=0x7f1c28154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.823+0000 7f1c26800640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c28157e10 0x7f1c28154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:05.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.823+0000 7f1c26800640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c281e67e0 con 0x7f1c2815ae60 2024-09-16T01:52:05.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.824+0000 7f1c26800640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 0x7f1c281548c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800b410 tx=0x7f1c1800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.832+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18015020 con 0x7f1c2815ae60 2024-09-16T01:52:05.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.832+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c18011830 con 0x7f1c2815ae60 2024-09-16T01:52:05.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.832+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18011c20 con 0x7f1c2815ae60 2024-09-16T01:52:05.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.833+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c280b0b80 con 0x7f1c2815ae60 2024-09-16T01:52:05.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.833+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c282049e0 con 0x7f1c2815ae60 2024-09-16T01:52:05.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.836+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c00005290 con 0x7f1c2815ae60 2024-09-16T01:52:05.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.840+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1c18018020 con 0x7f1c2815ae60 2024-09-16T01:52:05.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.842+0000 7f1c1e000640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c0806c300 0x7f1c0806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:05.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.842+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5770+0+0 (secure 0 0 0) 0x7f1c1808b1b0 con 0x7f1c2815ae60 2024-09-16T01:52:05.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.844+0000 7f1c25e00640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c0806c300 0x7f1c0806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:05.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.845+0000 7f1c25e00640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c0806c300 0x7f1c0806e7c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006fd0 tx=0x7f1c14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:05.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:05.849+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1c1801f020 con 0x7f1c2815ae60 2024-09-16T01:52:05.899 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:05.897+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:05.983 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:05.982+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:06.086 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:06.085+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:06.143 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:06.143 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:06.143 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:06.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:06.185 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:06.184+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:06.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.236+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f1c00005850 con 0x7f1c2815ae60 2024-09-16T01:52:06.286 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:06.285+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:06.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.347+0000 7f1c1e000640 1 -- 172.21.15.146:0/2094261155 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v93) ==== 87+0+0 (secure 0 0 0) 0x7f1c180172c0 con 0x7f1c2815ae60 2024-09-16T01:52:06.349 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:52:06.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.353+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c0806c300 msgr2=0x7f1c0806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:06.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.353+0000 7f1c27200640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c0806c300 0x7f1c0806e7c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006fd0 tx=0x7f1c14008040 comp rx=0 tx=0).stop 2024-09-16T01:52:06.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.353+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 msgr2=0x7f1c281548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:06.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.353+0000 7f1c27200640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 0x7f1c281548c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800b410 tx=0x7f1c1800b8d0 comp rx=0 tx=0).stop 2024-09-16T01:52:06.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 shutdown_connections 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1c0806c300 0x7f1c0806e7c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2815ae60 0x7f1c281548c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c28157e10 0x7f1c28154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 --2- 172.21.15.146:0/2094261155 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c280b1200 0x7f1c282051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 >> 172.21.15.146:0/2094261155 conn(0x7f1c28067950 msgr2=0x7f1c28071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.357+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 shutdown_connections 2024-09-16T01:52:06.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:06.358+0000 7f1c27200640 1 -- 172.21.15.146:0/2094261155 wait complete. 2024-09-16T01:52:06.388 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:06.387+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:06.489 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:06.488+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.673065887999996, 2024-09-16T01:52:06.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:06.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:06.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:06.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:06.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:06.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.536 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.530606223000007, 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.537 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:06.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:06.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:06.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:06.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.183139999999995, 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.548 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:06.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:06.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:06.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:06.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:06.560 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.298848636000002, 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:06.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:06.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.261109133000001, 2024-09-16T01:52:06.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:06.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.402122132999999, 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.576 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.578 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:06.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:06.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:06.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:06.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:06.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:06.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1620 3.13 3:c81354c0:::100000001f8.00000251:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:06.587 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:45.341779+0000", 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.180023626000001, 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.104539276, 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.588 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1620 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.341779+0000", 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.341779+0000", 2024-09-16T01:52:06.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.341782+0000", 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7800000000000001e-06 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425499+0000", 2024-09-16T01:52:06.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083716384000000005 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425530+0000", 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1145000000000001e-05 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425544+0000", 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3835999999999999e-05 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425678+0000", 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013432799999999999 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425852+0000", 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000174286 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.426035+0000", 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183191 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.442798+0000", 2024-09-16T01:52:06.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016762945000000001 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.446209+0000", 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003410344 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.446269+0000", 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0217999999999998e-05 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.446319+0000", 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9818999999999999e-05 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.535083158999999, 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.596 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.598 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:06.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:06.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:06.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:06.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:06.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.339130938, 2024-09-16T01:52:06.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:06.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.608 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:06.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:06.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:06.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:06.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:06.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:06.617 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.298128994999999, 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:06.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.073901435, 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:06.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2007 3.2f 3:f45f7969:::100000001f8.000003d4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.065925+0000", 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.455878126000002, 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097559495999999996, 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.632 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2007 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.065925+0000", 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.065925+0000", 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.065928+0000", 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.137908+0000", 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071980743999999999 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.137922+0000", 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4158000000000001e-05 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.137934+0000", 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1304999999999999e-05 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.138006+0000", 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1860999999999995e-05 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.138164+0000", 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158362 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.138347+0000", 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000182573 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.155916+0000", 2024-09-16T01:52:06.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017569784000000001 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.163388+0000", 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074713109999999996 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.163444+0000", 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6474000000000002e-05 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.163484+0000", 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0339999999999997e-05 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2045 3.13 3:c8562531:::100000001f8.000003fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.595706+0000", 2024-09-16T01:52:06.639 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.926096942000001, 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097724925000000004, 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2045 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.595706+0000", 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.595706+0000", 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.595708+0000", 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624248+0000", 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028539653000000002 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624266+0000", 2024-09-16T01:52:06.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7598e-05 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624278+0000", 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2189e-05 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624857+0000", 2024-09-16T01:52:06.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000579015 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.625007+0000", 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015013799999999999 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.625164+0000", 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156646 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.646055+0000", 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020891528999999999 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.693173+0000", 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047118243999999997 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.693235+0000", 2024-09-16T01:52:06.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1975000000000002e-05 2024-09-16T01:52:06.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.693431+0000", 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195739 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2093 3.2d 3:b5a9a28c:::100000001f8.0000042a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.271671+0000", 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.250132607999999, 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105370546, 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2093 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.271671+0000", 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.271671+0000", 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.271672+0000", 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9709999999999998e-06 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.299969+0000", 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028296847 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.299999+0000", 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9637000000000001e-05 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300011+0000", 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2320000000000001e-05 2024-09-16T01:52:06.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300086+0000", 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5153000000000004e-05 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300236+0000", 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149457 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300379+0000", 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143184 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330169+0000", 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029790146 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.376928+0000", 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046758358999999999 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.376990+0000", 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2483999999999993e-05 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.377041+0000", 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0988e-05 2024-09-16T01:52:06.656 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.117119882000001, 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.657 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:06.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:06.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:06.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.664 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2504 3.13 3:c93be02c:::100000001f8.000005c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:56.120715+0000", 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.401088223, 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097015065999999997, 2024-09-16T01:52:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2504 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.120715+0000", 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.120715+0000", 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.120717+0000", 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:06.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.160791+0000", 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040074578999999999 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.160823+0000", 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1588999999999999e-05 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.160836+0000", 2024-09-16T01:52:06.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3373999999999999e-05 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.162741+0000", 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019051859999999999 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.162890+0000", 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148527 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.163051+0000", 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016161499999999999 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.173965+0000", 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010913516 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.217622+0000", 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043656860999999998 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.217687+0000", 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5116000000000005e-05 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.217730+0000", 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.206139533, 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.675 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:06.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:06.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:06.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:06.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3331 3.28 3:16856b74:::100000001f8.00000900:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.370244+0000", 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.151559405, 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106986303, 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.685 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.370244+0000", 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.370244+0000", 2024-09-16T01:52:06.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.370246+0000", 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.869e-06 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.421465+0000", 2024-09-16T01:52:06.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051219301000000002 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.421489+0000", 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3935e-05 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.421502+0000", 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2734999999999999e-05 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422439+0000", 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093792500000000004 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422637+0000", 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000197107 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422789+0000", 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015251500000000001 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.460201+0000", 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037412182000000002 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.477140+0000", 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016938624999999999 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.477194+0000", 2024-09-16T01:52:06.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3961000000000001e-05 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.477230+0000", 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6148e-05 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3376 3.2d 3:b76041b2:::100000001f8.0000092d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.898410+0000", 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.62339293699999998, 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.102945044, 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.694 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3376 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.898410+0000", 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.898410+0000", 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.898412+0000", 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.278e-06 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973350+0000", 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074937918000000006 2024-09-16T01:52:06.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973367+0000", 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6629999999999998e-05 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973386+0000", 2024-09-16T01:52:06.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9298999999999999e-05 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973468+0000", 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1682999999999997e-05 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973646+0000", 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178507 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.974262+0000", 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061593699999999999 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.992922+0000", 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018659960999999999 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.001253+0000", 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083307729999999997 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.001306+0000", 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3062000000000001e-05 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.001355+0000", 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8996000000000003e-05 2024-09-16T01:52:06.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3384 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:06.037542+0000", 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.48426121100000002, 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105116187, 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:06.704 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3384 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.037542+0000", 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.037542+0000", 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.037543+0000", 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5859999999999999e-06 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108364+0000", 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070820314999999995 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108449+0000", 2024-09-16T01:52:06.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4876000000000005e-05 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108493+0000", 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4561000000000002e-05 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108666+0000", 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017283899999999999 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108859+0000", 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019284799999999999 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.109039+0000", 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000179958 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.123383+0000", 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014343631000000001 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:06.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.142572+0000", 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019189364 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.142623+0000", 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0973000000000001e-05 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:06.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.142658+0000", 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5235999999999999e-05 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:06.712 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:06.713 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:06.714 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:06.617+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:06.715 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:06.714+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:06.816 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:06.816+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:06.844 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:06.844 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:06.844 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:06.844 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:06.855 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:06.916 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:06.915+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:06.978 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:06.978 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:06.978 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:06.978 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:06.978 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:06.990 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:07.031 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:07.030+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:07.121 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:07.120+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 76.302993036000004, 2024-09-16T01:52:07.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:52:07.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:07.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.168 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 75.391414724000001, 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:07.174 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 75.347154493999994, 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:07.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:52:07.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.180 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 75.275099151999996, 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:52:07.182 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:52:07.185 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.186 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 67.018092730999996, 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.187 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:52:07.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:07.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:52:07.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.194 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 66.983296432000003, 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:52:07.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:52:07.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:52:07.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:52:07.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:07.204 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 66.899847479000002, 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:07.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:52:07.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.211 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:52:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.700544692000001, 2024-09-16T01:52:07.213 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:07.214 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:52:07.215 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.216 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:52:07.217 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:52:07.218 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:52:07.219 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.219 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.219 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.219 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:52:07.219 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:52:07.219 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.220 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.220 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.220 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.555667753999998, 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.221 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.222 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.223 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:52:07.224 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.225 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:52:07.226 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.227 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.228 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:52:07.228 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:52:07.228 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.228 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.228 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.229 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:52:07.229 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:52:07.229 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.229 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.229 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.229 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.230 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.230 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:07.230 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:07.230 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.515023329999998, 2024-09-16T01:52:07.230 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:52:07.230 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:52:07.231 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.232 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.232 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.232 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.232 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:07.232 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.233 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.233 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.233 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:07.233 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.233 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.234 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.234 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.234 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:52:07.234 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:52:07.234 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.235 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.235 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:52:07.235 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:52:07.235 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.235 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:52:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:52:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:52:07.237 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:52:07.237 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.237 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.237 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.237 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:52:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:52:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:52:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:52:07.239 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.239 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.239 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.239 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:52:07.239 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:52:07.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.240 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.240 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.240 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:52:07.240 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:52:07.240 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.241 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.241 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:52:07.241 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:52:07.241 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.241 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:52:07.243 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:52:07.243 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.243 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.243 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.243 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.243 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.357080496000002, 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.244 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.245 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.246 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:52:07.247 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:52:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.249 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.250 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:52:07.251 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.231953234999999, 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:52:07.252 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.253 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:07.254 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.255 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.256 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:52:07.257 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.258 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:52:07.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:52:07.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.195153304000002, 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.262 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:52:07.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.032127662000001, 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:52:07.268 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:52:07.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.274 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.898389872999999, 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:52:07.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:52:07.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:52:07.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:52:07.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:52:07.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:52:07.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:52:07.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:52:07.282 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.203999758999998, 2024-09-16T01:52:07.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.285 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.286 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:52:07.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.877025193000001, 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.291 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:52:07.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:52:07.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:52:07.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.736115052999999, 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.298 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:07.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:52:07.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.304 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.683312108999999, 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:07.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:52:07.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.311 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.364053568999999, 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:52:07.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:52:07.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.318 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.319 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.319 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:07.320 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:07.321 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:07.319+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:07.340 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:07.340 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:07.340 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 5, 6, 7] 2024-09-16T01:52:07.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-16T01:52:07.421 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:07.420+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:07.486 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:07.486 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:07.486 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:07.486 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:07.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:07.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:07.557 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:07.555+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:07.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.606+0000 7ff4c2c00640 1 Processor -- start 2024-09-16T01:52:07.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.606+0000 7ff4c2c00640 1 -- start start 2024-09-16T01:52:07.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4c2c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4bc0cc0c0 0x7ff4bc15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4c2c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4c2c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4bc15b1d0 con 0x7ff4bc15e9d0 2024-09-16T01:52:07.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4c2c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4bc0cbd00 con 0x7ff4bc0cc0c0 2024-09-16T01:52:07.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4c2c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc15ba70 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4bb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4bb400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56274/0 (socket says 172.21.15.146:56274) 2024-09-16T01:52:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.609+0000 7ff4bb400640 1 -- 172.21.15.146:0/176143196 learned_addr learned my addr 172.21.15.146:0/176143196 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.610+0000 7ff4bbe00640 1 --2- 172.21.15.146:0/176143196 >> v2:172.21.15.161:3300/0 conn(0x7ff4bc0cc0c0 0x7ff4bc15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.610+0000 7ff4bbe00640 1 -- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 msgr2=0x7ff4bc0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.610+0000 7ff4bbe00640 1 --2- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.610+0000 7ff4bbe00640 1 -- 172.21.15.146:0/176143196 >> v1:172.21.15.161:6789/0 conn(0x7ff4bc15e9d0 legacy=0x7ff4bc0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.610+0000 7ff4bbe00640 1 -- 172.21.15.146:0/176143196 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc15b4b0 con 0x7ff4bc0cc0c0 2024-09-16T01:52:07.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.611+0000 7ff4bbe00640 1 --2- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc15aab0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0097a0 tx=0x7ff4ac02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9fdd151857df19e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.612+0000 7ff4baa00640 1 -- 172.21.15.146:0/176143196 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac037070 con 0x7ff4bc0cc0c0 2024-09-16T01:52:07.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.612+0000 7ff4baa00640 1 -- 172.21.15.146:0/176143196 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ac032440 con 0x7ff4bc0cc0c0 2024-09-16T01:52:07.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.612+0000 7ff4baa00640 1 -- 172.21.15.146:0/176143196 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac032830 con 0x7ff4bc0cc0c0 2024-09-16T01:52:07.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 -- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 msgr2=0x7ff4bc15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc15aab0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0097a0 tx=0x7ff4ac02a1c0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 -- 172.21.15.146:0/176143196 shutdown_connections 2024-09-16T01:52:07.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/176143196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc15aab0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 -- 172.21.15.146:0/176143196 >> 172.21.15.146:0/176143196 conn(0x7ff4bc06d560 msgr2=0x7ff4bc06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:07.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.613+0000 7ff4c2c00640 1 -- 172.21.15.146:0/176143196 shutdown_connections 2024-09-16T01:52:07.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.614+0000 7ff4c2c00640 1 -- 172.21.15.146:0/176143196 wait complete. 2024-09-16T01:52:07.619 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:07.619 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:07.619 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:07.620 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:07.620 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:07.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.616+0000 7ff4c2c00640 1 Processor -- start 2024-09-16T01:52:07.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.616+0000 7ff4c2c00640 1 -- start start 2024-09-16T01:52:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c2c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c2c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc14e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4bc15e9d0 0x7ff4bc14b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc15e1d0 con 0x7ff4bc15e9d0 2024-09-16T01:52:07.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c2c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc15e540 con 0x7ff4bc0cc0c0 2024-09-16T01:52:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c2c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc0c5560 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4bbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4bbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0b5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56288/0 (socket says 172.21.15.146:56288) 2024-09-16T01:52:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4bbe00640 1 -- 172.21.15.146:0/3983221926 learned_addr learned my addr 172.21.15.146:0/3983221926 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4c0c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4bc15e9d0 0x7ff4bc14b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.618+0000 7ff4bb400640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc14e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4bbe00640 1 -- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 msgr2=0x7ff4bc14e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4bbe00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc14e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4bbe00640 1 -- 172.21.15.146:0/3983221926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4bc15e9d0 msgr2=0x7ff4bc14b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4bbe00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4bc15e9d0 0x7ff4bc14b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4bbe00640 1 -- 172.21.15.146:0/3983221926 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc0c8580 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4bb400640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc14e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:07.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.619+0000 7ff4c0c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4bc15e9d0 0x7ff4bc14b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:07.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.620+0000 7ff4bbe00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0b5e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.620+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac037040 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.620+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ac042070 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.620+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4bc0c8840 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.621+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac032830 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.621+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4bc0c8e00 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.622+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4bc0cbea0 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.623+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff4ac02ad10 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.624+0000 7ff4b3e00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff49c06c2b0 0x7ff49c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.624+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 5770+0+0 (secure 0 0 0) 0x7ff4ac0aba70 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.625+0000 7ff4bb400640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff49c06c2b0 0x7ff49c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.625+0000 7ff4bb400640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff49c06c2b0 0x7ff49c06e770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0c7bc0 tx=0x7ff4a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.629 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:07.627+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:07.630 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:07.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.630+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff4ac07a2b0 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.729 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:07.728+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:07.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.758+0000 7fe75b600640 1 Processor -- start 2024-09-16T01:52:07.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.758+0000 7fe75b600640 1 -- start start 2024-09-16T01:52:07.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe75b600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe7540c2e20 0x7fe7540c7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe75b600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7540c3280 0x7fe7540c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe75b600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7540c37e0 con 0x7fe754151120 2024-09-16T01:52:07.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe75b600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe75415b140 con 0x7fe7540c3280 2024-09-16T01:52:07.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe75b600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe75415a530 con 0x7fe7540c2e20 2024-09-16T01:52:07.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe758c00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe754151120 0x7fe7540c2710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46708/0 (socket says 172.21.15.146:46708) 2024-09-16T01:52:07.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe759600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7540c3280 0x7fe7540c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.760+0000 7fe758c00640 1 -- 172.21.15.146:0/147312820 learned_addr learned my addr 172.21.15.146:0/147312820 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe753e00640 1 --2- 172.21.15.146:0/147312820 >> v2:172.21.15.161:3300/0 conn(0x7fe7540c2e20 0x7fe7540c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe759600640 1 -- 172.21.15.146:0/147312820 >> v2:172.21.15.161:3300/0 conn(0x7fe7540c2e20 msgr2=0x7fe7540c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe759600640 1 --2- 172.21.15.146:0/147312820 >> v2:172.21.15.161:3300/0 conn(0x7fe7540c2e20 0x7fe7540c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe759600640 1 -- 172.21.15.146:0/147312820 >> v1:172.21.15.146:6789/0 conn(0x7fe754151120 legacy=0x7fe7540c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe759600640 1 -- 172.21.15.146:0/147312820 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75415a810 con 0x7fe7540c3280 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe753400640 1 -- 172.21.15.146:0/147312820 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54794736 0 0) 0x7fe7540c37e0 con 0x7fe754151120 2024-09-16T01:52:07.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.761+0000 7fe753e00640 1 --2- 172.21.15.146:0/147312820 >> v2:172.21.15.161:3300/0 conn(0x7fe7540c2e20 0x7fe7540c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.762+0000 7fe759600640 1 --2- 172.21.15.146:0/147312820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7540c3280 0x7fe7540c7a50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe7540c8f80 tx=0x7fe74c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c659705b1a71f1c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.762+0000 7fe753400640 1 -- 172.21.15.146:0/147312820 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c007740 con 0x7fe7540c3280 2024-09-16T01:52:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.762+0000 7fe753400640 1 -- 172.21.15.146:0/147312820 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe74c0078e0 con 0x7fe7540c3280 2024-09-16T01:52:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.762+0000 7fe753400640 1 -- 172.21.15.146:0/147312820 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c007cb0 con 0x7fe7540c3280 2024-09-16T01:52:07.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 -- 172.21.15.146:0/147312820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7540c3280 msgr2=0x7fe7540c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 --2- 172.21.15.146:0/147312820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7540c3280 0x7fe7540c7a50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe7540c8f80 tx=0x7fe74c02a710 comp rx=0 tx=0).stop 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 -- 172.21.15.146:0/147312820 shutdown_connections 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 --2- 172.21.15.146:0/147312820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7540c3280 0x7fe7540c7a50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 --2- 172.21.15.146:0/147312820 >> v2:172.21.15.161:3300/0 conn(0x7fe7540c2e20 0x7fe7540c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 -- 172.21.15.146:0/147312820 >> 172.21.15.146:0/147312820 conn(0x7fe7540ccb10 msgr2=0x7fe7540ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.763+0000 7fe75b600640 1 -- 172.21.15.146:0/147312820 shutdown_connections 2024-09-16T01:52:07.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.764+0000 7fe75b600640 1 -- 172.21.15.146:0/147312820 wait complete. 2024-09-16T01:52:07.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.765+0000 7fe75b600640 1 Processor -- start 2024-09-16T01:52:07.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.766+0000 7fe75b600640 1 -- start start 2024-09-16T01:52:07.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.767+0000 7fe75b600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7540c3280 0x7fe7540b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.767+0000 7fe75b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe754151120 0x7fe75406bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.767+0000 7fe75b600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 0x7fe754141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.767+0000 7fe75b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe754074110 con 0x7fe754151120 2024-09-16T01:52:07.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.767+0000 7fe75b600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe75415add0 con 0x7fe7540c3280 2024-09-16T01:52:07.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.767+0000 7fe75b600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe754151500 con 0x7fe75406d9d0 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 0x7fe754141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 0x7fe754141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56292/0 (socket says 172.21.15.146:56292) 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 learned_addr learned my addr 172.21.15.146:0/3625070147 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe754151120 0x7fe75406bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7540c3280 0x7fe7540b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7540c3280 msgr2=0x7fe7540b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7540c3280 0x7fe7540b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe754151120 msgr2=0x7fe75406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe754151120 0x7fe75406bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7541f4dc0 con 0x7fe75406d9d0 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.768+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 0x7fe754141170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009e50 tx=0x7fe74c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.769+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c040070 con 0x7fe75406d9d0 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.769+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe74c031c30 con 0x7fe75406d9d0 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.769+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c045460 con 0x7fe75406d9d0 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.769+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7541f4ae0 con 0x7fe75406d9d0 2024-09-16T01:52:07.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.769+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7540cc5e0 con 0x7fe75406d9d0 2024-09-16T01:52:07.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.770+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe7540cc7e0 con 0x7fe75406d9d0 2024-09-16T01:52:07.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.772+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe74c039040 con 0x7fe75406d9d0 2024-09-16T01:52:07.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.773+0000 7fe750c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73806c2e0 0x7fe73806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.773+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 5770+0+0 (secure 0 0 0) 0x7fe74c0acc80 con 0x7fe75406d9d0 2024-09-16T01:52:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.773+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73806c2e0 0x7fe73806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.774+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v93) ==== 54+0+16 (secure 0 0 0) 0x7fe7540cc7e0 con 0x7fe75406d9d0 2024-09-16T01:52:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.774+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe75406c150 0x7fe7540c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.774+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe754143e90 con 0x7fe75406c150 2024-09-16T01:52:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.774+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73806c2e0 0x7fe73806e7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe744000f60 tx=0x7fe744007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.774+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe75406c150 0x7fe7540c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.775+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe75406c150 0x7fe7540c2c00 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.776+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe754143e90 con 0x7fe75406c150 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 77.256551732000005, 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:52:07.782 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:52:07.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.789 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:574 3.27 3:e46e0c7f:::100000001f7.0000003b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.775095786999998, 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084267634999999994, 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.790 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 574 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004656+0000", 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.004659+0000", 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5129999999999999e-06 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032641+0000", 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027982080999999999 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032656+0000", 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5699999999999999e-05 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032666+0000", 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5599999999999999e-06 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032719+0000", 2024-09-16T01:52:07.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3066999999999999e-05 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032827+0000", 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010841900000000001 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.032929+0000", 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101128 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.036882+0000", 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039538949999999998 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088843+0000", 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.051960682000000001 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088878+0000", 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.485e-05 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.088924+0000", 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5739999999999999e-05 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1033 3.29 3:94178d8a:::100000001f8.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 29.588884751999998, 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086462834000000002, 2024-09-16T01:52:07.796 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1033 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190867+0000", 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.190869+0000", 2024-09-16T01:52:07.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3099999999999999e-06 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224163+0000", 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.033293469999999999 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224185+0000", 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1987000000000001e-05 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224197+0000", 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.222e-05 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.224896+0000", 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00069902199999999999 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.225064+0000", 2024-09-16T01:52:07.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016828399999999999 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.225208+0000", 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014386800000000001 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.231378+0000", 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006169796 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277260+0000", 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045882091999999999 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277295+0000", 2024-09-16T01:52:07.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4505999999999998e-05 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.277330+0000", 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5278999999999999e-05 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.803 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1137 3.29 3:9673bc10:::100000001f8.0000006e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.374890723, 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084134489000000007, 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1137 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404861+0000", 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.404862+0000", 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0270000000000001e-06 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458066+0000", 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053203817 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458080+0000", 2024-09-16T01:52:07.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3536999999999999e-05 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458085+0000", 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8130000000000001e-06 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458167+0000", 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1797000000000003e-05 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458306+0000", 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013863699999999999 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.458472+0000", 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016593700000000001 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.462073+0000", 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003601131 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488893+0000", 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026819912000000001 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488949+0000", 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5889e-05 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.488996+0000", 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6992e-05 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1246 3.31 3:8c5c37ef:::100000001f8.000000db:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.710465+0000", 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.069286967, 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084411384000000006, 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.810 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1246 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.811 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.710465+0000", 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.710465+0000", 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.710468+0000", 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.729e-06 2024-09-16T01:52:07.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738714+0000", 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028245936999999999 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738735+0000", 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1080000000000001e-05 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738746+0000", 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1751e-05 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738820+0000", 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3627999999999994e-05 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.738976+0000", 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015629000000000001 2024-09-16T01:52:07.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.739147+0000", 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017062499999999999 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.744247+0000", 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0051001279999999998 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.794797+0000", 2024-09-16T01:52:07.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050549627 2024-09-16T01:52:07.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.794842+0000", 2024-09-16T01:52:07.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5664999999999997e-05 2024-09-16T01:52:07.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.794876+0000", 2024-09-16T01:52:07.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3924000000000002e-05 2024-09-16T01:52:07.818 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.818 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.818 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1656 3.29 3:976aaeee:::100000001f8.00000275:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.729286+0000", 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.050465569, 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.085174584999999997, 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.819 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.820 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.820 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.820 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.820 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1656 2024-09-16T01:52:07.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.820 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.729286+0000", 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.729286+0000", 2024-09-16T01:52:07.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.729288+0000", 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1339999999999998e-06 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754379+0000", 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025090527000000001 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754397+0000", 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7966999999999999e-05 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754406+0000", 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3340000000000008e-06 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754469+0000", 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2571999999999997e-05 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754573+0000", 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103675 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.754691+0000", 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118885 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.757993+0000", 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033016740000000001 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.814386+0000", 2024-09-16T01:52:07.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056392721999999999 2024-09-16T01:52:07.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.814425+0000", 2024-09-16T01:52:07.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8741000000000001e-05 2024-09-16T01:52:07.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.814461+0000", 2024-09-16T01:52:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6353999999999998e-05 2024-09-16T01:52:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1663 3.17 3:ea019aa2:::100000001f8.0000027c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.003060902000001, 2024-09-16T01:52:07.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124529129, 2024-09-16T01:52:07.833 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.833 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.833 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.833 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.833 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.833 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1663 2024-09-16T01:52:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:07.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:07.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776694+0000", 2024-09-16T01:52:07.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8320000000000002e-06 2024-09-16T01:52:07.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866207+0000", 2024-09-16T01:52:07.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089513588000000005 2024-09-16T01:52:07.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866225+0000", 2024-09-16T01:52:07.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7325000000000001e-05 2024-09-16T01:52:07.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866232+0000", 2024-09-16T01:52:07.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6149999999999999e-06 2024-09-16T01:52:07.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866286+0000", 2024-09-16T01:52:07.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3863000000000003e-05 2024-09-16T01:52:07.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866393+0000", 2024-09-16T01:52:07.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106443 2024-09-16T01:52:07.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:07.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866499+0000", 2024-09-16T01:52:07.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010595299999999999 2024-09-16T01:52:07.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.869501+0000", 2024-09-16T01:52:07.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030023570000000002 2024-09-16T01:52:07.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901146+0000", 2024-09-16T01:52:07.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031644671999999999 2024-09-16T01:52:07.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901180+0000", 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4277999999999999e-05 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901220+0000", 2024-09-16T01:52:07.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0203000000000001e-05 2024-09-16T01:52:07.845 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.845 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.845 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.845 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2039 3.31 3:8f066b95:::100000001f8.000003f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:07.846 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:07.846 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.257454965000001, 2024-09-16T01:52:07.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12858615900000001, 2024-09-16T01:52:07.846 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.846 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.846 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2039 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522299+0000", 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0109999999999999e-06 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586678+0000", 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064378764000000005 2024-09-16T01:52:07.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586695+0000", 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7159999999999998e-05 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586705+0000", 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9839999999999996e-06 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586760+0000", 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5099999999999998e-05 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586894+0000", 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013428999999999999 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.587008+0000", 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114178 2024-09-16T01:52:07.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.590811+0000", 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038024529999999999 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650793+0000", 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059982029999999999 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650843+0000", 2024-09-16T01:52:07.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9876999999999997e-05 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650883+0000", 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0312000000000003e-05 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2052 3.29 3:96415e08:::100000001f8.00000401:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.683050+0000", 2024-09-16T01:52:07.852 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.09670234, 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088613109999999995, 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2052 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.683050+0000", 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.683050+0000", 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.683051+0000", 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.46e-06 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730050+0000", 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046998637000000003 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730068+0000", 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8521000000000001e-05 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730078+0000", 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0282000000000001e-05 2024-09-16T01:52:07.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730147+0000", 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8736000000000001e-05 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730311+0000", 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016330899999999999 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.730465+0000", 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015406299999999999 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.735811+0000", 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0053466750000000004 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.771561+0000", 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.035749360000000001 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.771615+0000", 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3891999999999998e-05 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.771663+0000", 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8174999999999999e-05 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.858 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2092 3.1e 3:7aa4b9dd:::100000001f8.00000429:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.515860911000001, 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10603082799999999, 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2092 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263893+0000", 2024-09-16T01:52:07.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2249999999999999e-06 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292346+0000", 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028453092999999999 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292359+0000", 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2306999999999999e-05 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292364+0000", 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292417+0000", 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2673000000000002e-05 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292522+0000", 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010544600000000001 2024-09-16T01:52:07.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292624+0000", 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101473 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.295728+0000", 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031039010000000001 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369848+0000", 2024-09-16T01:52:07.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074120309999999995 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369888+0000", 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9465000000000002e-05 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369922+0000", 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4193999999999998e-05 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.864 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2103 3.31 3:8c8a0fd7:::100000001f8.00000434:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.388257+0000", 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.391495206999998, 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086636273, 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2103 2024-09-16T01:52:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.388257+0000", 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.388257+0000", 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.388258+0000", 2024-09-16T01:52:07.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4079999999999999e-06 2024-09-16T01:52:07.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431717+0000", 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.043458745 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431731+0000", 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3971000000000001e-05 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431738+0000", 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8460000000000002e-06 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431797+0000", 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9125000000000001e-05 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.431909+0000", 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000112417 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.432018+0000", 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108305 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.435049+0000", 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030311919999999998 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.474790+0000", 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039741548000000002 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.474855+0000", 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4480000000000006e-05 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.474893+0000", 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8235999999999998e-05 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2489 3.27 3:e6c61997:::100000001f8.000005b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:55.921405+0000", 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.858347050000001, 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.084631479999999995, 2024-09-16T01:52:07.872 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2489 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.921405+0000", 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.921405+0000", 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.921406+0000", 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.638e-06 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973442+0000", 2024-09-16T01:52:07.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052035855999999998 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973459+0000", 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6721000000000001e-05 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973484+0000", 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4807000000000002e-05 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973549+0000", 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5158000000000002e-05 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973702+0000", 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015326700000000001 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.973876+0000", 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017350699999999999 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:55.977774+0000", 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038979319999999998 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.005962+0000", 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028188063999999999 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.006000+0000", 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8355999999999997e-05 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.006036+0000", 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6174000000000003e-05 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.879 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2506 3.31 3:8dd8e9cf:::100000001f8.000005c7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.155701+0000", 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.624051334000001, 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.085502210999999995, 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.880 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2506 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.155701+0000", 2024-09-16T01:52:07.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.155701+0000", 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.155702+0000", 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.606e-06 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214365+0000", 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058663172999999999 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.883 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214381+0000", 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5695000000000001e-05 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214391+0000", 2024-09-16T01:52:07.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8560000000000002e-06 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214481+0000", 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9932000000000006e-05 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214591+0000", 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110648 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.214689+0000", 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7826999999999997e-05 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.217696+0000", 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003006908 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.241115+0000", 2024-09-16T01:52:07.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023418575000000001 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.241161+0000", 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5768e-05 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.241203+0000", 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2222999999999999e-05 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2901 3.1f 3:fb8355dd:::100000001f8.00000752:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:07.889 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:00.356814+0000", 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 7.4229379639999999, 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089405411000000004, 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.890 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2901 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.356814+0000", 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.356814+0000", 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.356816+0000", 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.063e-06 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430011+0000", 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.073194542000000001 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430029+0000", 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8349999999999999e-05 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430037+0000", 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4440000000000008e-06 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430095+0000", 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7432999999999997e-05 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430198+0000", 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010282599999999999 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.430291+0000", 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3115999999999994e-05 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.433796+0000", 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035057759999999999 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.446128+0000", 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012331811 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.446172+0000", 2024-09-16T01:52:07.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4150999999999998e-05 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:00.446219+0000", 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6898999999999999e-05 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3029 3.17 3:ebfd80ec:::100000001f8.000007d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:07.896 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.0459436640000002, 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.099718444000000003, 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3029 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733810+0000", 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1560000000000002e-06 2024-09-16T01:52:07.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763618+0000", 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029807628999999999 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763642+0000", 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3655999999999998e-05 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763651+0000", 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5259999999999999e-06 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763711+0000", 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9721e-05 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763830+0000", 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118792 2024-09-16T01:52:07.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763940+0000", 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110046 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.767357+0000", 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034171829999999999 2024-09-16T01:52:07.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833431+0000", 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066074421999999994 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833483+0000", 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1261999999999998e-05 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833527+0000", 2024-09-16T01:52:07.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4051000000000002e-05 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3080 3.38 3:1d0a07e7:::100000001f8.00000805:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:02.266234+0000", 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.5135180400000001, 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.090907420000000003, 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.903 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3080 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.266234+0000", 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.266234+0000", 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.266235+0000", 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.221e-06 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307326+0000", 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041091181999999997 2024-09-16T01:52:07.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307338+0000", 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1874e-05 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307363+0000", 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4973999999999998e-05 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307424+0000", 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1161999999999995e-05 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307540+0000", 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011523600000000001 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307641+0000", 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101649 2024-09-16T01:52:07.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.310544+0000", 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029028539999999999 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.908 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.357039+0000", 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046494704999999997 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.357094+0000", 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5618000000000002e-05 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.357141+0000", 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6944999999999999e-05 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3344 3.17 3:e83df825:::100000001f8.0000090d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 2.2208844320000001, 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096291749999999995, 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.910 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3344 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558869+0000", 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614237+0000", 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055368217999999997 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614250+0000", 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3259e-05 2024-09-16T01:52:07.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614255+0000", 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2100000000000001e-06 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614314+0000", 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8730999999999997e-05 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614411+0000", 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7116999999999999e-05 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614507+0000", 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5575999999999994e-05 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.617642+0000", 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003135166 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.914 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655055+0000", 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037413257999999998 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655113+0000", 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7726999999999998e-05 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655159+0000", 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6292999999999999e-05 2024-09-16T01:52:07.915 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3358 3.27 3:e7c6e76b:::100000001f8.0000091b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 2.046010678, 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096871567000000006, 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.916 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3358 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733744+0000", 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.886e-06 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787499+0000", 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053755360000000002 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787517+0000", 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7645000000000001e-05 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787525+0000", 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787592+0000", 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6891999999999996e-05 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787751+0000", 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158975 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787911+0000", 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016023400000000001 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.793715+0000", 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058038480000000003 2024-09-16T01:52:07.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830517+0000", 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036802435000000001 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830571+0000", 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3409000000000003e-05 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830613+0000", 2024-09-16T01:52:07.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2070000000000002e-05 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3464 3.0 3:03c3962b:::100000001f8.00000985:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e93)", 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:07.018491+0000", 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.76126073299999997, 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086867555999999999, 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.922 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3464 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.018491+0000", 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.018491+0000", 2024-09-16T01:52:07.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.018493+0000", 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.598e-06 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.079488+0000", 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060994828000000001 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.079503+0000", 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5291999999999999e-05 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.079510+0000", 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7050000000000002e-06 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.079570+0000", 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0718999999999999e-05 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.079697+0000", 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012694200000000001 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.079808+0000", 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110569 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.083383+0000", 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035753730000000002 2024-09-16T01:52:07.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.105153+0000", 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.021769954000000001 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.105223+0000", 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9546000000000001e-05 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.105359+0000", 2024-09-16T01:52:07.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013603000000000001 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3510 3.31 3:8c78e186:::100000001f8.000009b3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e93)", 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:07.467732+0000", 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.31202001200000001, 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.090051679999999995, 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:07.928 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3510 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.467732+0000", 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:07.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.467732+0000", 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.467734+0000", 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9470000000000002e-06 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.531566+0000", 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.063831758000000002 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.531582+0000", 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6497000000000001e-05 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.531591+0000", 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6130000000000004e-06 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.531660+0000", 2024-09-16T01:52:07.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9315999999999998e-05 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.531771+0000", 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110839 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.531892+0000", 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000120674 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:07.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.535716+0000", 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038242770000000001 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.557670+0000", 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.021953770000000001 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.557733+0000", 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3058000000000005e-05 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:07.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.557784+0000", 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0930999999999997e-05 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:07.934 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:07.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.793+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe754165940 con 0x7fe75406c150 2024-09-16T01:52:07.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.796+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe754165940 con 0x7fe75406c150 2024-09-16T01:52:07.936 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:07.937 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.796+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe7540cbc60 0x7fe7541416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.796+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe75414f7a0 con 0x7fe7540cbc60 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.796+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe7540cbc60 0x7fe7541416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.797+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe7540cbc60 0x7fe7541416b0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.798+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe75414f7a0 con 0x7fe7540cbc60 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.831+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe75414fd60 con 0x7fe7540cbc60 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.834+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe75414fd60 con 0x7fe7540cbc60 2024-09-16T01:52:07.938 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.836+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe75413ddf0 0x7fe75413e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.836+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe7540cd2e0 con 0x7fe75413ddf0 2024-09-16T01:52:07.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.838+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe75413ddf0 0x7fe75413e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.839+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe75413ddf0 0x7fe75413e1d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.841+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7540cd2e0 con 0x7fe75413ddf0 2024-09-16T01:52:07.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.869+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7540cd2e0 con 0x7fe75413ddf0 2024-09-16T01:52:07.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.872+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7540cd2e0 con 0x7fe75413ddf0 2024-09-16T01:52:07.940 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:07.941 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:07.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.873+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe754062b40 0x7fe754062f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.873+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe7540cd2e0 con 0x7fe754062b40 2024-09-16T01:52:07.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.873+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe754062b40 0x7fe754062f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.874+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe754062b40 0x7fe754062f40 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.875+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7540cd2e0 con 0x7fe754062b40 2024-09-16T01:52:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.891+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7540cd2e0 con 0x7fe754062b40 2024-09-16T01:52:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.894+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7540cd2e0 con 0x7fe754062b40 2024-09-16T01:52:07.942 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.894+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe7540c7690 0x7fe7540c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.894+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe75413ead0 con 0x7fe7540c7690 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.895+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe7540c7690 0x7fe7540c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.895+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe7540c7690 0x7fe7540c7a90 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.896+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe75413ead0 con 0x7fe7540c7690 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.920+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe75413ead0 con 0x7fe7540c7690 2024-09-16T01:52:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.922+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe75413ead0 con 0x7fe7540c7690 2024-09-16T01:52:07.944 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:07.945 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:07.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.923+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe75402cb50 0x7fe75402cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.923+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe7540c7fd0 con 0x7fe75402cb50 2024-09-16T01:52:07.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.923+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe75402cb50 0x7fe75402cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.924+0000 7fe753e00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe75402cb50 0x7fe75402cfb0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.924+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7540c7fd0 con 0x7fe75402cb50 2024-09-16T01:52:07.945 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:07.936+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:07.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.939+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd out", "ids": ["0"]} v 0) -- 0x7ff4bc0cc6b0 con 0x7ff4bc0cb2f0 2024-09-16T01:52:07.946 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:52:07.941+0000 7f49bee00640 -1 mon.a@0(leader).osd e93 definitely_dead 0 2024-09-16T01:52:07.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.944+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe7540c7fd0 con 0x7fe75402cb50 2024-09-16T01:52:07.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.946+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe7540c7fd0 con 0x7fe75402cb50 2024-09-16T01:52:07.948 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:07.948 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:07.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.946+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe7542074b0 0x7fe7542098a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.946+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe75402db40 con 0x7fe7542074b0 2024-09-16T01:52:07.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.947+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe7542074b0 0x7fe7542098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.947+0000 7fe759600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe7542074b0 0x7fe7542098a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.948+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe75402db40 con 0x7fe7542074b0 2024-09-16T01:52:07.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.969+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe754000950 con 0x7fe7542074b0 2024-09-16T01:52:07.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.973+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe754000950 con 0x7fe7542074b0 2024-09-16T01:52:07.974 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:07.974 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:07.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.973+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe75420a2f0 0x7fe75420a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:07.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.973+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe75420f880 con 0x7fe75420a2f0 2024-09-16T01:52:07.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.974+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe75420a2f0 0x7fe75420a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:07.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.974+0000 7fe758c00640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe75420a2f0 0x7fe75420a750 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:07.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.975+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe75420f880 con 0x7fe75420a2f0 2024-09-16T01:52:07.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.992+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe75420ada0 con 0x7fe75420a2f0 2024-09-16T01:52:07.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe750c00640 1 -- 172.21.15.146:0/3625070147 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe75420ada0 con 0x7fe75420a2f0 2024-09-16T01:52:07.995 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:07.996 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:07.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe75406c150 msgr2=0x7fe7540c2c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe75406c150 0x7fe7540c2c00 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe7540cbc60 msgr2=0x7fe7541416b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe7540cbc60 0x7fe7541416b0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe75413ddf0 msgr2=0x7fe75413e1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe75413ddf0 0x7fe75413e1d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe754062b40 msgr2=0x7fe754062f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe754062b40 0x7fe754062f40 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe7540c7690 msgr2=0x7fe7540c7a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.994+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe7540c7690 0x7fe7540c7a90 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe75402cb50 msgr2=0x7fe75402cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe75402cb50 0x7fe75402cfb0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe7542074b0 msgr2=0x7fe7542098a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe7542074b0 0x7fe7542098a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 reap_dead start 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe75420a2f0 msgr2=0x7fe75420a750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe75420a2f0 0x7fe75420a750 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 reap_dead start 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 reap_dead start 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe759600640 1 -- 172.21.15.146:0/3625070147 reap_dead start 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73806c2e0 msgr2=0x7fe73806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73806c2e0 0x7fe73806e7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe744000f60 tx=0x7fe744007e40 comp rx=0 tx=0).stop 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 msgr2=0x7fe754141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 0x7fe754141170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009e50 tx=0x7fe74c004a40 comp rx=0 tx=0).stop 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 shutdown_connections 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73806c2e0 0x7fe73806e7a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 --2- 172.21.15.146:0/3625070147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe75406d9d0 0x7fe754141170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 >> 172.21.15.146:0/3625070147 conn(0x7fe7540ccb10 msgr2=0x7fe7540c2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 shutdown_connections 2024-09-16T01:52:07.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:07.995+0000 7fe75b600640 1 -- 172.21.15.146:0/3625070147 wait complete. 2024-09-16T01:52:08.060 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:08.059+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:08.068 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.068 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.081 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:08.138 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:08.137+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:08.207 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.207 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.217 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:08.262 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:08.261+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:08.339 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:08.338+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:08.340 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.340 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.349 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:08.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.365+0000 7ff4b3e00640 1 -- 172.21.15.146:0/3983221926 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["0"]}]=0 marked out osd.0. v94) ==== 87+0+0 (secure 0 0 0) 0x7ff4ac079bb0 con 0x7ff4bc0cb2f0 2024-09-16T01:52:08.367 INFO:teuthology.orchestra.run.smithi146.stderr:marked out osd.0. 2024-09-16T01:52:08.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.371+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff49c06c2b0 msgr2=0x7ff49c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.371+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff49c06c2b0 0x7ff49c06e770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0c7bc0 tx=0x7ff4a8008040 comp rx=0 tx=0).stop 2024-09-16T01:52:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.371+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 msgr2=0x7ff4bc0b5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.371+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0b5e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.372+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 shutdown_connections 2024-09-16T01:52:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.372+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff49c06c2b0 0x7ff49c06e770 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.372+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4bc15e9d0 0x7ff4bc14b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.372+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4bc0cc0c0 0x7ff4bc14e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.372+0000 7ff4c2c00640 1 --2- 172.21.15.146:0/3983221926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4bc0cb2f0 0x7ff4bc0b5e20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.372+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 >> 172.21.15.146:0/3983221926 conn(0x7ff4bc06d560 msgr2=0x7ff4bc15a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:08.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.374+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 shutdown_connections 2024-09-16T01:52:08.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.376+0000 7ff4c2c00640 1 -- 172.21.15.146:0/3983221926 wait complete. 2024-09-16T01:52:08.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:52:08.456 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:08.455+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:08.502 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.502 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.511 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:08.542 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:08.541+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:08.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.580+0000 7f200f000640 1 Processor -- start 2024-09-16T01:52:08.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.580+0000 7f200f000640 1 -- start start 2024-09-16T01:52:08.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.581+0000 7f200f000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f200815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:08.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.581+0000 7f200f000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f200806d140 con 0x7f20080cc040 2024-09-16T01:52:08.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200f000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f200806d420 con 0x7f20080c75a0 2024-09-16T01:52:08.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200f000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f200815b880 con 0x7f20080c6b70 2024-09-16T01:52:08.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200dc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f200815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:08.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200d200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f20080c75a0 0x7f20080cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:35906/0 (socket says 172.21.15.146:35906) 2024-09-16T01:52:08.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200d200640 1 -- 172.21.15.146:0/641722287 learned_addr learned my addr 172.21.15.146:0/641722287 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:08.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200e600640 1 --1- 172.21.15.146:0/641722287 >> v1:172.21.15.146:6789/0 conn(0x7f20080cc040 0x7f2008165160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46714/0 (socket says 172.21.15.146:46714) 2024-09-16T01:52:08.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200dc00640 1 -- 172.21.15.146:0/641722287 >> v1:172.21.15.161:6789/0 conn(0x7f20080c75a0 legacy=0x7f20080cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200dc00640 1 -- 172.21.15.146:0/641722287 >> v1:172.21.15.146:6789/0 conn(0x7f20080cc040 legacy=0x7f2008165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f2007e00640 1 -- 172.21.15.146:0/641722287 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366036297 0 0) 0x7f200806d420 con 0x7f20080c75a0 2024-09-16T01:52:08.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.582+0000 7f200dc00640 1 -- 172.21.15.146:0/641722287 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200806d700 con 0x7f20080c6b70 2024-09-16T01:52:08.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.583+0000 7f200dc00640 1 --2- 172.21.15.146:0/641722287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f200815b340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0097d0 tx=0x7f1ffc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f6e3043f88d218c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:08.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.583+0000 7f2007e00640 1 -- 172.21.15.146:0/641722287 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc037070 con 0x7f20080c6b70 2024-09-16T01:52:08.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.583+0000 7f2007e00640 1 -- 172.21.15.146:0/641722287 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc032440 con 0x7f20080c6b70 2024-09-16T01:52:08.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.583+0000 7f2007e00640 1 -- 172.21.15.146:0/641722287 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc032810 con 0x7f20080c6b70 2024-09-16T01:52:08.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 -- 172.21.15.146:0/641722287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 msgr2=0x7f200815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 --2- 172.21.15.146:0/641722287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f200815b340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0097d0 tx=0x7f1ffc02a180 comp rx=0 tx=0).stop 2024-09-16T01:52:08.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 -- 172.21.15.146:0/641722287 shutdown_connections 2024-09-16T01:52:08.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 --2- 172.21.15.146:0/641722287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f200815b340 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 -- 172.21.15.146:0/641722287 >> 172.21.15.146:0/641722287 conn(0x7f20080ccc20 msgr2=0x7f20080cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:08.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 -- 172.21.15.146:0/641722287 shutdown_connections 2024-09-16T01:52:08.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.584+0000 7f200f000640 1 -- 172.21.15.146:0/641722287 wait complete. 2024-09-16T01:52:08.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.586+0000 7f200f000640 1 Processor -- start 2024-09-16T01:52:08.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.586+0000 7f200f000640 1 -- start start 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.586+0000 7f200f000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f20081e4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200f000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20080c75a0 0x7f20081523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20080cc040 0x7f20081528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20080cb050 con 0x7f20080cc040 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200f000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20080cab20 con 0x7f20080c75a0 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200f000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f20080cbc80 con 0x7f20080c6b70 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f20081e4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f20081e4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56318/0 (socket says 172.21.15.146:56318) 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 -- 172.21.15.146:0/1825133631 learned_addr learned my addr 172.21.15.146:0/1825133631 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:08.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200d200640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20080c75a0 0x7f20081523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200e600640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20080cc040 0x7f20081528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 -- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20080c75a0 msgr2=0x7f20081523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20080c75a0 0x7f20081523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 -- 172.21.15.146:0/1825133631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20080cc040 msgr2=0x7f20081528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20080cc040 0x7f20081528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 -- 172.21.15.146:0/1825133631 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20081f09a0 con 0x7f20080c6b70 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200e600640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20080cc040 0x7f20081528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.587+0000 7f200dc00640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f20081e4fc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc035040 tx=0x7f1ffc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.588+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc037040 con 0x7f20080c6b70 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.588+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc004250 con 0x7f20080c6b70 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.588+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20081e6d80 con 0x7f20080c6b70 2024-09-16T01:52:08.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.588+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc032e40 con 0x7f20080c6b70 2024-09-16T01:52:08.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.588+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20081580f0 con 0x7f20080c6b70 2024-09-16T01:52:08.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.589+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200806cd20 con 0x7f20080c6b70 2024-09-16T01:52:08.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.590+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1ffc0306f0 con 0x7f20080c6b70 2024-09-16T01:52:08.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.591+0000 7f2005600640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1fe806c2e0 0x7f1fe806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:08.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.591+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(94..94 src has 1..94) ==== 5741+0+0 (secure 0 0 0) 0x7f1ffc0ac6c0 con 0x7f20080c6b70 2024-09-16T01:52:08.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.591+0000 7f200d200640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1fe806c2e0 0x7f1fe806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:08.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.591+0000 7f200d200640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1fe806c2e0 0x7f1fe806e7a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f20080c32d0 tx=0x7f1ff4007f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:08.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.595+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1ffc07ad70 con 0x7f20080c6b70 2024-09-16T01:52:08.616 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.616 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.625 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:08.643 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:08.642+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:08.728 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.729 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:08.743 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:08.742+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:08.781+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f200806da90 con 0x7f20080c6b70 2024-09-16T01:52:08.845 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:08.844+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:08.895 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:08.895 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:08.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:08.946 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:08.945+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:09.047 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:09.046+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:09.103 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:09.103 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:09.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:09.157 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:09.156+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:09.248 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:09.247+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:09.314 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:09.314 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:09.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:09.377 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:09.367+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:09.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.369+0000 7f2005600640 1 -- 172.21.15.146:0/1825133631 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v95) ==== 97+0+0 (secure 0 0 0) 0x7f1ffc07a670 con 0x7f20080c6b70 2024-09-16T01:52:09.377 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:52:09.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.379+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1fe806c2e0 msgr2=0x7f1fe806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:09.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.379+0000 7f200f000640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1fe806c2e0 0x7f1fe806e7a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f20080c32d0 tx=0x7f1ff4007f00 comp rx=0 tx=0).stop 2024-09-16T01:52:09.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.379+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 msgr2=0x7f20081e4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:09.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.379+0000 7f200f000640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f20081e4fc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc035040 tx=0x7f1ffc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:09.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.382+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 shutdown_connections 2024-09-16T01:52:09.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.382+0000 7f200f000640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1fe806c2e0 0x7f1fe806e7a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:09.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.382+0000 7f200f000640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20080cc040 0x7f20081528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:09.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.382+0000 7f200f000640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20080c75a0 0x7f20081523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:09.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.382+0000 7f200f000640 1 --2- 172.21.15.146:0/1825133631 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f20080c6b70 0x7f20081e4fc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:09.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.382+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 >> 172.21.15.146:0/1825133631 conn(0x7f20080ccc20 msgr2=0x7f20080c82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:09.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.384+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 shutdown_connections 2024-09-16T01:52:09.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:09.384+0000 7f200f000640 1 -- 172.21.15.146:0/1825133631 wait complete. 2024-09-16T01:52:09.444 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:52:09.445 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 initial writers = 306556.41 kB/sec 2024-09-16T01:52:09.445 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 initial writers = 306546.60 kB/sec 2024-09-16T01:52:09.445 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 306556.41 kB/sec 2024-09-16T01:52:09.445 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 306556.41 kB/sec 2024-09-16T01:52:09.445 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 306556.41 kB/sec 2024-09-16T01:52:09.445 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 10485760.00 kB 2024-09-16T01:52:09.450 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:09.449+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:09.551 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:09.550+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:09.637 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:09.638 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:09.652 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:09.651+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:09.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:09.753 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:09.752+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:09.853 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:09.852+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:09.943 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:09.943 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:09.955 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:09.954+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:09.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:10.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:10.095 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:10.094+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:10.156 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:10.155+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:10.257 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:10.255+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:10.264 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:10.265 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:10.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.293+0000 7fcab9600640 1 Processor -- start 2024-09-16T01:52:10.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.294+0000 7fcab9600640 1 -- start start 2024-09-16T01:52:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab9600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcab4152a10 0x7fcab40c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab9600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcab40c6bb0 0x7fcab40c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab4158bd0 0x7fcab4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab9600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcab41586d0 con 0x7fcab40c6bb0 2024-09-16T01:52:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab9600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcab406de70 con 0x7fcab4152a10 2024-09-16T01:52:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab9600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcab406e150 con 0x7fcab4158bd0 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.295+0000 7fcab3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcab4152a10 0x7fcab40c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcab4152a10 0x7fcab40c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:35116/0 (socket says 172.21.15.146:35116) 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 -- 172.21.15.146:0/3688097108 learned_addr learned my addr 172.21.15.146:0/3688097108 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab8c00640 1 --2- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab4158bd0 0x7fcab4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 -- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab4158bd0 msgr2=0x7fcab4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 --2- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab4158bd0 0x7fcab4158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 -- 172.21.15.146:0/3688097108 >> v2:172.21.15.146:3300/0 conn(0x7fcab40c6bb0 msgr2=0x7fcab40c5e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 --2- 172.21.15.146:0/3688097108 >> v2:172.21.15.146:3300/0 conn(0x7fcab40c6bb0 0x7fcab40c5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.296+0000 7fcab3e00640 1 -- 172.21.15.146:0/3688097108 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab406e680 con 0x7fcab4152a10 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.298+0000 7fcab3e00640 1 --2- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4152a10 0x7fcab40c58c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcab40c9cc0 tx=0x7fcaa402a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=141abb017d6c2935 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.298+0000 7fcab2a00640 1 -- 172.21.15.146:0/3688097108 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4037070 con 0x7fcab4152a10 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.298+0000 7fcab2a00640 1 -- 172.21.15.146:0/3688097108 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaa4032440 con 0x7fcab4152a10 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.298+0000 7fcab2a00640 1 -- 172.21.15.146:0/3688097108 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4032810 con 0x7fcab4152a10 2024-09-16T01:52:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.299+0000 7fcab9600640 1 -- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4152a10 msgr2=0x7fcab40c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.299+0000 7fcab9600640 1 --2- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4152a10 0x7fcab40c58c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcab40c9cc0 tx=0x7fcaa402a160 comp rx=0 tx=0).stop 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 -- 172.21.15.146:0/3688097108 shutdown_connections 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 --2- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab4158bd0 0x7fcab4158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 --2- 172.21.15.146:0/3688097108 >> v2:172.21.15.146:3300/0 conn(0x7fcab40c6bb0 0x7fcab40c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 --2- 172.21.15.146:0/3688097108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4152a10 0x7fcab40c58c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 -- 172.21.15.146:0/3688097108 >> 172.21.15.146:0/3688097108 conn(0x7fcab40cc1b0 msgr2=0x7fcab40cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 -- 172.21.15.146:0/3688097108 shutdown_connections 2024-09-16T01:52:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.300+0000 7fcab9600640 1 -- 172.21.15.146:0/3688097108 wait complete. 2024-09-16T01:52:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.302+0000 7fcab9600640 1 Processor -- start 2024-09-16T01:52:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.303+0000 7fcab9600640 1 -- start start 2024-09-16T01:52:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.304+0000 7fcab9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab40c6bb0 0x7fcab41e1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.304+0000 7fcab9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcab4152a10 0x7fcab41514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.304+0000 7fcab9600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4158bd0 0x7fcab41e2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.304+0000 7fcab9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4159530 con 0x7fcab4152a10 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.304+0000 7fcab9600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcab406ec40 con 0x7fcab4158bd0 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.304+0000 7fcab9600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcab4152df0 con 0x7fcab40c6bb0 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab40c6bb0 0x7fcab41e1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab40c6bb0 0x7fcab41e1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56336/0 (socket says 172.21.15.146:56336) 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab8c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4158bd0 0x7fcab41e2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 -- 172.21.15.146:0/44790147 learned_addr learned my addr 172.21.15.146:0/44790147 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcab4152a10 0x7fcab41514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4158bd0 msgr2=0x7fcab41e2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4158bd0 0x7fcab41e2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcab4152a10 msgr2=0x7fcab41514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcab4152a10 0x7fcab41514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab3e00640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab41e44c0 con 0x7fcab40c6bb0 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.305+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcab4158bd0 0x7fcab41e2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.306+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab40c6bb0 0x7fcab41e1bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4000c00 tx=0x7fcaa40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.307+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4037040 con 0x7fcab40c6bb0 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.307+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaa4042070 con 0x7fcab40c6bb0 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.307+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa4030880 con 0x7fcab40c6bb0 2024-09-16T01:52:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.307+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab41e4a70 con 0x7fcab40c6bb0 2024-09-16T01:52:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.307+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab414ca30 con 0x7fcab40c6bb0 2024-09-16T01:52:10.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.309+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fca8c005270 con 0x7fcab40c6bb0 2024-09-16T01:52:10.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.310+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcaa4030a20 con 0x7fcab40c6bb0 2024-09-16T01:52:10.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.312+0000 7fcaabe00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fca9406c2b0 0x7fca9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.312+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(95..95 src has 1..95) ==== 5741+0+0 (secure 0 0 0) 0x7fcaa40abb80 con 0x7fcab40c6bb0 2024-09-16T01:52:10.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.313+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fca9406c2b0 0x7fca9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.313+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v95) ==== 54+0+16 (secure 0 0 0) 0x7fcaa4079c70 con 0x7fcab40c6bb0 2024-09-16T01:52:10.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.314+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fca8c000fa0 0x7fca8c0045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.314+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca8c004b20 con 0x7fca8c000fa0 2024-09-16T01:52:10.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.314+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fca9406c2b0 0x7fca9406e770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcab40c3170 tx=0x7fcaa0009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.314+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fca8c000fa0 0x7fca8c0045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.315+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fca8c000fa0 0x7fca8c0045e0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.317+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c004b20 con 0x7fca8c000fa0 2024-09-16T01:52:10.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.339+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c005090 con 0x7fca8c000fa0 2024-09-16T01:52:10.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.343+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c005090 con 0x7fca8c000fa0 2024-09-16T01:52:10.344 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:10.345 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:10.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.343+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fca8c001c70 0x7fca8c002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.344+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fca8c0022b0 con 0x7fca8c001c70 2024-09-16T01:52:10.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.344+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fca8c001c70 0x7fca8c002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.345+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fca8c001c70 0x7fca8c002990 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.346+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c0022b0 con 0x7fca8c001c70 2024-09-16T01:52:10.358 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:10.357+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:10.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.370+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c0031a0 con 0x7fca8c001c70 2024-09-16T01:52:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.374+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c0031a0 con 0x7fca8c001c70 2024-09-16T01:52:10.375 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.374+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fca8c009cd0 0x7fca8c00c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.374+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fca8c00ee90 con 0x7fca8c009cd0 2024-09-16T01:52:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.375+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fca8c009cd0 0x7fca8c00c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.376+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fca8c009cd0 0x7fca8c00c0c0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.378+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c00ee90 con 0x7fca8c009cd0 2024-09-16T01:52:10.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.405+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c00c8d0 con 0x7fca8c009cd0 2024-09-16T01:52:10.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.408+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c00c8d0 con 0x7fca8c009cd0 2024-09-16T01:52:10.409 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:10.410 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:10.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.409+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fca8c00d390 0x7fca8c00d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.409+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fca8c00df30 con 0x7fca8c00d390 2024-09-16T01:52:10.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.411+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fca8c00d390 0x7fca8c00d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.412+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fca8c00d390 0x7fca8c00d7f0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.413+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c00df30 con 0x7fca8c00d390 2024-09-16T01:52:10.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.435+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c00e440 con 0x7fca8c00d390 2024-09-16T01:52:10.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.440+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c00e440 con 0x7fca8c00d390 2024-09-16T01:52:10.442 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:10.442 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:10.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.441+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fca8c011050 0x7fca8c013440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.441+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fca8c00eca0 con 0x7fca8c011050 2024-09-16T01:52:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.442+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fca8c011050 0x7fca8c013440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.442+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fca8c011050 0x7fca8c013440 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.444+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c00eca0 con 0x7fca8c011050 2024-09-16T01:52:10.459 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:10.458+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:10.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.474+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c00ec70 con 0x7fca8c011050 2024-09-16T01:52:10.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.476+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c00ec70 con 0x7fca8c011050 2024-09-16T01:52:10.478 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:10.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.477+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fca8c013e90 0x7fca8c0142f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.477+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fca8c019400 con 0x7fca8c013e90 2024-09-16T01:52:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.478+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fca8c013e90 0x7fca8c0142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.479+0000 7fcab3400640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fca8c013e90 0x7fca8c0142f0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.480+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c019400 con 0x7fca8c013e90 2024-09-16T01:52:10.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.512+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c014970 con 0x7fca8c013e90 2024-09-16T01:52:10.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.514+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c014970 con 0x7fca8c013e90 2024-09-16T01:52:10.516 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:10.516 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:10.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.515+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fca8c015090 0x7fca8c0154d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.515+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fca8c015bc0 con 0x7fca8c015090 2024-09-16T01:52:10.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.515+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fca8c015090 0x7fca8c0154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.516+0000 7fcab8c00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fca8c015090 0x7fca8c0154d0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.517+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c015bc0 con 0x7fca8c015090 2024-09-16T01:52:10.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.539+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c0008d0 con 0x7fca8c015090 2024-09-16T01:52:10.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.541+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c0008d0 con 0x7fca8c015090 2024-09-16T01:52:10.542 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.541+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fca8c01a7a0 0x7fca8c01cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.541+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fca8c01f9e0 con 0x7fca8c01a7a0 2024-09-16T01:52:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.542+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fca8c01a7a0 0x7fca8c01cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:10.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.543+0000 7fcab3e00640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fca8c01a7a0 0x7fca8c01cb90 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:10.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.544+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c01f9e0 con 0x7fca8c01a7a0 2024-09-16T01:52:10.553 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:10.553 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:10.553 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:10.553 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:10.560 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:10.559+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:10.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.564+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fca8c01d150 con 0x7fca8c01a7a0 2024-09-16T01:52:10.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.566+0000 7fcaabe00640 1 -- 172.21.15.146:0/44790147 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fca8c01d150 con 0x7fca8c01a7a0 2024-09-16T01:52:10.567 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:10.568 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:10.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fca8c000fa0 msgr2=0x7fca8c0045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fca8c000fa0 0x7fca8c0045e0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fca8c001c70 msgr2=0x7fca8c002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fca8c001c70 0x7fca8c002990 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fca8c009cd0 msgr2=0x7fca8c00c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fca8c009cd0 0x7fca8c00c0c0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fca8c00d390 msgr2=0x7fca8c00d7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fca8c00d390 0x7fca8c00d7f0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fca8c011050 msgr2=0x7fca8c013440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fca8c011050 0x7fca8c013440 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fca8c013e90 msgr2=0x7fca8c0142f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fca8c013e90 0x7fca8c0142f0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fca8c015090 msgr2=0x7fca8c0154d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fca8c015090 0x7fca8c0154d0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fca8c01a7a0 msgr2=0x7fca8c01cb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fca8c01a7a0 0x7fca8c01cb90 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab8c00640 1 -- 172.21.15.146:0/44790147 reap_dead start 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab8c00640 1 -- 172.21.15.146:0/44790147 reap_dead start 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fca9406c2b0 msgr2=0x7fca9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fca9406c2b0 0x7fca9406e770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcab40c3170 tx=0x7fcaa0009060 comp rx=0 tx=0).stop 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab40c6bb0 msgr2=0x7fcab41e1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab8c00640 1 -- 172.21.15.146:0/44790147 reap_dead start 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 --2- 172.21.15.146:0/44790147 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcab40c6bb0 0x7fcab41e1bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4000c00 tx=0x7fcaa40046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab8c00640 1 -- 172.21.15.146:0/44790147 reap_dead start 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab8c00640 1 -- 172.21.15.146:0/44790147 reap_dead start 2024-09-16T01:52:10.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 shutdown_connections 2024-09-16T01:52:10.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 >> 172.21.15.146:0/44790147 conn(0x7fcab40cc1b0 msgr2=0x7fcab41485d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:10.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 shutdown_connections 2024-09-16T01:52:10.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:10.567+0000 7fcab9600640 1 -- 172.21.15.146:0/44790147 wait complete. 2024-09-16T01:52:10.574 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:10.682 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:10.681+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:10.762 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:10.761+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:10.863 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:10.862+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:10.891 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:10.892 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:10.892 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:10.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:11.001 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:11.000+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:11.064 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:11.063+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:11.165 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:11.164+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:11.214 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.547941659999999, 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:11.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1031 3.e 3:73e84e43:::100000001f8.00000004:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.038326116, 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.093093621000000001, 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1031 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171939+0000", 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.171941+0000", 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2230000000000001e-06 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182392+0000", 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010451225999999999 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182418+0000", 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5561000000000001e-05 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.182431+0000", 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3012e-05 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186260+0000", 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0038291340000000001 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186410+0000", 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014975200000000001 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.186556+0000", 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000146587 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.210170+0000", 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023614109000000001 2024-09-16T01:52:11.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.264915+0000", 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054744835999999998 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.264984+0000", 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8753999999999999e-05 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:38.265032+0000", 2024-09-16T01:52:11.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8427000000000002e-05 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.919082393, 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:11.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:11.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.889921537999999, 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:11.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:11.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:11.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.85756997, 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.244 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.841622703999999, 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:11.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:11.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1132 3.5 3:a0fd78c8:::100000001f8.00000069:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.819088007000001, 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095513232000000003, 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1132 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391179+0000", 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.66e-06 2024-09-16T01:52:11.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408341+0000", 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017162686999999999 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408396+0000", 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5034000000000002e-05 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408412+0000", 2024-09-16T01:52:11.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5685e-05 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408710+0000", 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029794899999999999 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408870+0000", 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000160168 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.409033+0000", 2024-09-16T01:52:11.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016284700000000001 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480660+0000", 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071627535000000006 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486552+0000", 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0058913079999999996 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486634+0000", 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2051000000000004e-05 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486690+0000", 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6307999999999997e-05 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:11.256 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.294547539, 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:11.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:11.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:11.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.109618182999998, 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:11.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:11.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:11.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.107080374999999, 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:11.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:11.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:11.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1966 3.3c 3:3c832e37:::100000001f8.000003ab:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.754045634000001, 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10596815800000001, 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1966 2024-09-16T01:52:11.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456221+0000", 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.125e-06 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520630+0000", 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064408699 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520660+0000", 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0122000000000001e-05 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520672+0000", 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2179e-05 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520772+0000", 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.993e-05 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520970+0000", 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198097 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.521122+0000", 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151065 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.535969+0000", 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014847075 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562086+0000", 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026117475000000001 2024-09-16T01:52:11.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562144+0000", 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7806000000000002e-05 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562187+0000", 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3584999999999997e-05 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.497669143, 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:11.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.288 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2096 3.16 3:68301556:::100000001f8.0000042d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.937130807999999, 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096328759999999999, 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2096 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273136+0000", 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306295+0000", 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033158316 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306329+0000", 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4656999999999997e-05 2024-09-16T01:52:11.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306353+0000", 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4244999999999999e-05 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306437+0000", 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3885000000000001e-05 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306614+0000", 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000176656 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306762+0000", 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014849399999999999 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.368128+0000", 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061365140999999998 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369361+0000", 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001232934 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369413+0000", 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2123000000000002e-05 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369463+0000", 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0229000000000003e-05 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.294 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2449 3.2 3:4180d341:::100000001f8.0000058e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.446705+0000", 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.763560255, 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092182652000000004, 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.295 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2449 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.446705+0000", 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.446705+0000", 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.446707+0000", 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3659999999999999e-06 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.482929+0000", 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036221824 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485444+0000", 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025151800000000001 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485459+0000", 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4813e-05 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485569+0000", 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011008199999999999 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485728+0000", 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000159476 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.485886+0000", 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157608 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.518770+0000", 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032883901 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.538767+0000", 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019997451999999999 2024-09-16T01:52:11.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.538835+0000", 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7733999999999999e-05 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.538887+0000", 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2216000000000003e-05 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2989 3.e 3:715a0782:::100000001f8.000007aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.298775+0000", 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.9114895759999992, 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092600565999999995, 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2989 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.301 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.298775+0000", 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.298775+0000", 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.298778+0000", 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5900000000000002e-06 2024-09-16T01:52:11.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342873+0000", 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044094792000000001 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342890+0000", 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7659000000000002e-05 2024-09-16T01:52:11.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342902+0000", 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1389999999999999e-05 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.342982+0000", 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9728999999999995e-05 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.343135+0000", 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015343999999999999 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.343278+0000", 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014258000000000001 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.386435+0000", 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043157790000000001 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.391251+0000", 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0048159270000000002 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.391316+0000", 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4589000000000001e-05 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.391376+0000", 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0080000000000001e-05 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3056 3.16 3:68f2fc0a:::100000001f8.000007ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.2020379749999996, 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094053392, 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3056 2024-09-16T01:52:11.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008229+0000", 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.756e-06 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062258+0000", 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054028822999999997 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062288+0000", 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0896999999999999e-05 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062302+0000", 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3112000000000001e-05 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062499+0000", 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019759500000000001 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062685+0000", 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186281 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062841+0000", 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155847 2024-09-16T01:52:11.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.074322+0000", 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011480959000000001 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102173+0000", 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027851130000000002 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102236+0000", 2024-09-16T01:52:11.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2749000000000005e-05 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102280+0000", 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4243000000000003e-05 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3350 3.3c 3:3f688ec3:::100000001f8.00000913:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.639516+0000", 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5707485190000003, 2024-09-16T01:52:11.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097703430999999993, 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3350 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.639516+0000", 2024-09-16T01:52:11.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.639516+0000", 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.639519+0000", 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3039999999999999e-06 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674045+0000", 2024-09-16T01:52:11.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034526172000000001 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674061+0000", 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5896000000000002e-05 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674071+0000", 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7340000000000003e-06 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674458+0000", 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038717899999999997 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674612+0000", 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015388700000000001 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674756+0000", 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144704 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.717031+0000", 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042275158 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.737101+0000", 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020069872999999998 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.737169+0000", 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7393999999999999e-05 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.737220+0000", 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.113e-05 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3362 3.3c 3:3f474b19:::100000001f8.0000091f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.768356+0000", 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.4419091770000003, 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095600888999999994, 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.319 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3362 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.768356+0000", 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.768356+0000", 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.768358+0000", 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3e-06 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.799066+0000", 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030707463000000001 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.799079+0000", 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3635000000000001e-05 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.799087+0000", 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6650000000000003e-06 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.801339+0000", 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022524950000000002 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.801477+0000", 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000137706 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.801631+0000", 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015402999999999999 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.854713+0000", 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053081642999999998 2024-09-16T01:52:11.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.863843+0000", 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091305929999999993 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.863912+0000", 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8715999999999997e-05 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.863957+0000", 2024-09-16T01:52:11.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4642999999999999e-05 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3377 3.26 3:6604db39:::100000001f8.0000092e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2842785760000002, 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.104633035, 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3377 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925988+0000", 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0549999999999998e-06 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983358+0000", 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057369575999999999 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983375+0000", 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7356000000000001e-05 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983387+0000", 2024-09-16T01:52:11.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1277000000000001e-05 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983509+0000", 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012268399999999999 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983659+0000", 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014946499999999999 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:11.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983799+0000", 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139787 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.000668+0000", 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016869359 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030498+0000", 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029829855999999998 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030562+0000", 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4199e-05 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030619+0000", 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7420999999999998e-05 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3378 3.e 3:709bbee6:::100000001f8.0000092f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:11.332 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2268547610000002, 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108434431, 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3378 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983412+0000", 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.528e-06 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.059475+0000", 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076063617 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060086+0000", 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061040300000000003 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060097+0000", 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0842999999999999e-05 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060152+0000", 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5025999999999998e-05 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060248+0000", 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6351e-05 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060334+0000", 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6216000000000003e-05 2024-09-16T01:52:11.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.077660+0000", 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017325871 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091714+0000", 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014053566 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091794+0000", 2024-09-16T01:52:11.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0434999999999998e-05 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091845+0000", 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0575000000000003e-05 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.338 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:11.339 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:11.339 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:11.328+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:11.429 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:11.428+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:11.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:52:11.531 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:11.530+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "MOSDPGPushReply(3.3f 95/94 [PushReplyOp(3:fea993d3:::100000001f8.00000773:head)])", 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:11.620740+0000", 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.00114905, 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011755540000000001, 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "reached pg", 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:11.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620740+0000", 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620740+0000", 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620740+0000", 2024-09-16T01:52:11.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5799999999999998e-07 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620745+0000", 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2379999999999999e-06 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620758+0000", 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3821e-05 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:11.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620767+0000", 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4400000000000005e-06 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:11.620835+0000", 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7939000000000001e-05 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:52:11.626 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:11.632 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:11.631+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:11.643 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:11.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.675+0000 7f8feaa00640 1 Processor -- start 2024-09-16T01:52:11.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.676+0000 7f8feaa00640 1 -- start start 2024-09-16T01:52:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.677+0000 7f8feaa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8fe40cc0c0 0x7f8fe415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.677+0000 7f8feaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fe40cc4a0 0x7f8fe4165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.677+0000 7f8fe3400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8fe40cb2f0 0x7f8fe415e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:46724/0 (socket says 172.21.15.146:46724) 2024-09-16T01:52:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.677+0000 7f8fe3400640 1 -- 172.21.15.146:0/2374912605 learned_addr learned my addr 172.21.15.146:0/2374912605 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.677+0000 7f8fe3e00640 1 --2- 172.21.15.146:0/2374912605 >> v2:172.21.15.161:3300/0 conn(0x7f8fe40cc0c0 0x7f8fe415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8fe8a00640 1 --2- 172.21.15.146:0/2374912605 >> v2:172.21.15.146:3300/0 conn(0x7f8fe40cc4a0 0x7f8fe4165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.677+0000 7f8feaa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe415b1d0 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe40cbd00 con 0x7f8fe40cc4a0 2024-09-16T01:52:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe415ba70 con 0x7f8fe40cc0c0 2024-09-16T01:52:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8fe8a00640 1 -- 172.21.15.146:0/2374912605 >> v2:172.21.15.161:3300/0 conn(0x7f8fe40cc0c0 msgr2=0x7f8fe415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8fe8a00640 1 --2- 172.21.15.146:0/2374912605 >> v2:172.21.15.161:3300/0 conn(0x7f8fe40cc0c0 0x7f8fe415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8fe8a00640 1 -- 172.21.15.146:0/2374912605 >> v1:172.21.15.146:6789/0 conn(0x7f8fe40cb2f0 legacy=0x7f8fe415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.678+0000 7f8fe8a00640 1 -- 172.21.15.146:0/2374912605 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe415b4b0 con 0x7f8fe40cc4a0 2024-09-16T01:52:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.680+0000 7f8fe8a00640 1 --2- 172.21.15.146:0/2374912605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cc4a0 0x7f8fe4165430 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40ce150 tx=0x7f8fd402a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbaf2c15646a66a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.680+0000 7f8fe2a00640 1 -- 172.21.15.146:0/2374912605 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4037070 con 0x7f8fe40cc4a0 2024-09-16T01:52:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.680+0000 7f8fe2a00640 1 -- 172.21.15.146:0/2374912605 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd402ad30 con 0x7f8fe40cc4a0 2024-09-16T01:52:11.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.681+0000 7f8fe2a00640 1 -- 172.21.15.146:0/2374912605 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4032660 con 0x7f8fe40cc4a0 2024-09-16T01:52:11.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.681+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cc4a0 msgr2=0x7f8fe4165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:11.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.681+0000 7f8feaa00640 1 --2- 172.21.15.146:0/2374912605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cc4a0 0x7f8fe4165430 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40ce150 tx=0x7f8fd402a0a0 comp rx=0 tx=0).stop 2024-09-16T01:52:11.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.682+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 shutdown_connections 2024-09-16T01:52:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.682+0000 7f8feaa00640 1 --2- 172.21.15.146:0/2374912605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cc4a0 0x7f8fe4165430 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.682+0000 7f8feaa00640 1 --2- 172.21.15.146:0/2374912605 >> v2:172.21.15.161:3300/0 conn(0x7f8fe40cc0c0 0x7f8fe415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.682+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 >> 172.21.15.146:0/2374912605 conn(0x7f8fe406d560 msgr2=0x7f8fe406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.682+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 shutdown_connections 2024-09-16T01:52:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.683+0000 7f8feaa00640 1 -- 172.21.15.146:0/2374912605 wait complete. 2024-09-16T01:52:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.685+0000 7f8feaa00640 1 Processor -- start 2024-09-16T01:52:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.685+0000 7f8feaa00640 1 -- start start 2024-09-16T01:52:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.686+0000 7f8feaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 0x7f8fe415cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.686+0000 7f8feaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe40cc0c0 0x7f8fe4157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.686+0000 7f8feaa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8fe40cc4a0 0x7f8fe4154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.686+0000 7f8feaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe415e1d0 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.686+0000 7f8feaa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe415e540 con 0x7f8fe40cc0c0 2024-09-16T01:52:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.686+0000 7f8feaa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe40cc8b0 con 0x7f8fe40cc4a0 2024-09-16T01:52:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 0x7f8fe415cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 0x7f8fe415cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37466/0 (socket says 172.21.15.146:37466) 2024-09-16T01:52:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 -- 172.21.15.146:0/3285687484 learned_addr learned my addr 172.21.15.146:0/3285687484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3400640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe40cc0c0 0x7f8fe4157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe8a00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8fe40cc4a0 0x7f8fe4154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 -- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8fe40cc4a0 msgr2=0x7f8fe4154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8fe40cc4a0 0x7f8fe4154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 -- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe40cc0c0 msgr2=0x7f8fe4157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe40cc0c0 0x7f8fe4157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3e00640 1 -- 172.21.15.146:0/3285687484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe415d3e0 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.687+0000 7f8fe3400640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe40cc0c0 0x7f8fe4157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8fe8a00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8fe40cc4a0 0x7f8fe4154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8fe3e00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 0x7f8fe415cca0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009db0 tx=0x7f8fdc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc018020 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fdc014480 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc014830 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe41f06c0 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.688+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe41f0c80 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.690+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe40c5560 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.693+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8fdc005d10 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.695+0000 7f8fdbe00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8fc006c3b0 0x7f8fc006e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.695+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5741+0+0 (secure 0 0 0) 0x7f8fdc08bf00 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.695+0000 7f8fe3400640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8fc006c3b0 0x7f8fc006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.696+0000 7f8fe3400640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8fc006c3b0 0x7f8fc006e870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40d0a50 tx=0x7f8fcc007ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:11.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:11.702+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8fdc022070 con 0x7f8fe40cb2f0 2024-09-16T01:52:11.733 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:11.732+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:11.834 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:11.833+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:11.899 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:11.900 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:11.900 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:11.900 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:11.900 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:11.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:11.961 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:11.960+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:12.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.009+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8fe40cbea0 con 0x7f8fe40cb2f0 2024-09-16T01:52:12.034 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:12.033+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:12.137 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:12.136+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:12.209 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.356931853000006, 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:12.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:12.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.214472188000002, 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.217 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:12.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:12.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 71.867005965000004, 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:12.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:12.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.982714600999998, 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.232 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:12.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.944975098, 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:12.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:12.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.085988098000001, 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.244 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:12.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1620 3.13 3:c81354c0:::100000001f8.00000251:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:45.341779+0000", 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.863889591, 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.104539276, 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.250 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1620 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.341779+0000", 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.341779+0000", 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.341782+0000", 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7800000000000001e-06 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425499+0000", 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083716384000000005 2024-09-16T01:52:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425530+0000", 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1145000000000001e-05 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425544+0000", 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3835999999999999e-05 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425678+0000", 2024-09-16T01:52:12.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013432799999999999 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.425852+0000", 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000174286 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.426035+0000", 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000183191 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.442798+0000", 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016762945000000001 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.446209+0000", 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003410344 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.446269+0000", 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0217999999999998e-05 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:45.446319+0000", 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9818999999999999e-05 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.218949124000002, 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.256 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:12.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:12.262 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.022996902999999, 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:12.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:12.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.981994960000002, 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:12.269 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:12.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:12.275 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.757767399999999, 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:12.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:12.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2007 3.2f 3:f45f7969:::100000001f8.000003d4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.065925+0000", 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.139744091000001, 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097559495999999996, 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2007 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.065925+0000", 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.065925+0000", 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.065928+0000", 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5840000000000002e-06 2024-09-16T01:52:12.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.137908+0000", 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071980743999999999 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.137922+0000", 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4158000000000001e-05 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.137934+0000", 2024-09-16T01:52:12.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1304999999999999e-05 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.138006+0000", 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1860999999999995e-05 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.138164+0000", 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000158362 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.138347+0000", 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000182573 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.155916+0000", 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017569784000000001 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.163388+0000", 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074713109999999996 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.163444+0000", 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6474000000000002e-05 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.163484+0000", 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0339999999999997e-05 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.287 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2045 3.13 3:c8562531:::100000001f8.000003fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.595706+0000", 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.609962907, 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097724925000000004, 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2045 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.595706+0000", 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.595706+0000", 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.595708+0000", 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.199e-06 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624248+0000", 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028539653000000002 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624266+0000", 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7598e-05 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624278+0000", 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2189e-05 2024-09-16T01:52:12.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.624857+0000", 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000579015 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.625007+0000", 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015013799999999999 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.625164+0000", 2024-09-16T01:52:12.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156646 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.646055+0000", 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020891528999999999 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.693173+0000", 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047118243999999997 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.693235+0000", 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1975000000000002e-05 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.693431+0000", 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000195739 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2093 3.2d 3:b5a9a28c:::100000001f8.0000042a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:12.293 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.271671+0000", 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.933998573, 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105370546, 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2093 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.271671+0000", 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.271671+0000", 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.271672+0000", 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9709999999999998e-06 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.299969+0000", 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028296847 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.299999+0000", 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9637000000000001e-05 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300011+0000", 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2320000000000001e-05 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300086+0000", 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5153000000000004e-05 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300236+0000", 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149457 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.300379+0000", 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000143184 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330169+0000", 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029790146 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.376928+0000", 2024-09-16T01:52:12.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.046758358999999999 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.376990+0000", 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2483999999999993e-05 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.377041+0000", 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0988e-05 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.800985847, 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:12.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:12.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2504 3.13 3:c93be02c:::100000001f8.000005c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:56.120715+0000", 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.084954188000001, 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097015065999999997, 2024-09-16T01:52:12.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2504 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.120715+0000", 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.120715+0000", 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.120717+0000", 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.160791+0000", 2024-09-16T01:52:12.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040074578999999999 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.160823+0000", 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1588999999999999e-05 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.160836+0000", 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3373999999999999e-05 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.162741+0000", 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019051859999999999 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.162890+0000", 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148527 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.163051+0000", 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016161499999999999 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.173965+0000", 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010913516 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.217622+0000", 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043656860999999998 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.217687+0000", 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5116000000000005e-05 2024-09-16T01:52:12.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:56.217730+0000", 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.890005498000001, 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:12.312 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:12.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:12.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3331 3.28 3:16856b74:::100000001f8.00000900:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.370244+0000", 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.8354253700000003, 2024-09-16T01:52:12.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106986303, 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3331 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.370244+0000", 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.370244+0000", 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.370246+0000", 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.869e-06 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.421465+0000", 2024-09-16T01:52:12.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051219301000000002 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.421489+0000", 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3935e-05 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.421502+0000", 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2734999999999999e-05 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422439+0000", 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00093792500000000004 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422637+0000", 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000197107 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422789+0000", 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015251500000000001 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.460201+0000", 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037412182000000002 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.477140+0000", 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016938624999999999 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.477194+0000", 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3961000000000001e-05 2024-09-16T01:52:12.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.477230+0000", 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6148e-05 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3376 3.2d 3:b76041b2:::100000001f8.0000092d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.898410+0000", 2024-09-16T01:52:12.324 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.3072589020000001, 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.102945044, 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3376 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.898410+0000", 2024-09-16T01:52:12.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.898410+0000", 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.898412+0000", 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.278e-06 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973350+0000", 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074937918000000006 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973367+0000", 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6629999999999998e-05 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973386+0000", 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9298999999999999e-05 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973468+0000", 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1682999999999997e-05 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.973646+0000", 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000178507 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.974262+0000", 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061593699999999999 2024-09-16T01:52:12.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.992922+0000", 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018659960999999999 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.001253+0000", 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083307729999999997 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.001306+0000", 2024-09-16T01:52:12.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3062000000000001e-05 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.001355+0000", 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8996000000000003e-05 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3384 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:06.037542+0000", 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.1681271759999996, 2024-09-16T01:52:12.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105116187, 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3384 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.037542+0000", 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.037542+0000", 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.037543+0000", 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5859999999999999e-06 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108364+0000", 2024-09-16T01:52:12.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070820314999999995 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108449+0000", 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4876000000000005e-05 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108493+0000", 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4561000000000002e-05 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:12.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108666+0000", 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017283899999999999 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.108859+0000", 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019284799999999999 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.109039+0000", 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000179958 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.123383+0000", 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014343631000000001 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.142572+0000", 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019189364 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.142623+0000", 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0973000000000001e-05 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:12.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.142658+0000", 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5235999999999999e-05 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:12.336 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:12.337 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:12.338 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:12.336+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:12.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.390+0000 7f8fdbe00640 1 -- 172.21.15.146:0/3285687484 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v96) ==== 91+0+0 (secure 0 0 0) 0x7f8fdc010040 con 0x7f8fe40cb2f0 2024-09-16T01:52:12.392 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:52:12.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.394+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8fc006c3b0 msgr2=0x7f8fc006e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:12.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.394+0000 7f8feaa00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8fc006c3b0 0x7f8fc006e870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40d0a50 tx=0x7f8fcc007ce0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.394+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 msgr2=0x7f8fe415cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:12.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.394+0000 7f8feaa00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 0x7f8fe415cca0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009db0 tx=0x7f8fdc00c490 comp rx=0 tx=0).stop 2024-09-16T01:52:12.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.395+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 shutdown_connections 2024-09-16T01:52:12.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.395+0000 7f8feaa00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8fc006c3b0 0x7f8fc006e870 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.395+0000 7f8feaa00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8fe40cc4a0 0x7f8fe4154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.395+0000 7f8feaa00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe40cc0c0 0x7f8fe4157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.395+0000 7f8feaa00640 1 --2- 172.21.15.146:0/3285687484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fe40cb2f0 0x7f8fe415cca0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.395+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 >> 172.21.15.146:0/3285687484 conn(0x7f8fe406d560 msgr2=0x7f8fe40ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:12.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.396+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 shutdown_connections 2024-09-16T01:52:12.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.396+0000 7f8feaa00640 1 -- 172.21.15.146:0/3285687484 wait complete. 2024-09-16T01:52:12.438 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:12.437+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:12.453 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:12.453 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:12.453 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:12.454 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:12.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:12.539 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:12.538+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:12.594 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:12.616 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:12.617 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:12.617 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:12.617 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:12.617 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:12.626 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:12.670 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:12.669+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:12.739 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 81.881145967999998, 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.740 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:12.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:52:12.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:52:12.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 80.969567655999995, 2024-09-16T01:52:12.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:52:12.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 80.925307426000003, 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.753 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:52:12.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 80.853252084000005, 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:52:12.759 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:52:12.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:52:12.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.765 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 72.596245663000005, 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:52:12.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:52:12.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:52:12.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:52:12.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 72.561449363999998, 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.773 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:52:12.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:12.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:52:12.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 72.478000410999996, 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.781 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:52:12.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:52:12.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.786 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.278697624000003, 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.787 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:52:12.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:52:12.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.133820686, 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.793 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:52:12.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:52:12.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.093176262, 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:52:12.799 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.935233427999997, 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:52:12.805 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:52:12.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:52:12.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:52:12.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.810106167000001, 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:52:12.811 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:52:12.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:52:12.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:12.817 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 56.773306236000003, 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:52:12.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:52:12.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.610280594000002, 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:52:12.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:52:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.829 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.476542805000001, 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.830 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:52:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:52:12.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.835 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.782152691, 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:52:12.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.455178125, 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.842 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:52:12.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.314267985000001, 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:52:12.848 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:52:12.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:52:12.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 28.261465041000001, 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.854 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:52:12.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:52:12.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:52:12.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.942206501000001, 2024-09-16T01:52:12.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:12.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:52:12.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:12.866 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:12.869 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.811+0000 7f0ae3e00640 1 Processor -- start 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.811+0000 7f0ae3e00640 1 -- start start 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.812+0000 7f0ae3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0ae40d0490 0x7f0ae40c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.812+0000 7f0ae3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae40d0870 0x7f0ae40c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.812+0000 7f0ae3e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae415e220 con 0x7f0ae4154690 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.812+0000 7f0ae3e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae40c6160 con 0x7f0ae40d0490 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae3e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae415e500 con 0x7f0ae40d0870 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae3400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0ae4154690 0x7f0ae4154ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:35912/0 (socket says 172.21.15.146:35912) 2024-09-16T01:52:12.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae3400640 1 -- 172.21.15.146:0/3763758234 learned_addr learned my addr 172.21.15.146:0/3763758234 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae40d0870 0x7f0ae40c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/3763758234 >> v2:172.21.15.161:3300/0 conn(0x7f0ae40d0490 0x7f0ae40c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae1600640 1 -- 172.21.15.146:0/3763758234 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28308324 0 0) 0x7f0ae415e220 con 0x7f0ae4154690 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae2000640 1 -- 172.21.15.146:0/3763758234 >> v2:172.21.15.161:3300/0 conn(0x7f0ae40d0490 msgr2=0x7f0ae40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae2000640 1 --2- 172.21.15.146:0/3763758234 >> v2:172.21.15.161:3300/0 conn(0x7f0ae40d0490 0x7f0ae40c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae2000640 1 -- 172.21.15.146:0/3763758234 >> v1:172.21.15.161:6789/0 conn(0x7f0ae4154690 legacy=0x7f0ae4154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.813+0000 7f0ae2000640 1 -- 172.21.15.146:0/3763758234 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae415e7e0 con 0x7f0ae40d0870 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.814+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/3763758234 >> v2:172.21.15.161:3300/0 conn(0x7f0ae40d0490 0x7f0ae40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.815+0000 7f0ae2000640 1 --2- 172.21.15.146:0/3763758234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae40d0870 0x7f0ae40c6b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a95f0d34994e918 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.815+0000 7f0ae1600640 1 -- 172.21.15.146:0/3763758234 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc037070 con 0x7f0ae40d0870 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.815+0000 7f0ae1600640 1 -- 172.21.15.146:0/3763758234 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0acc032440 con 0x7f0ae40d0870 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.815+0000 7f0ae1600640 1 -- 172.21.15.146:0/3763758234 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc032850 con 0x7f0ae40d0870 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.816+0000 7f0ae3e00640 1 -- 172.21.15.146:0/3763758234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae40d0870 msgr2=0x7f0ae40c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:12.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.816+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/3763758234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae40d0870 0x7f0ae40c6b30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02a250 comp rx=0 tx=0).stop 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.817+0000 7f0ae3e00640 1 -- 172.21.15.146:0/3763758234 shutdown_connections 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.817+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/3763758234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae40d0870 0x7f0ae40c6b30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.817+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/3763758234 >> v2:172.21.15.161:3300/0 conn(0x7f0ae40d0490 0x7f0ae40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.817+0000 7f0ae3e00640 1 -- 172.21.15.146:0/3763758234 >> 172.21.15.146:0/3763758234 conn(0x7f0ae406d6f0 msgr2=0x7f0ae406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.817+0000 7f0ae3e00640 1 -- 172.21.15.146:0/3763758234 shutdown_connections 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.817+0000 7f0ae3e00640 1 -- 172.21.15.146:0/3763758234 wait complete. 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.820+0000 7f0ae3e00640 1 Processor -- start 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.820+0000 7f0ae3e00640 1 -- start start 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae40d0490 0x7f0ae40b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ae40d0870 0x7f0ae41f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae4154690 0x7f0ae41e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae415dec0 con 0x7f0ae40d0870 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae3e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae415eda0 con 0x7f0ae40d0490 2024-09-16T01:52:12.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae3e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae40c51f0 con 0x7f0ae4154690 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ae40d0870 0x7f0ae41f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ae40d0870 0x7f0ae41f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37480/0 (socket says 172.21.15.146:37480) 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae2000640 1 -- 172.21.15.146:0/1667611269 learned_addr learned my addr 172.21.15.146:0/1667611269 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.821+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae40d0490 0x7f0ae40b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0ae2000640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae4154690 msgr2=0x7f0ae41e6a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ae4154690 0x7f0ae41e6a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0ae2000640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae40d0490 msgr2=0x7f0ae40b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ae40d0490 0x7f0ae40b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0ae2000640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae40b55e0 con 0x7f0ae40d0870 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ae40d0870 0x7f0ae41f3a60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02a840 tx=0x7f0acc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.822+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc037050 con 0x7f0ae40d0870 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.823+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0acc043070 con 0x7f0ae40d0870 2024-09-16T01:52:12.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.823+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc03cb30 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.823+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae40b5300 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.823+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41f4670 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.825+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0ae415e220 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.826+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0acc03ccd0 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.827+0000 7f0adaa00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ac006c2b0 0x7f0ac006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.827+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5741+0+0 (secure 0 0 0) 0x7f0acc0ac0c0 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.827+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ac006c2b0 0x7f0ac006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.827+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v96) ==== 54+0+16 (secure 0 0 0) 0x7f0ae415e220 con 0x7f0ae40d0870 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.828+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0ae41e7650 0x7f0ae40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.828+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ae40cbe80 con 0x7f0ae41e7650 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.828+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ac006c2b0 0x7f0ac006e770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009bd0 tx=0x7f0ad400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.829+0000 7f0ae3400640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0ae41e7650 0x7f0ae40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.830+0000 7f0ae3400640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0ae41e7650 0x7f0ae40b3c10 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.831+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae40cbe80 con 0x7f0ae41e7650 2024-09-16T01:52:12.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.858+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae40cbba0 con 0x7f0ae41e7650 2024-09-16T01:52:12.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.860+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae40cbba0 con 0x7f0ae41e7650 2024-09-16T01:52:12.878 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:12.879 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:12.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.860+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f0ae41e6f90 0x7f0ae414d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.860+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0ae41f4110 con 0x7f0ae41e6f90 2024-09-16T01:52:12.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.861+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f0ae41e6f90 0x7f0ae414d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.862+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f0ae41e6f90 0x7f0ae414d770 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.863+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae41f4110 con 0x7f0ae41e6f90 2024-09-16T01:52:12.879 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:12.867+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:12.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.885+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae414dcb0 con 0x7f0ae41e6f90 2024-09-16T01:52:12.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.888+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae414dcb0 con 0x7f0ae41e6f90 2024-09-16T01:52:12.890 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:12.890 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:12.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.889+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0ae41f0790 0x7f0ae41f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.889+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0ae414dcb0 con 0x7f0ae41f0790 2024-09-16T01:52:12.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.889+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0ae41f0790 0x7f0ae41f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.890+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0ae41f0790 0x7f0ae41f0bc0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.891+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae414dcb0 con 0x7f0ae41f0790 2024-09-16T01:52:12.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.914+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae414dcb0 con 0x7f0ae41f0790 2024-09-16T01:52:12.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.916+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae414dcb0 con 0x7f0ae41f0790 2024-09-16T01:52:12.918 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:12.918 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:12.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.917+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0ae4063230 0x7f0ae4150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.917+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0ae414dcb0 con 0x7f0ae4063230 2024-09-16T01:52:12.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.917+0000 7f0ae3400640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0ae4063230 0x7f0ae4150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.918+0000 7f0ae3400640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0ae4063230 0x7f0ae4150c30 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.919+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae414dcb0 con 0x7f0ae4063230 2024-09-16T01:52:12.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.942+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae41513c0 con 0x7f0ae4063230 2024-09-16T01:52:12.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.944+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae41513c0 con 0x7f0ae4063230 2024-09-16T01:52:12.945 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:12.946 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:12.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.944+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f0ae402c640 0x7f0ae402ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.944+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0ae41f1100 con 0x7f0ae402c640 2024-09-16T01:52:12.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.945+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f0ae402c640 0x7f0ae402ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.946+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f0ae402c640 0x7f0ae402ca80 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.947+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae41f1100 con 0x7f0ae402c640 2024-09-16T01:52:12.968 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:12.967+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:12.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.968+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae41f1100 con 0x7f0ae402c640 2024-09-16T01:52:12.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.970+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae41f1100 con 0x7f0ae402c640 2024-09-16T01:52:12.971 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:12.972 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:12.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.971+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f0ae402d4d0 0x7f0ae4206b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:12.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.971+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0ae414dcb0 con 0x7f0ae402d4d0 2024-09-16T01:52:12.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.971+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f0ae402d4d0 0x7f0ae4206b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:12.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.972+0000 7f0ae2000640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f0ae402d4d0 0x7f0ae4206b80 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:12.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.973+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae414dcb0 con 0x7f0ae402d4d0 2024-09-16T01:52:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:12.997+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae414dcb0 con 0x7f0ae402d4d0 2024-09-16T01:52:13.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.000+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae414dcb0 con 0x7f0ae402d4d0 2024-09-16T01:52:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:13.002 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:13.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.001+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f0ae402d930 0x7f0ae42095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.001+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0ae420c3b0 con 0x7f0ae402d930 2024-09-16T01:52:13.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.001+0000 7f0ae3400640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f0ae402d930 0x7f0ae42095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.002+0000 7f0ae3400640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f0ae402d930 0x7f0ae42095e0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:13.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.003+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae420c3b0 con 0x7f0ae402d930 2024-09-16T01:52:13.024 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.024 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.024+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae4209b20 con 0x7f0ae402d930 2024-09-16T01:52:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.027+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae4209b20 con 0x7f0ae402d930 2024-09-16T01:52:13.028 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.028+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f0ae420a240 0x7f0ae420a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.028+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0ae420f720 con 0x7f0ae420a240 2024-09-16T01:52:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.029+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f0ae420a240 0x7f0ae420a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.030+0000 7f0ae2a00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f0ae420a240 0x7f0ae420a6a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:13.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.031+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ae420f720 con 0x7f0ae420a240 2024-09-16T01:52:13.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:13.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.053+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0ae420acf0 con 0x7f0ae420a240 2024-09-16T01:52:13.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.054+0000 7f0adaa00640 1 -- 172.21.15.146:0/1667611269 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0ae420acf0 con 0x7f0ae420a240 2024-09-16T01:52:13.056 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:13.057 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:13.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0ae41e7650 msgr2=0x7f0ae40b3c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f0ae41e7650 0x7f0ae40b3c10 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f0ae41e6f90 msgr2=0x7f0ae414d770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f0ae41e6f90 0x7f0ae414d770 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0ae41f0790 msgr2=0x7f0ae41f0bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0ae41f0790 0x7f0ae41f0bc0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0ae4063230 msgr2=0x7f0ae4150c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f0ae4063230 0x7f0ae4150c30 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f0ae402c640 msgr2=0x7f0ae402ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f0ae402c640 0x7f0ae402ca80 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f0ae402d4d0 msgr2=0x7f0ae4206b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f0ae402d4d0 0x7f0ae4206b80 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f0ae402d930 msgr2=0x7f0ae42095e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f0ae402d930 0x7f0ae42095e0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f0ae420a240 msgr2=0x7f0ae420a6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f0ae420a240 0x7f0ae420a6a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ac006c2b0 msgr2=0x7f0ac006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ac006c2b0 0x7f0ac006e770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009bd0 tx=0x7f0ad400b040 comp rx=0 tx=0).stop 2024-09-16T01:52:13.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ae40d0870 msgr2=0x7f0ae41f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 --2- 172.21.15.146:0/1667611269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ae40d0870 0x7f0ae41f3a60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02a840 tx=0x7f0acc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3400640 1 -- 172.21.15.146:0/1667611269 reap_dead start 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3400640 1 -- 172.21.15.146:0/1667611269 reap_dead start 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3400640 1 -- 172.21.15.146:0/1667611269 reap_dead start 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3400640 1 -- 172.21.15.146:0/1667611269 reap_dead start 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3400640 1 -- 172.21.15.146:0/1667611269 reap_dead start 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3400640 1 -- 172.21.15.146:0/1667611269 reap_dead start 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 shutdown_connections 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.056+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 >> 172.21.15.146:0/1667611269 conn(0x7f0ae406d6f0 msgr2=0x7f0ae4152560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.057+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 shutdown_connections 2024-09-16T01:52:13.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.057+0000 7f0ae3e00640 1 -- 172.21.15.146:0/1667611269 wait complete. 2024-09-16T01:52:13.078 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:13.077+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:13.140 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.140 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.149 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:13.191 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:13.190+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:13.271 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:13.270+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:13.294 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.294 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.303 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:13.371 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:13.370+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:13.414 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:13.414 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:13.414 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T01:52:13.414 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T01:52:13.415 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.415 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.424 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:13.473 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:13.472+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:13.574 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:13.573+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:13.585 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.585 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.596 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:13.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.652+0000 7fceca800640 1 Processor -- start 2024-09-16T01:52:13.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.652+0000 7fceca800640 1 -- start start 2024-09-16T01:52:13.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fceca800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcec415aab0 0x7fcec40cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fceca800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec415b1d0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fceca800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec415b4b0 con 0x7fcec40cb2f0 2024-09-16T01:52:13.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fceca800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcec40cbd00 con 0x7fcec415aab0 2024-09-16T01:52:13.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fcec2a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcec40cb2f0 0x7fcec4163420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34896/0 (socket says 172.21.15.146:34896) 2024-09-16T01:52:13.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fcec2a00640 1 -- 172.21.15.146:0/3453490890 learned_addr learned my addr 172.21.15.146:0/3453490890 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:13.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.654+0000 7fcec3e00640 1 --2- 172.21.15.146:0/3453490890 >> v2:172.21.15.146:3300/0 conn(0x7fcec415aab0 0x7fcec40cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.655+0000 7fcec3e00640 1 -- 172.21.15.146:0/3453490890 >> v1:172.21.15.161:6789/0 conn(0x7fcec40cb2f0 legacy=0x7fcec4163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.655+0000 7fcec3e00640 1 -- 172.21.15.146:0/3453490890 >> v1:172.21.15.146:6789/0 conn(0x7fcec40cc0c0 legacy=0x7fcec415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.655+0000 7fcec3e00640 1 -- 172.21.15.146:0/3453490890 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec415b790 con 0x7fcec415aab0 2024-09-16T01:52:13.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.656+0000 7fcec3e00640 1 --2- 172.21.15.146:0/3453490890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec415aab0 0x7fcec40cc4a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009d90 tx=0x7fcebc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd21c7a2f5166b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:13.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.658+0000 7fcec2000640 1 -- 172.21.15.146:0/3453490890 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc007740 con 0x7fcec415aab0 2024-09-16T01:52:13.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.658+0000 7fcec2000640 1 -- 172.21.15.146:0/3453490890 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcebc0078e0 con 0x7fcec415aab0 2024-09-16T01:52:13.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.658+0000 7fcec2000640 1 -- 172.21.15.146:0/3453490890 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc007cb0 con 0x7fcec415aab0 2024-09-16T01:52:13.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.659+0000 7fceca800640 1 -- 172.21.15.146:0/3453490890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec415aab0 msgr2=0x7fcec40cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.659+0000 7fceca800640 1 --2- 172.21.15.146:0/3453490890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec415aab0 0x7fcec40cc4a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009d90 tx=0x7fcebc02a610 comp rx=0 tx=0).stop 2024-09-16T01:52:13.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.659+0000 7fceca800640 1 -- 172.21.15.146:0/3453490890 shutdown_connections 2024-09-16T01:52:13.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.659+0000 7fceca800640 1 --2- 172.21.15.146:0/3453490890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec415aab0 0x7fcec40cc4a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.659+0000 7fceca800640 1 -- 172.21.15.146:0/3453490890 >> 172.21.15.146:0/3453490890 conn(0x7fcec406d560 msgr2=0x7fcec406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:13.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.660+0000 7fceca800640 1 -- 172.21.15.146:0/3453490890 shutdown_connections 2024-09-16T01:52:13.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.660+0000 7fceca800640 1 -- 172.21.15.146:0/3453490890 wait complete. 2024-09-16T01:52:13.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.662+0000 7fceca800640 1 Processor -- start 2024-09-16T01:52:13.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.662+0000 7fceca800640 1 -- start start 2024-09-16T01:52:13.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.663+0000 7fceca800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcec40cb2f0 0x7fcec4151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.663+0000 7fceca800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 0x7fcec414e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.663+0000 7fceca800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec415aab0 0x7fcec41f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.663+0000 7fceca800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcec415e540 con 0x7fcec40cc0c0 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.663+0000 7fceca800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcec415ba70 con 0x7fcec415aab0 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.663+0000 7fceca800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcec415aec0 con 0x7fcec40cb2f0 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 0x7fcec414e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 0x7fcec414e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41686/0 (socket says 172.21.15.146:41686) 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 -- 172.21.15.146:0/2557571011 learned_addr learned my addr 172.21.15.146:0/2557571011 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec3400640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcec40cb2f0 0x7fcec4151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec3e00640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec415aab0 0x7fcec41f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 -- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcec40cb2f0 msgr2=0x7fcec4151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcec40cb2f0 0x7fcec4151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 -- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec415aab0 msgr2=0x7fcec41f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec415aab0 0x7fcec41f3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:13.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.664+0000 7fcec2a00640 1 -- 172.21.15.146:0/2557571011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec4151fb0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.665+0000 7fcec2a00640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 0x7fcec414e760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fceb000c750 tx=0x7fceb000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:13.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.665+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0018070 con 0x7fcec40cc0c0 2024-09-16T01:52:13.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.665+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb00072f0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.665+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb00047e0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.666+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec4152260 con 0x7fcec40cc0c0 2024-09-16T01:52:13.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.667+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41f0740 con 0x7fcec40cc0c0 2024-09-16T01:52:13.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.669+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec40c5560 con 0x7fcec40cc0c0 2024-09-16T01:52:13.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.671+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fceb0004980 con 0x7fcec40cc0c0 2024-09-16T01:52:13.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.673+0000 7fcebb400640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcea406c2b0 0x7fcea406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:13.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.673+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5741+0+0 (secure 0 0 0) 0x7fceb008a560 con 0x7fcec40cc0c0 2024-09-16T01:52:13.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.673+0000 7fcec3400640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcea406c2b0 0x7fcea406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:13.675 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:13.673+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:13.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.674+0000 7fcec3400640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcea406c2b0 0x7fcea406e770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fceb4001780 tx=0x7fceb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:13.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.677+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fceb00586f0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.710 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.711 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.721 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:13.775 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:13.774+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:13.839 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:13.839 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:13.849 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:13.893 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:13.892+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:13.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.956+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fcec40cbea0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.958+0000 7fcebb400640 1 -- 172.21.15.146:0/2557571011 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v96) ==== 81+0+34533 (secure 0 0 0) 0x7fceb005c9f0 con 0x7fcec40cc0c0 2024-09-16T01:52:13.960 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:13.960 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:13.960 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 96, 2024-09-16T01:52:13.960 INFO:teuthology.orchestra.run.smithi146.stdout: "fsid": "4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d", 2024-09-16T01:52:13.960 INFO:teuthology.orchestra.run.smithi146.stdout: "created": "2024-09-16T01:48:34.399638+0000", 2024-09-16T01:52:13.960 INFO:teuthology.orchestra.run.smithi146.stdout: "modified": "2024-09-16T01:52:12.389068+0000", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "last_up_change": "2024-09-16T01:48:45.748664+0000", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "last_in_change": "2024-09-16T01:52:08.361334+0000", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_num": 5804032, 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_set": [ 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "nodeep-scrub", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "pglog_hardlimit", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "purged_snapdirs", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_deletes", 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "sortbitwise" 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:13.961 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_version": 6, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_snapshot": "", 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_max": 3, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd": 8, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "require_min_compat_client": "luminous", 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "min_compat_client": "jewel", 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "require_osd_release": "squid", 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_crimson": false, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "pools": [ 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 1, 2024-09-16T01:52:13.962 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": ".mgr", 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:46.570355+0000", 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:52:13.963 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 1, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 1, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 1, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 1, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 1, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:52:13.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "18", 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:52:13.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:52:13.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:52:13.967 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_max": 32, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 1 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr": {} 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 6.9000000953674316, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 6.9000000953674316, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.28999999165534973, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:52:13.968 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 2, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_metadata", 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:56.008811+0000", 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:52:13.969 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 32, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 32, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 32, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 32, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 32, 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:52:13.970 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "21", 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:52:13.971 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:52:13.972 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:52:13.973 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_bias": 4, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 16, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_priority": 5 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:52:13.974 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": "cephfs" 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 3.059999942779541, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 3.059999942779541, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.86000001430511475, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 2.630000114440918, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 2.630000114440918, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:52:13.975 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "pool": 3, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": "cephfs_data", 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "create_time": "2024-09-16T01:48:57.207121+0000", 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "flags_names": "hashpspool", 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "type": 1, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 2, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": 1, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_rule": 0, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_count": 0, 2024-09-16T01:52:13.976 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_target": 0, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "is_stretch_pool": false, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "object_hash": 2, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscale_mode": "off", 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": 64, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num": 64, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_placement_num_target": 64, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_target": 64, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_pending": 64, 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "last_pg_merge_meta": { 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "source_pgid": "0.0", 2024-09-16T01:52:13.977 INFO:teuthology.orchestra.run.smithi146.stdout: "ready_epoch": 0, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_started": 0, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "last_epoch_clean": 0, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "source_version": "0'0", 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "target_version": "0'0" 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "last_change": "22", 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend": "0", 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "auid": 0, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_mode": "selfmanaged", 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_seq": 0, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_epoch": 0, 2024-09-16T01:52:13.978 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_snaps": [], 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps": "[]", 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_bytes": 0, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "quota_max_objects": 0, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tiers": [], 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tier_of": -1, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "read_tier": -1, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "write_tier": -1, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_mode": "none", 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_bytes": 0, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "target_max_objects": 0, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T01:52:13.979 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_flush_age": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "cache_min_evict_age": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profile": "", 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_params": { 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "none" 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_period": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_count": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "use_gmt_hitset": true, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "min_read_recency_for_promote": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "min_write_recency_for_promote": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "hit_set_search_last_n": 0, 2024-09-16T01:52:13.980 INFO:teuthology.orchestra.run.smithi146.stdout: "grade_table": [], 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "stripe_width": 0, 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "expected_num_objects": 0, 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "fast_read": false, 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "options": { 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num_min": 64, 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "target_size_ratio": 0.89999997615814209 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "application_metadata": { 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "cephfs": { 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "data": "cephfs" 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "read_balance": { 2024-09-16T01:52:13.981 INFO:teuthology.orchestra.run.smithi146.stdout: "score_type": "Fair distribution", 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "score_acting": 1.7400000095367432, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "score_stable": 1.7400000095367432, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "optimal_score": 0.86000001430511475, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_acting": 1.5, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "raw_score_stable": 1.5, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity_weighted": 0.875, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity": 1, 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osds": [ 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "e7758ddf-75ff-43c7-abae-3647b170f7c0", 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 91, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6824", 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6825", 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:13.984 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6826", 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6827", 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:13.985 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6830", 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6831", 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:13.986 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6828", 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6829", 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 61440064 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6825/61440064", 2024-09-16T01:52:13.987 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6827/61440064", 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6831/61440064", 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6829/61440064", 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "a065e77e-26f0-4245-abe8-e36c2fb4436b", 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:52:13.988 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0.93414306640625, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 94, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6808", 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6809", 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6810", 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6811", 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6814", 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6815", 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6812", 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6813", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2375201615 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6809/2375201615", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6811/2375201615", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6815/2375201615", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6813/2375201615", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:13.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "fefb6d8f-6799-4344-a977-85ab983ec0cb", 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 94, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:13.994 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6800", 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6801", 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6802", 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6803", 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6806", 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6807", 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6804", 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6805", 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1818492127 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.998 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6801/1818492127", 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6803/1818492127", 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6807/1818492127", 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6805/1818492127", 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "9d1a2f96-efcb-4a73-b5db-ec5e2a63122a", 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:52:13.999 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 15, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 94, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6816", 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6817", 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6818", 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6819", 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6822", 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6823", 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6820", 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.003 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6821", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1633804394 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.146:6817/1633804394", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.146:6819/1633804394", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.146:6823/1633804394", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.146:6821/1633804394", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:14.004 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c", 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 0, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 13, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 67, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:14.005 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6800", 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6801", 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.006 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6802", 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6803", 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.007 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6806", 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6807", 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.008 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6804", 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6805", 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 154390265 2024-09-16T01:52:14.009 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6801/154390265", 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6803/154390265", 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6807/154390265", 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6805/154390265", 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.010 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "11562dd5-d537-4d88-a282-e8687caebad6", 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 94, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6824", 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6825", 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6826", 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6827", 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6830", 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6831", 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6828", 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6829", 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 356357279 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6825/356357279", 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6827/356357279", 2024-09-16T01:52:14.015 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6831/356357279", 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6829/356357279", 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "b31adf01-6768-49cf-a232-127b9d0d2dc8", 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 1, 2024-09-16T01:52:14.016 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 1, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 94, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6808", 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6809", 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6810", 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6811", 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6814", 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6815", 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6812", 2024-09-16T01:52:14.020 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6813", 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3661001615 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6809/3661001615", 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6811/3661001615", 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6815/3661001615", 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6813/3661001615", 2024-09-16T01:52:14.021 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "uuid": "67c95c28-b600-430c-9474-a902cf46a73b", 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "up": 1, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "in": 1, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "weight": 0.433685302734375, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_affinity": 0, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_begin": 0, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "last_clean_end": 0, 2024-09-16T01:52:14.022 INFO:teuthology.orchestra.run.smithi146.stdout: "up_from": 14, 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "up_thru": 35, 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "down_at": 0, 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "lost_at": 0, 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addrs": { 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6816", 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.023 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6817", 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addrs": { 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6818", 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.024 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6819", 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addrs": { 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6822", 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6823", 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addrs": { 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6820", 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.161:6821", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1279166573 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "public_addr": "172.21.15.161:6817/1279166573", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_addr": "172.21.15.161:6819/1279166573", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_back_addr": "172.21.15.161:6823/1279166573", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "heartbeat_front_addr": "172.21.15.161:6821/1279166573", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "state": [ 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "exists", 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: "up" 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:14.027 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_xinfo": [ 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 0, 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 65536, 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.135308+0000", 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 1, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.290750+0000", 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 2, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.029 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.712744+0000", 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 3, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.988678+0000", 2024-09-16T01:52:14.030 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 4, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 56585, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:42.198114+0000", 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 5, 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.031 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.085332+0000", 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 6, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.032 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.394321+0000", 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "osd": 7, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "down_stamp": "0.000000", 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_probability": 0, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "laggy_interval": 0, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "features": 4540701547738038271, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "old_weight": 0, 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "last_purged_snaps_scrub": "2024-09-16T01:48:43.236983+0000", 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: "dead_epoch": 0 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.033 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap": [], 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_items": [], 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_upmap_primaries": [], 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_temp": [], 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "primary_temp": [], 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklist": {}, 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "range_blocklist": {}, 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "erasure_code_profiles": { 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "default": { 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-failure-domain": "osd", 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "k": "2", 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "m": "1", 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "plugin": "jerasure", 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: "technique": "reed_sol_van" 2024-09-16T01:52:14.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "removed_snaps_queue": [], 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "new_removed_snaps": [], 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "new_purged_snaps": [], 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_node_flags": {}, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "device_class_flags": {}, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode": { 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_enabled": false, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_bucket_count": 0, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "degraded_stretch_mode": 0, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "recovering_stretch_mode": 0, 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: "stretch_mode_bucket": 0 2024-09-16T01:52:14.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:14.036 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:14.037 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:14.038 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:14.038 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:14.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.961+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcea406c2b0 msgr2=0x7fcea406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.961+0000 7fceca800640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcea406c2b0 0x7fcea406e770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fceb4001780 tx=0x7fceb4009040 comp rx=0 tx=0).stop 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 msgr2=0x7fcec414e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 0x7fcec414e760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fceb000c750 tx=0x7fceb000cc10 comp rx=0 tx=0).stop 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 shutdown_connections 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcea406c2b0 0x7fcea406e770 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec415aab0 0x7fcec41f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec40cc0c0 0x7fcec414e760 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 --2- 172.21.15.146:0/2557571011 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcec40cb2f0 0x7fcec4151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 >> 172.21.15.146:0/2557571011 conn(0x7fcec406d560 msgr2=0x7fcec40cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 shutdown_connections 2024-09-16T01:52:14.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:13.962+0000 7fceca800640 1 -- 172.21.15.146:0/2557571011 wait complete. 2024-09-16T01:52:14.040 INFO:tasks.thrashosds.thrasher:j is {'epoch': 96, 'fsid': '4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d', 'created': '2024-09-16T01:48:34.399638+0000', 'modified': '2024-09-16T01:52:12.389068+0000', 'last_up_change': '2024-09-16T01:48:45.748664+0000', 'last_in_change': '2024-09-16T01:52:08.361334+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:48:46.570355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:48:56.008811+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:48:57.207121+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'e7758ddf-75ff-43c7-abae-3647b170f7c0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6824', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6825', 'nonce': 61440064}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6826', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6827', 'nonce': 61440064}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6830', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6831', 'nonce': 61440064}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6828', 'nonce': 61440064}, {'type': 'v1', 'addr': '172.21.15.146:6829', 'nonce': 61440064}]}, 'public_addr': '172.21.15.146:6825/61440064', 'cluster_addr': '172.21.15.146:6827/61440064', 'heartbeat_back_addr': '172.21.15.146:6831/61440064', 'heartbeat_front_addr': '172.21.15.146:6829/61440064', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a065e77e-26f0-4245-abe8-e36c2fb4436b', 'up': 1, 'in': 1, 'weight': 0.93414306640625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6808', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6809', 'nonce': 2375201615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6810', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6811', 'nonce': 2375201615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6814', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6815', 'nonce': 2375201615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6812', 'nonce': 2375201615}, {'type': 'v1', 'addr': '172.21.15.146:6813', 'nonce': 2375201615}]}, 'public_addr': '172.21.15.146:6809/2375201615', 'cluster_addr': '172.21.15.146:6811/2375201615', 'heartbeat_back_addr': '172.21.15.146:6815/2375201615', 'heartbeat_front_addr': '172.21.15.146:6813/2375201615', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fefb6d8f-6799-4344-a977-85ab983ec0cb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6800', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6801', 'nonce': 1818492127}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6802', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6803', 'nonce': 1818492127}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6806', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6807', 'nonce': 1818492127}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6804', 'nonce': 1818492127}, {'type': 'v1', 'addr': '172.21.15.146:6805', 'nonce': 1818492127}]}, 'public_addr': '172.21.15.146:6801/1818492127', 'cluster_addr': '172.21.15.146:6803/1818492127', 'heartbeat_back_addr': '172.21.15.146:6807/1818492127', 'heartbeat_front_addr': '172.21.15.146:6805/1818492127', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9d1a2f96-efcb-4a73-b5db-ec5e2a63122a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6816', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6817', 'nonce': 1633804394}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6818', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6819', 'nonce': 1633804394}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6822', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6823', 'nonce': 1633804394}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6820', 'nonce': 1633804394}, {'type': 'v1', 'addr': '172.21.15.146:6821', 'nonce': 1633804394}]}, 'public_addr': '172.21.15.146:6817/1633804394', 'cluster_addr': '172.21.15.146:6819/1633804394', 'heartbeat_back_addr': '172.21.15.146:6823/1633804394', 'heartbeat_front_addr': '172.21.15.146:6821/1633804394', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6800', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6801', 'nonce': 154390265}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6802', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6803', 'nonce': 154390265}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6806', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6807', 'nonce': 154390265}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6804', 'nonce': 154390265}, {'type': 'v1', 'addr': '172.21.15.161:6805', 'nonce': 154390265}]}, 'public_addr': '172.21.15.161:6801/154390265', 'cluster_addr': '172.21.15.161:6803/154390265', 'heartbeat_back_addr': '172.21.15.161:6807/154390265', 'heartbeat_front_addr': '172.21.15.161:6805/154390265', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '11562dd5-d537-4d88-a282-e8687caebad6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6824', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6825', 'nonce': 356357279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6826', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6827', 'nonce': 356357279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6830', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6831', 'nonce': 356357279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6828', 'nonce': 356357279}, {'type': 'v1', 'addr': '172.21.15.161:6829', 'nonce': 356357279}]}, 'public_addr': '172.21.15.161:6825/356357279', 'cluster_addr': '172.21.15.161:6827/356357279', 'heartbeat_back_addr': '172.21.15.161:6831/356357279', 'heartbeat_front_addr': '172.21.15.161:6829/356357279', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b31adf01-6768-49cf-a232-127b9d0d2dc8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6808', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6809', 'nonce': 3661001615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6810', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6811', 'nonce': 3661001615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6814', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6815', 'nonce': 3661001615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6812', 'nonce': 3661001615}, {'type': 'v1', 'addr': '172.21.15.161:6813', 'nonce': 3661001615}]}, 'public_addr': '172.21.15.161:6809/3661001615', 'cluster_addr': '172.21.15.161:6811/3661001615', 'heartbeat_back_addr': '172.21.15.161:6815/3661001615', 'heartbeat_front_addr': '172.21.15.161:6813/3661001615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '67c95c28-b600-430c-9474-a902cf46a73b', 'up': 1, 'in': 1, 'weight': 0.433685302734375, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6816', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6817', 'nonce': 1279166573}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6818', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6819', 'nonce': 1279166573}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6822', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6823', 'nonce': 1279166573}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6820', 'nonce': 1279166573}, {'type': 'v1', 'addr': '172.21.15.161:6821', 'nonce': 1279166573}]}, 'public_addr': '172.21.15.161:6817/1279166573', 'cluster_addr': '172.21.15.161:6819/1279166573', 'heartbeat_back_addr': '172.21.15.161:6823/1279166573', 'heartbeat_front_addr': '172.21.15.161:6821/1279166573', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.135308+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.290750+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.712744+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.988678+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 56585, 'last_purged_snaps_scrub': '2024-09-16T01:48:42.198114+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.085332+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.394321+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T01:48:43.236983+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T01:52:14.040 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T01:52:14.041 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:14.036+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:14.138 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:14.137+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:14.189 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:14.189 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:14.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:14.246 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:14.245+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:14.340 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:14.339+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:14.397 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:14.397 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:14.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:14.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:52:14.441 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:14.440+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:14.542 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:14.541+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:14.645 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:14.642+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:14.743 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:14.742+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:14.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.775+0000 7fe337800640 1 Processor -- start 2024-09-16T01:52:14.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.775+0000 7fe337800640 1 -- start start 2024-09-16T01:52:14.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.776+0000 7fe337800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300d0560 0x7fe33015b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.776+0000 7fe337800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe3300c7cb0 0x7fe33015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.776+0000 7fe337800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3300d0940 0x7fe33015c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.776+0000 7fe337800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe33015bb40 con 0x7fe3300c7cb0 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.776+0000 7fe337800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe3300c86f0 con 0x7fe3300d0940 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.776+0000 7fe337800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3300c89d0 con 0x7fe3300d0560 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe335800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3300d0940 0x7fe33015c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe334e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300d0560 0x7fe33015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe335800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3300d0940 0x7fe33015c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51796/0 (socket says 172.21.15.146:51796) 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe335800640 1 -- 172.21.15.146:0/71693585 learned_addr learned my addr 172.21.15.146:0/71693585 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe334e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300d0560 0x7fe33015b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36284/0 (socket says 172.21.15.146:36284) 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe32fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe3300c7cb0 0x7fe33015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.777+0000 7fe335800640 1 -- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300d0560 msgr2=0x7fe33015b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:14.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.778+0000 7fe335800640 1 --2- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300d0560 0x7fe33015b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.778+0000 7fe335800640 1 -- 172.21.15.146:0/71693585 >> v2:172.21.15.146:3300/0 conn(0x7fe3300c7cb0 msgr2=0x7fe33015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:14.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.778+0000 7fe335800640 1 --2- 172.21.15.146:0/71693585 >> v2:172.21.15.146:3300/0 conn(0x7fe3300c7cb0 0x7fe33015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.778+0000 7fe335800640 1 -- 172.21.15.146:0/71693585 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3300c8cb0 con 0x7fe3300d0940 2024-09-16T01:52:14.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.778+0000 7fe32fe00640 1 --2- 172.21.15.146:0/71693585 >> v2:172.21.15.146:3300/0 conn(0x7fe3300c7cb0 0x7fe33015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:14.780 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:14.781 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:14.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.779+0000 7fe335800640 1 --2- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe33015c810 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe33006d110 tx=0x7fe31c02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1e2dde1e3497dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:14.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.783+0000 7fe32f400640 1 -- 172.21.15.146:0/71693585 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c037070 con 0x7fe3300d0940 2024-09-16T01:52:14.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.783+0000 7fe32f400640 1 -- 172.21.15.146:0/71693585 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c032440 con 0x7fe3300d0940 2024-09-16T01:52:14.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.783+0000 7fe32f400640 1 -- 172.21.15.146:0/71693585 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c032810 con 0x7fe3300d0940 2024-09-16T01:52:14.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 -- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 msgr2=0x7fe33015c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:14.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 --2- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe33015c810 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe33006d110 tx=0x7fe31c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 -- 172.21.15.146:0/71693585 shutdown_connections 2024-09-16T01:52:14.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 --2- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe33015c810 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 --2- 172.21.15.146:0/71693585 >> v2:172.21.15.146:3300/0 conn(0x7fe3300c7cb0 0x7fe33015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 --2- 172.21.15.146:0/71693585 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300d0560 0x7fe33015b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 -- 172.21.15.146:0/71693585 >> 172.21.15.146:0/71693585 conn(0x7fe33006cb00 msgr2=0x7fe33006e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:14.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.784+0000 7fe337800640 1 -- 172.21.15.146:0/71693585 shutdown_connections 2024-09-16T01:52:14.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.785+0000 7fe337800640 1 -- 172.21.15.146:0/71693585 wait complete. 2024-09-16T01:52:14.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.811+0000 7fe337800640 1 Processor -- start 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 -- start start 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300c7cb0 0x7fe3300c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 0x7fe330071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe330071560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe3300c9550 con 0x7fe3300d0560 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3300c9270 con 0x7fe3300d0940 2024-09-16T01:52:14.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.812+0000 7fe337800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3300c8120 con 0x7fe3300c7cb0 2024-09-16T01:52:14.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.813+0000 7fe334e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300c7cb0 0x7fe3300c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.813+0000 7fe32fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 0x7fe330071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.813+0000 7fe32fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 0x7fe330071020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41710/0 (socket says 172.21.15.146:41710) 2024-09-16T01:52:14.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.813+0000 7fe32fe00640 1 -- 172.21.15.146:0/3420499297 learned_addr learned my addr 172.21.15.146:0/3420499297 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:14.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe335800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe330071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe334e00640 1 -- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 msgr2=0x7fe330071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:14.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe334e00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe330071560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe334e00640 1 -- 172.21.15.146:0/3420499297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 msgr2=0x7fe330071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:14.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe334e00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 0x7fe330071020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:14.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe334e00640 1 -- 172.21.15.146:0/3420499297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3301559c0 con 0x7fe3300c7cb0 2024-09-16T01:52:14.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.814+0000 7fe32fe00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 0x7fe330071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:14.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.816+0000 7fe334e00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300c7cb0 0x7fe3300c6760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe32800efc0 tx=0x7fe32800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:14.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.816+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328018020 con 0x7fe3300c7cb0 2024-09-16T01:52:14.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.817+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe328014480 con 0x7fe3300c7cb0 2024-09-16T01:52:14.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.817+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328014850 con 0x7fe3300c7cb0 2024-09-16T01:52:14.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.817+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe330155c30 con 0x7fe3300c7cb0 2024-09-16T01:52:14.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.818+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3301561f0 con 0x7fe3300c7cb0 2024-09-16T01:52:14.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.823+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe33015b600 con 0x7fe3300c7cb0 2024-09-16T01:52:14.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.826+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe32801b020 con 0x7fe3300c7cb0 2024-09-16T01:52:14.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.827+0000 7fe32cc00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe31006c2b0 0x7fe31006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:14.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.827+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(96..96 src has 1..96) ==== 5741+0+0 (secure 0 0 0) 0x7fe32808b540 con 0x7fe3300c7cb0 2024-09-16T01:52:14.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.829+0000 7fe32fe00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe31006c2b0 0x7fe31006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:14.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.830+0000 7fe32fe00640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe31006c2b0 0x7fe31006e770 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe330070f80 tx=0x7fe320008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:14.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:14.835+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe328010040 con 0x7fe3300c7cb0 2024-09-16T01:52:14.844 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:14.843+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:14.944 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:14.943+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:15.045 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:15.045+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:15.086 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:15.146 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:15.145+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:15.155 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:15.155 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:15.184 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:15.247 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:15.247+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:15.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.247+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fe3300cb360 con 0x7fe3300c7cb0 2024-09-16T01:52:15.348 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:15.348+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:15.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.399+0000 7feecaa00640 1 Processor -- start 2024-09-16T01:52:15.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.399+0000 7feecaa00640 1 -- start start 2024-09-16T01:52:15.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.402+0000 7feecaa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc007d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.402+0000 7feecaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feebc04c7f0 0x7feebc04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.402+0000 7feecaa00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feebc04d500 con 0x7feebc0e0510 2024-09-16T01:52:15.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.402+0000 7feecaa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7feebc009220 con 0x7feebc04c7f0 2024-09-16T01:52:15.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.402+0000 7feecaa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7feebc048280 con 0x7feebc046f10 2024-09-16T01:52:15.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.403+0000 7feec8c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.403+0000 7feec8c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc007d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36296/0 (socket says 172.21.15.146:36296) 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.403+0000 7feec8c00640 1 -- 172.21.15.146:0/2450343028 learned_addr learned my addr 172.21.15.146:0/2450343028 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.403+0000 7feeca000640 1 --2- 172.21.15.146:0/2450343028 >> v2:172.21.15.146:3300/0 conn(0x7feebc04c7f0 0x7feebc04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.404+0000 7feec8c00640 1 -- 172.21.15.146:0/2450343028 >> v2:172.21.15.146:3300/0 conn(0x7feebc04c7f0 msgr2=0x7feebc04cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.404+0000 7feec8c00640 1 --2- 172.21.15.146:0/2450343028 >> v2:172.21.15.146:3300/0 conn(0x7feebc04c7f0 0x7feebc04cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.404+0000 7feec8c00640 1 -- 172.21.15.146:0/2450343028 >> v1:172.21.15.161:6789/0 conn(0x7feebc0e0510 legacy=0x7feebc04e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.404+0000 7feec8c00640 1 -- 172.21.15.146:0/2450343028 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc0e08f0 con 0x7feebc046f10 2024-09-16T01:52:15.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.410+0000 7feec8c00640 1 --2- 172.21.15.146:0/2450343028 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc007d70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009870 tx=0x7feeb002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d272de1f9e7edc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.414+0000 7feec3e00640 1 -- 172.21.15.146:0/2450343028 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0037070 con 0x7feebc046f10 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.414+0000 7feec3e00640 1 -- 172.21.15.146:0/2450343028 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb002acf0 con 0x7feebc046f10 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.414+0000 7feec3e00640 1 -- 172.21.15.146:0/2450343028 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0032660 con 0x7feebc046f10 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.415+0000 7feecaa00640 1 -- 172.21.15.146:0/2450343028 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 msgr2=0x7feebc007d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.415+0000 7feecaa00640 1 --2- 172.21.15.146:0/2450343028 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc007d70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009870 tx=0x7feeb002a060 comp rx=0 tx=0).stop 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.415+0000 7feecaa00640 1 -- 172.21.15.146:0/2450343028 shutdown_connections 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.415+0000 7feecaa00640 1 --2- 172.21.15.146:0/2450343028 >> v2:172.21.15.146:3300/0 conn(0x7feebc04c7f0 0x7feebc04cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.415+0000 7feecaa00640 1 --2- 172.21.15.146:0/2450343028 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc007d70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.415+0000 7feecaa00640 1 -- 172.21.15.146:0/2450343028 >> 172.21.15.146:0/2450343028 conn(0x7feebc00c390 msgr2=0x7feebc00c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:15.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.418+0000 7feecaa00640 1 -- 172.21.15.146:0/2450343028 shutdown_connections 2024-09-16T01:52:15.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.418+0000 7feecaa00640 1 -- 172.21.15.146:0/2450343028 wait complete. 2024-09-16T01:52:15.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.422+0000 7feecaa00640 1 Processor -- start 2024-09-16T01:52:15.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.422+0000 7feecaa00640 1 -- start start 2024-09-16T01:52:15.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.424+0000 7feecaa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc0097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feecaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feebc04c7f0 0x7feebc178f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feecaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feebc0e0510 0x7feebc0d3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feecaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feebc047c30 con 0x7feebc0e0510 2024-09-16T01:52:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feecaa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feebc04da80 con 0x7feebc04c7f0 2024-09-16T01:52:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feecaa00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7feebc047f10 con 0x7feebc046f10 2024-09-16T01:52:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feeca000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feebc0e0510 0x7feebc0d3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.425+0000 7feec9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc0097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.426+0000 7feec9600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc0097e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36310/0 (socket says 172.21.15.146:36310) 2024-09-16T01:52:15.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.426+0000 7feec9600640 1 -- 172.21.15.146:0/3595306694 learned_addr learned my addr 172.21.15.146:0/3595306694 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:15.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.426+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feebc04c7f0 0x7feebc178f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.427+0000 7feec8c00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 msgr2=0x7feebc0097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.427+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7feebc046f10 0x7feebc0097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.427+0000 7feec8c00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feebc0e0510 msgr2=0x7feebc0d3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.427+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feebc0e0510 0x7feebc0d3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.427+0000 7feec8c00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc17d320 con 0x7feebc04c7f0 2024-09-16T01:52:15.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.427+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feebc0e0510 0x7feebc0d3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:15.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.428+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feebc04c7f0 0x7feebc178f50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7feebc04d6d0 tx=0x7feeb00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.428+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0037040 con 0x7feebc04c7f0 2024-09-16T01:52:15.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.428+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb0004250 con 0x7feebc04c7f0 2024-09-16T01:52:15.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.428+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0003610 con 0x7feebc04c7f0 2024-09-16T01:52:15.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.431+0000 7fe32cc00640 1 -- 172.21.15.146:0/3420499297 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v97) ==== 101+0+0 (secure 0 0 0) 0x7fe328059630 con 0x7fe3300c7cb0 2024-09-16T01:52:15.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.431+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc17c560 con 0x7feebc04c7f0 2024-09-16T01:52:15.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.433+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc17cb20 con 0x7feebc04c7f0 2024-09-16T01:52:15.437 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:52:15.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.437+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7feeb002abb0 con 0x7feebc04c7f0 2024-09-16T01:52:15.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.438+0000 7feec1600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feea406c2b0 0x7feea406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5741+0+0 (secure 0 0 0) 0x7feeb00b0060 con 0x7feebc04c7f0 2024-09-16T01:52:15.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe31006c2b0 msgr2=0x7fe31006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe31006c2b0 0x7fe31006e770 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe330070f80 tx=0x7fe320008040 comp rx=0 tx=0).stop 2024-09-16T01:52:15.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300c7cb0 msgr2=0x7fe3300c6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300c7cb0 0x7fe3300c6760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe32800efc0 tx=0x7fe32800c490 comp rx=0 tx=0).stop 2024-09-16T01:52:15.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feea406c2b0 0x7feea406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 shutdown_connections 2024-09-16T01:52:15.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe31006c2b0 0x7fe31006e770 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3300d0940 0x7fe330071560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3300d0560 0x7fe330071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 --2- 172.21.15.146:0/3420499297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe3300c7cb0 0x7fe3300c6760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.439+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 >> 172.21.15.146:0/3420499297 conn(0x7fe33006cb00 msgr2=0x7fe3300ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.440+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 shutdown_connections 2024-09-16T01:52:15.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.440+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7feebc047330 con 0x7feebc04c7f0 2024-09-16T01:52:15.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.440+0000 7fe337800640 1 -- 172.21.15.146:0/3420499297 wait complete. 2024-09-16T01:52:15.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.440+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feea406c2b0 0x7feea406e770 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feeb8001980 tx=0x7feeb8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.441+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v97) ==== 54+0+16 (secure 0 0 0) 0x7feeb0079220 con 0x7feebc04c7f0 2024-09-16T01:52:15.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.442+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feebc179490 0x7feebc034db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.442+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feebc007690 con 0x7feebc179490 2024-09-16T01:52:15.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.442+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feebc179490 0x7feebc034db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.445+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feebc179490 0x7feebc034db0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.449+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc007690 con 0x7feebc179490 2024-09-16T01:52:15.452 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:15.450+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.484+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc04d2e0 con 0x7feebc179490 2024-09-16T01:52:15.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.492+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc04d2e0 con 0x7feebc179490 2024-09-16T01:52:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:15.499 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:15.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.498+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feebc0d6390 0x7feebc0d6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.498+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7feebc0d6cb0 con 0x7feebc0d6390 2024-09-16T01:52:15.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.502+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feebc0d6390 0x7feebc0d6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.509+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feebc0d6390 0x7feebc0d6770 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.513+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc0d6cb0 con 0x7feebc0d6390 2024-09-16T01:52:15.552 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:15.551+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:15.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.555+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc16ebf0 con 0x7feebc0d6390 2024-09-16T01:52:15.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.579+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc16ebf0 con 0x7feebc0d6390 2024-09-16T01:52:15.580 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:15.580 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:15.591 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:15.591 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:15.591 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:15.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.593+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feebc16f700 0x7feebc16fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.593+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7feebc04e430 con 0x7feebc16f700 2024-09-16T01:52:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.594+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feebc16f700 0x7feebc16fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.595+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feebc16f700 0x7feebc16fae0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.601+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc04e430 con 0x7feebc16f700 2024-09-16T01:52:15.630 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:15.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.633+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc168db0 con 0x7feebc16f700 2024-09-16T01:52:15.641 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:15.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.639+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc168db0 con 0x7feebc16f700 2024-09-16T01:52:15.642 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:15.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.640+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feebc187810 0x7feebc189c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.640+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7feebc16c420 con 0x7feebc187810 2024-09-16T01:52:15.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.645+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feebc187810 0x7feebc189c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.645+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feebc187810 0x7feebc189c00 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.650+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feec40741d0 con 0x7feebc187810 2024-09-16T01:52:15.653 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:15.652+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:15.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.675+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc17ce90 con 0x7feebc187810 2024-09-16T01:52:15.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.690+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc17ce90 con 0x7feebc187810 2024-09-16T01:52:15.691 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:15.691 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:15.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.690+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feebc18a650 0x7feebc18ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.690+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7feebc1706a0 con 0x7feebc18a650 2024-09-16T01:52:15.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.691+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feebc18a650 0x7feebc18ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.692+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feebc18a650 0x7feebc18ca40 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.695+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc1706a0 con 0x7feebc18a650 2024-09-16T01:52:15.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.728+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc1706a0 con 0x7feebc18a650 2024-09-16T01:52:15.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.730+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc1706a0 con 0x7feebc18a650 2024-09-16T01:52:15.731 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:15.731 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:15.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.731+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feebc18d490 0x7feebc18f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.731+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7feebc1706a0 con 0x7feebc18d490 2024-09-16T01:52:15.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.732+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feebc18d490 0x7feebc18f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.733+0000 7feec8c00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feebc18d490 0x7feebc18f880 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.734+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc1706a0 con 0x7feebc18d490 2024-09-16T01:52:15.754 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:15.753+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.792+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc1706a0 con 0x7feebc18d490 2024-09-16T01:52:15.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.795+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc1706a0 con 0x7feebc18d490 2024-09-16T01:52:15.797 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:15.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.795+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feebc1902d0 0x7feebc1926c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.795+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7feebc0037b0 con 0x7feebc1902d0 2024-09-16T01:52:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.796+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feebc1902d0 0x7feebc1926c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.796+0000 7feeca000640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feebc1902d0 0x7feebc1926c0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.797+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc0037b0 con 0x7feebc1902d0 2024-09-16T01:52:15.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.858+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc192c00 con 0x7feebc1902d0 2024-09-16T01:52:15.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.860+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc192c00 con 0x7feebc1902d0 2024-09-16T01:52:15.861 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:15.862 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:15.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.861+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feebc193110 0x7feebc195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:15.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.861+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7feebc198480 con 0x7feebc193110 2024-09-16T01:52:15.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.861+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feebc193110 0x7feebc195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:15.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.862+0000 7feec9600640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feebc193110 0x7feebc195500 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:15.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.863+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feebc198480 con 0x7feebc193110 2024-09-16T01:52:15.865 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:15.857+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:15.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.894+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7feebc195ae0 con 0x7feebc193110 2024-09-16T01:52:15.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.903+0000 7feec1600640 1 -- 172.21.15.146:0/3595306694 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7feebc195ae0 con 0x7feebc193110 2024-09-16T01:52:15.904 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:15.904 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feebc179490 msgr2=0x7feebc034db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7feebc179490 0x7feebc034db0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feebc0d6390 msgr2=0x7feebc0d6770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7feebc0d6390 0x7feebc0d6770 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feebc16f700 msgr2=0x7feebc16fae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7feebc16f700 0x7feebc16fae0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feebc187810 msgr2=0x7feebc189c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7feebc187810 0x7feebc189c00 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feebc18a650 msgr2=0x7feebc18ca40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7feebc18a650 0x7feebc18ca40 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feebc18d490 msgr2=0x7feebc18f880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7feebc18d490 0x7feebc18f880 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feebc1902d0 msgr2=0x7feebc1926c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7feebc1902d0 0x7feebc1926c0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feebc193110 msgr2=0x7feebc195500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.908+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7feebc193110 0x7feebc195500 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.909+0000 7feeca000640 1 -- 172.21.15.146:0/3595306694 reap_dead start 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.909+0000 7feeca000640 1 -- 172.21.15.146:0/3595306694 reap_dead start 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.909+0000 7feeca000640 1 -- 172.21.15.146:0/3595306694 reap_dead start 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.909+0000 7feeca000640 1 -- 172.21.15.146:0/3595306694 reap_dead start 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.909+0000 7feeca000640 1 -- 172.21.15.146:0/3595306694 reap_dead start 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.911+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feea406c2b0 msgr2=0x7feea406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.911+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feea406c2b0 0x7feea406e770 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feeb8001980 tx=0x7feeb8009040 comp rx=0 tx=0).stop 2024-09-16T01:52:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.911+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feebc04c7f0 msgr2=0x7feebc178f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.911+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feebc04c7f0 0x7feebc178f50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7feebc04d6d0 tx=0x7feeb00046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.912+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 shutdown_connections 2024-09-16T01:52:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.912+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7feea406c2b0 0x7feea406e770 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.912+0000 7feecaa00640 1 --2- 172.21.15.146:0/3595306694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feebc04c7f0 0x7feebc178f50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:15.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.912+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 >> 172.21.15.146:0/3595306694 conn(0x7feebc00c390 msgr2=0x7feebc0360f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:15.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.915+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 shutdown_connections 2024-09-16T01:52:15.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:15.915+0000 7feecaa00640 1 -- 172.21.15.146:0/3595306694 wait complete. 2024-09-16T01:52:15.955 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:15.955+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:15.969 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:15.969 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:15.969 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:15.969 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:15.969 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:15.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:16.091 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:16.090+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:16.157 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:16.155+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:16.258 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:16.258+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:16.358 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:16.357+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:16.392 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 208.62004995800001, 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.393 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:52:16.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:52:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:52:16.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:16.400 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.002597617999996, 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:52:16.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:52:16.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:16.411 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.210580669000002, 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:16.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:16.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.417 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.175048046000001, 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:52:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:52:16.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1461 3.1c 3:39a1ca69:::100000001f8.000001b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:52:16.423 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.977707250999998, 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105254967, 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1461 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410699+0000", 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.232e-06 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476134+0000", 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065435190000000004 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476173+0000", 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8878999999999998e-05 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476186+0000", 2024-09-16T01:52:16.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3016e-05 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476284+0000", 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7583000000000004e-05 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476440+0000", 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015621799999999999 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476629+0000", 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018937399999999999 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.505615+0000", 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028986212000000001 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515833+0000", 2024-09-16T01:52:16.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010217944 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515901+0000", 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7250999999999997e-05 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515952+0000", 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1068e-05 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.808959109, 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:52:16.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:52:16.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:52:16.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.244747571000001, 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.436 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:52:16.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:52:16.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.182960268999999, 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:52:16.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:52:16.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1931 3.39 e84/83 3:9d369e22:::100000001f8.00000388:head v 84'56, mlcod=0'0)", 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:16.446 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.160340612999999, 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110959082, 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228064+0000", 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5499999999999997e-07 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229687+0000", 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00162302 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229706+0000", 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8459999999999999e-05 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229719+0000", 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2986000000000001e-05 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229824+0000", 2024-09-16T01:52:16.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105163 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229850+0000", 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6621000000000001e-05 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.338768+0000", 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108917369 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339022+0000", 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025460800000000003 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1932 3.39 e84/83 3:9da8c7a0:::100000001f8.00000389:head v 84'57, mlcod=0'0)", 2024-09-16T01:52:16.451 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.153358672, 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10422508900000001, 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235046+0000", 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0589999999999999e-06 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245188+0000", 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010141739 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245228+0000", 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9755999999999997e-05 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245244+0000", 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6475000000000001e-05 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.249380+0000", 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041353170000000003 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.249407+0000", 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7586e-05 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339027+0000", 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089619521999999993 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339270+0000", 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024363499999999999 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1938 3.39 e84/83 3:9faf1682:::100000001f8.0000038f:head v 84'58, mlcod=0'0)", 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.143113856999999, 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.101947118, 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.456 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245292+0000", 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5030000000000001e-06 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253345+0000", 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080530840000000003 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253363+0000", 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8281999999999999e-05 2024-09-16T01:52:16.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253379+0000", 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6252999999999998e-05 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254278+0000", 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089897700000000002 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254304+0000", 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5559e-05 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347185+0000", 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092881693000000001 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347237+0000", 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1767000000000001e-05 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1963 3.19 3:990e9bf2:::100000001f8.000003a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.450671+0000", 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.937732953000001, 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095157485999999999, 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.461 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1963 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.450671+0000", 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.450671+0000", 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.450673+0000", 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2280000000000001e-06 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491124+0000", 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040450429000000003 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491156+0000", 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2715000000000001e-05 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491170+0000", 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3899e-05 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491383+0000", 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021269099999999999 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491542+0000", 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015885699999999999 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.491709+0000", 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166825 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.541315+0000", 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049606712999999997 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.542828+0000", 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015124610000000001 2024-09-16T01:52:16.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.542889+0000", 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1555000000000003e-05 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.545828+0000", 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029391130000000001 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.341177822999999, 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:52:16.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:52:16.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:16.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.472 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2089 3.37 3:ee65e5cb:::100000001f8.00000426:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.224209+0000", 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.164195285000002, 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105946261, 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2089 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.224209+0000", 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.224209+0000", 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.224211+0000", 2024-09-16T01:52:16.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7549999999999999e-06 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.247234+0000", 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023022918 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.247265+0000", 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0639999999999998e-05 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.247278+0000", 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2976000000000001e-05 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.250654+0000", 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003375992 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.250827+0000", 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173251 2024-09-16T01:52:16.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.250979+0000", 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151731 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.304799+0000", 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053819909999999999 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330020+0000", 2024-09-16T01:52:16.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025221323 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330093+0000", 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2653999999999999e-05 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330155+0000", 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2111000000000002e-05 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.478 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.597739069999999, 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.479 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:52:16.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:52:16.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2857 3.1c 3:39a72f86:::100000001f8.00000726:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.526107668000002, 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11170522400000001, 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2857 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.485 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862299+0000", 2024-09-16T01:52:16.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.351e-06 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890188+0000", 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027888969999999999 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890218+0000", 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0593999999999999e-05 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890230+0000", 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1688999999999999e-05 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890467+0000", 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023695699999999999 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.891129+0000", 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066218699999999995 2024-09-16T01:52:16.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.891811+0000", 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068177899999999998 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912951+0000", 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021140665999999999 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973903+0000", 2024-09-16T01:52:16.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060951181 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973961+0000", 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7989999999999999e-05 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974001+0000", 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0859999999999998e-05 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3328 3.37 3:ee2380e2:::100000001f8.000008fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.374289+0000", 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.014115, 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.098760725999999993, 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3328 2024-09-16T01:52:16.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.374289+0000", 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.374289+0000", 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.374291+0000", 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.978e-06 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.431333+0000", 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057042064000000003 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.431352+0000", 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.927e-05 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.431363+0000", 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0937e-05 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.431458+0000", 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4405000000000002e-05 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.431625+0000", 2024-09-16T01:52:16.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000167002 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.431774+0000", 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014987900000000001 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.453242+0000", 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021467515999999999 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.472929+0000", 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01968667 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.472991+0000", 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2736000000000004e-05 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.473050+0000", 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8269000000000001e-05 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.553518124, 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.497 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:52:16.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.545101638, 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.503 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:52:16.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3678 3.3d e97/91 3:bd5b01a1:::100000001f8.0000005b:head v 97'72, mlcod=96'70)", 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.879403+0000", 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.50900094500000004, 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106394118, 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.879403+0000", 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.879403+0000", 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.879405+0000", 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9410000000000001e-06 2024-09-16T01:52:16.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.939178+0000", 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059773197 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.940634+0000", 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001455708 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.940709+0000", 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5426000000000006e-05 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.944640+0000", 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039306360000000004 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.944670+0000", 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0599999999999998e-05 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.985736+0000", 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041065958 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.985797+0000", 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0652000000000001e-05 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.514 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:16.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:16.516 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:16.514+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:16.616 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:16.616+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:16.717 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:16.716+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:16.819 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:16.818+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:16.855 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:16.855 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3752 3.36 3:6f9f445e:::100000001f8.000000a5:head [write 0~4194304 [1@-1] in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:16.787424+0000", 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.064515632000000003, 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064552511000000007, 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for degraded object", 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:16.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3752 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.787424+0000", 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.787424+0000", 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.787427+0000", 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.103e-06 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.833377+0000", 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045950026999999997 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.833399+0000", 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2555999999999999e-05 2024-09-16T01:52:16.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.833410+0000", 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1226999999999999e-05 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.833540+0000", 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129839 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:16.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.835551+0000", 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020107409999999999 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:52:16.860 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:16.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:16.921 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:16.920+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:17.021 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:17.020+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:17.122 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:17.121+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:17.223 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:17.222+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:17.255 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:17.255 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:17.255 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:17.255 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:17.256 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:17.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:17.363 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:17.362+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:17.425 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:17.424+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:17.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:52:17.540 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:17.539+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:17.605 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:17.605 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:17.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:17.605 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:17.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:548 3.5 3:a1d42e8b:::100000001f7.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.939333581, 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095160324000000004, 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 548 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.606 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662323+0000", 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.662326+0000", 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5560000000000002e-06 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748231+0000", 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.085904741000000007 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748263+0000", 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2697999999999998e-05 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748276+0000", 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.254e-05 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748359+0000", 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2991999999999994e-05 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748511+0000", 2024-09-16T01:52:17.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015181600000000001 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.748677+0000", 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166613 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.753332+0000", 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046552090000000004 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757358+0000", 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040256880000000004 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757427+0000", 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9295000000000007e-05 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:14.757484+0000", 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6175999999999999e-05 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.611 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.310474313999997, 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.612 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:17.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:17.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.281313459000003, 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:17.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:17.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.248961891, 2024-09-16T01:52:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:17.625 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.625 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:17.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.233014625000003, 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:17.634 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:17.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:17.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:17.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1132 3.5 3:a0fd78c8:::100000001f8.00000069:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.210479927999998, 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095513232000000003, 2024-09-16T01:52:17.640 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1132 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391177+0000", 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391179+0000", 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.66e-06 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408341+0000", 2024-09-16T01:52:17.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017162686999999999 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408396+0000", 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5034000000000002e-05 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408412+0000", 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5685e-05 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408710+0000", 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029794899999999999 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.408870+0000", 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000160168 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.409033+0000", 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016284700000000001 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480660+0000", 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071627535000000006 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486552+0000", 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0058913079999999996 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486634+0000", 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2051000000000004e-05 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.486690+0000", 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6307999999999997e-05 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.68593946, 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:17.646 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:17.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.501010103999999, 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:17.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:17.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.498472295999999, 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:17.660 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:17.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:17.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1966 3.3c 3:3c832e37:::100000001f8.000003ab:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.145437555000001, 2024-09-16T01:52:17.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10596815800000001, 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1966 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456221+0000", 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.125e-06 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520630+0000", 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064408699 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520660+0000", 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0122000000000001e-05 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520672+0000", 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2179e-05 2024-09-16T01:52:17.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520772+0000", 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.993e-05 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520970+0000", 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198097 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.521122+0000", 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151065 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.535969+0000", 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014847075 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562086+0000", 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026117475000000001 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562144+0000", 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7806000000000002e-05 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562187+0000", 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3584999999999997e-05 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.889061064, 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:17.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:17.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2096 3.16 3:68301556:::100000001f8.0000042d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.328522728999999, 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096328759999999999, 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2096 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273134+0000", 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.273136+0000", 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.08e-06 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306295+0000", 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033158316 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306329+0000", 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4656999999999997e-05 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306353+0000", 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4244999999999999e-05 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306437+0000", 2024-09-16T01:52:17.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3885000000000001e-05 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306614+0000", 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000176656 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.306762+0000", 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014849399999999999 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.368128+0000", 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061365140999999998 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369361+0000", 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001232934 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369413+0000", 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2123000000000002e-05 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.369463+0000", 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0229000000000003e-05 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3056 3.16 3:68f2fc0a:::100000001f8.000007ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.593429896, 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.094053392, 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3056 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008227+0000", 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.008229+0000", 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.756e-06 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062258+0000", 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054028822999999997 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062288+0000", 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0896999999999999e-05 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062302+0000", 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3112000000000001e-05 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062499+0000", 2024-09-16T01:52:17.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019759500000000001 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062685+0000", 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000186281 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.062841+0000", 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155847 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.074322+0000", 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.011480959000000001 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102173+0000", 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027851130000000002 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102236+0000", 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2749000000000005e-05 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:02.102280+0000", 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4243000000000003e-05 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3350 3.3c 3:3f688ec3:::100000001f8.00000913:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.639516+0000", 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.962140440000001, 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097703430999999993, 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3350 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.639516+0000", 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.639516+0000", 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.639519+0000", 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3039999999999999e-06 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674045+0000", 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034526172000000001 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674061+0000", 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5896000000000002e-05 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674071+0000", 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7340000000000003e-06 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674458+0000", 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038717899999999997 2024-09-16T01:52:17.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674612+0000", 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015388700000000001 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.674756+0000", 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144704 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.717031+0000", 2024-09-16T01:52:17.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042275158 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.737101+0000", 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020069872999999998 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.737169+0000", 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7393999999999999e-05 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.737220+0000", 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.113e-05 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3362 3.3c 3:3f474b19:::100000001f8.0000091f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.768356+0000", 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.833301098, 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.095600888999999994, 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.697 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3362 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.768356+0000", 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.768356+0000", 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.768358+0000", 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3e-06 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.799066+0000", 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030707463000000001 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.799079+0000", 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3635000000000001e-05 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.799087+0000", 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6650000000000003e-06 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.801339+0000", 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022524950000000002 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.801477+0000", 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000137706 2024-09-16T01:52:17.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.801631+0000", 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015402999999999999 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.854713+0000", 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053081642999999998 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.863843+0000", 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0091305929999999993 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.863912+0000", 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8715999999999997e-05 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.863957+0000", 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4642999999999999e-05 2024-09-16T01:52:17.703 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3377 3.26 3:6604db39:::100000001f8.0000092e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.675670497, 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.104633035, 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.704 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3377 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925988+0000", 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0549999999999998e-06 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983358+0000", 2024-09-16T01:52:17.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057369575999999999 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983375+0000", 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7356000000000001e-05 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983387+0000", 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1277000000000001e-05 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983509+0000", 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012268399999999999 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983659+0000", 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014946499999999999 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983799+0000", 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139787 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.000668+0000", 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016869359 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030498+0000", 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029829855999999998 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030562+0000", 2024-09-16T01:52:17.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4199e-05 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030619+0000", 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7420999999999998e-05 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3378 3.e 3:709bbee6:::100000001f8.0000092f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:17.710 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.618246682000001, 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108434431, 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3378 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983412+0000", 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.528e-06 2024-09-16T01:52:17.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.059475+0000", 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076063617 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060086+0000", 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061040300000000003 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060097+0000", 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0842999999999999e-05 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060152+0000", 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5025999999999998e-05 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060248+0000", 2024-09-16T01:52:17.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6351e-05 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060334+0000", 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6216000000000003e-05 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.077660+0000", 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017325871 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091714+0000", 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014053566 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091794+0000", 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0434999999999998e-05 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091845+0000", 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0575000000000003e-05 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.716 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.850804629, 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:52:17.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:52:17.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.7757328939999999, 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:52:17.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:52:17.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3682 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.894687+0000", 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.7069697210000001, 2024-09-16T01:52:17.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10534299599999999, 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3682 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.894687+0000", 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.894687+0000", 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.894689+0000", 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.902e-06 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919371+0000", 2024-09-16T01:52:17.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024682101000000001 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919396+0000", 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4736000000000002e-05 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919410+0000", 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3859000000000001e-05 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:17.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.921989+0000", 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025788249999999999 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.922099+0000", 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110003 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.922251+0000", 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015253900000000001 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.951255+0000", 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029004268999999999 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.999912+0000", 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048656491000000003 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.999980+0000", 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7875999999999999e-05 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:17.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.000030+0000", 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0395000000000001e-05 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:17.735 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:17.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:17.740 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:17.739+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:17.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.767+0000 7f455ce00640 1 Processor -- start 2024-09-16T01:52:17.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.767+0000 7f455ce00640 1 -- start start 2024-09-16T01:52:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.769+0000 7f455ce00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580c75a0 0x7f455815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.769+0000 7f455ce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f45580cc040 0x7f45580cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.769+0000 7f455ce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f455806d140 con 0x7f45580c6b70 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.769+0000 7f455ce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f455815b880 con 0x7f45580cc040 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.769+0000 7f455ce00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f45580cab20 con 0x7f45580c75a0 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.770+0000 7f4557400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f45580c6b70 0x7f45580cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38108/0 (socket says 172.21.15.146:38108) 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.770+0000 7f4557400640 1 -- 172.21.15.146:0/3985737795 learned_addr learned my addr 172.21.15.146:0/3985737795 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.770+0000 7f454fe00640 1 --2- 172.21.15.146:0/3985737795 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580c75a0 0x7f455815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.770+0000 7f4556a00640 1 -- 172.21.15.146:0/3985737795 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665047179 0 0) 0x7f455806d140 con 0x7f45580c6b70 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.771+0000 7f4556a00640 1 -- 172.21.15.146:0/3985737795 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f453c003520 con 0x7f45580c6b70 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.771+0000 7f454fe00640 1 -- 172.21.15.146:0/3985737795 >> v2:172.21.15.146:3300/0 conn(0x7f45580cc040 msgr2=0x7f45580cc4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.771+0000 7f454fe00640 1 --2- 172.21.15.146:0/3985737795 >> v2:172.21.15.146:3300/0 conn(0x7f45580cc040 0x7f45580cc4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:17.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.771+0000 7f454fe00640 1 -- 172.21.15.146:0/3985737795 >> v1:172.21.15.146:6789/0 conn(0x7f45580c6b70 legacy=0x7f45580cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:17.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.771+0000 7f454fe00640 1 -- 172.21.15.146:0/3985737795 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455806d420 con 0x7f45580c75a0 2024-09-16T01:52:17.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.773+0000 7f454fe00640 1 --2- 172.21.15.146:0/3985737795 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580c75a0 0x7f455815b340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f45400097a0 tx=0x7f454002a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15763e0035e88550 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.774+0000 7f4556a00640 1 -- 172.21.15.146:0/3985737795 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540037070 con 0x7f45580c75a0 2024-09-16T01:52:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.774+0000 7f4556a00640 1 -- 172.21.15.146:0/3985737795 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4540032440 con 0x7f45580c75a0 2024-09-16T01:52:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.775+0000 7f4556a00640 1 -- 172.21.15.146:0/3985737795 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454002ae60 con 0x7f45580c75a0 2024-09-16T01:52:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.776+0000 7f455ce00640 1 -- 172.21.15.146:0/3985737795 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580c75a0 msgr2=0x7f455815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.776+0000 7f455ce00640 1 --2- 172.21.15.146:0/3985737795 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580c75a0 0x7f455815b340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f45400097a0 tx=0x7f454002a110 comp rx=0 tx=0).stop 2024-09-16T01:52:17.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.777+0000 7f455ce00640 1 -- 172.21.15.146:0/3985737795 shutdown_connections 2024-09-16T01:52:17.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.777+0000 7f455ce00640 1 --2- 172.21.15.146:0/3985737795 >> v2:172.21.15.146:3300/0 conn(0x7f45580cc040 0x7f45580cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:17.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.777+0000 7f455ce00640 1 --2- 172.21.15.146:0/3985737795 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580c75a0 0x7f455815b340 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:17.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.777+0000 7f455ce00640 1 -- 172.21.15.146:0/3985737795 >> 172.21.15.146:0/3985737795 conn(0x7f45580ccc20 msgr2=0x7f45580cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:17.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.778+0000 7f455ce00640 1 -- 172.21.15.146:0/3985737795 shutdown_connections 2024-09-16T01:52:17.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.778+0000 7f455ce00640 1 -- 172.21.15.146:0/3985737795 wait complete. 2024-09-16T01:52:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.781+0000 7f455ce00640 1 Processor -- start 2024-09-16T01:52:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.781+0000 7f455ce00640 1 -- start start 2024-09-16T01:52:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.784+0000 7f455ce00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 0x7f455815a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:17.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.784+0000 7f455ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45580c75a0 0x7f45581ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.784+0000 7f4557400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 0x7f455815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f455ce00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580cc040 0x7f4558158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f4557400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 0x7f455815a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51820/0 (socket says 172.21.15.146:51820) 2024-09-16T01:52:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f454fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45580c75a0 0x7f45581ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:17.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f45580cb050 con 0x7f45580c75a0 2024-09-16T01:52:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45580cad70 con 0x7f45580c6b70 2024-09-16T01:52:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f45580cbc80 con 0x7f45580cc040 2024-09-16T01:52:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.785+0000 7f4557400640 1 -- 172.21.15.146:0/3144426640 learned_addr learned my addr 172.21.15.146:0/3144426640 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:17.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.786+0000 7f4557e00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580cc040 0x7f4558158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.786+0000 7f4557400640 1 -- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580cc040 msgr2=0x7f4558158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.787+0000 7f4557400640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580cc040 0x7f4558158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.787+0000 7f4557400640 1 -- 172.21.15.146:0/3144426640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45580c75a0 msgr2=0x7f45581ea720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:17.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.787+0000 7f4557400640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45580c75a0 0x7f45581ea720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:17.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.787+0000 7f4557400640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45581f5500 con 0x7f45580c6b70 2024-09-16T01:52:17.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.787+0000 7f454fe00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45580c75a0 0x7f45581ea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:17.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.788+0000 7f4557e00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580cc040 0x7f4558158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:17.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.788+0000 7f4557400640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 0x7f455815a540 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f454800c390 tx=0x7f454800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:17.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.789+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4548017070 con 0x7f45580c6b70 2024-09-16T01:52:17.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.789+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4548005ce0 con 0x7f45580c6b70 2024-09-16T01:52:17.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.789+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455815aa80 con 0x7f45580c6b70 2024-09-16T01:52:17.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.789+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45581eb210 con 0x7f45580c6b70 2024-09-16T01:52:17.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.789+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45480072f0 con 0x7f45580c6b70 2024-09-16T01:52:17.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.791+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4530005290 con 0x7f45580c6b70 2024-09-16T01:52:17.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.793+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f454801a070 con 0x7f45580c6b70 2024-09-16T01:52:17.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.795+0000 7f454f400640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f453406c380 0x7f453406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:17.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.796+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5741+0+0 (secure 0 0 0) 0x7f454808a800 con 0x7f45580c6b70 2024-09-16T01:52:17.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.796+0000 7f454fe00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f453406c380 0x7f453406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:17.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.797+0000 7f454fe00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f453406c380 0x7f453406e840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4540034000 tx=0x7f454002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:17.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:17.798+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f45480588e0 con 0x7f45580c6b70 2024-09-16T01:52:17.839 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:17.837+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:17.939 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:17.938+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:17.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3838 3.1b 3:da93b623:::100000001f8.000000fb:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:17.961686+0000", 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.064442396999999998, 2024-09-16T01:52:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064482822999999995, 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3838 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.961686+0000", 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.961686+0000", 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.961688+0000", 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8819999999999999e-06 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:18.017930+0000", 2024-09-16T01:52:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056241799000000002 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:18.017952+0000", 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2229e-05 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:18.017963+0000", 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.077e-05 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:18.030 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:18.021358+0000", 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033948060000000002 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:18.021445+0000", 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7383e-05 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:18.021602+0000", 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015658299999999999 2024-09-16T01:52:18.031 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:18.032 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:52:18.032 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:18.040 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:18.039+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:18.054 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:18.141 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:18.140+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:18.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.165+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f4530005580 con 0x7f45580c6b70 2024-09-16T01:52:18.244 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:18.243+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:18.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.280+0000 7f99ff200640 1 Processor -- start 2024-09-16T01:52:18.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.281+0000 7f99ff200640 1 -- start start 2024-09-16T01:52:18.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99ff200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9a000d0490 0x7f9a000c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99ff200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a000d0870 0x7f9a000c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99ff200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0015e220 con 0x7f9a00154690 2024-09-16T01:52:18.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99ff200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9a000c6160 con 0x7f9a000d0870 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99ff200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9a0015e500 con 0x7f9a000d0490 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99fde00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9a000d0490 0x7f9a000c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99fde00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9a000d0490 0x7f9a000c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51824/0 (socket says 172.21.15.146:51824) 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.282+0000 7f99fde00640 1 -- 172.21.15.146:0/2912841789 learned_addr learned my addr 172.21.15.146:0/2912841789 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.283+0000 7f99fde00640 1 -- 172.21.15.146:0/2912841789 >> v2:172.21.15.146:3300/0 conn(0x7f9a000d0870 msgr2=0x7f9a000c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.283+0000 7f99fde00640 1 --2- 172.21.15.146:0/2912841789 >> v2:172.21.15.146:3300/0 conn(0x7f9a000d0870 0x7f9a000c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.283+0000 7f99fde00640 1 -- 172.21.15.146:0/2912841789 >> v1:172.21.15.146:6789/0 conn(0x7f9a00154690 legacy=0x7f9a00154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:18.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.283+0000 7f99fde00640 1 -- 172.21.15.146:0/2912841789 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0015e7e0 con 0x7f9a000d0490 2024-09-16T01:52:18.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.284+0000 7f99fde00640 1 --2- 172.21.15.146:0/2912841789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a000d0490 0x7f9a000c65f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9a000cb260 tx=0x7f99f002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d729540a599efd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.284+0000 7f99fca00640 1 -- 172.21.15.146:0/2912841789 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0037070 con 0x7f9a000d0490 2024-09-16T01:52:18.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.284+0000 7f99fca00640 1 -- 172.21.15.146:0/2912841789 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0032440 con 0x7f9a000d0490 2024-09-16T01:52:18.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.284+0000 7f99fca00640 1 -- 172.21.15.146:0/2912841789 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0032810 con 0x7f9a000d0490 2024-09-16T01:52:18.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 -- 172.21.15.146:0/2912841789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a000d0490 msgr2=0x7f9a000c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 --2- 172.21.15.146:0/2912841789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a000d0490 0x7f9a000c65f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9a000cb260 tx=0x7f99f002a2c0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 -- 172.21.15.146:0/2912841789 shutdown_connections 2024-09-16T01:52:18.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 --2- 172.21.15.146:0/2912841789 >> v2:172.21.15.146:3300/0 conn(0x7f9a000d0870 0x7f9a000c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 --2- 172.21.15.146:0/2912841789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a000d0490 0x7f9a000c65f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 -- 172.21.15.146:0/2912841789 >> 172.21.15.146:0/2912841789 conn(0x7f9a0006d6f0 msgr2=0x7f9a0006e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:18.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 -- 172.21.15.146:0/2912841789 shutdown_connections 2024-09-16T01:52:18.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.285+0000 7f99ff200640 1 -- 172.21.15.146:0/2912841789 wait complete. 2024-09-16T01:52:18.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.287+0000 7f99ff200640 1 Processor -- start 2024-09-16T01:52:18.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.287+0000 7f99ff200640 1 -- start start 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.288+0000 7f99ff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a000d0490 0x7f9a001533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.288+0000 7f99ff200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a000d0870 0x7f9a001f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.288+0000 7f99ff200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 0x7f9a001f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.288+0000 7f99ff200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0015dec0 con 0x7f9a000d0490 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.288+0000 7f99ff200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0015eda0 con 0x7f9a00154690 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.288+0000 7f99ff200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9a000c51f0 con 0x7f9a000d0870 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fd400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a000d0870 0x7f9a001f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 0x7f9a001f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 0x7f9a001f4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51838/0 (socket says 172.21.15.146:51838) 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 learned_addr learned my addr 172.21.15.146:0/4132722279 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:18.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fd400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a000d0870 0x7f9a001f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36332/0 (socket says 172.21.15.146:36332) 2024-09-16T01:52:18.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a000d0490 0x7f9a001533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a000d0870 msgr2=0x7f9a001f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9a000d0870 0x7f9a001f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a000d0490 msgr2=0x7f9a001533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a000d0490 0x7f9a001533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.289+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a000b3650 con 0x7f9a00154690 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.290+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 0x7f9a001f4770 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f99f800b990 tx=0x7f99f800bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.290+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8019070 con 0x7f9a00154690 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.290+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f800caa0 con 0x7f9a00154690 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.290+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8014440 con 0x7f9a00154690 2024-09-16T01:52:18.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.290+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a001f18e0 con 0x7f9a00154690 2024-09-16T01:52:18.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.291+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a000b3bd0 con 0x7f9a00154690 2024-09-16T01:52:18.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.292+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9a00154460 con 0x7f9a00154690 2024-09-16T01:52:18.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.293+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f99f801c070 con 0x7f9a00154690 2024-09-16T01:52:18.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.294+0000 7f99f6000640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f99e006c2e0 0x7f99e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.294+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5741+0+0 (secure 0 0 0) 0x7f99f808afc0 con 0x7f9a00154690 2024-09-16T01:52:18.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.294+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f99e006c2e0 0x7f99e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.294+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v97) ==== 54+0+16 (secure 0 0 0) 0x7f9a00154460 con 0x7f9a00154690 2024-09-16T01:52:18.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.294+0000 7f99ff200640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a001f4cb0 0x7f9a000c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.295+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a001f5090 con 0x7f9a001f4cb0 2024-09-16T01:52:18.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.295+0000 7f99fd400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a001f4cb0 0x7f9a000c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.295+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f99e006c2e0 0x7f99e006e7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f99f0034000 tx=0x7f99f002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.295+0000 7f99fd400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a001f4cb0 0x7f9a000c65d0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.296+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a001f5090 con 0x7f9a001f4cb0 2024-09-16T01:52:18.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.316+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a000ca900 con 0x7f9a001f4cb0 2024-09-16T01:52:18.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.318+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a000ca900 con 0x7f9a001f4cb0 2024-09-16T01:52:18.320 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:18.320 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:18.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.319+0000 7f99ff200640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a00062b40 0x7f9a00062f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.319+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9a000cc940 con 0x7f9a00062b40 2024-09-16T01:52:18.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.319+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a00062b40 0x7f9a00062f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.321+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a00062b40 0x7f9a00062f20 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.322+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a000cc940 con 0x7f9a00062b40 2024-09-16T01:52:18.344 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:18.343+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:18.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.350+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a000cc3f0 con 0x7f9a00062b40 2024-09-16T01:52:18.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.357+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a000cc3f0 con 0x7f9a00062b40 2024-09-16T01:52:18.359 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:18.359 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:18.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.358+0000 7f99ff200640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a001e74d0 0x7f9a001e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.358+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9a000cd810 con 0x7f9a001e74d0 2024-09-16T01:52:18.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.358+0000 7f99fd400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a001e74d0 0x7f9a001e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.359+0000 7f99fd400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a001e74d0 0x7f9a001e78b0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.360+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a000cd810 con 0x7f9a001e74d0 2024-09-16T01:52:18.366 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:18.366 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:18.366 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:18.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:18.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.386+0000 7f99ff200640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a000cd530 con 0x7f9a001e74d0 2024-09-16T01:52:18.392 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.389+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a000cd530 con 0x7f9a001e74d0 2024-09-16T01:52:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:18.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.389+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a0015bfa0 0x7f9a0015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.389+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9a000b40b0 con 0x7f9a0015bfa0 2024-09-16T01:52:18.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.390+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a0015bfa0 0x7f9a0015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.391+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a0015bfa0 0x7f9a0015c3a0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.394+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a000b40b0 con 0x7f9a0015bfa0 2024-09-16T01:52:18.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.425+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a001e7df0 con 0x7f9a0015bfa0 2024-09-16T01:52:18.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.426+0000 7f454f400640 1 -- 172.21.15.146:0/3144426640 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v98) ==== 87+0+0 (secure 0 0 0) 0x7f454805cbe0 con 0x7f45580c6b70 2024-09-16T01:52:18.428 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:52:18.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.433+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f453406c380 msgr2=0x7f453406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.433+0000 7f455ce00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f453406c380 0x7f453406e840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4540034000 tx=0x7f454002e000 comp rx=0 tx=0).stop 2024-09-16T01:52:18.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.433+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 msgr2=0x7f455815a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.433+0000 7f455ce00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 0x7f455815a540 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f454800c390 tx=0x7f454800c850 comp rx=0 tx=0).stop 2024-09-16T01:52:18.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 shutdown_connections 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f453406c380 0x7f453406e840 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f45580cc040 0x7f4558158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45580c75a0 0x7f45581ea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 --2- 172.21.15.146:0/3144426640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45580c6b70 0x7f455815a540 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 >> 172.21.15.146:0/3144426640 conn(0x7f45580ccc20 msgr2=0x7f45580c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 shutdown_connections 2024-09-16T01:52:18.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.434+0000 7f455ce00640 1 -- 172.21.15.146:0/3144426640 wait complete. 2024-09-16T01:52:18.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.438+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a001e7df0 con 0x7f9a0015bfa0 2024-09-16T01:52:18.440 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:18.441 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:18.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.439+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a002035a0 0x7f9a00205990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.439+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9a0015dcd0 con 0x7f9a002035a0 2024-09-16T01:52:18.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.439+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a002035a0 0x7f9a00205990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.440+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a002035a0 0x7f9a00205990 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.440+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a0015dcd0 con 0x7f9a002035a0 2024-09-16T01:52:18.447 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:18.445+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:18.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.475+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a0015dcd0 con 0x7f9a002035a0 2024-09-16T01:52:18.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.477+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a0015dcd0 con 0x7f9a002035a0 2024-09-16T01:52:18.478 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:18.478 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:18.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.479+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a002063e0 0x7f9a002087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.480+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9a0015cec0 con 0x7f9a002063e0 2024-09-16T01:52:18.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.480+0000 7f99fd400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a002063e0 0x7f9a002087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.481+0000 7f99fd400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a002063e0 0x7f9a002087d0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.485+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a0015cec0 con 0x7f9a002063e0 2024-09-16T01:52:18.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.504+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a00208d10 con 0x7f9a002063e0 2024-09-16T01:52:18.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.507+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a00208d10 con 0x7f9a002063e0 2024-09-16T01:52:18.508 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:18.508 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:18.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.507+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a00209220 0x7f9a0020b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.507+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9a0020e5f0 con 0x7f9a00209220 2024-09-16T01:52:18.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.508+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a00209220 0x7f9a0020b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.509+0000 7f99fe800640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a00209220 0x7f9a0020b610 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.509+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a0020e5f0 con 0x7f9a00209220 2024-09-16T01:52:18.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.528+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a0020d330 con 0x7f9a00209220 2024-09-16T01:52:18.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.530+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a0020d330 con 0x7f9a00209220 2024-09-16T01:52:18.531 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:18.531 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.530+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a0020da50 0x7f9a0020de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.530+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9a002119c0 con 0x7f9a0020da50 2024-09-16T01:52:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.530+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a0020da50 0x7f9a0020de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.531+0000 7f99fde00640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a0020da50 0x7f9a0020de70 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:18.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.532+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a002119c0 con 0x7f9a0020da50 2024-09-16T01:52:18.548 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:18.547+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:18.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.553+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9a00210840 con 0x7f9a0020da50 2024-09-16T01:52:18.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99f6000640 1 -- 172.21.15.146:0/4132722279 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a00210840 con 0x7f9a0020da50 2024-09-16T01:52:18.556 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:18.556 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:18.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a001f4cb0 msgr2=0x7f9a000c65d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f9a001f4cb0 0x7f9a000c65d0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a00062b40 msgr2=0x7f9a00062f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f9a00062b40 0x7f9a00062f20 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a001e74d0 msgr2=0x7f9a001e78b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f9a001e74d0 0x7f9a001e78b0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a0015bfa0 msgr2=0x7f9a0015c3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f9a0015bfa0 0x7f9a0015c3a0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a002035a0 msgr2=0x7f9a00205990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f9a002035a0 0x7f9a00205990 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a002063e0 msgr2=0x7f9a002087d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f9a002063e0 0x7f9a002087d0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a00209220 msgr2=0x7f9a0020b610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f9a00209220 0x7f9a0020b610 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a0020da50 msgr2=0x7f9a0020de70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.555+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9a0020da50 0x7f9a0020de70 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.556+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 reap_dead start 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.556+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 reap_dead start 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.556+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 reap_dead start 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.556+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 reap_dead start 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.556+0000 7f99fe800640 1 -- 172.21.15.146:0/4132722279 reap_dead start 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f99e006c2e0 msgr2=0x7f99e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f99e006c2e0 0x7f99e006e7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f99f0034000 tx=0x7f99f002e000 comp rx=0 tx=0).stop 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 msgr2=0x7f9a001f4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 0x7f9a001f4770 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f99f800b990 tx=0x7f99f800bd90 comp rx=0 tx=0).stop 2024-09-16T01:52:18.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 shutdown_connections 2024-09-16T01:52:18.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f99e006c2e0 0x7f99e006e7a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 --2- 172.21.15.146:0/4132722279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9a00154690 0x7f9a001f4770 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:18.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.557+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 >> 172.21.15.146:0/4132722279 conn(0x7f9a0006d6f0 msgr2=0x7f9a000ce1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:18.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.558+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 shutdown_connections 2024-09-16T01:52:18.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:18.558+0000 7f99eb400640 1 -- 172.21.15.146:0/4132722279 wait complete. 2024-09-16T01:52:18.649 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:18.648+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:18.662 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:18.662 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:18.677 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:18.749 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:18.748+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:18.791 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:18.792 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:18.803 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:18.887 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:18.886+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:18.954 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:18.953+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:18.972 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:18.973 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:18.984 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:19.041 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:19.041 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:19.041 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.23443588279267719 2024-09-16T01:52:19.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 0.23443588279267719 2024-09-16T01:52:19.068 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:19.067+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:19.145 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:19.145 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:19.155 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:19.156 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:19.152+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:19.253 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:19.252+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:19.312 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:19.312 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:19.325 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:19.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.340+0000 7f8653400640 1 Processor -- start 2024-09-16T01:52:19.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.340+0000 7f8653400640 1 -- start start 2024-09-16T01:52:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.342+0000 7f8653400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c0cc480 0x7f864c15e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.342+0000 7f8653400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f864c15ba20 con 0x7f864c0cc0a0 2024-09-16T01:52:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.342+0000 7f8653400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f864c15aea0 con 0x7f864c15eba0 2024-09-16T01:52:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.342+0000 7f8653400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f864c15f050 con 0x7f864c0cc480 2024-09-16T01:52:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.342+0000 7f8650a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f864c0cc0a0 0x7f864c0d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38118/0 (socket says 172.21.15.146:38118) 2024-09-16T01:52:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.342+0000 7f8650a00640 1 -- 172.21.15.146:0/3824887561 learned_addr learned my addr 172.21.15.146:0/3824887561 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823406836 0 0) 0x7f864c15ba20 con 0x7f864c0cc0a0 2024-09-16T01:52:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864be00640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c0cc480 0x7f864c15e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8634003520 con 0x7f864c0cc0a0 2024-09-16T01:52:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157488448 0 0) 0x7f864c15aea0 con 0x7f864c15eba0 2024-09-16T01:52:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f864c15ba20 con 0x7f864c15eba0 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1448446623 0 0) 0x7f864c15ba20 con 0x7f864c15eba0 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c0cc480 msgr2=0x7f864c15e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.343+0000 7f864b400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c0cc480 0x7f864c15e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 >> v1:172.21.15.146:6789/0 conn(0x7f864c0cc0a0 legacy=0x7f864c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c15b180 con 0x7f864c15eba0 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f86440019f0 con 0x7f864c15eba0 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 >> v1:172.21.15.161:6789/0 conn(0x7f864c15eba0 legacy=0x7f864c165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8634004ba0 0x7f8634004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86340037a0 0x7f8634003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f86340141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f864c15f050 con 0x7f8634004ba0 2024-09-16T01:52:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8634014710 con 0x7f8634004160 2024-09-16T01:52:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8634014910 con 0x7f86340037a0 2024-09-16T01:52:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f8651400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f86340141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f864be00640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86340037a0 0x7f8634003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.344+0000 7f8650a00640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8634004ba0 0x7f8634004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.345+0000 7f8651400640 1 -- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86340037a0 msgr2=0x7f8634003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.345+0000 7f8651400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86340037a0 0x7f8634003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.345+0000 7f8651400640 1 -- 172.21.15.146:0/3824887561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8634004ba0 msgr2=0x7f8634004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.345+0000 7f8651400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8634004ba0 0x7f8634004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.346+0000 7f8651400640 1 -- 172.21.15.146:0/3824887561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c15b180 con 0x7f8634004160 2024-09-16T01:52:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.347+0000 7f8650a00640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8634004ba0 0x7f8634004fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:19.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.347+0000 7f8651400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f86340141d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f864400c510 tx=0x7f8634005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b28234463a69ccf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:19.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.347+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864400c0e0 con 0x7f8634004160 2024-09-16T01:52:19.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.348+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864402e650 con 0x7f8634004160 2024-09-16T01:52:19.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.348+0000 7f864b400640 1 -- 172.21.15.146:0/3824887561 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864402ea60 con 0x7f8634004160 2024-09-16T01:52:19.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.349+0000 7f8653400640 1 -- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 msgr2=0x7f86340141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:19.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.349+0000 7f8653400640 1 --2- 172.21.15.146:0/3824887561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f86340141d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f864400c510 tx=0x7f8634005830 comp rx=0 tx=0).stop 2024-09-16T01:52:19.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.349+0000 7f8651400640 1 -- 172.21.15.146:0/3824887561 reap_dead start 2024-09-16T01:52:19.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.349+0000 7f8653400640 1 -- 172.21.15.146:0/3824887561 shutdown_connections 2024-09-16T01:52:19.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.349+0000 7f8653400640 1 -- 172.21.15.146:0/3824887561 >> 172.21.15.146:0/3824887561 conn(0x7f864c06d560 msgr2=0x7f864c073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:19.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.351+0000 7f8653400640 1 -- 172.21.15.146:0/3824887561 shutdown_connections 2024-09-16T01:52:19.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.351+0000 7f8653400640 1 -- 172.21.15.146:0/3824887561 wait complete. 2024-09-16T01:52:19.354 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:19.354+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:19.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.355+0000 7f8653400640 1 Processor -- start 2024-09-16T01:52:19.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.357+0000 7f8653400640 1 -- start start 2024-09-16T01:52:19.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.358+0000 7f8653400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f864c1efee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.358+0000 7f8653400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c1501b0 0x7f864c0b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8653400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f864c0c9650 0x7f864c0c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8653400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f864c15b740 con 0x7f864c0c9650 2024-09-16T01:52:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8653400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f864c15e1d0 con 0x7f8634004160 2024-09-16T01:52:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8653400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f864c0cc8a0 con 0x7f864c1501b0 2024-09-16T01:52:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f864c1efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f864be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c1501b0 0x7f864c0b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f864c1efee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51850/0 (socket says 172.21.15.146:51850) 2024-09-16T01:52:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 -- 172.21.15.146:0/3435850479 learned_addr learned my addr 172.21.15.146:0/3435850479 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 -- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c1501b0 msgr2=0x7f864c0b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c1501b0 0x7f864c0b5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 -- 172.21.15.146:0/3435850479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f864c0c9650 msgr2=0x7f864c0c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f864c0c9650 0x7f864c0c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:19.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.359+0000 7f8650a00640 1 -- 172.21.15.146:0/3435850479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c1f05d0 con 0x7f8634004160 2024-09-16T01:52:19.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.360+0000 7f8650a00640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f864c1efee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f863800d720 tx=0x7f863800dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:19.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.360+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638005ce0 con 0x7f8634004160 2024-09-16T01:52:19.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.360+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8638017070 con 0x7f8634004160 2024-09-16T01:52:19.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.360+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638007290 con 0x7f8634004160 2024-09-16T01:52:19.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.360+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c1f08b0 con 0x7f8634004160 2024-09-16T01:52:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.361+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c0c8e50 con 0x7f8634004160 2024-09-16T01:52:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.362+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864c1f0dd0 con 0x7f8634004160 2024-09-16T01:52:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.365+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8638007430 con 0x7f8634004160 2024-09-16T01:52:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.366+0000 7f8648c00640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f862c06c2e0 0x7f862c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.367+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5741+0+0 (secure 0 0 0) 0x7f863808b670 con 0x7f8634004160 2024-09-16T01:52:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.368+0000 7f864be00640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f862c06c2e0 0x7f862c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:19.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.369+0000 7f864be00640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f862c06c2e0 0x7f862c06e7a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f864c1ea5a0 tx=0x7f8640008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:19.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.369+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8638059da0 con 0x7f8634004160 2024-09-16T01:52:19.458 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:19.454+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:19.493 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:19.494 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:19.506 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:19.557 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:19.556+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:19.619 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:19.619 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:19.629 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:19.659 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:19.657+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:19.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:19.724+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 0.23443588279267719} v 0) -- 0x7f864c0cbde0 con 0x7f8634004160 2024-09-16T01:52:19.747 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:19.747 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:19.757 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:19.758 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:19.757+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:19.860 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:19.859+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:19.873 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:19.874 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:19.886 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:19.961 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:19.960+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:20.001 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:20.001 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:20.010 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:20.066 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:20.064+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:20.131 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:20.132 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:20.142 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:20.164 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:20.163+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:20.259 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:20.260 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:20.264 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:20.263+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:20.269 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:20.365 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:20.364+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:20.422 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:20.422 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:20.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:20.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.450+0000 7f8648c00640 1 -- 172.21.15.146:0/3435850479 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 0.23443588279267719}]=0 reweighted osd.7 to 0.234436 (3c03) v99) ==== 135+0+0 (secure 0 0 0) 0x7f86380596a0 con 0x7f8634004160 2024-09-16T01:52:20.453 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.7 to 0.234436 (3c03) 2024-09-16T01:52:20.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.461+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f862c06c2e0 msgr2=0x7f862c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.461+0000 7f8653400640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f862c06c2e0 0x7f862c06e7a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f864c1ea5a0 tx=0x7f8640008040 comp rx=0 tx=0).stop 2024-09-16T01:52:20.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.461+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 msgr2=0x7f864c1efee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.461+0000 7f8653400640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f864c1efee0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f863800d720 tx=0x7f863800dbe0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.462+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 shutdown_connections 2024-09-16T01:52:20.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.462+0000 7f8653400640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f862c06c2e0 0x7f862c06e7a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.462+0000 7f8653400640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f864c0c9650 0x7f864c0c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.462+0000 7f8653400640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f864c1501b0 0x7f864c0b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.462+0000 7f8653400640 1 --2- 172.21.15.146:0/3435850479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8634004160 0x7f864c1efee0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.462+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 >> 172.21.15.146:0/3435850479 conn(0x7f864c06d560 msgr2=0x7f864c1521b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:20.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.463+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 shutdown_connections 2024-09-16T01:52:20.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.463+0000 7f8653400640 1 -- 172.21.15.146:0/3435850479 wait complete. 2024-09-16T01:52:20.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:52:20.473 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:20.472+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:20.574 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:20.566+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:20.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:20.668 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:20.667+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:20.781 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:20.780+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:20.788 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:20.789 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:20.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.818+0000 7f4f2fc00640 1 Processor -- start 2024-09-16T01:52:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.819+0000 7f4f2fc00640 1 -- start start 2024-09-16T01:52:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2fc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4f200d8920 0x7f4f200df130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2fc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f2004a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2fc00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f2004adc0 con 0x7f4f20050260 2024-09-16T01:52:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2fc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4f2004a3f0 con 0x7f4f200d8920 2024-09-16T01:52:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2fc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4f20049b00 con 0x7f4f20005db0 2024-09-16T01:52:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2dc00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4f20050260 0x7f4f20048fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:34928/0 (socket says 172.21.15.146:34928) 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2dc00640 1 -- 172.21.15.146:0/4144688743 learned_addr learned my addr 172.21.15.146:0/4144688743 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2d200640 1 --2- 172.21.15.146:0/4144688743 >> v2:172.21.15.161:3300/0 conn(0x7f4f200d8920 0x7f4f200df130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f27e00640 1 --2- 172.21.15.146:0/4144688743 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f2004a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f27e00640 1 -- 172.21.15.146:0/4144688743 >> v2:172.21.15.161:3300/0 conn(0x7f4f200d8920 msgr2=0x7f4f200df130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f27e00640 1 --2- 172.21.15.146:0/4144688743 >> v2:172.21.15.161:3300/0 conn(0x7f4f200d8920 0x7f4f200df130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f27e00640 1 -- 172.21.15.146:0/4144688743 >> v1:172.21.15.161:6789/0 conn(0x7f4f20050260 legacy=0x7f4f20048fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f27e00640 1 -- 172.21.15.146:0/4144688743 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2004f0a0 con 0x7f4f20005db0 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.821+0000 7f4f2d200640 1 --2- 172.21.15.146:0/4144688743 >> v2:172.21.15.161:3300/0 conn(0x7f4f200d8920 0x7f4f200df130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.823+0000 7f4f27e00640 1 --2- 172.21.15.146:0/4144688743 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f2004a880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4f140097a0 tx=0x7f4f1402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb1ac56168c15382 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.823+0000 7f4f27400640 1 -- 172.21.15.146:0/4144688743 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14037070 con 0x7f4f20005db0 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.823+0000 7f4f27400640 1 -- 172.21.15.146:0/4144688743 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f14032440 con 0x7f4f20005db0 2024-09-16T01:52:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.823+0000 7f4f27400640 1 -- 172.21.15.146:0/4144688743 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14032810 con 0x7f4f20005db0 2024-09-16T01:52:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.824+0000 7f4f2fc00640 1 -- 172.21.15.146:0/4144688743 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 msgr2=0x7f4f2004a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.824+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/4144688743 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f2004a880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4f140097a0 tx=0x7f4f1402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.824+0000 7f4f2fc00640 1 -- 172.21.15.146:0/4144688743 shutdown_connections 2024-09-16T01:52:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.824+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/4144688743 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f2004a880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.824+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/4144688743 >> v2:172.21.15.161:3300/0 conn(0x7f4f200d8920 0x7f4f200df130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.824+0000 7f4f2fc00640 1 -- 172.21.15.146:0/4144688743 >> 172.21.15.146:0/4144688743 conn(0x7f4f20054080 msgr2=0x7f4f20054460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.826+0000 7f4f2fc00640 1 -- 172.21.15.146:0/4144688743 shutdown_connections 2024-09-16T01:52:20.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.826+0000 7f4f2fc00640 1 -- 172.21.15.146:0/4144688743 wait complete. 2024-09-16T01:52:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.829+0000 7f4f2fc00640 1 Processor -- start 2024-09-16T01:52:20.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.829+0000 7f4f2fc00640 1 -- start start 2024-09-16T01:52:20.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.831+0000 7f4f2fc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f20174500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.832+0000 7f4f2fc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f20050260 0x7f4f200cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.832+0000 7f4f2fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 0x7f4f200d3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.833+0000 7f4f2dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 0x7f4f200d3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.833+0000 7f4f2dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 0x7f4f200d3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41800/0 (socket says 172.21.15.146:41800) 2024-09-16T01:52:20.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.833+0000 7f4f2dc00640 1 -- 172.21.15.146:0/1687549136 learned_addr learned my addr 172.21.15.146:0/1687549136 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:20.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.833+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2004a080 con 0x7f4f200d8920 2024-09-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.833+0000 7f4f2d200640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f20174500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.833+0000 7f4f27e00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f20050260 0x7f4f200cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.834+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4f20049da0 con 0x7f4f20050260 2024-09-16T01:52:20.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.834+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4f200d8d00 con 0x7f4f20005db0 2024-09-16T01:52:20.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.835+0000 7f4f2dc00640 1 -- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 msgr2=0x7f4f20174500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.835+0000 7f4f2dc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f20174500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.835+0000 7f4f2dc00640 1 -- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f20050260 msgr2=0x7f4f200cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.835+0000 7f4f2dc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f20050260 0x7f4f200cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.835+0000 7f4f2dc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f200cda90 con 0x7f4f200d8920 2024-09-16T01:52:20.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.835+0000 7f4f2dc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 0x7f4f200d3ba0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4f2806b730 tx=0x7f4f280714b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.836+0000 7f4f2d200640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f20174500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:20.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.836+0000 7f4f27e00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f20050260 0x7f4f200cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:20.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.836+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28091070 con 0x7f4f200d8920 2024-09-16T01:52:20.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.836+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f28074260 con 0x7f4f200d8920 2024-09-16T01:52:20.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.836+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28071b90 con 0x7f4f200d8920 2024-09-16T01:52:20.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.837+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f200cd4d0 con 0x7f4f200d8920 2024-09-16T01:52:20.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.837+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f200d2a70 con 0x7f4f200d8920 2024-09-16T01:52:20.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.839+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f20050710 con 0x7f4f200d8920 2024-09-16T01:52:20.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.841+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4f28094070 con 0x7f4f200d8920 2024-09-16T01:52:20.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.842+0000 7f4f24c00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4f0806c300 0x7f4f0806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.845+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5741+0+0 (secure 0 0 0) 0x7f4f28071df0 con 0x7f4f200d8920 2024-09-16T01:52:20.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.846+0000 7f4f2d200640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4f0806c300 0x7f4f0806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.848+0000 7f4f2d200640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4f0806c300 0x7f4f0806e7c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009ac0 tx=0x7f4f1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.853+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4f280d5d90 con 0x7f4f200d8920 2024-09-16T01:52:20.874 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:20.873+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:20.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.934+0000 7f3856600640 1 Processor -- start 2024-09-16T01:52:20.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.934+0000 7f3856600640 1 -- start start 2024-09-16T01:52:20.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.935+0000 7f3856600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3850154c10 0x7f385015b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f3856600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385015e0b0 con 0x7f38500cafa0 2024-09-16T01:52:20.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f3856600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385015e390 con 0x7f38500cca70 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f3856600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f385015de60 con 0x7f3850154c10 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f384f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3850154c10 0x7f385015b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f384f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3850154c10 0x7f385015b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51864/0 (socket says 172.21.15.146:51864) 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f384f400640 1 -- 172.21.15.146:0/1794362404 learned_addr learned my addr 172.21.15.146:0/1794362404 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f384f400640 1 -- 172.21.15.146:0/1794362404 >> v1:172.21.15.161:6789/0 conn(0x7f38500cca70 legacy=0x7f3850073340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f384f400640 1 -- 172.21.15.146:0/1794362404 >> v1:172.21.15.146:6789/0 conn(0x7f38500cafa0 legacy=0x7f38500cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.936+0000 7f384f400640 1 -- 172.21.15.146:0/1794362404 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385015e670 con 0x7f3850154c10 2024-09-16T01:52:20.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.938+0000 7f384f400640 1 --2- 172.21.15.146:0/1794362404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 0x7f385015b370 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3840009870 tx=0x7f384002a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f5abf2720496f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.938+0000 7f384e000640 1 -- 172.21.15.146:0/1794362404 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840037070 con 0x7f3850154c10 2024-09-16T01:52:20.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.938+0000 7f384e000640 1 -- 172.21.15.146:0/1794362404 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3840032640 con 0x7f3850154c10 2024-09-16T01:52:20.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.938+0000 7f384e000640 1 -- 172.21.15.146:0/1794362404 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840032a10 con 0x7f3850154c10 2024-09-16T01:52:20.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 -- 172.21.15.146:0/1794362404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 msgr2=0x7f385015b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 --2- 172.21.15.146:0/1794362404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 0x7f385015b370 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3840009870 tx=0x7f384002a300 comp rx=0 tx=0).stop 2024-09-16T01:52:20.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 -- 172.21.15.146:0/1794362404 shutdown_connections 2024-09-16T01:52:20.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 --2- 172.21.15.146:0/1794362404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 0x7f385015b370 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 -- 172.21.15.146:0/1794362404 >> 172.21.15.146:0/1794362404 conn(0x7f385006ce80 msgr2=0x7f385006d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:20.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 -- 172.21.15.146:0/1794362404 shutdown_connections 2024-09-16T01:52:20.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.939+0000 7f3856600640 1 -- 172.21.15.146:0/1794362404 wait complete. 2024-09-16T01:52:20.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.941+0000 7f3856600640 1 Processor -- start 2024-09-16T01:52:20.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.941+0000 7f3856600640 1 -- start start 2024-09-16T01:52:20.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.942+0000 7f3856600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 0x7f38500b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.942+0000 7f3856600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38500cca70 0x7f385015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.942+0000 7f3856600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 0x7f38500cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.942+0000 7f3856600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3850154ff0 con 0x7f38500cca70 2024-09-16T01:52:20.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.942+0000 7f3856600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f385015ec30 con 0x7f3850154c10 2024-09-16T01:52:20.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.942+0000 7f3856600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3850073a50 con 0x7f38500cafa0 2024-09-16T01:52:20.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 0x7f38500b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 0x7f38500b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:36364/0 (socket says 172.21.15.146:36364) 2024-09-16T01:52:20.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 -- 172.21.15.146:0/1879998342 learned_addr learned my addr 172.21.15.146:0/1879998342 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:20.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 0x7f38500cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38500cca70 0x7f385015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 msgr2=0x7f38500cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3850154c10 0x7f38500cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38500cca70 msgr2=0x7f385015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38500cca70 0x7f385015d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384f400640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385015c450 con 0x7f38500cafa0 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.943+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38500cca70 0x7f385015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:20.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.944+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 0x7f38500b5b60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3840000c00 tx=0x7f38400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.944+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840041070 con 0x7f38500cafa0 2024-09-16T01:52:20.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.944+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384003c870 con 0x7f38500cafa0 2024-09-16T01:52:20.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.944+0000 7f3856600640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38500b5610 con 0x7f38500cafa0 2024-09-16T01:52:20.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.944+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840037040 con 0x7f38500cafa0 2024-09-16T01:52:20.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.944+0000 7f3856600640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f385015baf0 con 0x7f38500cafa0 2024-09-16T01:52:20.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.945+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3824005270 con 0x7f38500cafa0 2024-09-16T01:52:20.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.951+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3840044070 con 0x7f38500cafa0 2024-09-16T01:52:20.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.952+0000 7f3847400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f383006c2b0 0x7f383006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.953+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(99..99 src has 1..99) ==== 5741+0+0 (secure 0 0 0) 0x7f38400ac410 con 0x7f38500cafa0 2024-09-16T01:52:20.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.953+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v99) ==== 54+0+16 (secure 0 0 0) 0x7f38400ac800 con 0x7f38500cafa0 2024-09-16T01:52:20.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.954+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f383006c2b0 0x7f383006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.956+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f383006c2b0 0x7f383006e770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f383c006fd0 tx=0x7f383c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.956+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3824000fa0 0x7f38240045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.956+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3824005550 con 0x7f3824000fa0 2024-09-16T01:52:20.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.958+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3824000fa0 0x7f38240045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.959+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3824000fa0 0x7f38240045e0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.961+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3824005550 con 0x7f3824000fa0 2024-09-16T01:52:20.973 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:20.972+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:20.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.989+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3824004f70 con 0x7f3824000fa0 2024-09-16T01:52:20.994 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:20.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.992+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3824004f70 con 0x7f3824000fa0 2024-09-16T01:52:20.995 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:20.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.994+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f3824001cb0 0x7f3824002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:20.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.994+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3824002300 con 0x7f3824001cb0 2024-09-16T01:52:20.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.994+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f3824001cb0 0x7f3824002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:20.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.995+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f3824001cb0 0x7f3824002990 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:20.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:20.996+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3824002300 con 0x7f3824001cb0 2024-09-16T01:52:21.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.030+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3824003150 con 0x7f3824001cb0 2024-09-16T01:52:21.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.032+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3824003150 con 0x7f3824001cb0 2024-09-16T01:52:21.033 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:21.034 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:21.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.033+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f3824009c80 0x7f382400c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:21.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.033+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f382400ee40 con 0x7f3824009c80 2024-09-16T01:52:21.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.033+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f3824009c80 0x7f382400c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:21.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.034+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f3824009c80 0x7f382400c070 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:21.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.035+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f382400ee40 con 0x7f3824009c80 2024-09-16T01:52:21.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.063+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f382400c880 con 0x7f3824009c80 2024-09-16T01:52:21.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.067+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f382400c880 con 0x7f3824009c80 2024-09-16T01:52:21.068 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:21.068 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:21.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.067+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f382400d0d0 0x7f382400f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:21.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.067+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3824012610 con 0x7f382400d0d0 2024-09-16T01:52:21.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.067+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f382400d0d0 0x7f382400f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:21.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.069+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f382400d0d0 0x7f382400f570 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:21.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.070+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3824012610 con 0x7f382400d0d0 2024-09-16T01:52:21.072 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:21.071+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:21.077 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:21.077 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:21.077 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:21.077 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:21.077 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:21.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.099+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f38240113b0 con 0x7f382400d0d0 2024-09-16T01:52:21.107 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.101+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f38240113b0 con 0x7f382400d0d0 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.101+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3824011ef0 0x7f382400fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.101+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3824015f70 con 0x7f3824011ef0 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.101+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3824011ef0 0x7f382400fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.102+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3824011ef0 0x7f382400fbf0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.103+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3824015f70 con 0x7f3824011ef0 2024-09-16T01:52:21.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.128+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3824010130 con 0x7f3824011ef0 2024-09-16T01:52:21.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.130+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3824010130 con 0x7f3824011ef0 2024-09-16T01:52:21.131 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:21.131 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:21.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.130+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f3824010850 0x7f3824010cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:21.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.130+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3824019180 con 0x7f3824010850 2024-09-16T01:52:21.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.131+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f3824010850 0x7f3824010cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:21.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.132+0000 7f384ea00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f3824010850 0x7f3824010cd0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:21.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.133+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3824019180 con 0x7f3824010850 2024-09-16T01:52:21.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.155+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f38240168f0 con 0x7f3824010850 2024-09-16T01:52:21.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.157+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f38240168f0 con 0x7f3824010850 2024-09-16T01:52:21.158 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:21.159 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:21.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.158+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3824017010 0x7f3824017430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:21.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.158+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f382401c5a0 con 0x7f3824017010 2024-09-16T01:52:21.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.159+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3824017010 0x7f3824017430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:21.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.160+0000 7f384fe00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3824017010 0x7f3824017430 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:21.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.161+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f382401c5a0 con 0x7f3824017010 2024-09-16T01:52:21.173 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:21.172+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:21.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.188+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3824017a80 con 0x7f3824017010 2024-09-16T01:52:21.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.190+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3824017a80 con 0x7f3824017010 2024-09-16T01:52:21.191 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:21.191 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:21.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.190+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f3824018100 0x7f3824018540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:21.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.190+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3824018bc0 con 0x7f3824018100 2024-09-16T01:52:21.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.190+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f3824018100 0x7f3824018540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:21.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.191+0000 7f384f400640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f3824018100 0x7f3824018540 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:21.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.192+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3824018bc0 con 0x7f3824018100 2024-09-16T01:52:21.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.225+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3824018b90 con 0x7f3824018100 2024-09-16T01:52:21.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.227+0000 7f3847400640 1 -- 172.21.15.146:0/1879998342 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3824018b90 con 0x7f3824018100 2024-09-16T01:52:21.228 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3824000fa0 msgr2=0x7f38240045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3824000fa0 0x7f38240045e0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f3824001cb0 msgr2=0x7f3824002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f3824001cb0 0x7f3824002990 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f3824009c80 msgr2=0x7f382400c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f3824009c80 0x7f382400c070 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f382400d0d0 msgr2=0x7f382400f570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f382400d0d0 0x7f382400f570 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3824011ef0 msgr2=0x7f382400fbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f3824011ef0 0x7f382400fbf0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f3824010850 msgr2=0x7f3824010cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f3824010850 0x7f3824010cd0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3824017010 msgr2=0x7f3824017430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3824017010 0x7f3824017430 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f3824018100 msgr2=0x7f3824018540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f3824018100 0x7f3824018540 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f384fe00640 1 -- 172.21.15.146:0/1879998342 reap_dead start 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f384fe00640 1 -- 172.21.15.146:0/1879998342 reap_dead start 2024-09-16T01:52:21.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f384fe00640 1 -- 172.21.15.146:0/1879998342 reap_dead start 2024-09-16T01:52:21.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f384fe00640 1 -- 172.21.15.146:0/1879998342 reap_dead start 2024-09-16T01:52:21.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f384fe00640 1 -- 172.21.15.146:0/1879998342 reap_dead start 2024-09-16T01:52:21.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f383006c2b0 msgr2=0x7f383006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f383006c2b0 0x7f383006e770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f383c006fd0 tx=0x7f383c008040 comp rx=0 tx=0).stop 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 msgr2=0x7f38500b5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.228+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 0x7f38500b5b60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3840000c00 tx=0x7f38400046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.229+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 shutdown_connections 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.229+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f383006c2b0 0x7f383006e770 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.229+0000 7f3844c00640 1 --2- 172.21.15.146:0/1879998342 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38500cafa0 0x7f38500b5b60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.229+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 >> 172.21.15.146:0/1879998342 conn(0x7f385006ce80 msgr2=0x7f38500c6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.229+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 shutdown_connections 2024-09-16T01:52:21.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.229+0000 7f3844c00640 1 -- 172.21.15.146:0/1879998342 wait complete. 2024-09-16T01:52:21.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.235+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f4f2004faf0 con 0x7f4f200d8920 2024-09-16T01:52:21.273 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:21.272+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:21.375 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:21.374+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:21.416 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:21.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:478 3.c 3:32521e74:::10000000002.000000db:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 81.386058814999998, 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.88174018200000004, 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 478 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025090+0000", 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2699999999999998e-07 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047858+0000", 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022768360000000001 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047892+0000", 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3855e-05 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047918+0000", 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5595000000000001e-05 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048026+0000", 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108242 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048035+0000", 2024-09-16T01:52:21.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3940000000000002e-06 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828462+0000", 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.78042635999999999 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831445+0000", 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029837000000000002 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882498+0000", 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051052510000000002 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882597+0000", 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8874000000000002e-05 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882746+0000", 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149457 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.903108+0000", 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020361516 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906632+0000", 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035239960000000002 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906720+0000", 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7952000000000001e-05 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906829+0000", 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109444 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:777 3.2c 3:36199bb1:::100000001f7.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.198682902999998, 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106547593, 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.424 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 777 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212467+0000", 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1399999999999998e-06 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221218+0000", 2024-09-16T01:52:21.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087507980000000006 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221238+0000", 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9729999999999999e-05 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221249+0000", 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1462e-05 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222669+0000", 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001420017 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222797+0000", 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127572 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222974+0000", 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017753699999999999 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.258753+0000", 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035779068999999997 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318898+0000", 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060144465000000001 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318963+0000", 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5098999999999995e-05 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.319012+0000", 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9703999999999997e-05 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1127 3.36 3:6e2a8853:::100000001f8.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.106954512999998, 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15931991200000001, 2024-09-16T01:52:21.430 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1127 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304196+0000", 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.526e-06 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354927+0000", 2024-09-16T01:52:21.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050730866999999999 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355561+0000", 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063468300000000004 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355571+0000", 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6199999999999994e-06 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355665+0000", 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3875000000000005e-05 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355819+0000", 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015463900000000001 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355964+0000", 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144241 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.393080+0000", 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037116647000000003 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463398+0000", 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070317222999999998 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463456+0000", 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8749999999999998e-05 2024-09-16T01:52:21.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463513+0000", 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6841e-05 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1218 3.3f 3:ffd44cf4:::100000001f8.000000bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.130773472000001, 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.121928352, 2024-09-16T01:52:21.436 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1218 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280376+0000", 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8980000000000001e-06 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360384+0000", 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080007935000000002 2024-09-16T01:52:21.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360415+0000", 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1007999999999999e-05 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360429+0000", 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4233000000000001e-05 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360518+0000", 2024-09-16T01:52:21.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8465000000000001e-05 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360685+0000", 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016704299999999999 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360842+0000", 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157496 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.379376+0000", 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018533701999999999 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402192+0000", 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022815932000000001 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402254+0000", 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2378000000000005e-05 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402303+0000", 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8262e-05 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1460 3.2a 3:574d3868:::100000001f8.000001b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.024780638000003, 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114279699, 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.442 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1460 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386369+0000", 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1969999999999999e-06 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447453+0000", 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061083239999999997 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447479+0000", 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5941000000000001e-05 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447490+0000", 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1967000000000001e-05 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447627+0000", 2024-09-16T01:52:21.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013671700000000001 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447784+0000", 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015693 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447948+0000", 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016412100000000001 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.467094+0000", 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019145590000000001 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.497455+0000", 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030361091 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.499539+0000", 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020837780000000001 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.500647+0000", 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108127 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1474 3.39 3:9e943776:::100000001f8.000001bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.829467190999999, 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.116146155, 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.448 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1474 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581683+0000", 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6740000000000001e-06 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639340+0000", 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057656583999999997 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639448+0000", 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107707 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639499+0000", 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1391999999999999e-05 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648061+0000", 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085624870000000006 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648374+0000", 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000312307 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648681+0000", 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030683700000000002 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.693784+0000", 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045103740000000003 2024-09-16T01:52:21.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697710+0000", 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039253650000000001 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697780+0000", 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0134000000000003e-05 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697827+0000", 2024-09-16T01:52:21.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6928000000000002e-05 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1925 3.39 3:9ea071c3:::100000001f8.00000382:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.531084583000002, 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40497902499999999, 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.454 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1925 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880065+0000", 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7120000000000001e-06 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905598+0000", 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025532966000000001 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905629+0000", 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0797999999999999e-05 2024-09-16T01:52:21.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905642+0000", 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905790+0000", 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014842100000000001 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:21.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905804+0000", 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3781000000000001e-05 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184322+0000", 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27851797299999997 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184507+0000", 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185203 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184707+0000", 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199287 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274277+0000", 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089570446999999997 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.284842+0000", 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010564756999999999 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.284919+0000", 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6735000000000003e-05 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.285042+0000", 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123626 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1928 3.1b 3:dae8eecd:::100000001f8.00000385:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.501911194000002, 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44792130200000002, 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1928 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909238+0000", 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.804e-06 2024-09-16T01:52:21.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937404+0000", 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028165758999999999 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937432+0000", 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7718000000000001e-05 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937443+0000", 2024-09-16T01:52:21.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0923e-05 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937583+0000", 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014051800000000001 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937597+0000", 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.367e-05 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321633+0000", 2024-09-16T01:52:21.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.38403599100000002 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321793+0000", 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015996000000000001 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321969+0000", 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017598599999999999 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340671+0000", 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018701613999999998 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.356957+0000", 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016286421999999998 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.357041+0000", 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3857999999999996e-05 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.357158+0000", 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011707899999999999 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1931 3.39 3:9d369e22:::100000001f8.00000388:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.473378332000003, 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40258944099999999, 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.467 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1931 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:52:21.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937771+0000", 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2839999999999999e-06 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961925+0000", 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02415401 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961962+0000", 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6869000000000002e-05 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961975+0000", 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.361e-05 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.962078+0000", 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102604 2024-09-16T01:52:21.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.962091+0000", 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3216e-05 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.194767+0000", 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.23267554400000001 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.194914+0000", 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147501 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.195084+0000", 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016995000000000001 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274589+0000", 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079504732999999994 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340176+0000", 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065587200999999998 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340241+0000", 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4921999999999999e-05 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340359+0000", 2024-09-16T01:52:21.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117997 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1932 3.39 3:9da8c7a0:::100000001f8.00000389:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.443487628, 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.37297202400000001, 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.474 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1932 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967663+0000", 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4660000000000002e-06 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990244+0000", 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022581615999999999 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990262+0000", 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7672999999999998e-05 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990271+0000", 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8410000000000006e-06 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990369+0000", 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7793000000000004e-05 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990377+0000", 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5040000000000002e-06 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217601+0000", 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22722361699999999 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217755+0000", 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015434699999999999 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217927+0000", 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017202499999999999 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274595+0000", 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056667704999999999 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340459+0000", 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065864332999999997 2024-09-16T01:52:21.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340504+0000", 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5265000000000001e-05 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340632+0000", 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012783900000000001 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1938 3.39 3:9faf1682:::100000001f8.0000038f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.360500483000003, 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.297091671, 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1938 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050649+0000", 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6559999999999999e-06 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067265+0000", 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016615959 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067308+0000", 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3154000000000001e-05 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067333+0000", 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5364e-05 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067439+0000", 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106051 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067453+0000", 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3093e-05 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226329+0000", 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15887599599999999 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226492+0000", 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016327299999999999 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226687+0000", 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019515299999999999 2024-09-16T01:52:21.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.323930+0000", 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097242771000000006 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347547+0000", 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023617610000000001 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347630+0000", 2024-09-16T01:52:21.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2270999999999999e-05 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347739+0000", 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010932 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1944 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.295330645, 2024-09-16T01:52:21.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.23495701299999999, 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1944 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115819+0000", 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.26e-06 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189079+0000", 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073259493999999994 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189092+0000", 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2648e-05 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189102+0000", 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0377000000000001e-05 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191220+0000", 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021178830000000001 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191347+0000", 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127391 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191478+0000", 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013041699999999999 2024-09-16T01:52:21.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.211598+0000", 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020119978 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350671+0000", 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.139073536 2024-09-16T01:52:21.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350725+0000", 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4144000000000002e-05 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350774+0000", 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8884999999999997e-05 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1949 3.2f e84/20 3:f7d79abc:::100000001f8.0000039a:head v 84'35, mlcod=81'33)", 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.182413515999997, 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107175486, 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228736+0000", 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.522e-06 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230216+0000", 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001480592 2024-09-16T01:52:21.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230232+0000", 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5648999999999999e-05 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230241+0000", 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7749999999999998e-06 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230314+0000", 2024-09-16T01:52:21.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3069000000000001e-05 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230342+0000", 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.828e-05 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.335832+0000", 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10549030299999999 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.335910+0000", 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7296e-05 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2088 3.3b 3:de25c16a:::100000001f8.00000425:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.184932252999999, 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.113643656, 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2088 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226217+0000", 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8899999999999999e-06 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268408+0000", 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042191008000000002 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268436+0000", 2024-09-16T01:52:21.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7183999999999999e-05 2024-09-16T01:52:21.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268447+0000", 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1233000000000001e-05 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268657+0000", 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020985900000000001 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.269211+0000", 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055385500000000002 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.269780+0000", 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00056933999999999999 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.333074+0000", 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063294230000000007 2024-09-16T01:52:21.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339717+0000", 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066431490000000001 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339798+0000", 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0920000000000005e-05 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339859+0000", 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0988e-05 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2445 3.2a 3:566ba4fe:::100000001f8.0000058a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.982997464, 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10738498, 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.509 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2445 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428152+0000", 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5209999999999999e-06 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455890+0000", 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027737906 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455910+0000", 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0023000000000001e-05 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455920+0000", 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0628999999999999e-05 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455993+0000", 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2358000000000001e-05 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.456334+0000", 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034162400000000001 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.456560+0000", 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022543699999999999 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.489902+0000", 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033341844000000002 2024-09-16T01:52:21.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535444+0000", 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045542250999999999 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535495+0000", 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0973000000000001e-05 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535535+0000", 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0414000000000003e-05 2024-09-16T01:52:21.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2832 3.1b 3:da939a2a:::100000001f8.0000070d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.625776+0000", 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.785372221999999, 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12403525999999999, 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.515 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2832 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.625776+0000", 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.625776+0000", 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.625777+0000", 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7e-06 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681277+0000", 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055500081999999999 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681304+0000", 2024-09-16T01:52:21.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6222999999999999e-05 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681316+0000", 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2237e-05 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681401+0000", 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4767999999999999e-05 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681596+0000", 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019540000000000001 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681778+0000", 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000181726 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.725894+0000", 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044116796999999999 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.749693+0000", 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02379848 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.749761+0000", 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8441000000000004e-05 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.749811+0000", 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9406000000000001e-05 2024-09-16T01:52:21.520 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2855 3.f 3:f27ce28a:::100000001f8.00000724:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.873661+0000", 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.537487039999998, 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.100443244, 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.521 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2855 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.873661+0000", 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.873661+0000", 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.873662+0000", 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2219999999999999e-06 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914147+0000", 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040484617000000001 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914171+0000", 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914180+0000", 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4999999999999999e-06 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914282+0000", 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102024 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914451+0000", 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016926899999999999 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914626+0000", 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175427 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.939551+0000", 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024924512999999999 2024-09-16T01:52:21.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973794+0000", 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034243705999999999 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974058+0000", 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026316300000000001 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974104+0000", 2024-09-16T01:52:21.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6349e-05 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3319 3.c 3:31034f07:::100000001f8.000008f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.215954+0000", 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.195193920000001, 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109990009, 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.215954+0000", 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.215954+0000", 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.215956+0000", 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0619999999999999e-06 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290610+0000", 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074654478999999996 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290645+0000", 2024-09-16T01:52:21.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4718999999999997e-05 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290660+0000", 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5279000000000001e-05 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290746+0000", 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5953999999999997e-05 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290930+0000", 2024-09-16T01:52:21.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018327799999999999 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.291117+0000", 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018774800000000001 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.300434+0000", 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093171179999999992 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.325761+0000", 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025326284000000001 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.325824+0000", 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3721999999999997e-05 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.325944+0000", 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119366 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3326 3.3b 3:dcfe8cbf:::100000001f8.000008fb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.299731+0000", 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.111416471999998, 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12301342799999999, 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3326 2024-09-16T01:52:21.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.299731+0000", 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.299731+0000", 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.299733+0000", 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.968e-06 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.355860+0000", 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056127019 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356421+0000", 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00056061000000000001 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356511+0000", 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9658000000000001e-05 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356639+0000", 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012836799999999999 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356807+0000", 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016850900000000001 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356970+0000", 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.406342+0000", 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049371874000000003 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422625+0000", 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016282613000000001 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422689+0000", 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4302999999999997e-05 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422745+0000", 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5506e-05 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3500 3.2a 3:55cec617:::100000001f8.000009a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e93)", 2024-09-16T01:52:21.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:07.403698+0000", 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.007449825, 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.111072217, 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3500 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.403698+0000", 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.403698+0000", 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.403700+0000", 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2869999999999999e-06 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439887+0000", 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036186907999999997 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439911+0000", 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4117e-05 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439921+0000", 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3740000000000004e-06 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439996+0000", 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5003e-05 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.440179+0000", 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018289599999999999 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.440354+0000", 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175179 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.455926+0000", 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015572236 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.514650+0000", 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058723643999999998 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.514719+0000", 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-09-16T01:52:21.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.514770+0000", 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0957e-05 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.544 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.446+0000 7f4f24c00640 1 -- 172.21.15.146:0/1687549136 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v100) ==== 97+0+0 (secure 0 0 0) 0x7f4f280d5690 con 0x7f4f200d8920 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.452+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4f0806c300 msgr2=0x7f4f0806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.452+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4f0806c300 0x7f4f0806e7c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009ac0 tx=0x7f4f1c008040 comp rx=0 tx=0).stop 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.452+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 msgr2=0x7f4f200d3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.452+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 0x7f4f200d3ba0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4f2806b730 tx=0x7f4f280714b0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.456+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 shutdown_connections 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.456+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4f0806c300 0x7f4f0806e7c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.456+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f200d8920 0x7f4f200d3ba0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.456+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4f20050260 0x7f4f200cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.456+0000 7f4f2fc00640 1 --2- 172.21.15.146:0/1687549136 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f20005db0 0x7f4f20174500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:21.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.456+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 >> 172.21.15.146:0/1687549136 conn(0x7f4f20054080 msgr2=0x7f4f200cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:21.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.461+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 shutdown_connections 2024-09-16T01:52:21.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:21.461+0000 7f4f2fc00640 1 -- 172.21.15.146:0/1687549136 wait complete. 2024-09-16T01:52:21.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:21.549 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:21.544+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:21.650 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:21.647+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:21.750 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:21.748+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:21.856 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:21.855+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:21.929 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:21.929 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:21.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.929 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4149 3.3d e100/91 3:bd018e47:::100000001f8.00000232:head v 100'79, mlcod=99'77)", 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.885946+0000", 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.042509311000000001, 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042541476000000002, 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.885946+0000", 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.885946+0000", 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.885946+0000", 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2600000000000001e-07 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.898958+0000", 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013011695 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.898973+0000", 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4688999999999999e-05 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.898981+0000", 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1659999999999994e-06 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.900725+0000", 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0017444470000000001 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.900751+0000", 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5752e-05 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.933 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "MOSDPGPush(3.b 100/99 [PushOp(3:d1e869e5:::100000001f8.00000397:head, version: 84'47, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:d1e869e5:::100000001f8.00000397:head@84'47, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.926964+0000", 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.001491341, 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016255149999999999, 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.926964+0000", 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.926964+0000", 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.926965+0000", 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.451e-06 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.927746+0000", 2024-09-16T01:52:21.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00078128300000000002 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.927781+0000", 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4193000000000003e-05 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.927799+0000", 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8442000000000001e-05 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.928094+0000", 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00029515200000000002 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.928103+0000", 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9239999999999996e-06 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 2 2024-09-16T01:52:21.937 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:21.950 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:21.949+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:21.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:22.050 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:22.049+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:22.151 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:22.151+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:22.255 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:22.253+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:22.301 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:22.301 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:22.301 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:22.302 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:22.302 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:22.326 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:22.353 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:22.352+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:22.455 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:22.453+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:22.556 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:22.555+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:22.656 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:22.655+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:22.721 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 214.94751865800001, 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.722 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:22.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:52:22.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:52:22.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.330066317999993, 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:52:22.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:52:22.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:52:22.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:52:22.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:22.736 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.538049368999999, 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:22.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:52:22.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.502516745999998, 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.743 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:52:22.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:22.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:52:22.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:52:22.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1461 3.1c 3:39a1ca69:::100000001f8.000001b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.305175951000002, 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105254967, 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.749 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1461 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410697+0000", 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.410699+0000", 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.232e-06 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476134+0000", 2024-09-16T01:52:22.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065435190000000004 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476173+0000", 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8878999999999998e-05 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476186+0000", 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3016e-05 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.752 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476284+0000", 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7583000000000004e-05 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476440+0000", 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015621799999999999 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.476629+0000", 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018937399999999999 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.505615+0000", 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028986212000000001 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515833+0000", 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010217944 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515901+0000", 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7250999999999997e-05 2024-09-16T01:52:22.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.515952+0000", 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1068e-05 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.136427808999997, 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:52:22.755 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:52:22.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:52:22.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:22.761 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.572216271000002, 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:52:22.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:52:22.767 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.510428969000003, 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:52:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:52:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1931 3.39 e84/83 3:9d369e22:::100000001f8.00000388:head v 84'56, mlcod=0'0)", 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.487809313, 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110959082, 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.772 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228064+0000", 2024-09-16T01:52:22.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5499999999999997e-07 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229687+0000", 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00162302 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229706+0000", 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8459999999999999e-05 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229719+0000", 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2986000000000001e-05 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229824+0000", 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105163 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229850+0000", 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6621000000000001e-05 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.338768+0000", 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108917369 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339022+0000", 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025460800000000003 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1932 3.39 e84/83 3:9da8c7a0:::100000001f8.00000389:head v 84'57, mlcod=0'0)", 2024-09-16T01:52:22.776 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.480827372, 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10422508900000001, 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235045+0000", 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.235046+0000", 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0589999999999999e-06 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245188+0000", 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010141739 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245228+0000", 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9755999999999997e-05 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245244+0000", 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6475000000000001e-05 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.249380+0000", 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041353170000000003 2024-09-16T01:52:22.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.249407+0000", 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7586e-05 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339027+0000", 2024-09-16T01:52:22.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089619521999999993 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339270+0000", 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024363499999999999 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1938 3.39 e84/83 3:9faf1682:::100000001f8.0000038f:head v 84'58, mlcod=0'0)", 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.470582557, 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.101947118, 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245290+0000", 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.245292+0000", 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5030000000000001e-06 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253345+0000", 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0080530840000000003 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253363+0000", 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8281999999999999e-05 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.253379+0000", 2024-09-16T01:52:22.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6252999999999998e-05 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254278+0000", 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00089897700000000002 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254304+0000", 2024-09-16T01:52:22.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5559e-05 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347185+0000", 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.092881693000000001 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347237+0000", 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1767000000000001e-05 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.668646523, 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:52:22.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:52:22.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.792 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2089 3.37 3:ee65e5cb:::100000001f8.00000426:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.224209+0000", 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.491663984999999, 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.105946261, 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.793 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2089 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.224209+0000", 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.224209+0000", 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.224211+0000", 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7549999999999999e-06 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.247234+0000", 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023022918 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.247265+0000", 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0639999999999998e-05 2024-09-16T01:52:22.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.247278+0000", 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2976000000000001e-05 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.250654+0000", 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003375992 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.250827+0000", 2024-09-16T01:52:22.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000173251 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.250979+0000", 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151731 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.304799+0000", 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053819909999999999 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330020+0000", 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025221323 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330093+0000", 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2653999999999999e-05 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.330155+0000", 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2111000000000002e-05 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.92520777, 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:52:22.804 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2857 3.1c 3:39a72f86:::100000001f8.00000726:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.853576367999999, 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11170522400000001, 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.805 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2857 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862299+0000", 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.351e-06 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890188+0000", 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027888969999999999 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890218+0000", 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0593999999999999e-05 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890230+0000", 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1688999999999999e-05 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890467+0000", 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023695699999999999 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.891129+0000", 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066218699999999995 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.891811+0000", 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068177899999999998 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912951+0000", 2024-09-16T01:52:22.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021140665999999999 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973903+0000", 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060951181 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973961+0000", 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7989999999999999e-05 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974001+0000", 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0859999999999998e-05 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.8809868239999998, 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.811 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:52:22.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.8725703380000001, 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.817 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:52:22.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:52:22.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3678 3.3d e97/91 3:bd5b01a1:::100000001f8.0000005b:head v 97'72, mlcod=96'70)", 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.879403+0000", 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.8364696450000002, 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106394118, 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.879403+0000", 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.879403+0000", 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.879405+0000", 2024-09-16T01:52:22.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9410000000000001e-06 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.939178+0000", 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059773197 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.940634+0000", 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001455708 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.940709+0000", 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5426000000000006e-05 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.944640+0000", 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039306360000000004 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.944670+0000", 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0599999999999998e-05 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.985736+0000", 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041065958 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.985797+0000", 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0652000000000001e-05 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3770 3.37 3:ec19b1da:::100000001f8.000000b7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:22.827 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:17.008166+0000", 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.707706935, 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109566236, 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3770 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.008166+0000", 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.008166+0000", 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.008167+0000", 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7829999999999999e-06 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050683+0000", 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042515404999999999 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050714+0000", 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1433000000000002e-05 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050726+0000", 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1919999999999999e-05 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050795+0000", 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9067999999999997e-05 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050856+0000", 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0884999999999997e-05 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050950+0000", 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3812000000000002e-05 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.063516+0000", 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012565952 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.117654+0000", 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054137941000000002 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.117698+0000", 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4268999999999999e-05 2024-09-16T01:52:22.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.117732+0000", 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3767999999999999e-05 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4107 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e99)", 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.439932851, 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31541509600000001, 2024-09-16T01:52:22.833 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4107 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275941+0000", 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6959999999999999e-06 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303032+0000", 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027090797 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303053+0000", 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1019999999999999e-05 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303063+0000", 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6479999999999998e-06 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303176+0000", 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011292 2024-09-16T01:52:22.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303185+0000", 2024-09-16T01:52:22.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:52:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.491825+0000", 2024-09-16T01:52:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.18864050500000001 2024-09-16T01:52:22.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.498314+0000", 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064884210000000003 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551379+0000", 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053065042999999999 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551446+0000", 2024-09-16T01:52:22.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7477999999999993e-05 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551537+0000", 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1055000000000001e-05 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.576787+0000", 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025250156999999999 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:22.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591154+0000", 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01436693 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591228+0000", 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4103000000000005e-05 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591355+0000", 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126367 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:22.842 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:22.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:22.843 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:22.837+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:22.940 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:22.939+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:23.041 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:23.040+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:23.124 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:23.125 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:23.125 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:23.142 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:23.140+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:23.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:23.242 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:23.241+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:23.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:23.342 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:23.341+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:23.450 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:23.443+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:23.482 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:23.482 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:23.482 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:23.482 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:23.482 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:23.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:23.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.525+0000 7fa7eac00640 1 Processor -- start 2024-09-16T01:52:23.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.525+0000 7fa7eac00640 1 -- start start 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.528+0000 7fa7eac00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7dc0d45f0 0x7fa7dc0da540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.530+0000 7fa7eac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa7dc04ae70 0x7fa7dc04b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.530+0000 7fa7eac00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7dc0d49d0 con 0x7fa7dc005a00 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.530+0000 7fa7eac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa7dc032c10 con 0x7fa7dc04ae70 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.530+0000 7fa7eac00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa7dc005fd0 con 0x7fa7dc0d45f0 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.531+0000 7fa7e8e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa7dc005a00 0x7fa7dc034250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:37598/0 (socket says 172.21.15.146:37598) 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.531+0000 7fa7e8e00640 1 -- 172.21.15.146:0/1277893817 learned_addr learned my addr 172.21.15.146:0/1277893817 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:23.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.531+0000 7fa7e9800640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7dc0d45f0 0x7fa7dc0da540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.531+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207003883 0 0) 0x7fa7dc0d49d0 con 0x7fa7dc005a00 2024-09-16T01:52:23.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.531+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7cc003520 con 0x7fa7dc005a00 2024-09-16T01:52:23.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.531+0000 7fa7ea200640 1 --2- 172.21.15.146:0/1277893817 >> v2:172.21.15.146:3300/0 conn(0x7fa7dc04ae70 0x7fa7dc04b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.532+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3394481844 0 0) 0x7fa7cc003520 con 0x7fa7dc005a00 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.532+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7dc0d45f0 msgr2=0x7fa7dc0da540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.532+0000 7fa7e3e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7dc0d45f0 0x7fa7dc0da540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.532+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 >> v2:172.21.15.146:3300/0 conn(0x7fa7dc04ae70 msgr2=0x7fa7dc04b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.533+0000 7fa7e3e00640 1 --2- 172.21.15.146:0/1277893817 >> v2:172.21.15.146:3300/0 conn(0x7fa7dc04ae70 0x7fa7dc04b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc0062b0 con 0x7fa7dc005a00 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fa7d0004f00 con 0x7fa7dc005a00 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 >> v1:172.21.15.161:6789/0 conn(0x7fa7dc005a00 legacy=0x7fa7dc034250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc0039b0 0x7fa7cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7cc004800 0x7fa7cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc0051c0 0x7fa7cc0145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc032c10 con 0x7fa7cc0039b0 2024-09-16T01:52:23.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc005fd0 con 0x7fa7cc004800 2024-09-16T01:52:23.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa7cc009bf0 con 0x7fa7cc0051c0 2024-09-16T01:52:23.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.534+0000 7fa7ea200640 1 --2- 172.21.15.146:0/1277893817 >> v2:172.21.15.146:3300/0 conn(0x7fa7dc04ae70 0x7fa7dc04b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:23.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.535+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7cc004800 0x7fa7cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.535+0000 7fa7ea200640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc0051c0 0x7fa7cc0145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.536+0000 7fa7e9800640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc0039b0 0x7fa7cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.538+0000 7fa7e8e00640 1 -- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc0051c0 msgr2=0x7fa7cc0145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:23.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.538+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc0051c0 0x7fa7cc0145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.538+0000 7fa7e8e00640 1 -- 172.21.15.146:0/1277893817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc0039b0 msgr2=0x7fa7cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:23.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.538+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc0039b0 0x7fa7cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.538+0000 7fa7e8e00640 1 -- 172.21.15.146:0/1277893817 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc0062b0 con 0x7fa7cc004800 2024-09-16T01:52:23.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.539+0000 7fa7e9800640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc0039b0 0x7fa7cc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:23.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.539+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7cc004800 0x7fa7cc004c80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0001eb0 tx=0x7fa7cc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b92b007387ac0703 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.540+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0001b80 con 0x7fa7cc004800 2024-09-16T01:52:23.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.540+0000 7fa7e3e00640 1 -- 172.21.15.146:0/1277893817 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d002f070 con 0x7fa7cc004800 2024-09-16T01:52:23.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7eac00640 1 -- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7cc004800 msgr2=0x7fa7cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7eac00640 1 --2- 172.21.15.146:0/1277893817 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7cc004800 0x7fa7cc004c80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0001eb0 tx=0x7fa7cc005830 comp rx=0 tx=0).stop 2024-09-16T01:52:23.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7ea200640 1 -- 172.21.15.146:0/1277893817 reap_dead start 2024-09-16T01:52:23.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7eac00640 1 -- 172.21.15.146:0/1277893817 shutdown_connections 2024-09-16T01:52:23.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7eac00640 1 -- 172.21.15.146:0/1277893817 >> 172.21.15.146:0/1277893817 conn(0x7fa7dc04fc50 msgr2=0x7fa7dc050060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:23.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7eac00640 1 -- 172.21.15.146:0/1277893817 shutdown_connections 2024-09-16T01:52:23.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.541+0000 7fa7eac00640 1 -- 172.21.15.146:0/1277893817 wait complete. 2024-09-16T01:52:23.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.543+0000 7fa7eac00640 1 Processor -- start 2024-09-16T01:52:23.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.543+0000 7fa7eac00640 1 -- start start 2024-09-16T01:52:23.547 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:23.545+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7eac00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc004800 0x7fa7dc0daa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7eac00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7dc0cbab0 0x7fa7dc0d95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7eac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 0x7fa7dc049920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7eac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc006b50 con 0x7fa7dc046f50 2024-09-16T01:52:23.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7eac00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc006870 con 0x7fa7dc0cbab0 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7eac00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa7dc006e30 con 0x7fa7cc004800 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7ea200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 0x7fa7dc049920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7ea200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 0x7fa7dc049920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35786/0 (socket says 172.21.15.146:35786) 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.545+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 learned_addr learned my addr 172.21.15.146:0/3847633353 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.546+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc004800 msgr2=0x7fa7dc0daa00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.546+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa7cc004800 0x7fa7dc0daa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.546+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7dc0cbab0 msgr2=0x7fa7dc0d95d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.546+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7dc0cbab0 0x7fa7dc0d95d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.546+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc0cc040 con 0x7fa7dc046f50 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.546+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 0x7fa7dc049920 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa7e406a760 tx=0x7fa7e4073590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.547+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e409a070 con 0x7fa7dc046f50 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.547+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7dc0d8530 con 0x7fa7dc046f50 2024-09-16T01:52:23.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.547+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7dc0d8af0 con 0x7fa7dc046f50 2024-09-16T01:52:23.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.547+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7e40697b0 con 0x7fa7dc046f50 2024-09-16T01:52:23.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.547+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800b520 con 0x7fa7dc046f50 2024-09-16T01:52:23.550 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:52:23.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.550+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa7e409d360 con 0x7fa7dc046f50 2024-09-16T01:52:23.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.550+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa7c0005270 con 0x7fa7dc046f50 2024-09-16T01:52:23.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.551+0000 7fa7e1600640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa7c406c2e0 0x7fa7c406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.551+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5741+0+0 (secure 0 0 0) 0x7fa7e41076f0 con 0x7fa7dc046f50 2024-09-16T01:52:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.552+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa7c406c2e0 0x7fa7c406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.552+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v100) ==== 54+0+16 (secure 0 0 0) 0x7fa7e4092070 con 0x7fa7dc046f50 2024-09-16T01:52:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.553+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa7c406c2e0 0x7fa7c406e7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00d7f0 tx=0x7fa7d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.555+0000 7fa7eac00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa7c0000fa0 0x7fa7c00045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.556+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa7c0000fa0 0x7fa7c00045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.557+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7c0005550 con 0x7fa7c0000fa0 2024-09-16T01:52:23.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.560+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa7c0000fa0 0x7fa7c00045e0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.562+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7c0005550 con 0x7fa7c0000fa0 2024-09-16T01:52:23.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.594+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7c0004f80 con 0x7fa7c0000fa0 2024-09-16T01:52:23.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.597+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7c0004f80 con 0x7fa7c0000fa0 2024-09-16T01:52:23.599 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:23.599 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:23.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.598+0000 7fa7eac00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa7c0001cb0 0x7fa7c0002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.598+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa7c0002320 con 0x7fa7c0001cb0 2024-09-16T01:52:23.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.599+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa7c0001cb0 0x7fa7c0002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.601+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa7c0001cb0 0x7fa7c0002990 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.603+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7c0002320 con 0x7fa7c0001cb0 2024-09-16T01:52:23.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.622+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7c0003200 con 0x7fa7c0001cb0 2024-09-16T01:52:23.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.627+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7c0003200 con 0x7fa7c0001cb0 2024-09-16T01:52:23.629 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:23.629 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:23.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.628+0000 7fa7eac00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa7c0009c80 0x7fa7c000c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.628+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa7c000ee40 con 0x7fa7c0009c80 2024-09-16T01:52:23.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.629+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa7c0009c80 0x7fa7c000c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.630+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa7c0009c80 0x7fa7c000c070 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.631+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7c000ee40 con 0x7fa7c0009c80 2024-09-16T01:52:23.649 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:23.648+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.660+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7c000c880 con 0x7fa7c0009c80 2024-09-16T01:52:23.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.668+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7c000c880 con 0x7fa7c0009c80 2024-09-16T01:52:23.669 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:23.669 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:23.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.668+0000 7fa7eac00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa7c000d300 0x7fa7c000d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.668+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa7c000df10 con 0x7fa7c000d300 2024-09-16T01:52:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.675+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa7c000d300 0x7fa7c000d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.676+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa7c000d300 0x7fa7c000d780 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.677+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7c000df10 con 0x7fa7c000d300 2024-09-16T01:52:23.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.728+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7c000e480 con 0x7fa7c000d300 2024-09-16T01:52:23.750 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:23.749+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:23.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.753+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7c000e480 con 0x7fa7c000d300 2024-09-16T01:52:23.757 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:23.758 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:23.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.756+0000 7fa7eac00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa7c0011000 0x7fa7c00133f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.757+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa7c00161c0 con 0x7fa7c0011000 2024-09-16T01:52:23.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.757+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa7c0011000 0x7fa7c00133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.758+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa7c0011000 0x7fa7c00133f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.760+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7c00161c0 con 0x7fa7c0011000 2024-09-16T01:52:23.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.807+0000 7fa7eac00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7c0013c00 con 0x7fa7c0011000 2024-09-16T01:52:23.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.809+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7c0013c00 con 0x7fa7c0011000 2024-09-16T01:52:23.813 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:23.813 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:23.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.812+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa7b4001650 0x7fa7b4003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.812+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa7b4006bb0 con 0x7fa7b4001650 2024-09-16T01:52:23.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.813+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa7b4001650 0x7fa7b4003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.816+0000 7fa7e8e00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa7b4001650 0x7fa7b4003b10 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.817+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7b4006bb0 con 0x7fa7b4001650 2024-09-16T01:52:23.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.843+0000 7f9d87e00640 1 Processor -- start 2024-09-16T01:52:23.845 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:23.845 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:23.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.844+0000 7f9d87e00640 1 -- start start 2024-09-16T01:52:23.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.846+0000 7f9d87e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9d8015aab0 0x7f9d800cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.846+0000 7f9d87e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8015b1d0 con 0x7f9d800cb2f0 2024-09-16T01:52:23.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.846+0000 7f9d87e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8015b4b0 con 0x7f9d800cc0c0 2024-09-16T01:52:23.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.846+0000 7f9d87e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9d800cbd00 con 0x7f9d8015aab0 2024-09-16T01:52:23.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.847+0000 7f9d84a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9d800cb2f0 0x7f9d80163420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52470/0 (socket says 172.21.15.146:52470) 2024-09-16T01:52:23.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.847+0000 7f9d84a00640 1 -- 172.21.15.146:0/1514344671 learned_addr learned my addr 172.21.15.146:0/1514344671 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:23.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.847+0000 7f9d85400640 1 --1- 172.21.15.146:0/1514344671 >> v1:172.21.15.161:6789/0 conn(0x7f9d800cc0c0 0x7f9d8015e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:37608/0 (socket says 172.21.15.146:37608) 2024-09-16T01:52:23.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.847+0000 7f9d85e00640 1 --2- 172.21.15.146:0/1514344671 >> v2:172.21.15.161:3300/0 conn(0x7f9d8015aab0 0x7f9d800cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.847+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113939473 0 0) 0x7f9d8015b1d0 con 0x7f9d800cb2f0 2024-09-16T01:52:23.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.847+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d68003520 con 0x7f9d800cb2f0 2024-09-16T01:52:23.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.848+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798857367 0 0) 0x7f9d8015b4b0 con 0x7f9d800cc0c0 2024-09-16T01:52:23.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.848+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d8015b1d0 con 0x7f9d800cc0c0 2024-09-16T01:52:23.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 168612058 0 0) 0x7f9d68003520 con 0x7f9d800cb2f0 2024-09-16T01:52:23.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 >> v2:172.21.15.161:3300/0 conn(0x7f9d8015aab0 msgr2=0x7f9d800cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:23.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 --2- 172.21.15.146:0/1514344671 >> v2:172.21.15.161:3300/0 conn(0x7f9d8015aab0 0x7f9d800cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 >> v1:172.21.15.161:6789/0 conn(0x7f9d800cc0c0 legacy=0x7f9d8015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.852 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:23.851+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8015b790 con 0x7f9d800cb2f0 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f9d74001670 con 0x7f9d800cb2f0 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.849+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 >> v1:172.21.15.146:6789/0 conn(0x7f9d800cb2f0 legacy=0x7f9d80163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d7fe00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d68003bc0 0x7f9d68003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d7fe00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d7fe00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680051c0 0x7f9d680141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d800cbd00 con 0x7f9d68003bc0 2024-09-16T01:52:23.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9d680148d0 con 0x7f9d68004800 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9d68014ad0 con 0x7f9d680051c0 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d84a00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.850+0000 7f9d85400640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d68003bc0 0x7f9d68003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d84a00640 1 -- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680051c0 msgr2=0x7f9d680141d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d84a00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680051c0 0x7f9d680141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d84a00640 1 -- 172.21.15.146:0/1514344671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d68003bc0 msgr2=0x7f9d68003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d84a00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d68003bc0 0x7f9d68003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d84a00640 1 -- 172.21.15.146:0/1514344671 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d68003520 con 0x7f9d68004800 2024-09-16T01:52:23.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d85e00640 1 --2- 172.21.15.146:0/1514344671 >> v2:172.21.15.161:3300/0 conn(0x7f9d8015aab0 0x7f9d800cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:23.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.851+0000 7f9d85e00640 1 -- 172.21.15.146:0/1514344671 reap_dead start 2024-09-16T01:52:23.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.852+0000 7f9d84a00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d68004c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9d7401d1a0 tx=0x7f9d68005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7536de86f5bd2880 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.853+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74007c70 con 0x7f9d68004800 2024-09-16T01:52:23.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.853+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7403f070 con 0x7f9d68004800 2024-09-16T01:52:23.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.853+0000 7f9d7fe00640 1 -- 172.21.15.146:0/1514344671 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74005d30 con 0x7f9d68004800 2024-09-16T01:52:23.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.854+0000 7f9d87e00640 1 -- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 msgr2=0x7f9d68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.854+0000 7f9d87e00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d68004c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9d7401d1a0 tx=0x7f9d68005830 comp rx=0 tx=0).stop 2024-09-16T01:52:23.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.854+0000 7f9d87e00640 1 -- 172.21.15.146:0/1514344671 shutdown_connections 2024-09-16T01:52:23.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.854+0000 7f9d87e00640 1 --2- 172.21.15.146:0/1514344671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d68004c80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.854+0000 7f9d87e00640 1 -- 172.21.15.146:0/1514344671 >> 172.21.15.146:0/1514344671 conn(0x7f9d8006d560 msgr2=0x7f9d8006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:23.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.855+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7b40058d0 con 0x7fa7b4001650 2024-09-16T01:52:23.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.855+0000 7f9d87e00640 1 -- 172.21.15.146:0/1514344671 shutdown_connections 2024-09-16T01:52:23.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.855+0000 7f9d87e00640 1 -- 172.21.15.146:0/1514344671 wait complete. 2024-09-16T01:52:23.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.858+0000 7f9d87e00640 1 Processor -- start 2024-09-16T01:52:23.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.858+0000 7f9d87e00640 1 -- start start 2024-09-16T01:52:23.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.859+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d80148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.859+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d800c9ee0 0x7f9d800ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.859+0000 7f9d87e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d800b4f20 0x7f9d800b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.859+0000 7f9d87e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8015e540 con 0x7f9d800b4f20 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.859+0000 7f9d87e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8015ba70 con 0x7f9d68004800 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.859+0000 7f9d87e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8015aec0 con 0x7f9d800c9ee0 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d80148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d80148400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44604/0 (socket says 172.21.15.146:44604) 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 -- 172.21.15.146:0/3198104320 learned_addr learned my addr 172.21.15.146:0/3198104320 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 -- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d800c9ee0 msgr2=0x7f9d800ca340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d800c9ee0 0x7f9d800ca340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 -- 172.21.15.146:0/3198104320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d800b4f20 msgr2=0x7f9d800b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d800b4f20 0x7f9d800b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.860+0000 7f9d85400640 1 -- 172.21.15.146:0/3198104320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d800b6900 con 0x7f9d68004800 2024-09-16T01:52:23.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.861+0000 7f9d85400640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d80148400 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00e680 tx=0x7f9d6c00eb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.861+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00c660 con 0x7f9d68004800 2024-09-16T01:52:23.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.861+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6c018070 con 0x7f9d68004800 2024-09-16T01:52:23.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.861+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00ca00 con 0x7f9d68004800 2024-09-16T01:52:23.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.861+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d800b4760 con 0x7f9d68004800 2024-09-16T01:52:23.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.861+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d800b6340 con 0x7f9d68004800 2024-09-16T01:52:23.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.862+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7b40058d0 con 0x7fa7b4001650 2024-09-16T01:52:23.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.862+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d800c5560 con 0x7f9d68004800 2024-09-16T01:52:23.865 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:23.865 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:23.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.864+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9d6c00cba0 con 0x7f9d68004800 2024-09-16T01:52:23.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.863+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa7b40048a0 0x7fa7b4004d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.864+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa7b4006690 con 0x7fa7b40048a0 2024-09-16T01:52:23.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:23.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.864+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa7b40048a0 0x7fa7b4004d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.864+0000 7fa7ea200640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa7b40048a0 0x7fa7b4004d00 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.864+0000 7f9d7d600640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d6006c2b0 0x7f9d6006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.865+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5741+0+0 (secure 0 0 0) 0x7f9d6c08b1b0 con 0x7f9d68004800 2024-09-16T01:52:23.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.865+0000 7f9d84a00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d6006c2b0 0x7f9d6006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.865+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7b4006690 con 0x7fa7b40048a0 2024-09-16T01:52:23.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.865+0000 7f9d84a00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d6006c2b0 0x7f9d6006e770 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9d74043000 tx=0x7f9d74007000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.869+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9d6c0112e0 con 0x7f9d68004800 2024-09-16T01:52:23.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.892+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7b40065e0 con 0x7fa7b40048a0 2024-09-16T01:52:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.894+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7b40065e0 con 0x7fa7b40048a0 2024-09-16T01:52:23.897 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.896+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa7b4005890 0x7fa7b4005d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.897+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa7b4006330 con 0x7fa7b4005890 2024-09-16T01:52:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.897+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa7b4005890 0x7fa7b4005d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.898+0000 7fa7e9800640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa7b4005890 0x7fa7b4005d40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:23.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.899+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7b4006330 con 0x7fa7b4005890 2024-09-16T01:52:23.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.918+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7b400afb0 con 0x7fa7b4005890 2024-09-16T01:52:23.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.920+0000 7fa7e1600640 1 -- 172.21.15.146:0/3847633353 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7b400afb0 con 0x7fa7b4005890 2024-09-16T01:52:23.924 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa7c0000fa0 msgr2=0x7fa7c00045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa7c0000fa0 0x7fa7c00045e0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa7c0001cb0 msgr2=0x7fa7c0002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa7c0001cb0 0x7fa7c0002990 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa7c0009c80 msgr2=0x7fa7c000c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa7c0009c80 0x7fa7c000c070 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa7c000d300 msgr2=0x7fa7c000d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa7c000d300 0x7fa7c000d780 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa7c0011000 msgr2=0x7fa7c00133f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa7c0011000 0x7fa7c00133f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa7b4001650 msgr2=0x7fa7b4003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa7b4001650 0x7fa7b4003b10 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa7b40048a0 msgr2=0x7fa7b4004d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa7b40048a0 0x7fa7b4004d00 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa7b4005890 msgr2=0x7fa7b4005d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.923+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa7b4005890 0x7fa7b4005d40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 reap_dead start 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 reap_dead start 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 reap_dead start 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 reap_dead start 2024-09-16T01:52:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7ea200640 1 -- 172.21.15.146:0/3847633353 reap_dead start 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa7c406c2e0 msgr2=0x7fa7c406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa7c406c2e0 0x7fa7c406e7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00d7f0 tx=0x7fa7d8009040 comp rx=0 tx=0).stop 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 msgr2=0x7fa7dc049920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.924+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 0x7fa7dc049920 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa7e406a760 tx=0x7fa7e4073590 comp rx=0 tx=0).stop 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.925+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 shutdown_connections 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.925+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa7c406c2e0 0x7fa7c406e7a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.925+0000 7fa7d6a00640 1 --2- 172.21.15.146:0/3847633353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7dc046f50 0x7fa7dc049920 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.925+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 >> 172.21.15.146:0/3847633353 conn(0x7fa7dc04fc50 msgr2=0x7fa7dc0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:23.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.926+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 shutdown_connections 2024-09-16T01:52:23.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:23.926+0000 7fa7d6a00640 1 -- 172.21.15.146:0/3847633353 wait complete. 2024-09-16T01:52:23.953 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:23.952+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:24.055 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:24.054+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:24.112 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:24.112 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:24.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:24.155 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:24.154+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:24.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.168+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f9d800b2fe0 con 0x7f9d68004800 2024-09-16T01:52:24.257 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:24.256+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:24.357 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:24.356+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:24.444 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:24.444 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:24.459 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:24.458+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:24.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:24.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.492+0000 7f9d7d600640 1 -- 172.21.15.146:0/3198104320 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v101) ==== 91+0+0 (secure 0 0 0) 0x7f9d6c059860 con 0x7f9d68004800 2024-09-16T01:52:24.494 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:52:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.499+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d6006c2b0 msgr2=0x7f9d6006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.499+0000 7f9d87e00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d6006c2b0 0x7f9d6006e770 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9d74043000 tx=0x7f9d74007000 comp rx=0 tx=0).stop 2024-09-16T01:52:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.499+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 msgr2=0x7f9d80148400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.499+0000 7f9d87e00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d80148400 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00e680 tx=0x7f9d6c00eb40 comp rx=0 tx=0).stop 2024-09-16T01:52:24.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.501+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 shutdown_connections 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.501+0000 7f9d87e00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d6006c2b0 0x7f9d6006e770 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.501+0000 7f9d87e00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d800b4f20 0x7f9d800b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.501+0000 7f9d87e00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d800c9ee0 0x7f9d800ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.501+0000 7f9d87e00640 1 --2- 172.21.15.146:0/3198104320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68004800 0x7f9d80148400 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.501+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 >> 172.21.15.146:0/3198104320 conn(0x7f9d8006d560 msgr2=0x7f9d8014b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.505+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 shutdown_connections 2024-09-16T01:52:24.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:24.505+0000 7f9d87e00640 1 -- 172.21.15.146:0/3198104320 wait complete. 2024-09-16T01:52:24.576 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:24.575+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:24.662 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:24.661+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:24.757 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:24.757 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:24.768 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:24.767+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:24.792 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:24.863 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:24.862+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:24.910 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:24.911 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:24.921 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:24.968 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:24.967+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:25.043 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.044 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.054 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:25.066 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:25.065+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:25.165 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.165 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.166 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:25.164+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:25.174 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:25.269 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:25.268+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:25.339 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.339 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.350 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:25.367 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:25.366+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:25.463 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.463 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.468 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:25.466+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:25.473 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:25.504 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:25.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:25.504 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-16T01:52:25.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-16T01:52:25.569 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:25.567+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:25.624 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.624 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.633 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:25.671 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:25.670+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:25.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.731+0000 7f1d05c00640 1 Processor -- start 2024-09-16T01:52:25.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.732+0000 7f1d05c00640 1 -- start start 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.733+0000 7f1d05c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 0x7f1d00072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1d05c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1d000c70e0 0x7f1d000727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1d05c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d001569d0 con 0x7f1d00072dd0 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1d05c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1d00157580 con 0x7f1d000c70e0 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1d05c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d00071ce0 con 0x7f1d00150c10 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1cfe000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1d000c70e0 0x7f1d000727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1cff400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1d00072dd0 0x7f1d0006da50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:37618/0 (socket says 172.21.15.146:37618) 2024-09-16T01:52:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1cfe000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1d000c70e0 0x7f1d000727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44608/0 (socket says 172.21.15.146:44608) 2024-09-16T01:52:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.734+0000 7f1cff400640 1 -- 172.21.15.146:0/272211508 learned_addr learned my addr 172.21.15.146:0/272211508 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:25.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.735+0000 7f1cfea00640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 0x7f1d00072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.735+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102452310 0 0) 0x7f1cec001110 con 0x7f1d00072dd0 2024-09-16T01:52:25.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.735+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ce4003520 con 0x7f1d00072dd0 2024-09-16T01:52:25.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 111436080 0 0) 0x7f1ce4003520 con 0x7f1d00072dd0 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 msgr2=0x7f1d00072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 0x7f1d00072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 >> v2:172.21.15.161:3300/0 conn(0x7f1d000c70e0 msgr2=0x7f1d000727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 --2- 172.21.15.146:0/272211508 >> v2:172.21.15.161:3300/0 conn(0x7f1d000c70e0 0x7f1d000727d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d00156c50 con 0x7f1d00072dd0 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfe000640 1 --2- 172.21.15.146:0/272211508 >> v2:172.21.15.161:3300/0 conn(0x7f1d000c70e0 0x7f1d000727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f1cec001840 con 0x7f1d00072dd0 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 >> v1:172.21.15.161:6789/0 conn(0x7f1d00072dd0 legacy=0x7f1d0006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.736+0000 7f1cfd600640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 0x7f1ce40039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfd600640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ce4003fa0 0x7f1ce4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfd600640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 0x7f1ce40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1d00157580 con 0x7f1ce4004d40 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d00071ce0 con 0x7f1ce4003fa0 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce400b9f0 con 0x7f1d00150c10 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfea00640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 0x7f1ce40039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cfe000640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ce4003fa0 0x7f1ce4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.737+0000 7f1cff400640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 0x7f1ce40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.738+0000 7f1cff400640 1 -- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 msgr2=0x7f1ce40039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.738+0000 7f1cff400640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00150c10 0x7f1ce40039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.738+0000 7f1cff400640 1 -- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ce4003fa0 msgr2=0x7f1ce4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.738+0000 7f1cff400640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ce4003fa0 0x7f1ce4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.738+0000 7f1cff400640 1 -- 172.21.15.146:0/272211508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce4003520 con 0x7f1ce4004d40 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.738+0000 7f1cfe000640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ce4003fa0 0x7f1ce4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.739+0000 7f1cff400640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 0x7f1ce40051c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c2f0 tx=0x7f1ce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c198538568c88a83 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.740+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec035070 con 0x7f1ce4004d40 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.740+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cec0058d0 con 0x7f1ce4004d40 2024-09-16T01:52:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.740+0000 7f1cfd600640 1 -- 172.21.15.146:0/272211508 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec035070 con 0x7f1ce4004d40 2024-09-16T01:52:25.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.743+0000 7f1d05c00640 1 -- 172.21.15.146:0/272211508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 msgr2=0x7f1ce40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:25.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.743+0000 7f1d05c00640 1 --2- 172.21.15.146:0/272211508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 0x7f1ce40051c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c2f0 tx=0x7f1ce4005830 comp rx=0 tx=0).stop 2024-09-16T01:52:25.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.743+0000 7f1cff400640 1 -- 172.21.15.146:0/272211508 reap_dead start 2024-09-16T01:52:25.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.745+0000 7f1d05c00640 1 -- 172.21.15.146:0/272211508 shutdown_connections 2024-09-16T01:52:25.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.745+0000 7f1d05c00640 1 -- 172.21.15.146:0/272211508 >> 172.21.15.146:0/272211508 conn(0x7f1d0006f3c0 msgr2=0x7f1d0006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:25.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.745+0000 7f1d05c00640 1 -- 172.21.15.146:0/272211508 shutdown_connections 2024-09-16T01:52:25.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.745+0000 7f1d05c00640 1 -- 172.21.15.146:0/272211508 wait complete. 2024-09-16T01:52:25.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.747+0000 7f1d05c00640 1 Processor -- start 2024-09-16T01:52:25.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.747+0000 7f1d05c00640 1 -- start start 2024-09-16T01:52:25.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.748+0000 7f1d05c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 0x7f1d00159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.748+0000 7f1d05c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d0014c4f0 0x7f1d00155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.748+0000 7f1d05c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00158160 0x7f1d001585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.748+0000 7f1d05c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1d000c9a90 con 0x7f1ce4004d40 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.748+0000 7f1d05c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d00157210 con 0x7f1d0014c4f0 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.748+0000 7f1d05c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d000718c0 con 0x7f1d00158160 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00158160 0x7f1d001585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cfe000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d0014c4f0 0x7f1d00155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00158160 0x7f1d001585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45128/0 (socket says 172.21.15.146:45128) 2024-09-16T01:52:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 -- 172.21.15.146:0/3021698411 learned_addr learned my addr 172.21.15.146:0/3021698411 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 -- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d0014c4f0 msgr2=0x7f1d00155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d0014c4f0 0x7f1d00155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 -- 172.21.15.146:0/3021698411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 msgr2=0x7f1d00159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce4004d40 0x7f1d00159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 -- 172.21.15.146:0/3021698411 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0015a760 con 0x7f1d00158160 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cfe000640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d0014c4f0 0x7f1d00155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.749+0000 7f1cff400640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d00158160 0x7f1d001585e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1cec02af30 tx=0x7f1d000c64f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.750+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf80104d0 con 0x7f1d00158160 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.750+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cec041070 con 0x7f1d00158160 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.750+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec035070 con 0x7f1d00158160 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.750+0000 7f1d05c00640 1 -- 172.21.15.146:0/3021698411 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0015aa40 con 0x7f1d00158160 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.750+0000 7f1d05c00640 1 -- 172.21.15.146:0/3021698411 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0014b790 con 0x7f1d00158160 2024-09-16T01:52:25.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.751+0000 7f1d05c00640 1 -- 172.21.15.146:0/3021698411 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1d0006fee0 con 0x7f1d00158160 2024-09-16T01:52:25.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.752+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1cec007480 con 0x7f1d00158160 2024-09-16T01:52:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.753+0000 7f1cf6a00640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1ce006c2b0 0x7f1ce006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.753+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7f1cec0ab1b0 con 0x7f1d00158160 2024-09-16T01:52:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.754+0000 7f1cf6a00640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1ce0074510 0x7f1ce0076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.754+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cec0058d0 con 0x7f1ce0074510 2024-09-16T01:52:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.754+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f1cec0b2050 con 0x7f1d00158160 2024-09-16T01:52:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.754+0000 7f1cfe000640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1ce0074510 0x7f1ce0076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.754+0000 7f1cfea00640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1ce006c2b0 0x7f1ce006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:25.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.755+0000 7f1cfe000640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1ce0074510 0x7f1ce0076930 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:25.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.756+0000 7f1cfea00640 1 --2- 172.21.15.146:0/3021698411 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1ce006c2b0 0x7f1ce006e770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009ab0 tx=0x7f1cf00095b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:25.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:25.756+0000 7f1cf6a00640 1 -- 172.21.15.146:0/3021698411 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cec0058d0 con 0x7f1ce0074510 2024-09-16T01:52:25.771 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:25.769+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:25.806 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.807 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.814 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-09-16T01:52:25.814 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-09-16T01:52:25.814 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:bench [] [] [] [] 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs debug_inject_read_zeros 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs files list 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs stats 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:bluestore allocator dump block 2024-09-16T01:52:25.815 INFO:teuthology.orchestra.run.smithi146.stderr:Error EINVAL: invalid command 2024-09-16T01:52:25.817 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T01:52:25.829 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T01:52:25.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-16T01:52:25.872 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:25.871+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:25.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:25.974 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:25.975 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:25.984 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:26.000 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:25.998+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:26.074 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:26.073+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:26.098 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:26.099 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:26.109 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:26.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.147+0000 7f61ad200640 1 Processor -- start 2024-09-16T01:52:26.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.147+0000 7f61ad200640 1 -- start start 2024-09-16T01:52:26.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.149+0000 7f61ad200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61a806d1d0 0x7f61a8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.149+0000 7f61ad200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a80c7280 con 0x7f61a806d5b0 2024-09-16T01:52:26.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.149+0000 7f61ad200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a80ca720 con 0x7f61a8072790 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.149+0000 7f61ad200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f61a80c7a40 con 0x7f61a806d1d0 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.150+0000 7f61a6a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f61a8072790 0x7f61a80736e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:37632/0 (socket says 172.21.15.146:37632) 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.150+0000 7f61a6a00640 1 -- 172.21.15.146:0/1253307259 learned_addr learned my addr 172.21.15.146:0/1253307259 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054502686 0 0) 0x7f61a80ca720 con 0x7f61a8072790 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f618c003520 con 0x7f61a8072790 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1620310556 0 0) 0x7f618c003520 con 0x7f61a8072790 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 >> v2:172.21.15.146:3300/0 conn(0x7f61a806d1d0 msgr2=0x7f61a8158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 --2- 172.21.15.146:0/1253307259 >> v2:172.21.15.146:3300/0 conn(0x7f61a806d1d0 0x7f61a8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 >> v1:172.21.15.146:6789/0 conn(0x7f61a806d5b0 legacy=0x7f61a8169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a80c7d20 con 0x7f61a8072790 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f6194001670 con 0x7f61a8072790 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 >> v1:172.21.15.161:6789/0 conn(0x7f61a8072790 legacy=0x7f61a80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f618c0039b0 0x7f618c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f618c004800 0x7f618c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f618c0052e0 0x7f618c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61a80c7a40 con 0x7f618c004800 2024-09-16T01:52:26.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f618c02e1e0 con 0x7f618c0039b0 2024-09-16T01:52:26.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.151+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f618c02e3e0 con 0x7f618c0052e0 2024-09-16T01:52:26.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.153+0000 7f61a7400640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f618c0039b0 0x7f618c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.154+0000 7f61a7400640 1 -- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f618c0052e0 msgr2=0x7f618c00b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.154+0000 7f61a7400640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f618c0052e0 0x7f618c00b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.154+0000 7f61a7400640 1 -- 172.21.15.146:0/1253307259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f618c004800 msgr2=0x7f618c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.154+0000 7f61a7400640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f618c004800 0x7f618c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.154+0000 7f61a7400640 1 -- 172.21.15.146:0/1253307259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a80c7d20 con 0x7f618c0039b0 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.156+0000 7f61a7e00640 1 -- 172.21.15.146:0/1253307259 reap_dead start 2024-09-16T01:52:26.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.156+0000 7f61a7400640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f618c0039b0 0x7f618c003db0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6198006fd0 tx=0x7f618c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6933a6cad7498a30 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.159+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198009460 con 0x7f618c0039b0 2024-09-16T01:52:26.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.159+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6198004250 con 0x7f618c0039b0 2024-09-16T01:52:26.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.159+0000 7f61a6000640 1 -- 172.21.15.146:0/1253307259 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619802a850 con 0x7f618c0039b0 2024-09-16T01:52:26.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.160+0000 7f61ad200640 1 -- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f618c0039b0 msgr2=0x7f618c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.160+0000 7f61ad200640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f618c0039b0 0x7f618c003db0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6198006fd0 tx=0x7f618c005830 comp rx=0 tx=0).stop 2024-09-16T01:52:26.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.161+0000 7f61ad200640 1 -- 172.21.15.146:0/1253307259 shutdown_connections 2024-09-16T01:52:26.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.161+0000 7f61ad200640 1 --2- 172.21.15.146:0/1253307259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f618c0039b0 0x7f618c003db0 secure :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6198006fd0 tx=0x7f618c005830 comp rx=0 tx=0).stop 2024-09-16T01:52:26.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.161+0000 7f61ad200640 1 -- 172.21.15.146:0/1253307259 >> 172.21.15.146:0/1253307259 conn(0x7f61a8067950 msgr2=0x7f61a8067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:26.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.163+0000 7f61ad200640 1 -- 172.21.15.146:0/1253307259 shutdown_connections 2024-09-16T01:52:26.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.166+0000 7f61ad200640 1 -- 172.21.15.146:0/1253307259 wait complete. 2024-09-16T01:52:26.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.168+0000 7f61ad200640 1 Processor -- start 2024-09-16T01:52:26.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.168+0000 7f61ad200640 1 -- start start 2024-09-16T01:52:26.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.169+0000 7f61ad200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a81480a0 0x7f61a8146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.169+0000 7f61ad200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61a8148480 0x7f61a8148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.169+0000 7f61ad200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61a814aae0 0x7f61a8149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.169+0000 7f61ad200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f61a8072bb0 con 0x7f61a814aae0 2024-09-16T01:52:26.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.169+0000 7f61ad200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61a8073310 con 0x7f61a81480a0 2024-09-16T01:52:26.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.169+0000 7f61ad200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f61a8072d30 con 0x7f61a8148480 2024-09-16T01:52:26.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.170+0000 7f61a7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61a814aae0 0x7f61a8149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.170+0000 7f61a6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61a8148480 0x7f61a8148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.170+0000 7f61a6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61a8148480 0x7f61a8148c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45132/0 (socket says 172.21.15.146:45132) 2024-09-16T01:52:26.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.170+0000 7f61a6a00640 1 -- 172.21.15.146:0/1529946870 learned_addr learned my addr 172.21.15.146:0/1529946870 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:26.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.171+0000 7f61a7e00640 1 -- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61a8148480 msgr2=0x7f61a8148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.171+0000 7f61a7e00640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f61a8148480 0x7f61a8148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.171+0000 7f61a7e00640 1 -- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a81480a0 msgr2=0x7f61a8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.171+0000 7f61a7e00640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a81480a0 0x7f61a8146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.171+0000 7f61a7e00640 1 -- 172.21.15.146:0/1529946870 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a8153250 con 0x7f61a814aae0 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.172+0000 7f61a7e00640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61a814aae0 0x7f61a8149160 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f61a80ce040 tx=0x7f61a000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.173+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0017070 con 0x7f61a814aae0 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.173+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a00070d0 con 0x7f61a814aae0 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.173+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a00074a0 con 0x7f61a814aae0 2024-09-16T01:52:26.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.173+0000 7f61ad200640 1 -- 172.21.15.146:0/1529946870 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a8153530 con 0x7f61a814aae0 2024-09-16T01:52:26.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.173+0000 7f61ad200640 1 -- 172.21.15.146:0/1529946870 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a8153af0 con 0x7f61a814aae0 2024-09-16T01:52:26.178 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:26.177+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:26.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.178+0000 7f61ad200640 1 -- 172.21.15.146:0/1529946870 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61a81541b0 con 0x7f61a814aae0 2024-09-16T01:52:26.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.179+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f61a001a070 con 0x7f61a814aae0 2024-09-16T01:52:26.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.180+0000 7f619f400640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f618806c3b0 0x7f618806e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.180+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7f61a008a520 con 0x7f61a814aae0 2024-09-16T01:52:26.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.181+0000 7f61a7400640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f618806c3b0 0x7f618806e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.181+0000 7f619f400640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6188074690 0x7f6188076ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.181+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61a0010e50 con 0x7f6188074690 2024-09-16T01:52:26.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.182+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f61a0091050 con 0x7f61a814aae0 2024-09-16T01:52:26.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.182+0000 7f61a7400640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f618806c3b0 0x7f618806e870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6198033000 tx=0x7f619802c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.182+0000 7f61a6a00640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6188074690 0x7f6188076ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.185+0000 7f61a6a00640 1 --2- 172.21.15.146:0/1529946870 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f6188074690 0x7f6188076ab0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.186+0000 7f619f400640 1 -- 172.21.15.146:0/1529946870 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61a0010e50 con 0x7f6188074690 2024-09-16T01:52:26.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.220+0000 7f389b800640 1 Processor -- start 2024-09-16T01:52:26.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.221+0000 7f389b800640 1 -- start start 2024-09-16T01:52:26.228 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:26.228 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:26.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f389b800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38940c8f80 0x7f38941577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f389b800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3894071120 0x7f3894157d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f389b800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38940cd3e0 con 0x7f3894070d40 2024-09-16T01:52:26.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f389b800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f38940c8af0 con 0x7f3894071120 2024-09-16T01:52:26.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f389b800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f38940cd6c0 con 0x7f38940c8f80 2024-09-16T01:52:26.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f3898e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3894070d40 0x7f389415ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52480/0 (socket says 172.21.15.146:52480) 2024-09-16T01:52:26.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f3898e00640 1 -- 172.21.15.146:0/1218969370 learned_addr learned my addr 172.21.15.146:0/1218969370 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:26.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.227+0000 7f3893e00640 1 --2- 172.21.15.146:0/1218969370 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38940c8f80 0x7f38941577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.228+0000 7f3899800640 1 --2- 172.21.15.146:0/1218969370 >> v2:172.21.15.146:3300/0 conn(0x7f3894071120 0x7f3894157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.228+0000 7f3893e00640 1 -- 172.21.15.146:0/1218969370 >> v2:172.21.15.146:3300/0 conn(0x7f3894071120 msgr2=0x7f3894157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.228+0000 7f3893e00640 1 --2- 172.21.15.146:0/1218969370 >> v2:172.21.15.146:3300/0 conn(0x7f3894071120 0x7f3894157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.228+0000 7f3893e00640 1 -- 172.21.15.146:0/1218969370 >> v1:172.21.15.146:6789/0 conn(0x7f3894070d40 legacy=0x7f389415ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.228+0000 7f3893e00640 1 -- 172.21.15.146:0/1218969370 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38940cd9a0 con 0x7f38940c8f80 2024-09-16T01:52:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.229+0000 7f3893e00640 1 --2- 172.21.15.146:0/1218969370 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38940c8f80 0x7f38941577d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38800097a0 tx=0x7f388002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c623f1d9ab40e72 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.229+0000 7f3893400640 1 -- 172.21.15.146:0/1218969370 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880037070 con 0x7f38940c8f80 2024-09-16T01:52:26.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.229+0000 7f3893400640 1 -- 172.21.15.146:0/1218969370 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3880032440 con 0x7f38940c8f80 2024-09-16T01:52:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.229+0000 7f3893400640 1 -- 172.21.15.146:0/1218969370 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880032810 con 0x7f38940c8f80 2024-09-16T01:52:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 -- 172.21.15.146:0/1218969370 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38940c8f80 msgr2=0x7f38941577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 --2- 172.21.15.146:0/1218969370 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38940c8f80 0x7f38941577d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f38800097a0 tx=0x7f388002a230 comp rx=0 tx=0).stop 2024-09-16T01:52:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 -- 172.21.15.146:0/1218969370 shutdown_connections 2024-09-16T01:52:26.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 --2- 172.21.15.146:0/1218969370 >> v2:172.21.15.146:3300/0 conn(0x7f3894071120 0x7f3894157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 --2- 172.21.15.146:0/1218969370 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38940c8f80 0x7f38941577d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 -- 172.21.15.146:0/1218969370 >> 172.21.15.146:0/1218969370 conn(0x7f3894072a60 msgr2=0x7f3894072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:26.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 -- 172.21.15.146:0/1218969370 shutdown_connections 2024-09-16T01:52:26.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.230+0000 7f389b800640 1 -- 172.21.15.146:0/1218969370 wait complete. 2024-09-16T01:52:26.241 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:26.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.233+0000 7f389b800640 1 Processor -- start 2024-09-16T01:52:26.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.233+0000 7f389b800640 1 -- start start 2024-09-16T01:52:26.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f389b800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 0x7f38941559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f389b800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3894071120 0x7f3894155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f389b800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38940c8f80 0x7f3894153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f389b800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f38940c7c40 con 0x7f38940c8f80 2024-09-16T01:52:26.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f389b800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f38940c8780 con 0x7f3894070d40 2024-09-16T01:52:26.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f389b800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f38940c7dc0 con 0x7f3894071120 2024-09-16T01:52:26.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f3898e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 0x7f38941559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f3893e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3894071120 0x7f3894155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f3899800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38940c8f80 0x7f3894153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f3898e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 0x7f38941559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44642/0 (socket says 172.21.15.146:44642) 2024-09-16T01:52:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f3893e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3894071120 0x7f3894155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45140/0 (socket says 172.21.15.146:45140) 2024-09-16T01:52:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.235+0000 7f3898e00640 1 -- 172.21.15.146:0/495310321 learned_addr learned my addr 172.21.15.146:0/495310321 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3898e00640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3894071120 msgr2=0x7f3894155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3894071120 0x7f3894155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3898e00640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38940c8f80 msgr2=0x7f3894153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38940c8f80 0x7f3894153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3898e00640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38941f42a0 con 0x7f3894070d40 2024-09-16T01:52:26.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3894071120 0x7f3894155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:26.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 0x7f38941559c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f388800c3f0 tx=0x7f388800c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888017070 con 0x7f3894070d40 2024-09-16T01:52:26.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3888005ce0 con 0x7f3894070d40 2024-09-16T01:52:26.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38880072f0 con 0x7f3894070d40 2024-09-16T01:52:26.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.236+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38941f4800 con 0x7f3894070d40 2024-09-16T01:52:26.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.237+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3894154c00 con 0x7f3894070d40 2024-09-16T01:52:26.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.238+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3894076300 con 0x7f3894070d40 2024-09-16T01:52:26.250 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-09-16T01:52:26.250 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-09-16T01:52:26.250 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-09-16T01:52:26.250 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-09-16T01:52:26.250 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-09-16T01:52:26.250 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-09-16T01:52:26.251 INFO:teuthology.orchestra.run.smithi146.stderr:bench [] [] [] [] 2024-09-16T01:52:26.251 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs debug_inject_read_zeros 2024-09-16T01:52:26.251 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs files list 2024-09-16T01:52:26.251 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs stats 2024-09-16T01:52:26.251 INFO:teuthology.orchestra.run.smithi146.stderr:bluestore allocator dump block 2024-09-16T01:52:26.251 INFO:teuthology.orchestra.run.smithi146.stderr:Error EINVAL: invalid command 2024-09-16T01:52:26.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.240+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f388801a070 con 0x7f3894070d40 2024-09-16T01:52:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.241+0000 7f3890c00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f387406c2b0 0x7f387406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.241+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7f388808a730 con 0x7f3894070d40 2024-09-16T01:52:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.242+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v101) ==== 54+0+16 (secure 0 0 0) 0x7f3894076300 con 0x7f3894070d40 2024-09-16T01:52:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.242+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f387406c2b0 0x7f387406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.242+0000 7f389b800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3894154090 0x7f389406cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.242+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f389406fc60 con 0x7f3894154090 2024-09-16T01:52:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.242+0000 7f3899800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3894154090 0x7f389406cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.243+0000 7f3899800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3894154090 0x7f389406cc20 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.243+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f387406c2b0 0x7f387406e770 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3880034000 tx=0x7f388002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.244+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f389406fc60 con 0x7f3894154090 2024-09-16T01:52:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T01:52:26.256 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T01:52:26.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-16T01:52:26.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.263+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f38940cb2e0 con 0x7f3894154090 2024-09-16T01:52:26.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.270+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f38940cb2e0 con 0x7f3894154090 2024-09-16T01:52:26.272 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:26.273 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:26.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.272+0000 7f389b800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f38941552e0 0x7f3894156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.272+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3894154470 con 0x7f38941552e0 2024-09-16T01:52:26.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.276+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f38941552e0 0x7f3894156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.277+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f38941552e0 0x7f3894156990 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.278+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3894154470 con 0x7f38941552e0 2024-09-16T01:52:26.299 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:26.297+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:26.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.307+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3894157120 con 0x7f38941552e0 2024-09-16T01:52:26.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.312+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3894157120 con 0x7f38941552e0 2024-09-16T01:52:26.314 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:26.315 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:26.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.314+0000 7f389b800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f38941f09b0 0x7f38941f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.314+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f38940c5270 con 0x7f38941f09b0 2024-09-16T01:52:26.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.315+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f38941f09b0 0x7f38941f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.316+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f38941f09b0 0x7f38941f0db0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.317+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38940c5270 con 0x7f38941f09b0 2024-09-16T01:52:26.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.337+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f38940c5750 con 0x7f38941f09b0 2024-09-16T01:52:26.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.339+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f38940c5750 con 0x7f38941f09b0 2024-09-16T01:52:26.340 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:26.340 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:26.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.339+0000 7f389b800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f3894062630 0x7f3894062a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.339+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f38940c6230 con 0x7f3894062630 2024-09-16T01:52:26.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.339+0000 7f3899800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f3894062630 0x7f3894062a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.340+0000 7f3899800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f3894062630 0x7f3894062a90 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.342+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38940c6230 con 0x7f3894062630 2024-09-16T01:52:26.363 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:26.363 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:26.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.365+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f38940632a0 con 0x7f3894062630 2024-09-16T01:52:26.372 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:26.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.371+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f38940632a0 con 0x7f3894062630 2024-09-16T01:52:26.373 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:26.374 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:26.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.372+0000 7f389b800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f389402cb50 0x7f389402cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.372+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f38940634b0 con 0x7f389402cb50 2024-09-16T01:52:26.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.372+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f389402cb50 0x7f389402cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.373+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f389402cb50 0x7f389402cf90 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.374+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38940634b0 con 0x7f389402cb50 2024-09-16T01:52:26.377 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:26.377+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:26.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.423+0000 7f389b800640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f38940634b0 con 0x7f389402cb50 2024-09-16T01:52:26.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.425+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f38940634b0 con 0x7f389402cb50 2024-09-16T01:52:26.428 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:26.428 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:26.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.427+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f38942063d0 0x7f38942087c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.427+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f389402d9e0 con 0x7f38942063d0 2024-09-16T01:52:26.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.428+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f38942063d0 0x7f38942087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.429+0000 7f3893e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f38942063d0 0x7f38942087c0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.430+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f389402d9e0 con 0x7f38942063d0 2024-09-16T01:52:26.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.455+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f389402d9e0 con 0x7f38942063d0 2024-09-16T01:52:26.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.460+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f389402d9e0 con 0x7f38942063d0 2024-09-16T01:52:26.462 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:26.462 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:26.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.461+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3894209210 0x7f389420b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.462+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f389420e3d0 con 0x7f3894209210 2024-09-16T01:52:26.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.472+0000 7f3899800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3894209210 0x7f389420b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.472+0000 7f3899800640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3894209210 0x7f389420b600 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.474+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f389420e3d0 con 0x7f3894209210 2024-09-16T01:52:26.477 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:26.476+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:26.516 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:26.516 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.512+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f389420bb40 con 0x7f3894209210 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.515+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f389420bb40 con 0x7f3894209210 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.515+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f389420c370 0x7f389420c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.515+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3894211890 con 0x7f389420c370 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.516+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f389420c370 0x7f389420c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.516+0000 7f3898e00640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f389420c370 0x7f389420c7b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.517+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3894211890 con 0x7f389420c370 2024-09-16T01:52:26.526 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:26.526 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:26.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:52:26.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:26.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.549+0000 7f57b9000640 1 Processor -- start 2024-09-16T01:52:26.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.549+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3894211890 con 0x7f389420c370 2024-09-16T01:52:26.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.552+0000 7f3890c00640 1 -- 172.21.15.146:0/495310321 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3894211890 con 0x7f389420c370 2024-09-16T01:52:26.554 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:26.554 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:26.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.551+0000 7f57b9000640 1 -- start start 2024-09-16T01:52:26.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.553+0000 7f57b9000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac0da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.553+0000 7f57b9000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57ac005a30 0x7f57ac034280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.553+0000 7f57b9000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57ac0d4a00 con 0x7f57ac0347c0 2024-09-16T01:52:26.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.553+0000 7f57b9000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f57ac032c40 con 0x7f57ac005a30 2024-09-16T01:52:26.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.553+0000 7f57b9000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f57ac006000 con 0x7f57ac0d4620 2024-09-16T01:52:26.556 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3894154090 msgr2=0x7f389406cc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f3894154090 0x7f389406cc20 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f38941552e0 msgr2=0x7f3894156990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f38941552e0 0x7f3894156990 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f38941f09b0 msgr2=0x7f38941f0db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f38941f09b0 0x7f38941f0db0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f3894062630 msgr2=0x7f3894062a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f3894062630 0x7f3894062a90 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f389402cb50 msgr2=0x7f389402cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f389402cb50 0x7f389402cf90 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f38942063d0 msgr2=0x7f38942087c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f38942063d0 0x7f38942087c0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3894209210 msgr2=0x7f389420b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f3894209210 0x7f389420b600 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f389420c370 msgr2=0x7f389420c7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f389420c370 0x7f389420c7b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3899800640 1 -- 172.21.15.146:0/495310321 reap_dead start 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3899800640 1 -- 172.21.15.146:0/495310321 reap_dead start 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f3899800640 1 -- 172.21.15.146:0/495310321 reap_dead start 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.555+0000 7f3899800640 1 -- 172.21.15.146:0/495310321 reap_dead start 2024-09-16T01:52:26.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.555+0000 7f3899800640 1 -- 172.21.15.146:0/495310321 reap_dead start 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f57b1e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac0da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.554+0000 7f57b1400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57ac005a30 0x7f57ac034280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.555+0000 7f57b1e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac0da570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45156/0 (socket says 172.21.15.146:45156) 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.555+0000 7f57b1e00640 1 -- 172.21.15.146:0/2155698786 learned_addr learned my addr 172.21.15.146:0/2155698786 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.555+0000 7f57b2800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f57ac0347c0 0x7f57ac04aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:37648/0 (socket says 172.21.15.146:37648) 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f57b1e00640 1 -- 172.21.15.146:0/2155698786 >> v2:172.21.15.161:3300/0 conn(0x7f57ac005a30 msgr2=0x7f57ac034280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f57b1e00640 1 --2- 172.21.15.146:0/2155698786 >> v2:172.21.15.161:3300/0 conn(0x7f57ac005a30 0x7f57ac034280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f57b1e00640 1 -- 172.21.15.146:0/2155698786 >> v1:172.21.15.161:6789/0 conn(0x7f57ac0347c0 legacy=0x7f57ac04aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f57b1e00640 1 -- 172.21.15.146:0/2155698786 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac0062e0 con 0x7f57ac0d4620 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f387406c2b0 msgr2=0x7f387406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f387406c2b0 0x7f387406e770 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3880034000 tx=0x7f388002e000 comp rx=0 tx=0).stop 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 msgr2=0x7f38941559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.556+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 0x7f38941559c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f388800c3f0 tx=0x7f388800c8b0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.557+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 shutdown_connections 2024-09-16T01:52:26.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.557+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f387406c2b0 0x7f387406e770 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.557+0000 7f3886000640 1 --2- 172.21.15.146:0/495310321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3894070d40 0x7f38941559c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.557+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 >> 172.21.15.146:0/495310321 conn(0x7f3894072a60 msgr2=0x7f389415ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:26.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.558+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 shutdown_connections 2024-09-16T01:52:26.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.558+0000 7f3886000640 1 -- 172.21.15.146:0/495310321 wait complete. 2024-09-16T01:52:26.574 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:52:26.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.560+0000 7f57b1e00640 1 --2- 172.21.15.146:0/2155698786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac0da570 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009870 tx=0x7f57a002a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b42224df8b238f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.560+0000 7f57b0a00640 1 -- 172.21.15.146:0/2155698786 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0037070 con 0x7f57ac0d4620 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.560+0000 7f57b0a00640 1 -- 172.21.15.146:0/2155698786 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57a0032440 con 0x7f57ac0d4620 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.560+0000 7f57b0a00640 1 -- 172.21.15.146:0/2155698786 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0032810 con 0x7f57ac0d4620 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.563+0000 7f57b9000640 1 -- 172.21.15.146:0/2155698786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 msgr2=0x7f57ac0da570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.563+0000 7f57b9000640 1 --2- 172.21.15.146:0/2155698786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac0da570 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009870 tx=0x7f57a002a290 comp rx=0 tx=0).stop 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.563+0000 7f57b9000640 1 -- 172.21.15.146:0/2155698786 shutdown_connections 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.563+0000 7f57b9000640 1 --2- 172.21.15.146:0/2155698786 >> v2:172.21.15.161:3300/0 conn(0x7f57ac005a30 0x7f57ac034280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.563+0000 7f57b9000640 1 --2- 172.21.15.146:0/2155698786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac0da570 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.563+0000 7f57b9000640 1 -- 172.21.15.146:0/2155698786 >> 172.21.15.146:0/2155698786 conn(0x7f57ac04fc80 msgr2=0x7f57ac050090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.567+0000 7f57b9000640 1 -- 172.21.15.146:0/2155698786 shutdown_connections 2024-09-16T01:52:26.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.568+0000 7f57b9000640 1 -- 172.21.15.146:0/2155698786 wait complete. 2024-09-16T01:52:26.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.578+0000 7f57b9000640 1 Processor -- start 2024-09-16T01:52:26.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.579+0000 7f57b9000640 1 -- start start 2024-09-16T01:52:26.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.581+0000 7f57b9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ac005a30 0x7f57ac165a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.582+0000 7f57b9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57ac0347c0 0x7f57ac0da4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.582+0000 7f57b1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ac005a30 0x7f57ac165a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.582+0000 7f57b1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ac005a30 0x7f57ac165a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44658/0 (socket says 172.21.15.146:44658) 2024-09-16T01:52:26.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.583+0000 7f57b9000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac184eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.586+0000 7f57b9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac006b80 con 0x7f57ac0347c0 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.586+0000 7f57b9000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac0068a0 con 0x7f57ac005a30 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.586+0000 7f57b9000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f57ac006e60 con 0x7f57ac0d4620 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.583+0000 7f57b1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57ac0347c0 0x7f57ac0da4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.586+0000 7f57b2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac184eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.586+0000 7f57b2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac184eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45158/0 (socket says 172.21.15.146:45158) 2024-09-16T01:52:26.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.586+0000 7f57b2800640 1 -- 172.21.15.146:0/3187200786 learned_addr learned my addr 172.21.15.146:0/3187200786 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b2800640 1 -- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ac005a30 msgr2=0x7f57ac165a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b2800640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ac005a30 0x7f57ac165a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b2800640 1 -- 172.21.15.146:0/3187200786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57ac0347c0 msgr2=0x7f57ac0da4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b2800640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57ac0347c0 0x7f57ac0da4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b2800640 1 -- 172.21.15.146:0/3187200786 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac160880 con 0x7f57ac0d4620 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b1e00640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ac005a30 0x7f57ac165a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b1400640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57ac0347c0 0x7f57ac0da4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:26.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57b2800640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f57ac0d4620 0x7f57ac184eb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57b406b730 tx=0x7f57b4069890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.587+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4091070 con 0x7f57ac0d4620 2024-09-16T01:52:26.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.588+0000 7f57b9000640 1 -- 172.21.15.146:0/3187200786 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac0d9df0 con 0x7f57ac0d4620 2024-09-16T01:52:26.590 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:26.583+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:26.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.588+0000 7f57b9000640 1 -- 172.21.15.146:0/3187200786 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac0d83e0 con 0x7f57ac0d4620 2024-09-16T01:52:26.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.588+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b4071890 con 0x7f57ac0d4620 2024-09-16T01:52:26.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.588+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4091070 con 0x7f57ac0d4620 2024-09-16T01:52:26.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.592+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f57b4094070 con 0x7f57ac0d4620 2024-09-16T01:52:26.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.592+0000 7f57b9000640 1 -- 172.21.15.146:0/3187200786 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f578c001030 con 0x7f57ac0d4620 2024-09-16T01:52:26.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.593+0000 7f57aa000640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f579406c4b0 0x7f579406e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.599+0000 7f57b1e00640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f579406c4b0 0x7f579406e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.599+0000 7f57b1e00640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f579406c4b0 0x7f579406e970 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034000 tx=0x7f57a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.600+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7f57b4107340 con 0x7f57ac0d4620 2024-09-16T01:52:26.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.600+0000 7f57aa000640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f57940746b0 0x7f5794076ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.601+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57940771a0 con 0x7f57940746b0 2024-09-16T01:52:26.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.601+0000 7f57b1400640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f57940746b0 0x7f5794076ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.603+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f57b410e050 con 0x7f57ac0d4620 2024-09-16T01:52:26.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.605+0000 7f57b1400640 1 --2- 172.21.15.146:0/3187200786 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f57940746b0 0x7f5794076ad0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.608+0000 7f57aa000640 1 -- 172.21.15.146:0/3187200786 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f57940771a0 con 0x7f57940746b0 2024-09-16T01:52:26.674 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:bench [] [] [] [] 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs debug_inject_read_zeros 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs files list 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs stats 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:bluestore allocator dump block 2024-09-16T01:52:26.675 INFO:teuthology.orchestra.run.smithi146.stderr:Error EINVAL: invalid command 2024-09-16T01:52:26.679 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:26.678+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T01:52:26.688 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T01:52:26.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-16T01:52:26.779 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:26.777+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:26.880 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:26.879+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:26.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.891+0000 7fd5e9a00640 1 Processor -- start 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.892+0000 7fd5e9a00640 1 -- start start 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e9a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d8bb0 0x7fd5dc04ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e9a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd5dc007340 0x7fd5dc005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5dc050520 con 0x7fd5dc04f870 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e9a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd5dc050800 con 0x7fd5dc0d8bb0 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e9a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd5dc04f070 con 0x7fd5dc007340 2024-09-16T01:52:26.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d8bb0 0x7fd5dc04ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d8bb0 0x7fd5dc04ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35866/0 (socket says 172.21.15.146:35866) 2024-09-16T01:52:26.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.894+0000 7fd5e3e00640 1 -- 172.21.15.146:0/3093546412 learned_addr learned my addr 172.21.15.146:0/3093546412 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:26.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.895+0000 7fd5e3400640 1 --2- 172.21.15.146:0/3093546412 >> v2:172.21.15.161:3300/0 conn(0x7fd5dc007340 0x7fd5dc005d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.895+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898535985 0 0) 0x7fd5dc050520 con 0x7fd5dc04f870 2024-09-16T01:52:26.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.895+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5cc003520 con 0x7fd5dc04f870 2024-09-16T01:52:26.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.896+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2575316144 0 0) 0x7fd5cc003520 con 0x7fd5dc04f870 2024-09-16T01:52:26.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.896+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 >> v2:172.21.15.161:3300/0 conn(0x7fd5dc007340 msgr2=0x7fd5dc005d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 --2- 172.21.15.146:0/3093546412 >> v2:172.21.15.161:3300/0 conn(0x7fd5dc007340 0x7fd5dc005d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d8bb0 msgr2=0x7fd5dc04ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 --2- 172.21.15.146:0/3093546412 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d8bb0 0x7fd5dc04ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc04f350 con 0x7fd5dc04f870 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fd5e4074810 con 0x7fd5dc04f870 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e3e00640 1 --2- 172.21.15.146:0/3093546412 >> v2:172.21.15.146:3300/0 conn(0x7fd5dc0d8bb0 0x7fd5dc04ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 >> v1:172.21.15.146:6789/0 conn(0x7fd5dc04f870 legacy=0x7fd5dc00e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5cc0039b0 0x7fd5cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc050800 con 0x7fd5cc004800 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc04f070 con 0x7fd5cc0039b0 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd5cc00bf80 con 0x7fd5cc0051c0 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e3400640 1 --2- 172.21.15.146:0/3093546412 >> v2:172.21.15.161:3300/0 conn(0x7fd5dc007340 0x7fd5dc005d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.899+0000 7fd5e9000640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.900+0000 7fd5e3400640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.900+0000 7fd5e9000640 1 -- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5cc0039b0 msgr2=0x7fd5cc003dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.900+0000 7fd5e9000640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5cc0039b0 0x7fd5cc003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.900+0000 7fd5e9000640 1 -- 172.21.15.146:0/3093546412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5cc004800 msgr2=0x7fd5cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.900+0000 7fd5e9000640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.900+0000 7fd5e9000640 1 -- 172.21.15.146:0/3093546412 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc04f350 con 0x7fd5cc0051c0 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.901+0000 7fd5e9000640 1 -- 172.21.15.146:0/3093546412 reap_dead start 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.903+0000 7fd5e3400640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5cc004800 0x7fd5cc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.904+0000 7fd5e9000640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5cc00b9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40504d0 tx=0x7fd5cc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95fe1e3a64ef11cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.904+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4050800 con 0x7fd5cc0051c0 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.904+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e40ab070 con 0x7fd5cc0051c0 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.904+0000 7fd5e2a00640 1 -- 172.21.15.146:0/3093546412 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4063020 con 0x7fd5cc0051c0 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.906+0000 7fd5e9a00640 1 -- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 msgr2=0x7fd5cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.906+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5cc00b9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40504d0 tx=0x7fd5cc005830 comp rx=0 tx=0).stop 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.906+0000 7fd5e9a00640 1 -- 172.21.15.146:0/3093546412 shutdown_connections 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.906+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/3093546412 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5cc00b9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.906+0000 7fd5e9a00640 1 -- 172.21.15.146:0/3093546412 >> 172.21.15.146:0/3093546412 conn(0x7fd5dc054470 msgr2=0x7fd5dc054880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:26.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.907+0000 7fd5e9a00640 1 -- 172.21.15.146:0/3093546412 shutdown_connections 2024-09-16T01:52:26.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.907+0000 7fd5e9a00640 1 -- 172.21.15.146:0/3093546412 wait complete. 2024-09-16T01:52:26.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.910+0000 7fd5e9a00640 1 Processor -- start 2024-09-16T01:52:26.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.910+0000 7fd5e9a00640 1 -- start start 2024-09-16T01:52:26.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.912+0000 7fd5e9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5dc04b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.912+0000 7fd5e9a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5dc174a70 0x7fd5dc04dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.913+0000 7fd5e3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5dc04b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.913+0000 7fd5e3e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5dc04b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45190/0 (socket says 172.21.15.146:45190) 2024-09-16T01:52:26.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.913+0000 7fd5e9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 0x7fd5dc0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.914+0000 7fd5e9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 0x7fd5dc0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.914+0000 7fd5e9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 0x7fd5dc0cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35880/0 (socket says 172.21.15.146:35880) 2024-09-16T01:52:26.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.914+0000 7fd5e3e00640 1 -- 172.21.15.146:0/590265626 learned_addr learned my addr 172.21.15.146:0/590265626 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:26.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.914+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc049340 con 0x7fd5dc0cfb40 2024-09-16T01:52:26.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.914+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc0491c0 con 0x7fd5dc174a70 2024-09-16T01:52:26.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.914+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd5dc04feb0 con 0x7fd5cc0051c0 2024-09-16T01:52:26.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5dc174a70 0x7fd5dc04dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 -- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 msgr2=0x7fd5dc04b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5dc04b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 -- 172.21.15.146:0/590265626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 msgr2=0x7fd5dc0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:26.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 0x7fd5dc0cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc04c110 con 0x7fd5dc174a70 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e9000640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 0x7fd5dc0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.915+0000 7fd5e3400640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5dc174a70 0x7fd5dc04dbe0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000f490 tx=0x7fd5d000f950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.916+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0018070 con 0x7fd5dc174a70 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.916+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d0004220 con 0x7fd5dc174a70 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.916+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0014690 con 0x7fd5dc174a70 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.916+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5dc04c3b0 con 0x7fd5dc174a70 2024-09-16T01:52:26.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.916+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5dc04d3e0 con 0x7fd5dc174a70 2024-09-16T01:52:26.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.918+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5dc04ea00 con 0x7fd5dc174a70 2024-09-16T01:52:26.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.919+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd5d0012070 con 0x7fd5dc174a70 2024-09-16T01:52:26.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.920+0000 7fd5d7e00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd5c406c380 0x7fd5c406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:26.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.920+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7fd5d008b070 con 0x7fd5dc174a70 2024-09-16T01:52:26.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.920+0000 7fd5e3e00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd5c406c380 0x7fd5c406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:26.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.921+0000 7fd5e3e00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd5c406c380 0x7fd5c406e840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009b10 tx=0x7fd5d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:26.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:26.926+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd5d00591e0 con 0x7fd5dc174a70 2024-09-16T01:52:26.982 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:26.981+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:27.055 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:27.055 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:27.055 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:27.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.080+0000 7eff9c600640 1 Processor -- start 2024-09-16T01:52:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.081+0000 7eff9c600640 1 -- start start 2024-09-16T01:52:27.082 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:27.081+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:27.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.086+0000 7eff9c600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7eff94154d50 0x7eff940c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.087+0000 7eff9c600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 0x7eff9415e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.087+0000 7eff99c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7eff94154d50 0x7eff940c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff9c600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff9415a9b0 0x7eff940c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff9c600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7eff94155130 con 0x7eff94154d50 2024-09-16T01:52:27.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff9c600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7eff9415ee20 con 0x7eff9415a9b0 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff9c600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7eff940c9db0 con 0x7eff9406bd30 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff99200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 0x7eff9415e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff99200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 0x7eff9415e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45204/0 (socket says 172.21.15.146:45204) 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.088+0000 7eff99200640 1 -- 172.21.15.146:0/2867044839 learned_addr learned my addr 172.21.15.146:0/2867044839 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.089+0000 7eff9a600640 1 --2- 172.21.15.146:0/2867044839 >> v2:172.21.15.161:3300/0 conn(0x7eff9415a9b0 0x7eff940c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.089+0000 7eff99200640 1 -- 172.21.15.146:0/2867044839 >> v2:172.21.15.161:3300/0 conn(0x7eff9415a9b0 msgr2=0x7eff940c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:27.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.089+0000 7eff99200640 1 --2- 172.21.15.146:0/2867044839 >> v2:172.21.15.161:3300/0 conn(0x7eff9415a9b0 0x7eff940c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.089+0000 7eff99200640 1 -- 172.21.15.146:0/2867044839 >> v2:172.21.15.146:3300/0 conn(0x7eff94154d50 msgr2=0x7eff940c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:27.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.089+0000 7eff99200640 1 --2- 172.21.15.146:0/2867044839 >> v2:172.21.15.146:3300/0 conn(0x7eff94154d50 0x7eff940c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.089+0000 7eff99200640 1 -- 172.21.15.146:0/2867044839 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff940ca090 con 0x7eff9406bd30 2024-09-16T01:52:27.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.091+0000 7eff99200640 1 --2- 172.21.15.146:0/2867044839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 0x7eff9415e8e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7eff940701d0 tx=0x7eff8002a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c6bd315c43059b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.091+0000 7eff93e00640 1 -- 172.21.15.146:0/2867044839 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80037070 con 0x7eff9406bd30 2024-09-16T01:52:27.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.091+0000 7eff93e00640 1 -- 172.21.15.146:0/2867044839 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff80032440 con 0x7eff9406bd30 2024-09-16T01:52:27.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.091+0000 7eff93e00640 1 -- 172.21.15.146:0/2867044839 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80032810 con 0x7eff9406bd30 2024-09-16T01:52:27.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.093+0000 7eff9c600640 1 -- 172.21.15.146:0/2867044839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 msgr2=0x7eff9415e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.093+0000 7eff9c600640 1 --2- 172.21.15.146:0/2867044839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 0x7eff9415e8e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7eff940701d0 tx=0x7eff8002a1d0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.094+0000 7eff9c600640 1 -- 172.21.15.146:0/2867044839 shutdown_connections 2024-09-16T01:52:27.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.094+0000 7eff9c600640 1 --2- 172.21.15.146:0/2867044839 >> v2:172.21.15.161:3300/0 conn(0x7eff9415a9b0 0x7eff940c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.094+0000 7eff9c600640 1 --2- 172.21.15.146:0/2867044839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9406bd30 0x7eff9415e8e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.094+0000 7eff9c600640 1 --2- 172.21.15.146:0/2867044839 >> v2:172.21.15.146:3300/0 conn(0x7eff94154d50 0x7eff940c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.094+0000 7eff9c600640 1 -- 172.21.15.146:0/2867044839 >> 172.21.15.146:0/2867044839 conn(0x7eff94068140 msgr2=0x7eff94070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:27.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.095+0000 7eff9c600640 1 -- 172.21.15.146:0/2867044839 shutdown_connections 2024-09-16T01:52:27.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.095+0000 7eff9c600640 1 -- 172.21.15.146:0/2867044839 wait complete. 2024-09-16T01:52:27.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.097+0000 7eff9c600640 1 Processor -- start 2024-09-16T01:52:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.099+0000 7eff9c600640 1 -- start start 2024-09-16T01:52:27.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.100+0000 7eff9c600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff9406bd30 0x7eff9415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff9c600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94154d50 0x7eff94147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff9c600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff9415a9b0 0x7eff9415cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff9c600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7eff9415e3c0 con 0x7eff9415a9b0 2024-09-16T01:52:27.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff9c600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff9415e0e0 con 0x7eff9406bd30 2024-09-16T01:52:27.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff9c600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7eff940c7320 con 0x7eff94154d50 2024-09-16T01:52:27.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff99c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff9406bd30 0x7eff9415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff99c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff9406bd30 0x7eff9415bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44714/0 (socket says 172.21.15.146:44714) 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff99c00640 1 -- 172.21.15.146:0/2213980288 learned_addr learned my addr 172.21.15.146:0/2213980288 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.101+0000 7eff99200640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94154d50 0x7eff94147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff9a600640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff9415a9b0 0x7eff9415cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff99c00640 1 -- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94154d50 msgr2=0x7eff94147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff99c00640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94154d50 0x7eff94147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff99c00640 1 -- 172.21.15.146:0/2213980288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff9415a9b0 msgr2=0x7eff9415cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff99c00640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff9415a9b0 0x7eff9415cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff99c00640 1 -- 172.21.15.146:0/2213980288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff94155570 con 0x7eff9406bd30 2024-09-16T01:52:27.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.102+0000 7eff9a600640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff9415a9b0 0x7eff9415cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:27.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.103+0000 7eff99c00640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff9406bd30 0x7eff9415bfb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7eff8800d800 tx=0x7eff8800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.105+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88018020 con 0x7eff9406bd30 2024-09-16T01:52:27.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.106+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff880096a0 con 0x7eff9406bd30 2024-09-16T01:52:27.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.106+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88009a70 con 0x7eff9406bd30 2024-09-16T01:52:27.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.106+0000 7eff9c600640 1 -- 172.21.15.146:0/2213980288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff94155850 con 0x7eff9406bd30 2024-09-16T01:52:27.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.106+0000 7eff9c600640 1 -- 172.21.15.146:0/2213980288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff94155e10 con 0x7eff9406bd30 2024-09-16T01:52:27.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.108+0000 7eff9c600640 1 -- 172.21.15.146:0/2213980288 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff94155320 con 0x7eff9406bd30 2024-09-16T01:52:27.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.110+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7eff8801b020 con 0x7eff9406bd30 2024-09-16T01:52:27.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.117+0000 7eff91600640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7eff7406c380 0x7eff7406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.117+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7eff8808b650 con 0x7eff9406bd30 2024-09-16T01:52:27.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.117+0000 7eff91600640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7eff740745e0 0x7eff74076a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.117+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff88010e50 con 0x7eff740745e0 2024-09-16T01:52:27.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.118+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7eff88092050 con 0x7eff9406bd30 2024-09-16T01:52:27.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.119+0000 7eff9a600640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7eff740745e0 0x7eff74076a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.121+0000 7eff99200640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7eff7406c380 0x7eff7406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.125+0000 7eff9a600640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7eff740745e0 0x7eff74076a00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.125+0000 7eff99200640 1 --2- 172.21.15.146:0/2213980288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7eff7406c380 0x7eff7406e840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7eff80034000 tx=0x7eff8002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.127+0000 7eff91600640 1 -- 172.21.15.146:0/2213980288 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff88010e50 con 0x7eff740745e0 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:bench [] [] [] [] 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs debug_inject_read_zeros 2024-09-16T01:52:27.179 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs files list 2024-09-16T01:52:27.180 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs stats 2024-09-16T01:52:27.180 INFO:teuthology.orchestra.run.smithi146.stderr:bluestore allocator dump block 2024-09-16T01:52:27.180 INFO:teuthology.orchestra.run.smithi146.stderr:Error EINVAL: invalid command 2024-09-16T01:52:27.183 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:27.182+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T01:52:27.189 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T01:52:27.190 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-16T01:52:27.284 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:27.283+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:27.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.308+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fd5dc04e4b0 con 0x7fd5dc174a70 2024-09-16T01:52:27.388 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:27.387+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:27.417 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:27.417 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:27.417 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:27.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:27.524 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:27.523+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:27.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.547+0000 7f4c09800640 1 Processor -- start 2024-09-16T01:52:27.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.547+0000 7f4c09800640 1 -- start start 2024-09-16T01:52:27.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.549+0000 7f4c09800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c040cc0c0 0x7f4c0415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.549+0000 7f4c09800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4c040cc4a0 0x7f4c04165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.549+0000 7f4c09800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c0415b1d0 con 0x7f4c040cb2f0 2024-09-16T01:52:27.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.549+0000 7f4c09800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4c040cbd00 con 0x7f4c040cc0c0 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.549+0000 7f4c09800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4c0415ba70 con 0x7f4c040cc4a0 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c02000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4c040cb2f0 0x7f4c0415e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52494/0 (socket says 172.21.15.146:52494) 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c02000640 1 -- 172.21.15.146:0/3806700609 learned_addr learned my addr 172.21.15.146:0/3806700609 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c03400640 1 --2- 172.21.15.146:0/3806700609 >> v2:172.21.15.161:3300/0 conn(0x7f4c040cc4a0 0x7f4c04165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c02a00640 1 --2- 172.21.15.146:0/3806700609 >> v2:172.21.15.146:3300/0 conn(0x7f4c040cc0c0 0x7f4c0415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c03400640 1 -- 172.21.15.146:0/3806700609 >> v2:172.21.15.146:3300/0 conn(0x7f4c040cc0c0 msgr2=0x7f4c0415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c03400640 1 --2- 172.21.15.146:0/3806700609 >> v2:172.21.15.146:3300/0 conn(0x7f4c040cc0c0 0x7f4c0415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.550+0000 7f4c03400640 1 -- 172.21.15.146:0/3806700609 >> v1:172.21.15.146:6789/0 conn(0x7f4c040cb2f0 legacy=0x7f4c0415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.553+0000 7f4c03400640 1 -- 172.21.15.146:0/3806700609 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0415b4b0 con 0x7f4c040cc4a0 2024-09-16T01:52:27.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.554+0000 7f4c02a00640 1 --2- 172.21.15.146:0/3806700609 >> v2:172.21.15.146:3300/0 conn(0x7f4c040cc0c0 0x7f4c0415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:27.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.556+0000 7f4c03400640 1 --2- 172.21.15.146:0/3806700609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cc4a0 0x7f4c04165430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406e100 tx=0x7f4bfc03b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8542de25bd03bb93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.556+0000 7f4c01600640 1 -- 172.21.15.146:0/3806700609 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc007740 con 0x7f4c040cc4a0 2024-09-16T01:52:27.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.556+0000 7f4c01600640 1 -- 172.21.15.146:0/3806700609 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bfc0078e0 con 0x7f4c040cc4a0 2024-09-16T01:52:27.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.556+0000 7f4c01600640 1 -- 172.21.15.146:0/3806700609 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc007cf0 con 0x7f4c040cc4a0 2024-09-16T01:52:27.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.557+0000 7f4c09800640 1 -- 172.21.15.146:0/3806700609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cc4a0 msgr2=0x7f4c04165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.557+0000 7f4c09800640 1 --2- 172.21.15.146:0/3806700609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cc4a0 0x7f4c04165430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406e100 tx=0x7f4bfc03b6b0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.558+0000 7f4c09800640 1 -- 172.21.15.146:0/3806700609 shutdown_connections 2024-09-16T01:52:27.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.558+0000 7f4c09800640 1 --2- 172.21.15.146:0/3806700609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cc4a0 0x7f4c04165430 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.558+0000 7f4c09800640 1 --2- 172.21.15.146:0/3806700609 >> v2:172.21.15.146:3300/0 conn(0x7f4c040cc0c0 0x7f4c0415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.558+0000 7f4c09800640 1 -- 172.21.15.146:0/3806700609 >> 172.21.15.146:0/3806700609 conn(0x7f4c0406d560 msgr2=0x7f4c0406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:27.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.558+0000 7f4c09800640 1 -- 172.21.15.146:0/3806700609 shutdown_connections 2024-09-16T01:52:27.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.558+0000 7f4c09800640 1 -- 172.21.15.146:0/3806700609 wait complete. 2024-09-16T01:52:27.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.560+0000 7f4c09800640 1 Processor -- start 2024-09-16T01:52:27.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.560+0000 7f4c09800640 1 -- start start 2024-09-16T01:52:27.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.561+0000 7f4c09800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cb2f0 0x7f4c040c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.561+0000 7f4c09800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c040cc0c0 0x7f4c040b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.561+0000 7f4c09800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c040cc4a0 0x7f4c041445c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.561+0000 7f4c09800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0415e1d0 con 0x7f4c040cc4a0 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.561+0000 7f4c09800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0415e540 con 0x7f4c040cb2f0 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.561+0000 7f4c09800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4c040cc8b0 con 0x7f4c040cc0c0 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cb2f0 0x7f4c040c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cb2f0 0x7f4c040c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44722/0 (socket says 172.21.15.146:44722) 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 -- 172.21.15.146:0/3094828351 learned_addr learned my addr 172.21.15.146:0/3094828351 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:27.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02000640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c040cc0c0 0x7f4c040b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c03400640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c040cc4a0 0x7f4c041445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 -- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c040cc0c0 msgr2=0x7f4c040b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4c040cc0c0 0x7f4c040b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 -- 172.21.15.146:0/3094828351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c040cc4a0 msgr2=0x7f4c041445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.562+0000 7f4c02a00640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c040cc4a0 0x7f4c041445c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4c02a00640 1 -- 172.21.15.146:0/3094828351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c040b6920 con 0x7f4c040cb2f0 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4c03400640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c040cc4a0 0x7f4c041445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:27.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4c02a00640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c040cb2f0 0x7f4c040c8380 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400c850 tx=0x7f4bf400cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4015070 con 0x7f4c040cb2f0 2024-09-16T01:52:27.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf4007a30 con 0x7f4c040cb2f0 2024-09-16T01:52:27.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4c09800640 1 -- 172.21.15.146:0/3094828351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0414f5a0 con 0x7f4c040cb2f0 2024-09-16T01:52:27.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.563+0000 7f4c09800640 1 -- 172.21.15.146:0/3094828351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c040b6640 con 0x7f4c040cb2f0 2024-09-16T01:52:27.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.564+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf400cea0 con 0x7f4c040cb2f0 2024-09-16T01:52:27.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.565+0000 7f4c09800640 1 -- 172.21.15.146:0/3094828351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c040c5560 con 0x7f4c040cb2f0 2024-09-16T01:52:27.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.566+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4bf4018070 con 0x7f4c040cb2f0 2024-09-16T01:52:27.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.571+0000 7f4bfaa00640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4be406c2b0 0x7f4be406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.571+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5741+0+0 (secure 0 0 0) 0x7f4bf408b4e0 con 0x7f4c040cb2f0 2024-09-16T01:52:27.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.571+0000 7f4bfaa00640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4be40744b0 0x7f4be40768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:27.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.572+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bf401ce50 con 0x7f4be40744b0 2024-09-16T01:52:27.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.572+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7f4bf4092050 con 0x7f4c040cb2f0 2024-09-16T01:52:27.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.572+0000 7f4c02000640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4be406c2b0 0x7f4be406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.572+0000 7f4c03400640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4be40744b0 0x7f4be40768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:27.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.574+0000 7f4c02000640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4be406c2b0 0x7f4be406e770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4c040cab00 tx=0x7f4bf0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.578+0000 7f4c03400640 1 --2- 172.21.15.146:0/3094828351 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f4be40744b0 0x7f4be40768d0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:27.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.579+0000 7f4bfaa00640 1 -- 172.21.15.146:0/3094828351 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bf401ce50 con 0x7f4be40744b0 2024-09-16T01:52:27.586 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:27.584+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:27.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.584+0000 7fd5d7e00640 1 -- 172.21.15.146:0/590265626 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v102) ==== 101+0+0 (secure 0 0 0) 0x7fd5d005d4e0 con 0x7fd5dc174a70 2024-09-16T01:52:27.587 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:52:27.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.597+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd5c406c380 msgr2=0x7fd5c406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd5c406c380 0x7fd5c406e840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009b10 tx=0x7fd5d8008040 comp rx=0 tx=0).stop 2024-09-16T01:52:27.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5dc174a70 msgr2=0x7fd5dc04dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:27.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5dc174a70 0x7fd5dc04dbe0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000f490 tx=0x7fd5d000f950 comp rx=0 tx=0).stop 2024-09-16T01:52:27.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 shutdown_connections 2024-09-16T01:52:27.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd5c406c380 0x7fd5c406e840 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009b10 tx=0x7fd5d8008040 comp rx=0 tx=0).stop 2024-09-16T01:52:27.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5dc0cfb40 0x7fd5dc0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5dc174a70 0x7fd5dc04dbe0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 --2- 172.21.15.146:0/590265626 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd5cc0051c0 0x7fd5dc04b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:27.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.598+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 >> 172.21.15.146:0/590265626 conn(0x7fd5dc054470 msgr2=0x7fd5dc006650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:27.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.601+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 shutdown_connections 2024-09-16T01:52:27.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:27.602+0000 7fd5e9a00640 1 -- 172.21.15.146:0/590265626 wait complete. 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:bench [] [] [] [] 2024-09-16T01:52:27.634 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs debug_inject_read_zeros 2024-09-16T01:52:27.635 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs files list 2024-09-16T01:52:27.635 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs stats 2024-09-16T01:52:27.635 INFO:teuthology.orchestra.run.smithi146.stderr:bluestore allocator dump block 2024-09-16T01:52:27.635 INFO:teuthology.orchestra.run.smithi146.stderr:Error EINVAL: invalid command 2024-09-16T01:52:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T01:52:27.655 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T01:52:27.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-16T01:52:27.687 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:27.685+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:478 3.c 3:32521e74:::10000000002.000000db:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.742942737000007, 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.88174018200000004, 2024-09-16T01:52:27.779 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 478 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025089+0000", 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.025090+0000", 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2699999999999998e-07 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047858+0000", 2024-09-16T01:52:27.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022768360000000001 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047892+0000", 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3855e-05 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.047918+0000", 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5595000000000001e-05 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048026+0000", 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108242 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.048035+0000", 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3940000000000002e-06 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828462+0000", 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.78042635999999999 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831445+0000", 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029837000000000002 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882498+0000", 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051052510000000002 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882597+0000", 2024-09-16T01:52:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8874000000000002e-05 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.882746+0000", 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000149457 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.903108+0000", 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020361516 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906632+0000", 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035239960000000002 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906720+0000", 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7952000000000001e-05 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.906829+0000", 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109444 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.786 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:777 3.2c 3:36199bb1:::100000001f7.00000006:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.555566825, 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.106547593, 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 777 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212465+0000", 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.212467+0000", 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1399999999999998e-06 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221218+0000", 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087507980000000006 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221238+0000", 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9729999999999999e-05 2024-09-16T01:52:27.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.221249+0000", 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1462e-05 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222669+0000", 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001420017 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.790 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222797+0000", 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127572 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222974+0000", 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017753699999999999 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.258753+0000", 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035779068999999997 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318898+0000", 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060144465000000001 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.318963+0000", 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5098999999999995e-05 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.319012+0000", 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9703999999999997e-05 2024-09-16T01:52:27.792 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1127 3.36 3:6e2a8853:::100000001f8.00000064:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.463838435, 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15931991200000001, 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.793 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1127 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304193+0000", 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.304196+0000", 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.526e-06 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354927+0000", 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050730866999999999 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355561+0000", 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00063468300000000004 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355571+0000", 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6199999999999994e-06 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355665+0000", 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3875000000000005e-05 2024-09-16T01:52:27.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355819+0000", 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015463900000000001 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355964+0000", 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000144241 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.393080+0000", 2024-09-16T01:52:27.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037116647000000003 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463398+0000", 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.070317222999999998 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463456+0000", 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8749999999999998e-05 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.463513+0000", 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6841e-05 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1218 3.3f 3:ffd44cf4:::100000001f8.000000bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.487657394000003, 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.121928352, 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1218 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280374+0000", 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.280376+0000", 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8980000000000001e-06 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360384+0000", 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080007935000000002 2024-09-16T01:52:27.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360415+0000", 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1007999999999999e-05 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360429+0000", 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4233000000000001e-05 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360518+0000", 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8465000000000001e-05 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360685+0000", 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016704299999999999 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.360842+0000", 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000157496 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.379376+0000", 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018533701999999999 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402192+0000", 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022815932000000001 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402254+0000", 2024-09-16T01:52:27.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2378000000000005e-05 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.402303+0000", 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8262e-05 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1460 3.2a 3:574d3868:::100000001f8.000001b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:52:27.805 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.381664559999997, 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114279699, 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1460 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:52:27.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386367+0000", 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.386369+0000", 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1969999999999999e-06 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447453+0000", 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.061083239999999997 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447479+0000", 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5941000000000001e-05 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447490+0000", 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1967000000000001e-05 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447627+0000", 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013671700000000001 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447784+0000", 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015693 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.447948+0000", 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016412100000000001 2024-09-16T01:52:27.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.467094+0000", 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019145590000000001 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.497455+0000", 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030361091 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.499539+0000", 2024-09-16T01:52:27.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020837780000000001 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.500647+0000", 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001108127 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1474 3.39 3:9e943776:::100000001f8.000001bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:52:27.811 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 44.186351113000001, 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.116146155, 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1474 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:52:27.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581681+0000", 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.581683+0000", 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6740000000000001e-06 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639340+0000", 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057656583999999997 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639448+0000", 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107707 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.639499+0000", 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1391999999999999e-05 2024-09-16T01:52:27.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648061+0000", 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085624870000000006 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648374+0000", 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000312307 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:27.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.648681+0000", 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00030683700000000002 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.693784+0000", 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045103740000000003 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697710+0000", 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039253650000000001 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697780+0000", 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0134000000000003e-05 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.697827+0000", 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6928000000000002e-05 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.817 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1925 3.39 3:9ea071c3:::100000001f8.00000382:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.887968505000003, 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40497902499999999, 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1925 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880063+0000", 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.880065+0000", 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7120000000000001e-06 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905598+0000", 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025532966000000001 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905629+0000", 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0797999999999999e-05 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905642+0000", 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:27.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905790+0000", 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014842100000000001 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.905804+0000", 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3781000000000001e-05 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184322+0000", 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27851797299999997 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184507+0000", 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000185203 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.184707+0000", 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199287 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274277+0000", 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.089570446999999997 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.284842+0000", 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010564756999999999 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.284919+0000", 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6735000000000003e-05 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.285042+0000", 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123626 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1928 3.1b 3:dae8eecd:::100000001f8.00000385:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.858795116000003, 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.44792130200000002, 2024-09-16T01:52:27.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1928 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909237+0000", 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.909238+0000", 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.804e-06 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937404+0000", 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028165758999999999 2024-09-16T01:52:27.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937432+0000", 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7718000000000001e-05 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937443+0000", 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0923e-05 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937583+0000", 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014051800000000001 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937597+0000", 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.367e-05 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321633+0000", 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.38403599100000002 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321793+0000", 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015996000000000001 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.321969+0000", 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017598599999999999 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340671+0000", 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018701613999999998 2024-09-16T01:52:27.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.356957+0000", 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016286421999999998 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.357041+0000", 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3857999999999996e-05 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.357158+0000", 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011707899999999999 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1931 3.39 3:9d369e22:::100000001f8.00000388:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.830262253999997, 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.40258944099999999, 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1931 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937769+0000", 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.937771+0000", 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2839999999999999e-06 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961925+0000", 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02415401 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961962+0000", 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6869000000000002e-05 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.961975+0000", 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.361e-05 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.962078+0000", 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102604 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.962091+0000", 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3216e-05 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.194767+0000", 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.23267554400000001 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.194914+0000", 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147501 2024-09-16T01:52:27.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.195084+0000", 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016995000000000001 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274589+0000", 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079504732999999994 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340176+0000", 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065587200999999998 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340241+0000", 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4921999999999999e-05 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340359+0000", 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000117997 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.837 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1932 3.39 3:9da8c7a0:::100000001f8.00000389:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.800371550000001, 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.37297202400000001, 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1932 2024-09-16T01:52:27.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967660+0000", 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.967663+0000", 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4660000000000002e-06 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990244+0000", 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022581615999999999 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990262+0000", 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7672999999999998e-05 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990271+0000", 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.8410000000000006e-06 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990369+0000", 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7793000000000004e-05 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.990377+0000", 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5040000000000002e-06 2024-09-16T01:52:27.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217601+0000", 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22722361699999999 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217755+0000", 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015434699999999999 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.217927+0000", 2024-09-16T01:52:27.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017202499999999999 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.274595+0000", 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056667704999999999 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340459+0000", 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065864332999999997 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340504+0000", 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5265000000000001e-05 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.340632+0000", 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012783900000000001 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1938 3.39 3:9faf1682:::100000001f8.0000038f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:52:27.844 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.717384404999997, 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.297091671, 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1938 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050647+0000", 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.050649+0000", 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6559999999999999e-06 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067265+0000", 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016615959 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067308+0000", 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3154000000000001e-05 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067333+0000", 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5364e-05 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067439+0000", 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106051 2024-09-16T01:52:27.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.067453+0000", 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3093e-05 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226329+0000", 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15887599599999999 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226492+0000", 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016327299999999999 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.226687+0000", 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019515299999999999 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.323930+0000", 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097242771000000006 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347547+0000", 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023617610000000001 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347630+0000", 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.2270999999999999e-05 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.347739+0000", 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010932 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1944 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.652214567000001, 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.23495701299999999, 2024-09-16T01:52:27.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1944 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115817+0000", 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.115819+0000", 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.26e-06 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189079+0000", 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073259493999999994 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189092+0000", 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2648e-05 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.189102+0000", 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0377000000000001e-05 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191220+0000", 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021178830000000001 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191347+0000", 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127391 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.191478+0000", 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013041699999999999 2024-09-16T01:52:27.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.211598+0000", 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.020119978 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350671+0000", 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.139073536 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350725+0000", 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.4144000000000002e-05 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350774+0000", 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8884999999999997e-05 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.858 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1949 3.2f e84/20 3:f7d79abc:::100000001f8.0000039a:head v 84'35, mlcod=81'33)", 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.539297437999998, 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107175486, 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228734+0000", 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228736+0000", 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.522e-06 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230216+0000", 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001480592 2024-09-16T01:52:27.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230232+0000", 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5648999999999999e-05 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230241+0000", 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7749999999999998e-06 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230314+0000", 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3069000000000001e-05 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230342+0000", 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.828e-05 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.335832+0000", 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10549030299999999 2024-09-16T01:52:27.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.335910+0000", 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7296e-05 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2088 3.3b 3:de25c16a:::100000001f8.00000425:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:52:27.863 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.541816175000001, 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.113643656, 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2088 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226216+0000", 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.226217+0000", 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8899999999999999e-06 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268408+0000", 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042191008000000002 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268436+0000", 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7183999999999999e-05 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268447+0000", 2024-09-16T01:52:27.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1233000000000001e-05 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.268657+0000", 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00020985900000000001 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.269211+0000", 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055385500000000002 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.269780+0000", 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00056933999999999999 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.333074+0000", 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.063294230000000007 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339717+0000", 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0066431490000000001 2024-09-16T01:52:27.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339798+0000", 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0920000000000005e-05 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:51.339859+0000", 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0988e-05 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2445 3.2a 3:566ba4fe:::100000001f8.0000058a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.339881386000002, 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10738498, 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2445 2024-09-16T01:52:27.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428150+0000", 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.428152+0000", 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5209999999999999e-06 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455890+0000", 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027737906 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455910+0000", 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0023000000000001e-05 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455920+0000", 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0628999999999999e-05 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.455993+0000", 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2358000000000001e-05 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.456334+0000", 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034162400000000001 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.456560+0000", 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00022543699999999999 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.489902+0000", 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033341844000000002 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535444+0000", 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045542250999999999 2024-09-16T01:52:27.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535495+0000", 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0973000000000001e-05 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.535535+0000", 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0414000000000003e-05 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2832 3.1b 3:da939a2a:::100000001f8.0000070d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.625776+0000", 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.142256144000001, 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12403525999999999, 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2832 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.625776+0000", 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.625776+0000", 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.625777+0000", 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7e-06 2024-09-16T01:52:27.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681277+0000", 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055500081999999999 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681304+0000", 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6222999999999999e-05 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681316+0000", 2024-09-16T01:52:27.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2237e-05 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681401+0000", 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4767999999999999e-05 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681596+0000", 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019540000000000001 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:27.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.681778+0000", 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000181726 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.725894+0000", 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044116796999999999 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.749693+0000", 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.02379848 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.749761+0000", 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.8441000000000004e-05 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.749811+0000", 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9406000000000001e-05 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.881 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2855 3.f 3:f27ce28a:::100000001f8.00000724:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.873661+0000", 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.894370962, 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.100443244, 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2855 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.873661+0000", 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.873661+0000", 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.873662+0000", 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2219999999999999e-06 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914147+0000", 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040484617000000001 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914171+0000", 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4454e-05 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914180+0000", 2024-09-16T01:52:27.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.4999999999999999e-06 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914282+0000", 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102024 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914451+0000", 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016926899999999999 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:27.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.914626+0000", 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175427 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.939551+0000", 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024924512999999999 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973794+0000", 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034243705999999999 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974058+0000", 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00026316300000000001 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974104+0000", 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6349e-05 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3319 3.c 3:31034f07:::100000001f8.000008f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.215954+0000", 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.552077841999999, 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109990009, 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3319 2024-09-16T01:52:27.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.215954+0000", 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.215954+0000", 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.215956+0000", 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0619999999999999e-06 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290610+0000", 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.074654478999999996 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290645+0000", 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4718999999999997e-05 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290660+0000", 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5279000000000001e-05 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290746+0000", 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5953999999999997e-05 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.290930+0000", 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018327799999999999 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.291117+0000", 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018774800000000001 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.300434+0000", 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0093171179999999992 2024-09-16T01:52:27.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.325761+0000", 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025326284000000001 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.325824+0000", 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3721999999999997e-05 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.325944+0000", 2024-09-16T01:52:27.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119366 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3326 3.3b 3:dcfe8cbf:::100000001f8.000008fb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.299731+0000", 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.468300394, 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12301342799999999, 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3326 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.299731+0000", 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.299731+0000", 2024-09-16T01:52:27.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.299733+0000", 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.968e-06 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.355860+0000", 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.056127019 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356421+0000", 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00056061000000000001 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356511+0000", 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9658000000000001e-05 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356639+0000", 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012836799999999999 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356807+0000", 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016850900000000001 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.356970+0000", 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000163 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.406342+0000", 2024-09-16T01:52:27.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049371874000000003 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422625+0000", 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016282613000000001 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422689+0000", 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4302999999999997e-05 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.422745+0000", 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5506e-05 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3500 3.2a 3:55cec617:::100000001f8.000009a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e93)", 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:07.403698+0000", 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.364333747, 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.111072217, 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:27.900 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3500 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.403698+0000", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.403698+0000", 2024-09-16T01:52:27.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.403700+0000", 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2869999999999999e-06 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439887+0000", 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036186907999999997 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439911+0000", 2024-09-16T01:52:27.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4117e-05 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439921+0000", 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3740000000000004e-06 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.439996+0000", 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5003e-05 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.440179+0000", 2024-09-16T01:52:27.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018289599999999999 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.440354+0000", 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000175179 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.455926+0000", 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.015572236 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.514650+0000", 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.058723643999999998 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.514719+0000", 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9616000000000006e-05 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:07.514770+0000", 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0957e-05 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.905 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:27.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:27.906 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:27.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:27.908 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:27.907+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:28.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.004+0000 7f6c93200640 1 Processor -- start 2024-09-16T01:52:28.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.004+0000 7f6c93200640 1 -- start start 2024-09-16T01:52:28.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.005+0000 7f6c93200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6c8c0cc0c0 0x7f6c8c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.005+0000 7f6c93200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.005+0000 7f6c93200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c15b1d0 con 0x7f6c8c15e9d0 2024-09-16T01:52:28.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.005+0000 7f6c93200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6c8c0cbd00 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.005+0000 7f6c93200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c15ba70 con 0x7f6c8c0cb2f0 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c8be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6c8c0cc0c0 0x7f6c8c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45236/0 (socket says 172.21.15.146:45236) 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 -- 172.21.15.146:0/575180437 learned_addr learned my addr 172.21.15.146:0/575180437 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 -- 172.21.15.146:0/575180437 >> v2:172.21.15.161:3300/0 conn(0x7f6c8c0cc0c0 msgr2=0x7f6c8c15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 --2- 172.21.15.146:0/575180437 >> v2:172.21.15.161:3300/0 conn(0x7f6c8c0cc0c0 0x7f6c8c15aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 -- 172.21.15.146:0/575180437 >> v1:172.21.15.146:6789/0 conn(0x7f6c8c15e9d0 legacy=0x7f6c8c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:28.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.006+0000 7f6c83e00640 1 -- 172.21.15.146:0/575180437 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c15b4b0 con 0x7f6c8c0cb2f0 2024-09-16T01:52:28.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.007+0000 7f6c83e00640 1 --2- 172.21.15.146:0/575180437 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0cc4a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009870 tx=0x7f6c7402a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=851d3a54ef4eafa8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.008+0000 7f6c8b400640 1 -- 172.21.15.146:0/575180437 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74037070 con 0x7f6c8c0cb2f0 2024-09-16T01:52:28.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.008+0000 7f6c8b400640 1 -- 172.21.15.146:0/575180437 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c74032440 con 0x7f6c8c0cb2f0 2024-09-16T01:52:28.009 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:28.007+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.008+0000 7f6c8b400640 1 -- 172.21.15.146:0/575180437 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74032810 con 0x7f6c8c0cb2f0 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.008+0000 7f6c93200640 1 -- 172.21.15.146:0/575180437 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 msgr2=0x7f6c8c0cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.008+0000 7f6c93200640 1 --2- 172.21.15.146:0/575180437 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0cc4a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009870 tx=0x7f6c7402a200 comp rx=0 tx=0).stop 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.009+0000 7f6c93200640 1 -- 172.21.15.146:0/575180437 shutdown_connections 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.009+0000 7f6c93200640 1 --2- 172.21.15.146:0/575180437 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0cc4a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.009+0000 7f6c93200640 1 --2- 172.21.15.146:0/575180437 >> v2:172.21.15.161:3300/0 conn(0x7f6c8c0cc0c0 0x7f6c8c15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.009+0000 7f6c93200640 1 -- 172.21.15.146:0/575180437 >> 172.21.15.146:0/575180437 conn(0x7f6c8c06d560 msgr2=0x7f6c8c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:28.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.009+0000 7f6c93200640 1 -- 172.21.15.146:0/575180437 shutdown_connections 2024-09-16T01:52:28.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.009+0000 7f6c93200640 1 -- 172.21.15.146:0/575180437 wait complete. 2024-09-16T01:52:28.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.011+0000 7f6c93200640 1 Processor -- start 2024-09-16T01:52:28.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.011+0000 7f6c93200640 1 -- start start 2024-09-16T01:52:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.013+0000 7f6c93200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.013+0000 7f6c93200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cc0c0 0x7f6c8c201080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.013+0000 7f6c93200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c8c15e9d0 0x7f6c8c2015c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.013+0000 7f6c93200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c15e1d0 con 0x7f6c8c15e9d0 2024-09-16T01:52:28.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.013+0000 7f6c93200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c15e540 con 0x7f6c8c0cb2f0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.014+0000 7f6c93200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c0c5560 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.014+0000 7f6c83e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cc0c0 0x7f6c8c201080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.014+0000 7f6c83e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cc0c0 0x7f6c8c201080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45250/0 (socket says 172.21.15.146:45250) 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.014+0000 7f6c83e00640 1 -- 172.21.15.146:0/1902741556 learned_addr learned my addr 172.21.15.146:0/1902741556 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c91200640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c8c15e9d0 0x7f6c8c2015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c83e00640 1 -- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c8c0cb2f0 msgr2=0x7f6c8c0b5c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c83e00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c8c0cb2f0 0x7f6c8c0b5c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c83e00640 1 -- 172.21.15.146:0/1902741556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c8c15e9d0 msgr2=0x7f6c8c2015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c83e00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c8c15e9d0 0x7f6c8c2015c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c83e00640 1 -- 172.21.15.146:0/1902741556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c15db10 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.015+0000 7f6c91200640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c8c15e9d0 0x7f6c8c2015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.016+0000 7f6c83e00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c8c0cc0c0 0x7f6c8c201080 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009990 tx=0x7f6c740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.016+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74041020 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.016+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c7403d6e0 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.016+0000 7f6c93200640 1 -- 172.21.15.146:0/1902741556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c15ddb0 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.016+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74037040 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.016+0000 7f6c93200640 1 -- 172.21.15.146:0/1902741556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c0c7d40 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.018+0000 7f6c93200640 1 -- 172.21.15.146:0/1902741556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c8c0c8670 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.019+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6c74044020 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.020+0000 7f6c88c00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c6806c2b0 0x7f6c6806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.020+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 5741+0+0 (secure 0 0 0) 0x7f6c740abc00 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.021+0000 7f6c88c00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6c68074510 0x7f6c68076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.021+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c74032440 con 0x7f6c68074510 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.021+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=102) ==== 24+0+0 (secure 0 0 0) 0x7f6c74075e60 con 0x7f6c8c0cc0c0 2024-09-16T01:52:28.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.021+0000 7f6c8be00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c6806c2b0 0x7f6c6806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.023+0000 7f6c8be00640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c6806c2b0 0x7f6c6806e770 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0099d0 tx=0x7f6c7c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.025+0000 7f6c91200640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6c68074510 0x7f6c68076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.026+0000 7f6c91200640 1 --2- 172.21.15.146:0/1902741556 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6c68074510 0x7f6c68076930 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.027+0000 7f6c88c00640 1 -- 172.21.15.146:0/1902741556 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c74032440 con 0x7f6c68074510 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:no valid command found; 10 closest matches: 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:0 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:1 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:2 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:abort 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:assert 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:bench [] [] [] [] 2024-09-16T01:52:28.066 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs debug_inject_read_zeros 2024-09-16T01:52:28.067 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs files list 2024-09-16T01:52:28.067 INFO:teuthology.orchestra.run.smithi146.stderr:bluefs stats 2024-09-16T01:52:28.067 INFO:teuthology.orchestra.run.smithi146.stderr:bluestore allocator dump block 2024-09-16T01:52:28.067 INFO:teuthology.orchestra.run.smithi146.stderr:Error EINVAL: invalid command 2024-09-16T01:52:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T01:52:28.091 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T01:52:28.132 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:28.131+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:28.181 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:28.181 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:28.181 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:28.181 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:28.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:28.209 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:28.207+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:28.309 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:28.308+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:28.412 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:28.409+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:28.494 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:28.495 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:28.495 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:28.495 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:28.495 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:28.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:28.511 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:28.509+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:28.622 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:28.621+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:28.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:28.713 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:28.712+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:28.814 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:28.812+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 221.05044452499999, 2024-09-16T01:52:28.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:28.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:52:28.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:52:28.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:28.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.432992185000003, 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.832 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:28.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:52:28.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:52:28.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:52:28.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.640975236000003, 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:52:28.841 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:52:28.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:52:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.605442612999994, 2024-09-16T01:52:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:52:28.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:52:28.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:52:28.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:52:28.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:52:28.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.239353676, 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.855 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:28.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:52:28.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.865 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.675142137999998, 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.866 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.613354835999999, 2024-09-16T01:52:28.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:28.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1931 3.39 e84/83 3:9d369e22:::100000001f8.00000388:head v 84'56, mlcod=0'0)", 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.590735180000003, 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110959082, 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228063+0000", 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228064+0000", 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5499999999999997e-07 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229687+0000", 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00162302 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229706+0000", 2024-09-16T01:52:28.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8459999999999999e-05 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229719+0000", 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2986000000000001e-05 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229824+0000", 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105163 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.229850+0000", 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6621000000000001e-05 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.338768+0000", 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108917369 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.339022+0000", 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00025460800000000003 2024-09-16T01:52:28.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.771572390000003, 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.882 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:52:28.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:52:28.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:52:28.887 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.028133637, 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.888 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:52:28.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:52:28.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2857 3.1c 3:39a72f86:::100000001f8.00000726:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.956502234999999, 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11170522400000001, 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.894 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2857 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862296+0000", 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.862299+0000", 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.351e-06 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890188+0000", 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027888969999999999 2024-09-16T01:52:28.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890218+0000", 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0593999999999999e-05 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890230+0000", 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1688999999999999e-05 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.890467+0000", 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023695699999999999 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.891129+0000", 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00066218699999999995 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.891811+0000", 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068177899999999998 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912951+0000", 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021140665999999999 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973903+0000", 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060951181 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.973961+0000", 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7989999999999999e-05 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.974001+0000", 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0859999999999998e-05 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.983912691, 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.900 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:52:28.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:52:28.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:28.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 12.975496205000001, 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:52:28.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:28.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.912 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3770 3.37 3:ec19b1da:::100000001f8.000000b7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:17.008166+0000", 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.810632802000001, 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109566236, 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3770 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.008166+0000", 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.008166+0000", 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.008167+0000", 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7829999999999999e-06 2024-09-16T01:52:28.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050683+0000", 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042515404999999999 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050714+0000", 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1433000000000002e-05 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050726+0000", 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1919999999999999e-05 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050795+0000", 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9067999999999997e-05 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050856+0000", 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0884999999999997e-05 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.050950+0000", 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3812000000000002e-05 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.063516+0000", 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012565952 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.117654+0000", 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054137941000000002 2024-09-16T01:52:28.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.117698+0000", 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4268999999999999e-05 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:17.117732+0000", 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3767999999999999e-05 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4107 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e99)", 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 7.5428587179999997, 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31541509600000001, 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4107 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275941+0000", 2024-09-16T01:52:28.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6959999999999999e-06 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303032+0000", 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027090797 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303053+0000", 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1019999999999999e-05 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303063+0000", 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6479999999999998e-06 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303176+0000", 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011292 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303185+0000", 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.491825+0000", 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.18864050500000001 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.498314+0000", 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064884210000000003 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551379+0000", 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053065042999999999 2024-09-16T01:52:28.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551446+0000", 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7477999999999993e-05 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551537+0000", 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1055000000000001e-05 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.576787+0000", 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025250156999999999 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591154+0000", 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01436693 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591228+0000", 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4103000000000005e-05 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591355+0000", 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126367 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4250 3.10 3:0a05e26b:::100000001f8.00000297:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.5086391800000003, 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.149078401, 2024-09-16T01:52:28.926 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4250 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310162+0000", 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6599999999999999e-06 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407037+0000", 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096875366000000004 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407073+0000", 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5969000000000001e-05 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407088+0000", 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5034999999999999e-05 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407218+0000", 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129462 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407332+0000", 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114183 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407516+0000", 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018366500000000001 2024-09-16T01:52:28.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.446008+0000", 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038492551999999999 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456958+0000", 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01095023 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.458896+0000", 2024-09-16T01:52:28.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019376650000000001 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.459238+0000", 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034161400000000001 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4504 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:28.932 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.2890119840000001, 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22042661699999999, 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4504 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:28.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529789+0000", 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5409999999999999e-06 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577075+0000", 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047285897 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577104+0000", 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9173999999999999e-05 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577115+0000", 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0895e-05 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577223+0000", 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108194 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577330+0000", 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107089 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:28.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577485+0000", 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154532 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.599121+0000", 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021636636000000001 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750098+0000", 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15097622499999999 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750164+0000", 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6172000000000003e-05 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750213+0000", 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9261999999999997e-05 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4512 3.19 3:987f889e:::100000001f8.0000039d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.1868183270000001, 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12513411299999999, 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4512 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631984+0000", 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8999999999999999e-06 2024-09-16T01:52:28.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679187+0000", 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047202698000000001 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679218+0000", 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1775e-05 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679233+0000", 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4182000000000001e-05 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682406+0000", 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031734889999999998 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682504+0000", 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7374999999999995e-05 2024-09-16T01:52:28.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.683089+0000", 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058535 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.705462+0000", 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022372670000000001 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757013+0000", 2024-09-16T01:52:28.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051551620999999999 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757066+0000", 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3044999999999998e-05 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757114+0000", 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8007999999999998e-05 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.944 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4553 3.6 3:62a9ab61:::100000001f8.000003c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.6521588949999999, 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.118770112, 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.945 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4553 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166642+0000", 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0470000000000001e-06 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206911+0000", 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040269395999999999 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206944+0000", 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3185000000000002e-05 2024-09-16T01:52:28.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206957+0000", 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2894999999999999e-05 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207157+0000", 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199908 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207262+0000", 2024-09-16T01:52:28.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105509 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207423+0000", 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016019999999999999 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.221084+0000", 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013661240999999999 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285289+0000", 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064205110999999995 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285358+0000", 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9015000000000002e-05 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285410+0000", 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1604999999999998e-05 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.950 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4560 3.39 3:9e09c0a8:::100000001f8.000003cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.5730733479999999, 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11688955500000001, 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:28.951 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4560 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245728+0000", 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.548e-06 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301561+0000", 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055832859999999998 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301606+0000", 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5423000000000002e-05 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301620+0000", 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.453e-05 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301728+0000", 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107836 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301847+0000", 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119127 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.302620+0000", 2024-09-16T01:52:28.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000772461 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.328630+0000", 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026010253000000001 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362241+0000", 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033611125999999998 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.956 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362382+0000", 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014118400000000001 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362615+0000", 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023220700000000001 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:28.957 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:28.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.870+0000 7f1c17000640 1 Processor -- start 2024-09-16T01:52:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.870+0000 7f1c17000640 1 -- start start 2024-09-16T01:52:28.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.872+0000 7f1c17000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c100c9150 0x7f1c1006bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.872+0000 7f1c17000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c100c95a0 con 0x7f1c1015e0d0 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.872+0000 7f1c17000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c100cba20 con 0x7f1c100cb470 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.872+0000 7f1c17000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1c100c89e0 con 0x7f1c100c9150 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0f400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1c100cb470 0x7f1c101633f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:37660/0 (socket says 172.21.15.146:37660) 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0f400640 1 -- 172.21.15.146:0/760120601 learned_addr learned my addr 172.21.15.146:0/760120601 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c15000640 1 --2- 172.21.15.146:0/760120601 >> v2:172.21.15.146:3300/0 conn(0x7f1c100c9150 0x7f1c1006bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890439635 0 0) 0x7f1c100cba20 con 0x7f1c100cb470 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf4003520 con 0x7f1c100cb470 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 435137920 0 0) 0x7f1bf4003520 con 0x7f1c100cb470 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 >> v2:172.21.15.146:3300/0 conn(0x7f1c100c9150 msgr2=0x7f1c1006bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0ea00640 1 --2- 172.21.15.146:0/760120601 >> v2:172.21.15.146:3300/0 conn(0x7f1c100c9150 0x7f1c1006bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.873+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 >> v1:172.21.15.146:6789/0 conn(0x7f1c1015e0d0 legacy=0x7f1c100c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:28.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c100cbf80 con 0x7f1c100cb470 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f1bfc001670 con 0x7f1c100cb470 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 >> v1:172.21.15.161:6789/0 conn(0x7f1c100cb470 legacy=0x7f1c101633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf40039b0 0x7f1bf4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4004800 0x7f1bf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1bf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100c89e0 con 0x7f1bf4004800 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf400c040 con 0x7f1bf40051c0 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf400c240 con 0x7f1bf40039b0 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0f400640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4004800 0x7f1bf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c15000640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1bf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf40039b0 0x7f1bf4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c15000640 1 -- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf40039b0 msgr2=0x7f1bf4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:28.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c15000640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf40039b0 0x7f1bf4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c15000640 1 -- 172.21.15.146:0/760120601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4004800 msgr2=0x7f1bf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c15000640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4004800 0x7f1bf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.874+0000 7f1c15000640 1 -- 172.21.15.146:0/760120601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf4003520 con 0x7f1bf40051c0 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.875+0000 7f1c15000640 1 -- 172.21.15.146:0/760120601 reap_dead start 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.875+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf40039b0 0x7f1bf4003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.876+0000 7f1c15000640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1bf400b9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009fc0 tx=0x7f1bf4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9552f3a49ca9ffc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.876+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08034030 con 0x7f1bf40051c0 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.876+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c08003990 con 0x7f1bf40051c0 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.876+0000 7f1c0ea00640 1 -- 172.21.15.146:0/760120601 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08009c10 con 0x7f1bf40051c0 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 -- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 msgr2=0x7f1bf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1bf400b9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009fc0 tx=0x7f1bf4005830 comp rx=0 tx=0).stop 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 -- 172.21.15.146:0/760120601 shutdown_connections 2024-09-16T01:52:28.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 --2- 172.21.15.146:0/760120601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1bf400b9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 -- 172.21.15.146:0/760120601 >> 172.21.15.146:0/760120601 conn(0x7f1c100d0360 msgr2=0x7f1c100d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 -- 172.21.15.146:0/760120601 shutdown_connections 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.877+0000 7f1c17000640 1 -- 172.21.15.146:0/760120601 wait complete. 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.879+0000 7f1c17000640 1 Processor -- start 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.880+0000 7f1c17000640 1 -- start start 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c17000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1c100c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c17000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100c2ec0 0x7f1c10156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c17000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 0x7f1c100bc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c17000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100cc540 con 0x7f1c100c2ec0 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c17000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c100c8cc0 con 0x7f1bf40051c0 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c17000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1015ed60 con 0x7f1c100beeb0 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c15000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 0x7f1c100bc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c15000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 0x7f1c100bc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45272/0 (socket says 172.21.15.146:45272) 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.881+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 learned_addr learned my addr 172.21.15.146:0/3498923623 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1c100c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 msgr2=0x7f1c100c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1c100c1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100c2ec0 msgr2=0x7f1c10156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100c2ec0 0x7f1c10156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c101f51f0 con 0x7f1c100beeb0 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.882+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1bf40051c0 0x7f1c100c1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.883+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 0x7f1c100bc4f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e200 tx=0x7f1c08009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.883+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08034040 con 0x7f1c100beeb0 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.883+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c08042070 con 0x7f1c100beeb0 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.883+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c08039640 con 0x7f1c100beeb0 2024-09-16T01:52:28.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.883+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c100c2650 con 0x7f1c100beeb0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.883+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c101f54b0 con 0x7f1c100beeb0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.886+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1c08037070 con 0x7f1c100beeb0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.886+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1c100cba20 con 0x7f1c100beeb0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.887+0000 7f1c07e00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bf006c2b0 0x7f1bf006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.887+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 5741+0+0 (secure 0 0 0) 0x7f1c080ab1b0 con 0x7f1c100beeb0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.888+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bf006c2b0 0x7f1bf006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.888+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v102) ==== 54+0+16 (secure 0 0 0) 0x7f1c100cba20 con 0x7f1c100beeb0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.888+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c10158300 0x7f1c101f8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.888+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c100cbd60 con 0x7f1c10158300 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.888+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bf006c2b0 0x7f1bf006e770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1c1006a450 tx=0x7f1c00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.888+0000 7f1c0f400640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c10158300 0x7f1c101f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.889+0000 7f1c0f400640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c10158300 0x7f1c101f8760 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.890+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c100cbd60 con 0x7f1c10158300 2024-09-16T01:52:28.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.905+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c100cc8a0 con 0x7f1c10158300 2024-09-16T01:52:28.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.907+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c100cc8a0 con 0x7f1c10158300 2024-09-16T01:52:28.972 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.907+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c101f91b0 0x7f1c10062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.907+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1c1006d6f0 con 0x7f1c101f91b0 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.908+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c101f91b0 0x7f1c10062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.908+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c101f91b0 0x7f1c10062630 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.910+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c1006d6f0 con 0x7f1c101f91b0 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.939+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c1006d410 con 0x7f1c101f91b0 2024-09-16T01:52:28.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.941+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c1006d410 con 0x7f1c101f91b0 2024-09-16T01:52:28.974 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:28.974 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:28.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.943+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c10063080 0x7f1c1002c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.943+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1c1006d130 con 0x7f1c10063080 2024-09-16T01:52:28.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.943+0000 7f1c0f400640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c10063080 0x7f1c1002c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.944+0000 7f1c0f400640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c10063080 0x7f1c1002c640 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.946+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c1006d130 con 0x7f1c10063080 2024-09-16T01:52:28.975 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:28.958+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:28.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.965+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c101f56b0 con 0x7f1c10063080 2024-09-16T01:52:28.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.966+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c101f56b0 con 0x7f1c10063080 2024-09-16T01:52:28.976 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:28.977 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:28.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.967+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c1002d090 0x7f1c1002d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.967+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1c1002da30 con 0x7f1c1002d090 2024-09-16T01:52:28.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.967+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c1002d090 0x7f1c1002d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.967+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c1002d090 0x7f1c1002d4f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.968+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c1002da30 con 0x7f1c1002d090 2024-09-16T01:52:28.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.989+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c101f56b0 con 0x7f1c1002d090 2024-09-16T01:52:28.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.991+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c101f56b0 con 0x7f1c1002d090 2024-09-16T01:52:28.992 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:28.992 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:28.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.991+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c10153300 0x7f1c10153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:28.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.991+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1c10153c60 con 0x7f1c10153300 2024-09-16T01:52:28.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.992+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c10153300 0x7f1c10153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:28.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.993+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c10153300 0x7f1c10153720 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:28.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:28.995+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c10153c60 con 0x7f1c10153300 2024-09-16T01:52:29.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.013+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c10153c60 con 0x7f1c10153300 2024-09-16T01:52:29.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.015+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c10153c60 con 0x7f1c10153300 2024-09-16T01:52:29.020 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:29.021 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:29.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.020+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c101543b0 0x7f1c1020baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:29.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.020+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1c1020e8c0 con 0x7f1c101543b0 2024-09-16T01:52:29.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.020+0000 7f1c0f400640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c101543b0 0x7f1c1020baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:29.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.022+0000 7f1c0f400640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c101543b0 0x7f1c1020baf0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:29.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.023+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c1020e8c0 con 0x7f1c101543b0 2024-09-16T01:52:29.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.057+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c1020c030 con 0x7f1c101543b0 2024-09-16T01:52:29.064 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:29.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.061+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c1020c030 con 0x7f1c101543b0 2024-09-16T01:52:29.064 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:29.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.061+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c1020c750 0x7f1c1020eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.061+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1c10211910 con 0x7f1c1020c750 2024-09-16T01:52:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.062+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c1020c750 0x7f1c1020eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:29.065 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:29.062+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.064+0000 7f1c15000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c1020c750 0x7f1c1020eb40 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:29.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.065+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c10211910 con 0x7f1c1020c750 2024-09-16T01:52:29.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.092+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c1020f080 con 0x7f1c1020c750 2024-09-16T01:52:29.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.094+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c1020f080 con 0x7f1c1020c750 2024-09-16T01:52:29.095 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:29.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.094+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c1020f700 0x7f1c10211af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:29.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.094+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1c102148c0 con 0x7f1c1020f700 2024-09-16T01:52:29.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.094+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c1020f700 0x7f1c10211af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:29.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.095+0000 7f1c0fe00640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c1020f700 0x7f1c10211af0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:29.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.096+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c102148c0 con 0x7f1c1020f700 2024-09-16T01:52:29.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.110+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1c10212030 con 0x7f1c1020f700 2024-09-16T01:52:29.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c07e00640 1 -- 172.21.15.146:0/3498923623 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c10212030 con 0x7f1c1020f700 2024-09-16T01:52:29.113 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c10158300 msgr2=0x7f1c101f8760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1c10158300 0x7f1c101f8760 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c101f91b0 msgr2=0x7f1c10062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1c101f91b0 0x7f1c10062630 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c10063080 msgr2=0x7f1c1002c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1c10063080 0x7f1c1002c640 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c1002d090 msgr2=0x7f1c1002d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1c1002d090 0x7f1c1002d4f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.112+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c10153300 msgr2=0x7f1c10153720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1c10153300 0x7f1c10153720 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c101543b0 msgr2=0x7f1c1020baf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1c101543b0 0x7f1c1020baf0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 reap_dead start 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c1020c750 msgr2=0x7f1c1020eb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1c1020c750 0x7f1c1020eb40 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 reap_dead start 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c1020f700 msgr2=0x7f1c10211af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1c1020f700 0x7f1c10211af0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 reap_dead start 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c15000640 1 -- 172.21.15.146:0/3498923623 reap_dead start 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bf006c2b0 msgr2=0x7f1bf006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bf006c2b0 0x7f1bf006e770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1c1006a450 tx=0x7f1c00009040 comp rx=0 tx=0).stop 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 msgr2=0x7f1c100bc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 0x7f1c100bc4f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e200 tx=0x7f1c08009750 comp rx=0 tx=0).stop 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 shutdown_connections 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1bf006c2b0 0x7f1bf006e770 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 --2- 172.21.15.146:0/3498923623 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1c100beeb0 0x7f1c100bc4f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 >> 172.21.15.146:0/3498923623 conn(0x7f1c100d0360 msgr2=0x7f1c100cea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 shutdown_connections 2024-09-16T01:52:29.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:29.113+0000 7f1c17000640 1 -- 172.21.15.146:0/3498923623 wait complete. 2024-09-16T01:52:29.160 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:29.158+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:29.261 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:29.260+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:29.295 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:29.296 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:29.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:29.363 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:29.360+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:29.464 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:29.463+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:29.564 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:29.562+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:29.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:52:29.665 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:29.663+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:29.706 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:29.706 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:29.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:29.791 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:29.790+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:29.868 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:29.867+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:29.969 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:29.968+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:30.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.015+0000 7f687e800640 1 Processor -- start 2024-09-16T01:52:30.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.015+0000 7f687e800640 1 -- start start 2024-09-16T01:52:30.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f687e800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68780c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f687e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68780c6410 0x7f6878158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f687e800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68780c5280 0x7f68780c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f687e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6878151e70 con 0x7f68780c6410 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f687e800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f68780c8130 con 0x7f68780c5280 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f687e800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f68780c8410 con 0x7f6878151a90 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f6876a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68780c6410 0x7f6878158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.017+0000 7f6876a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68780c6410 0x7f6878158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35976/0 (socket says 172.21.15.146:35976) 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6876a00640 1 -- 172.21.15.146:0/4290951691 learned_addr learned my addr 172.21.15.146:0/4290951691 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877e00640 1 --2- 172.21.15.146:0/4290951691 >> v2:172.21.15.161:3300/0 conn(0x7f68780c5280 0x7f68780c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877400640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68780c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877e00640 1 -- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 msgr2=0x7f68780c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877e00640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68780c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877e00640 1 -- 172.21.15.146:0/4290951691 >> v2:172.21.15.146:3300/0 conn(0x7f68780c6410 msgr2=0x7f6878158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:30.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877e00640 1 --2- 172.21.15.146:0/4290951691 >> v2:172.21.15.146:3300/0 conn(0x7f68780c6410 0x7f6878158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877e00640 1 -- 172.21.15.146:0/4290951691 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68780c86f0 con 0x7f68780c5280 2024-09-16T01:52:30.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.018+0000 7f6877400640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68780c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:30.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.020+0000 7f6877e00640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68780c5730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f687815bd40 tx=0x7f687002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ac242566451d5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.020+0000 7f6876000640 1 -- 172.21.15.146:0/4290951691 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870031600 con 0x7f68780c5280 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.020+0000 7f6876000640 1 -- 172.21.15.146:0/4290951691 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68700317a0 con 0x7f68780c5280 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.020+0000 7f6876000640 1 -- 172.21.15.146:0/4290951691 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870031b70 con 0x7f68780c5280 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.021+0000 7f687e800640 1 -- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 msgr2=0x7f68780c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.021+0000 7f687e800640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68780c5730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f687815bd40 tx=0x7f687002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.022+0000 7f687e800640 1 -- 172.21.15.146:0/4290951691 shutdown_connections 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.022+0000 7f687e800640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68780c5730 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.022+0000 7f687e800640 1 --2- 172.21.15.146:0/4290951691 >> v2:172.21.15.146:3300/0 conn(0x7f68780c6410 0x7f6878158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.022+0000 7f687e800640 1 --2- 172.21.15.146:0/4290951691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68780c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.022+0000 7f687e800640 1 -- 172.21.15.146:0/4290951691 >> 172.21.15.146:0/4290951691 conn(0x7f687806f3c0 msgr2=0x7f687806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:30.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.022+0000 7f687e800640 1 -- 172.21.15.146:0/4290951691 shutdown_connections 2024-09-16T01:52:30.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.025+0000 7f687e800640 1 -- 172.21.15.146:0/4290951691 wait complete. 2024-09-16T01:52:30.028 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:30.028 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:30.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.027+0000 7f687e800640 1 Processor -- start 2024-09-16T01:52:30.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.027+0000 7f687e800640 1 -- start start 2024-09-16T01:52:30.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f687e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68781f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f687e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68780c6410 0x7f68781f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f687e800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68781479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f687e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6878157b20 con 0x7f68780c6410 2024-09-16T01:52:30.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f6877e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68781479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:30.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f6877e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68781479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:45292/0 (socket says 172.21.15.146:45292) 2024-09-16T01:52:30.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.029+0000 7f6877e00640 1 -- 172.21.15.146:0/2054143458 learned_addr learned my addr 172.21.15.146:0/2054143458 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:30.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.030+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6878157840 con 0x7f68780c5280 2024-09-16T01:52:30.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.030+0000 7f6877400640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68781f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.030+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f68780c3bf0 con 0x7f6878151a90 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877e00640 1 -- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 msgr2=0x7f68781f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877e00640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68781f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877e00640 1 -- 172.21.15.146:0/2054143458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68780c6410 msgr2=0x7f68781f4750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877e00640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68780c6410 0x7f68781f4750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877e00640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68780ca620 con 0x7f6878151a90 2024-09-16T01:52:30.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877400640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68781f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:30.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f6877e00640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68781479f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6870009e50 tx=0x7f6870004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:30.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.031+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870040070 con 0x7f6878151a90 2024-09-16T01:52:30.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.032+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687003c830 con 0x7f6878151a90 2024-09-16T01:52:30.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.032+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687003cc00 con 0x7f6878151a90 2024-09-16T01:52:30.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.032+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68780ca900 con 0x7f6878151a90 2024-09-16T01:52:30.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.032+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68781f3f50 con 0x7f6878151a90 2024-09-16T01:52:30.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.035+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6850005290 con 0x7f6878151a90 2024-09-16T01:52:30.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.036+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6870039040 con 0x7f6878151a90 2024-09-16T01:52:30.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.037+0000 7f686f400640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f685806c2e0 0x7f685806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:30.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.037+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 5741+0+0 (secure 0 0 0) 0x7f68700ac9f0 con 0x7f6878151a90 2024-09-16T01:52:30.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.038+0000 7f6877400640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f685806c2e0 0x7f685806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:30.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.039+0000 7f6877400640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f685806c2e0 0x7f685806e7a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6864009560 tx=0x7f6864009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:30.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.046+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f687007b0c0 con 0x7f6878151a90 2024-09-16T01:52:30.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:30.069 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:30.067+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:30.176 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:30.172+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:30.278 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:30.276+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:30.377 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:30.376+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:30.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.457+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f6850005580 con 0x7f6878151a90 2024-09-16T01:52:30.505 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:30.504+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:30.575 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:30.573+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:30.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.623+0000 7f686f400640 1 -- 172.21.15.146:0/2054143458 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v103) ==== 87+0+0 (secure 0 0 0) 0x7f687007a9c0 con 0x7f6878151a90 2024-09-16T01:52:30.627 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:52:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.632+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f685806c2e0 msgr2=0x7f685806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.632+0000 7f687e800640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f685806c2e0 0x7f685806e7a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6864009560 tx=0x7f6864009190 comp rx=0 tx=0).stop 2024-09-16T01:52:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.632+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 msgr2=0x7f68781479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.632+0000 7f687e800640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68781479f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6870009e50 tx=0x7f6870004a40 comp rx=0 tx=0).stop 2024-09-16T01:52:30.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.633+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 shutdown_connections 2024-09-16T01:52:30.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.633+0000 7f687e800640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f685806c2e0 0x7f685806e7a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.633+0000 7f687e800640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6878151a90 0x7f68781479f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.633+0000 7f687e800640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68780c6410 0x7f68781f4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.633+0000 7f687e800640 1 --2- 172.21.15.146:0/2054143458 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68780c5280 0x7f68781f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:30.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.633+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 >> 172.21.15.146:0/2054143458 conn(0x7f687806f3c0 msgr2=0x7f6878156970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:30.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.634+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 shutdown_connections 2024-09-16T01:52:30.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:30.634+0000 7f687e800640 1 -- 172.21.15.146:0/2054143458 wait complete. 2024-09-16T01:52:30.706 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:30.704+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:30.724 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:30.724 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:30.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:30.786 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:30.785+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:30.879 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:30.877+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:30.982 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:30.981+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:31.050 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:31.050 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:31.074 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:31.119 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:31.117+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:31.139 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:31.186 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:31.185+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:31.187 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:31.187 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:31.196 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:31.281 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:31.279+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:31.344 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:31.344 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:31.354 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:31.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.374+0000 7f97dcc00640 1 Processor -- start 2024-09-16T01:52:31.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.375+0000 7f97dcc00640 1 -- start start 2024-09-16T01:52:31.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.376+0000 7f97dcc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97d80c5460 0x7f97d80719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97dcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d8151ae0 0x7f97d8161f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97dcc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d80c3850 con 0x7f97d80c5840 2024-09-16T01:52:31.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97dcc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f97d80c4e20 con 0x7f97d80c5460 2024-09-16T01:52:31.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97dcc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f97d81576e0 con 0x7f97d8151ae0 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97d7400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97d80c5460 0x7f97d80719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97cfe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f97d80c5840 0x7f97d81513d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:52514/0 (socket says 172.21.15.146:52514) 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97cfe00640 1 -- 172.21.15.146:0/590045541 learned_addr learned my addr 172.21.15.146:0/590045541 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97d7e00640 1 --2- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d8151ae0 0x7f97d8161f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97d7400640 1 -- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d8151ae0 msgr2=0x7f97d8161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97d7400640 1 --2- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d8151ae0 0x7f97d8161f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.377+0000 7f97d7400640 1 -- 172.21.15.146:0/590045541 >> v1:172.21.15.146:6789/0 conn(0x7f97d80c5840 legacy=0x7f97d81513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.378+0000 7f97d7400640 1 -- 172.21.15.146:0/590045541 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d81579c0 con 0x7f97d80c5460 2024-09-16T01:52:31.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.378+0000 7f97d6a00640 1 -- 172.21.15.146:0/590045541 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3745649776 0 0) 0x7f97d80c3850 con 0x7f97d80c5840 2024-09-16T01:52:31.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.379+0000 7f97d7400640 1 --2- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d80719a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97c80097a0 tx=0x7f97c802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adab8c4f7c65fcd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.379+0000 7f97d6a00640 1 -- 172.21.15.146:0/590045541 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8037070 con 0x7f97d80c5460 2024-09-16T01:52:31.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.379+0000 7f97d6a00640 1 -- 172.21.15.146:0/590045541 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c8032440 con 0x7f97d80c5460 2024-09-16T01:52:31.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.379+0000 7f97d6a00640 1 -- 172.21.15.146:0/590045541 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8032810 con 0x7f97d80c5460 2024-09-16T01:52:31.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 -- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 msgr2=0x7f97d80719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 --2- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d80719a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97c80097a0 tx=0x7f97c802a210 comp rx=0 tx=0).stop 2024-09-16T01:52:31.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 -- 172.21.15.146:0/590045541 shutdown_connections 2024-09-16T01:52:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 --2- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d8151ae0 0x7f97d8161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 --2- 172.21.15.146:0/590045541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d80719a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 -- 172.21.15.146:0/590045541 >> 172.21.15.146:0/590045541 conn(0x7f97d80cc1b0 msgr2=0x7f97d80cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 -- 172.21.15.146:0/590045541 shutdown_connections 2024-09-16T01:52:31.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.380+0000 7f97dcc00640 1 -- 172.21.15.146:0/590045541 wait complete. 2024-09-16T01:52:31.383 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:31.381+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:31.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.383+0000 7f97dcc00640 1 Processor -- start 2024-09-16T01:52:31.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.385+0000 7f97dcc00640 1 -- start start 2024-09-16T01:52:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97dcc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d8163a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97dcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d80c5840 0x7f97d8163f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97dcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97d8151ae0 0x7f97d81569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97dcc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81584a0 con 0x7f97d8151ae0 2024-09-16T01:52:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97dcc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f97d8157f80 con 0x7f97d80c5460 2024-09-16T01:52:31.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97dcc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f97d8071ee0 con 0x7f97d80c5840 2024-09-16T01:52:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97d7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d8163a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97d7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d8163a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:44794/0 (socket says 172.21.15.146:44794) 2024-09-16T01:52:31.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97d7400640 1 -- 172.21.15.146:0/2495596424 learned_addr learned my addr 172.21.15.146:0/2495596424 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.387+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97d8151ae0 0x7f97d81569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7400640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d80c5840 msgr2=0x7f97d8163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97d80c5840 0x7f97d8163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7400640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97d8151ae0 msgr2=0x7f97d81569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97d8151ae0 0x7f97d81569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7400640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d8156450 con 0x7f97d80c5460 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97d8151ae0 0x7f97d81569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d8163a30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.388+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8037050 con 0x7f97d80c5460 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.389+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c8038040 con 0x7f97d80c5460 2024-09-16T01:52:31.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.389+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d80cb7b0 con 0x7f97d80c5460 2024-09-16T01:52:31.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.389+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8004430 con 0x7f97d80c5460 2024-09-16T01:52:31.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.389+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d814f230 con 0x7f97d80c5460 2024-09-16T01:52:31.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.390+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f97d80c3b30 con 0x7f97d80c5460 2024-09-16T01:52:31.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.392+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f97c803c6d0 con 0x7f97d80c5460 2024-09-16T01:52:31.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.393+0000 7f97cf400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f97b406c2b0 0x7f97b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.393+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5741+0+0 (secure 0 0 0) 0x7f97c80abe40 con 0x7f97d80c5460 2024-09-16T01:52:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.394+0000 7f97cfe00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f97b406c2b0 0x7f97b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.394+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v103) ==== 54+0+16 (secure 0 0 0) 0x7f97c80ae3e0 con 0x7f97d80c5460 2024-09-16T01:52:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.394+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f97d806bc10 0x7f97d814fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.394+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97d814f630 con 0x7f97d806bc10 2024-09-16T01:52:31.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.394+0000 7f97cfe00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f97b406c2b0 0x7f97b406e770 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f97c0007fd0 tx=0x7f97c0007920 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.394+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f97d806bc10 0x7f97d814fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.395+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f97d806bc10 0x7f97d814fa30 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.396+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d814f630 con 0x7f97d806bc10 2024-09-16T01:52:31.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.418+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d8072fe0 con 0x7f97d806bc10 2024-09-16T01:52:31.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.420+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d8072fe0 con 0x7f97d806bc10 2024-09-16T01:52:31.421 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:31.422 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:31.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.421+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f97d806bff0 0x7f97d8062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.421+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f97d8072d00 con 0x7f97d806bff0 2024-09-16T01:52:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.423+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f97d806bff0 0x7f97d8062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.424+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f97d806bff0 0x7f97d8062630 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.425+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d8072d00 con 0x7f97d806bff0 2024-09-16T01:52:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.451+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d8062dc0 con 0x7f97d806bff0 2024-09-16T01:52:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.453+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d8062dc0 con 0x7f97d806bff0 2024-09-16T01:52:31.454 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.453+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f97d8063390 0x7f97d80ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.453+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f97d8062dc0 con 0x7f97d8063390 2024-09-16T01:52:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.454+0000 7f97cfe00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f97d8063390 0x7f97d80ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.454+0000 7f97cfe00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f97d8063390 0x7f97d80ca580 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.455+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d8062dc0 con 0x7f97d8063390 2024-09-16T01:52:31.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.477+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d8062dc0 con 0x7f97d8063390 2024-09-16T01:52:31.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.480+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d8062dc0 con 0x7f97d8063390 2024-09-16T01:52:31.481 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.480+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f97d8155720 0x7f97d8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.480+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f97d8062dc0 con 0x7f97d8155720 2024-09-16T01:52:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.481+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f97d8155720 0x7f97d8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.481+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f97d8155720 0x7f97d8155b20 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.482+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d8062dc0 con 0x7f97d8155720 2024-09-16T01:52:31.484 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:31.482+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:31.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.497+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d8072d00 con 0x7f97d8155720 2024-09-16T01:52:31.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.500+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d8072d00 con 0x7f97d8155720 2024-09-16T01:52:31.501 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:31.501 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:31.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.500+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f97d802cb50 0x7f97d802cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.500+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f97d8062dc0 con 0x7f97d802cb50 2024-09-16T01:52:31.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.502+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f97d802cb50 0x7f97d802cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.506 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:31.506 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:31.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.504+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f97d802cb50 0x7f97d802cf70 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.507+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d8062dc0 con 0x7f97d802cb50 2024-09-16T01:52:31.518 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:31.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.530+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d8062dc0 con 0x7f97d802cb50 2024-09-16T01:52:31.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.532+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d8062dc0 con 0x7f97d802cb50 2024-09-16T01:52:31.533 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:31.533 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:31.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.532+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f97d8208520 0x7f97d820a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.532+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f97d8072770 con 0x7f97d8208520 2024-09-16T01:52:31.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.532+0000 7f97cfe00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f97d8208520 0x7f97d820a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.533+0000 7f97cfe00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f97d8208520 0x7f97d820a910 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.534+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97c0021bb0 con 0x7f97d8208520 2024-09-16T01:52:31.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.556+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d802da50 con 0x7f97d8208520 2024-09-16T01:52:31.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.558+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d802da50 con 0x7f97d8208520 2024-09-16T01:52:31.559 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:31.560 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:31.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.558+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f97d820b360 0x7f97d820d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.558+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f97d8210660 con 0x7f97d820b360 2024-09-16T01:52:31.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.559+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f97d820b360 0x7f97d820d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.559+0000 7f97d7e00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f97d820b360 0x7f97d820d750 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.560+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d8210660 con 0x7f97d820b360 2024-09-16T01:52:31.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.580+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d820dc90 con 0x7f97d820b360 2024-09-16T01:52:31.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.582+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d820dc90 con 0x7f97d820b360 2024-09-16T01:52:31.584 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:31.584 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:31.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.583+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f97d820e3b0 0x7f97d820e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:31.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.583+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f97d8213a30 con 0x7f97d820e3b0 2024-09-16T01:52:31.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.583+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f97d820e3b0 0x7f97d820e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:31.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.584+0000 7f97d7400640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f97d820e3b0 0x7f97d820e810 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:31.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.584+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97d8213a30 con 0x7f97d820e3b0 2024-09-16T01:52:31.602 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:31.600+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:31.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.617+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f97d82128b0 con 0x7f97d820e3b0 2024-09-16T01:52:31.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.619+0000 7f97cf400640 1 -- 172.21.15.146:0/2495596424 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97d82128b0 con 0x7f97d820e3b0 2024-09-16T01:52:31.621 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:31.622 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f97d806bc10 msgr2=0x7f97d814fa30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f97d806bc10 0x7f97d814fa30 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f97d806bff0 msgr2=0x7f97d8062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f97d806bff0 0x7f97d8062630 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f97d8063390 msgr2=0x7f97d80ca580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f97d8063390 0x7f97d80ca580 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f97d8155720 msgr2=0x7f97d8155b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f97d8155720 0x7f97d8155b20 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f97d802cb50 msgr2=0x7f97d802cf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f97d802cb50 0x7f97d802cf70 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f97d8208520 msgr2=0x7f97d820a910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f97d8208520 0x7f97d820a910 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f97d820b360 msgr2=0x7f97d820d750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f97d820b360 0x7f97d820d750 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f97d820e3b0 msgr2=0x7f97d820e810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f97d820e3b0 0x7f97d820e810 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f97b406c2b0 msgr2=0x7f97b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f97b406c2b0 0x7f97b406e770 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f97c0007fd0 tx=0x7f97c0007920 comp rx=0 tx=0).stop 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 msgr2=0x7f97d8163a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.620+0000 7f97dcc00640 1 --2- 172.21.15.146:0/2495596424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97d80c5460 0x7f97d8163a30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c80046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.621+0000 7f97d7e00640 1 -- 172.21.15.146:0/2495596424 reap_dead start 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.621+0000 7f97d7e00640 1 -- 172.21.15.146:0/2495596424 reap_dead start 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.621+0000 7f97d7e00640 1 -- 172.21.15.146:0/2495596424 reap_dead start 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.621+0000 7f97d7e00640 1 -- 172.21.15.146:0/2495596424 reap_dead start 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.621+0000 7f97d7e00640 1 -- 172.21.15.146:0/2495596424 reap_dead start 2024-09-16T01:52:31.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.621+0000 7f97d7e00640 1 -- 172.21.15.146:0/2495596424 reap_dead start 2024-09-16T01:52:31.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.626+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 shutdown_connections 2024-09-16T01:52:31.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.626+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 >> 172.21.15.146:0/2495596424 conn(0x7f97d80cc1b0 msgr2=0x7f97d8142280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:31.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.627+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 shutdown_connections 2024-09-16T01:52:31.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:31.627+0000 7f97dcc00640 1 -- 172.21.15.146:0/2495596424 wait complete. 2024-09-16T01:52:31.672 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:31.672 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4919 3.31 3:8d9928e2:::100000001f8.00000534:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:31.597885+0000", 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.074138552999999996, 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074163947999999993, 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:31.673 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4919 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.597885+0000", 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.597885+0000", 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.597892+0000", 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1489999999999997e-06 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.638239+0000", 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040347109999999999 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:31.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.638262+0000", 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2966999999999999e-05 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.638273+0000", 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0648e-05 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:31.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.638368+0000", 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5148000000000004e-05 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.638437+0000", 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8843000000000005e-05 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.638546+0000", 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000108682 2024-09-16T01:52:31.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:31.643523+0000", 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049777980000000003 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 1 2024-09-16T01:52:31.678 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:31.688 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:31.689 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:31.685+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:31.788 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:31.786+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:31.841 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:31.841 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:31.841 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:31.841 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:31.841 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:31.850 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:31.890 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:31.889+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:31.991 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:31.988+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:32.005 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 101.480034836, 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.006 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:32.007 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.008 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.009 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:52:32.010 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:52:32.011 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1663 3.17 3:ea019aa2:::100000001f8.0000027c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.226544005999997, 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124529129, 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.012 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1663 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.013 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776694+0000", 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8320000000000002e-06 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866207+0000", 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089513588000000005 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.014 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866225+0000", 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7325000000000001e-05 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866232+0000", 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6149999999999999e-06 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866286+0000", 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3863000000000003e-05 2024-09-16T01:52:32.015 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866393+0000", 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106443 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866499+0000", 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010595299999999999 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.016 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.869501+0000", 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030023570000000002 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901146+0000", 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031644671999999999 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901180+0000", 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4277999999999999e-05 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.017 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901220+0000", 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0203000000000001e-05 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2039 3.31 3:8f066b95:::100000001f8.000003f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.480938068999997, 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12858615900000001, 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.018 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2039 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.019 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522299+0000", 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0109999999999999e-06 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586678+0000", 2024-09-16T01:52:32.020 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064378764000000005 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586695+0000", 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7159999999999998e-05 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586705+0000", 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9839999999999996e-06 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.021 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586760+0000", 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5099999999999998e-05 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586894+0000", 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013428999999999999 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.587008+0000", 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114178 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.022 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.590811+0000", 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038024529999999999 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650793+0000", 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059982029999999999 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650843+0000", 2024-09-16T01:52:32.023 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9876999999999997e-05 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650883+0000", 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0312000000000003e-05 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.024 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2092 3.1e 3:7aa4b9dd:::100000001f8.00000429:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 40.739344015, 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10603082799999999, 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2092 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.025 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:32.026 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263893+0000", 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2249999999999999e-06 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292346+0000", 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028453092999999999 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.027 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292359+0000", 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2306999999999999e-05 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292364+0000", 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.028 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292417+0000", 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2673000000000002e-05 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292522+0000", 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010544600000000001 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292624+0000", 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101473 2024-09-16T01:52:32.029 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.295728+0000", 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031039010000000001 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369848+0000", 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074120309999999995 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.030 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369888+0000", 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9465000000000002e-05 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369922+0000", 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4193999999999998e-05 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.031 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3029 3.17 3:ebfd80ec:::100000001f8.000007d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.269426767999999, 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.099718444000000003, 2024-09-16T01:52:32.032 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.033 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.033 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.033 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.033 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.033 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3029 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.034 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733810+0000", 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1560000000000002e-06 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763618+0000", 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029807628999999999 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.035 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763642+0000", 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3655999999999998e-05 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763651+0000", 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5259999999999999e-06 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.036 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763711+0000", 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9721e-05 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763830+0000", 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118792 2024-09-16T01:52:32.037 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763940+0000", 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110046 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.767357+0000", 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034171829999999999 2024-09-16T01:52:32.038 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833431+0000", 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066074421999999994 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833483+0000", 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1261999999999998e-05 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833527+0000", 2024-09-16T01:52:32.039 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4051000000000002e-05 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3080 3.38 3:1d0a07e7:::100000001f8.00000805:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:02.266234+0000", 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 29.737001144000001, 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.090907420000000003, 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.040 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3080 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.266234+0000", 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.266234+0000", 2024-09-16T01:52:32.041 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.266235+0000", 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.221e-06 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307326+0000", 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041091181999999997 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.042 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307338+0000", 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1874e-05 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307363+0000", 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4973999999999998e-05 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307424+0000", 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1161999999999995e-05 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.043 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307540+0000", 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011523600000000001 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.307641+0000", 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101649 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.310544+0000", 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0029028539999999999 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.044 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.357039+0000", 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046494704999999997 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.357094+0000", 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5618000000000002e-05 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:02.357141+0000", 2024-09-16T01:52:32.045 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6944999999999999e-05 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3344 3.17 3:e83df825:::100000001f8.0000090d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.444367536000001, 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096291749999999995, 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.046 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3344 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.047 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558869+0000", 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614237+0000", 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055368217999999997 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614250+0000", 2024-09-16T01:52:32.048 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3259e-05 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614255+0000", 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2100000000000001e-06 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614314+0000", 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8730999999999997e-05 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.049 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614411+0000", 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7116999999999999e-05 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614507+0000", 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5575999999999994e-05 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.617642+0000", 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003135166 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.050 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655055+0000", 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037413257999999998 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655113+0000", 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7726999999999998e-05 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655159+0000", 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6292999999999999e-05 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.051 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3358 3.27 3:e7c6e76b:::100000001f8.0000091b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.269493782000001, 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096871567000000006, 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.052 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3358 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:32.053 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733744+0000", 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.886e-06 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787499+0000", 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053755360000000002 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.054 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787517+0000", 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7645000000000001e-05 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787525+0000", 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787592+0000", 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6891999999999996e-05 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.055 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787751+0000", 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158975 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787911+0000", 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016023400000000001 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.793715+0000", 2024-09-16T01:52:32.056 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058038480000000003 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830517+0000", 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036802435000000001 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830571+0000", 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3409000000000003e-05 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.057 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830613+0000", 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2070000000000002e-05 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3535 3.0 3:0314dbed:::100000001f8.000009cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e93)", 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:07.806791+0000", 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 24.196444246999999, 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10098130900000001, 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.058 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3535 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.806791+0000", 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.806791+0000", 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.806793+0000", 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.216e-06 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889287+0000", 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.082494206 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.060 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889300+0000", 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2405999999999999e-05 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889307+0000", 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8180000000000004e-06 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889371+0000", 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3439e-05 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.061 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889478+0000", 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107463 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889565+0000", 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6687999999999994e-05 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.892897+0000", 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033321700000000002 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.062 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.907707+0000", 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014809666000000001 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.907742+0000", 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5358999999999998e-05 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.907772+0000", 2024-09-16T01:52:32.063 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9878000000000001e-05 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3678 3.3d 3:bd5b01a1:::100000001f8.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:15.845258+0000", 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.157977267, 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.14510701300000001, 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.064 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3678 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.845258+0000", 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.845258+0000", 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.065 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.845259+0000", 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.559e-06 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874177+0000", 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028917361999999999 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.066 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874191+0000", 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4112e-05 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874199+0000", 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8269999999999998e-06 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874263+0000", 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4771999999999996e-05 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.067 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874330+0000", 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6367999999999999e-05 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874428+0000", 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7827999999999999e-05 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.878015+0000", 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035875270000000001 2024-09-16T01:52:32.068 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.990289+0000", 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.11227349 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.990328+0000", 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9393e-05 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.990365+0000", 2024-09-16T01:52:32.069 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6774999999999999e-05 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3767 3.17 3:ea5aa73c:::100000001f8.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:16.992733+0000", 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.010501561, 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098425198000000005, 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.070 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3767 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.992733+0000", 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.071 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.992733+0000", 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.992736+0000", 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3420000000000002e-06 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.072 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020069+0000", 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027332838000000002 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020086+0000", 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7093000000000001e-05 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020093+0000", 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.605e-06 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.073 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020154+0000", 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0449999999999999e-05 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020224+0000", 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0395000000000006e-05 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020319+0000", 2024-09-16T01:52:32.074 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4666999999999993e-05 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.025192+0000", 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048735690000000003 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.091050+0000", 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065857268999999996 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.091108+0000", 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8604000000000003e-05 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.091159+0000", 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0365999999999999e-05 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3848 3.38 3:1fac8857:::100000001f8.00000105:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:32.076 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:18.087700+0000", 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 13.915534607, 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.095218909000000004, 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3848 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.077 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.087700+0000", 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.087700+0000", 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.087702+0000", 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5570000000000001e-06 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.078 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.133820+0000", 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.046118417000000002 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.133846+0000", 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5964000000000001e-05 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.133854+0000", 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3690000000000001e-06 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.079 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.133908+0000", 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4119999999999997e-05 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.133985+0000", 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6871000000000003e-05 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.134100+0000", 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000115464 2024-09-16T01:52:32.080 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.138388+0000", 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042879160000000001 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.182845+0000", 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044457231 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.081 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.182886+0000", 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0466000000000002e-05 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.182919+0000", 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3534000000000001e-05 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4143 3.38 3:1ff14137:::100000001f8.0000022c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:32.082 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:21.806808+0000", 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.196427155, 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.114230891, 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4143 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.083 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.806808+0000", 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.806808+0000", 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.806809+0000", 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.113e-06 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.084 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855271+0000", 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048461536 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855283+0000", 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2850000000000001e-05 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855289+0000", 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.088e-06 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.085 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855343+0000", 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3408000000000001e-05 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855441+0000", 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7683000000000006e-05 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855590+0000", 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014963800000000001 2024-09-16T01:52:32.086 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.860163+0000", 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045727429999999998 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.920937+0000", 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060774256999999998 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.920994+0000", 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6390000000000001e-05 2024-09-16T01:52:32.087 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.921039+0000", 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5185000000000002e-05 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4149 3.3d 3:bd018e47:::100000001f8.00000232:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:21.858586+0000", 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.144649178, 2024-09-16T01:52:32.088 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.120072939, 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4149 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.858586+0000", 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.858586+0000", 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.858588+0000", 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885427+0000", 2024-09-16T01:52:32.090 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026839279000000001 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885444+0000", 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6639000000000001e-05 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885450+0000", 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7730000000000002e-06 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.091 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885501+0000", 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0745999999999998e-05 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885589+0000", 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7706000000000004e-05 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885730+0000", 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014126199999999999 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.092 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.891308+0000", 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055783200000000003 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.978563+0000", 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.087254524 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.978612+0000", 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9527000000000002e-05 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.093 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.094 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.094 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.978659+0000", 2024-09-16T01:52:32.094 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6279000000000002e-05 2024-09-16T01:52:32.094 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.094 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.094 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.095 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.095 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4514 3.31 3:8d5ad0f7:::100000001f8.0000039f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:26.668664+0000", 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.3345709819999998, 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12311899900000001, 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4514 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.096 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.668664+0000", 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.668664+0000", 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.668666+0000", 2024-09-16T01:52:32.097 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4899999999999999e-06 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714354+0000", 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045688078999999999 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714371+0000", 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7238999999999999e-05 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.098 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714379+0000", 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0330000000000008e-06 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714433+0000", 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4403000000000001e-05 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714505+0000", 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1579e-05 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.099 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714619+0000", 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0001142 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.719684+0000", 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050654039999999999 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.791705+0000", 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.072020613999999997 2024-09-16T01:52:32.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.791744+0000", 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8818e-05 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.791783+0000", 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.101 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4516 3.1f 3:fbaaae67:::100000001f8.000003a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:26.714401+0000", 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.2888339860000002, 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10010822, 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4516 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.102 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714401+0000", 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714401+0000", 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714402+0000", 2024-09-16T01:52:32.103 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2389999999999999e-06 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742530+0000", 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028127632 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742547+0000", 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7448999999999998e-05 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.104 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742558+0000", 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0594999999999999e-05 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742622+0000", 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3960000000000004e-05 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742697+0000", 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5136999999999996e-05 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.105 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742787+0000", 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9806e-05 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.747912+0000", 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0051250610000000002 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.814442+0000", 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066529753999999997 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.106 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.814477+0000", 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.553e-05 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.814509+0000", 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2057000000000002e-05 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.107 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4556 3.17 3:e95f8c08:::100000001f8.000003c9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.184667+0000", 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.8185676810000002, 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.095277206000000003, 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4556 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.108 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.184667+0000", 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.184667+0000", 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.184668+0000", 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.145e-06 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.109 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224466+0000", 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039797470000000001 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224480+0000", 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3746e-05 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.110 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224504+0000", 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3986999999999999e-05 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224584+0000", 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0331e-05 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224670+0000", 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5536000000000002e-05 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.111 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224795+0000", 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012539199999999999 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.229058+0000", 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042626299999999999 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.279846+0000", 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050788341000000001 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.112 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.279895+0000", 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9468999999999997e-05 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.279945+0000", 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9159000000000001e-05 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.113 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4558 3.31 3:8c5060e5:::100000001f8.000003cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.224529+0000", 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.7787058, 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.110665689, 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4558 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.114 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224529+0000", 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224529+0000", 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224530+0000", 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8700000000000004e-07 2024-09-16T01:52:32.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280539+0000", 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056008702 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280550+0000", 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0834e-05 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280559+0000", 2024-09-16T01:52:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0929999999999992e-06 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280609+0000", 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9965e-05 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280680+0000", 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1489999999999995e-05 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280790+0000", 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0001098 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.285201+0000", 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044107510000000001 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.335088+0000", 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049887163999999998 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.335148+0000", 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9647e-05 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.335195+0000", 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7256000000000003e-05 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.119 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4579 3.3b 3:deddf43b:::100000001f8.000003e0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.536685+0000", 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.4665501110000001, 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.116318755, 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4579 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.536685+0000", 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.536685+0000", 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.536687+0000", 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616285+0000", 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.079598133000000001 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616302+0000", 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7014e-05 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616312+0000", 2024-09-16T01:52:32.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0515e-05 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616455+0000", 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014210199999999999 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616600+0000", 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014501299999999999 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616823+0000", 2024-09-16T01:52:32.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022324000000000001 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.622521+0000", 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056982409999999997 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.652897+0000", 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030375537000000001 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.652955+0000", 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8412999999999998e-05 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.653004+0000", 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8665999999999998e-05 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4642 3.29 3:963f87f9:::100000001f8.0000041f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e102)", 2024-09-16T01:52:32.125 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:28.282905+0000", 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 3.7203297439999998, 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096186552999999994, 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4642 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.282905+0000", 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.282905+0000", 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.282907+0000", 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.784e-06 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.310989+0000", 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028081608000000001 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311013+0000", 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3951000000000002e-05 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311025+0000", 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2268e-05 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311099+0000", 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4212e-05 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311199+0000", 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100348 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311357+0000", 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000157937 2024-09-16T01:52:32.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.316147+0000", 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047894620000000004 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.378946+0000", 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062799520999999997 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.379035+0000", 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8933999999999999e-05 2024-09-16T01:52:32.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.379092+0000", 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6527999999999998e-05 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.131 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:32.133 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:32.133 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:32.130+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:32.233 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:32.232+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:32.287 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:32.287 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:32.287 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:32.288 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:32.299 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:32.335 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:32.333+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:32.431 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:32.431 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:32.431 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:32.431 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:32.431 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:32.436 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:32.435+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:32.443 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:32.537 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:32.536+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:32.573 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:32.573 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:32.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:32.573 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:32.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.573 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:482 3.32 3:4c3c6424:::10000000002.000000df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.505480305999995, 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.78198721800000004, 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 482 2024-09-16T01:52:32.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064704+0000", 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.578e-06 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102892+0000", 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038187684999999999 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102910+0000", 2024-09-16T01:52:32.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8338999999999998e-05 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102918+0000", 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.259e-06 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102991+0000", 2024-09-16T01:52:32.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2638999999999994e-05 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.103003+0000", 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2269999999999999e-05 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.816160+0000", 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.71315740100000002 2024-09-16T01:52:32.578 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.820892+0000", 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047315539999999998 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835633+0000", 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014741232999999999 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835741+0000", 2024-09-16T01:52:32.579 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107768 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835903+0000", 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016157399999999999 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846577+0000", 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010674197 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.580 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846586+0000", 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5149999999999995e-06 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846627+0000", 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0513000000000003e-05 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846690+0000", 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2693000000000004e-05 2024-09-16T01:52:32.581 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:487 3.33 3:ce8011cb:::10000000002.000000e4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.465047671999997, 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.80604562300000004, 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.582 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.583 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 487 2024-09-16T01:52:32.583 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.583 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.583 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.583 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.583 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105136+0000", 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1429999999999999e-06 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.584 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131251+0000", 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026114741 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131264+0000", 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3325e-05 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131270+0000", 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1399999999999997e-06 2024-09-16T01:52:32.585 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131326+0000", 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5643000000000002e-05 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131334+0000", 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4020000000000002e-06 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.586 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.832106+0000", 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.700771794 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.833244+0000", 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011381679999999999 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890416+0000", 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057171588000000002 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.587 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890476+0000", 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0118999999999998e-05 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890571+0000", 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5011000000000002e-05 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.588 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.895250+0000", 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046789140000000002 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911079+0000", 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015829296999999999 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911119+0000", 2024-09-16T01:52:32.589 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9489000000000001e-05 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911181+0000", 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1848999999999997e-05 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.590 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:506 3.32 3:4c9ed383:::10000000002.000000f7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 92.28673268, 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.56375856099999999, 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.591 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 506 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.592 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283453+0000", 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9679999999999998e-06 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324488+0000", 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041035220999999997 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324503+0000", 2024-09-16T01:52:32.593 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5167000000000001e-05 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324512+0000", 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2019999999999993e-06 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324603+0000", 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494999999999993e-05 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:32.594 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324612+0000", 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0769999999999997e-06 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.821133+0000", 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.496521196 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.824031+0000", 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028972809999999998 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841705+0000", 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017674083 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841807+0000", 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102268 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841968+0000", 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016139600000000001 2024-09-16T01:52:32.596 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846962+0000", 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049941990000000004 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847135+0000", 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017284200000000001 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.597 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847161+0000", 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5193000000000001e-05 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847208+0000", 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7973000000000003e-05 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:538 3.33 3:cf13e063:::100000001f7.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:32.598 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 77.989699715, 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.109075775, 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 538 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:32.599 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580485+0000", 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1449999999999998e-06 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.600 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637145+0000", 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056659689999999999 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637165+0000", 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0092e-05 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637173+0000", 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0660000000000004e-06 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.601 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637263+0000", 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0059e-05 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637365+0000", 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101922 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.602 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637467+0000", 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101853 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.640930+0000", 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003463617 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689487+0000", 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048556903999999998 2024-09-16T01:52:32.603 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689524+0000", 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6776999999999997e-05 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689559+0000", 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4650000000000002e-05 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.604 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1239 3.33 3:cd5a60ff:::100000001f8.000000d4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.972804021000002, 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.093172247999999999, 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1239 2024-09-16T01:52:32.605 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597379+0000", 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.606 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.597381+0000", 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9139999999999998e-06 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647202+0000", 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049821439000000002 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.607 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647225+0000", 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3061000000000002e-05 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647234+0000", 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5620000000000004e-06 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647368+0000", 2024-09-16T01:52:32.608 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000134141 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647475+0000", 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106896 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.647575+0000", 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100463 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.609 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.650879+0000", 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033037740000000002 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690472+0000", 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039593415999999999 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690523+0000", 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0657e-05 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.610 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.690551+0000", 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7925000000000001e-05 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1501 3.24 3:26bdc0b7:::100000001f8.000001da:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.693746443999999, 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.092713806999999995, 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.611 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1501 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876436+0000", 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.876439+0000", 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4279999999999999e-06 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952215+0000", 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.075776423999999995 2024-09-16T01:52:32.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952227+0000", 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2218999999999999e-05 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952234+0000", 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.533e-06 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952285+0000", 2024-09-16T01:52:32.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1088000000000003e-05 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952410+0000", 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012510700000000001 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.952516+0000", 2024-09-16T01:52:32.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106437 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.957009+0000", 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044929569999999997 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969064+0000", 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012055066 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969104+0000", 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9687000000000001e-05 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:43.969150+0000", 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5861e-05 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1619 3.b 3:d196788b:::100000001f8.00000250:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:32.617 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 47.238850114999998, 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.094691490000000003, 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1619 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331332+0000", 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.331334+0000", 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4619999999999999e-06 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418738+0000", 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.087404198000000002 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418758+0000", 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9513000000000001e-05 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418768+0000", 2024-09-16T01:52:32.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0291e-05 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418832+0000", 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3544e-05 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.418975+0000", 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000143361 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:32.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.419119+0000", 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014366699999999999 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425110+0000", 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059917099999999999 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425951+0000", 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00084087400000000003 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.425990+0000", 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8868999999999997e-05 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.426024+0000", 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4001000000000002e-05 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1953 3.7 3:e1d5d0f0:::100000001f8.0000039e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.321177345000002, 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.108386528, 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1953 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249006+0000", 2024-09-16T01:52:32.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1179999999999999e-06 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307322+0000", 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058315141000000001 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307341+0000", 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9011000000000001e-05 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307349+0000", 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307416+0000", 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7377000000000002e-05 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307582+0000", 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016564800000000001 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307687+0000", 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104881 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.314441+0000", 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006754726 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357290+0000", 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042848857999999997 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357330+0000", 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9477000000000002e-05 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357392+0000", 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2009999999999998e-05 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1968 3.12 3:4ba9d71c:::100000001f8.000003ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.118929319999999, 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10484845199999999, 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1968 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451256+0000", 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:52:32.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483869+0000", 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032612532 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483888+0000", 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9029e-05 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483894+0000", 2024-09-16T01:52:32.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.674e-06 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483963+0000", 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8381999999999997e-05 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484065+0000", 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010199 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484168+0000", 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103727 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.489008+0000", 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048394880000000003 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556018+0000", 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067010583999999998 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556061+0000", 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3062999999999997e-05 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556102+0000", 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0246000000000001e-05 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.635 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1988 3.12 3:4924d375:::100000001f8.000003c1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.806728216000003, 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098484843000000002, 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1988 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763456+0000", 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7290000000000001e-06 2024-09-16T01:52:32.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819882+0000", 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056425945999999998 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819899+0000", 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6441000000000001e-05 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819905+0000", 2024-09-16T01:52:32.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8750000000000002e-06 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819953+0000", 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7774e-05 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820053+0000", 2024-09-16T01:52:32.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9702000000000002e-05 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820148+0000", 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5147000000000002e-05 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.823215+0000", 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030665369999999998 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861864+0000", 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038649944999999998 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861903+0000", 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8126000000000001e-05 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861939+0000", 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6621000000000001e-05 2024-09-16T01:52:32.641 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2037 3.12 3:486bd0c2:::100000001f8.000003f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 42.083112505999999, 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124209681, 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.642 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2037 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487072+0000", 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0590000000000001e-06 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523700+0000", 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036627383999999999 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523718+0000", 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8297000000000001e-05 2024-09-16T01:52:32.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523725+0000", 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523780+0000", 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4598000000000001e-05 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523909+0000", 2024-09-16T01:52:32.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012891700000000001 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.524030+0000", 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000120874 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.527427+0000", 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033977389999999999 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.646 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611203+0000", 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083775383999999994 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611243+0000", 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114000000000003e-05 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611280+0000", 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6965999999999998e-05 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.647 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2090 3.1 3:83370771:::100000001f8.00000427:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.337820034000003, 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.097529115999999999, 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.648 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2090 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232363+0000", 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.232365+0000", 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.945e-06 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264573+0000", 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032208891000000003 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264593+0000", 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9488999999999999e-05 2024-09-16T01:52:32.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264601+0000", 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2050000000000002e-06 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264714+0000", 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011313499999999999 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.651 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.264867+0000", 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000152805 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.265004+0000", 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013696499999999999 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.268362+0000", 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033575039999999999 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.652 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329806+0000", 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061444812000000001 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329852+0000", 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5367000000000001e-05 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.329892+0000", 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9997999999999999e-05 2024-09-16T01:52:32.653 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2987 3.1 3:8149885c:::100000001f8.000007a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:01.294923+0000", 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 31.275259844000001, 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.091602542999999995, 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.654 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2987 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.294923+0000", 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.294923+0000", 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.294925+0000", 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.863e-06 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.320982+0000", 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026057569999999999 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.321001+0000", 2024-09-16T01:52:32.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8847000000000001e-05 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.321025+0000", 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3561999999999999e-05 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.321101+0000", 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6688999999999997e-05 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.321237+0000", 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000135463 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.321358+0000", 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012139200000000001 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.325218+0000", 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038594409999999999 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.386449+0000", 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061231736000000002 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.386489+0000", 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9181999999999999e-05 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.386525+0000", 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6798000000000002e-05 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.659 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3374 3.33 3:cde6a495:::100000001f8.0000092b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.878092+0000", 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 26.692090187000002, 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.18611583200000001, 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.660 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3374 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.878092+0000", 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.878092+0000", 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.661 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.878096+0000", 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.163e-06 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990033+0000", 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.111937466 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990049+0000", 2024-09-16T01:52:32.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5953000000000001e-05 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990056+0000", 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6839999999999999e-06 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990108+0000", 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.198e-05 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990227+0000", 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119752 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990332+0000", 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010474799999999999 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.994827+0000", 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044950930000000004 2024-09-16T01:52:32.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.064103+0000", 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.069275854999999997 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.064150+0000", 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6845000000000003e-05 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.064208+0000", 2024-09-16T01:52:32.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8292999999999999e-05 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3836 3.24 3:268062e0:::100000001f8.000000f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:17.895041+0000", 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.675141543000001, 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.120890712, 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.666 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3836 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.895041+0000", 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.895041+0000", 2024-09-16T01:52:32.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.895042+0000", 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3500000000000005e-07 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984228+0000", 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089185981999999997 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984239+0000", 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1443e-05 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984245+0000", 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1869999999999996e-06 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984288+0000", 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3278000000000001e-05 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984355+0000", 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6909000000000006e-05 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984448+0000", 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3339999999999997e-05 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.988056+0000", 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036076229999999999 2024-09-16T01:52:32.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.015861+0000", 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027805407000000001 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.015899+0000", 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7626000000000002e-05 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.015932+0000", 2024-09-16T01:52:32.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2981999999999997e-05 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3865 3.1 3:81b781a2:::100000001f8.00000116:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:18.383704+0000", 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 14.186478803, 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.100186924, 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.672 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3865 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.383704+0000", 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.383704+0000", 2024-09-16T01:52:32.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.383706+0000", 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9199999999999998e-06 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442239+0000", 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058533636 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442254+0000", 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4221e-05 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442260+0000", 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7839999999999998e-06 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442360+0000", 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9674999999999997e-05 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442430+0000", 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0436000000000001e-05 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442530+0000", 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9454e-05 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.447395+0000", 2024-09-16T01:52:32.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048653280000000004 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.483813+0000", 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036417599000000002 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.483854+0000", 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0858000000000001e-05 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.483891+0000", 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7013e-05 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4141 3.1 3:8217cd07:::100000001f8.0000022a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:21.759930+0000", 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.810252874, 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12925631700000001, 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4141 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.759930+0000", 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.759930+0000", 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.759933+0000", 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7760000000000002e-06 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826535+0000", 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066602090000000003 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826548+0000", 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3106e-05 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826553+0000", 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6640000000000003e-06 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826599+0000", 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5812000000000001e-05 2024-09-16T01:52:32.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826671+0000", 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1588999999999995e-05 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826805+0000", 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000134358 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.835476+0000", 2024-09-16T01:52:32.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086709700000000001 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.889079+0000", 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053602740000000003 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.889140+0000", 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1122000000000002e-05 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.889186+0000", 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6090000000000001e-05 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4279 3.24 3:268558f9:::100000001f8.000002b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:23.725725+0000", 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 8.8444572109999999, 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098008718999999994, 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.684 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4279 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.725725+0000", 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.725725+0000", 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.725727+0000", 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.978e-06 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783117+0000", 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057389759999999998 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783138+0000", 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1134000000000001e-05 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783150+0000", 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1785e-05 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783274+0000", 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012360299999999999 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783398+0000", 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012403300000000001 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783530+0000", 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013236500000000001 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.789530+0000", 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059996290000000002 2024-09-16T01:52:32.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.823675+0000", 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034144865000000003 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.823707+0000", 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2746000000000001e-05 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.823734+0000", 2024-09-16T01:52:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6821e-05 2024-09-16T01:52:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4506 3.b 3:d1e869e5:::100000001f8.00000397:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:26.514914+0000", 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.0552684489999997, 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098290625000000006, 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4506 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.691 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.514914+0000", 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.514914+0000", 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.514915+0000", 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7899999999999997e-07 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563848+0000", 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048933046000000001 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563864+0000", 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5948999999999999e-05 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563872+0000", 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3340000000000002e-06 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563924+0000", 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1907999999999998e-05 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563995+0000", 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1198000000000005e-05 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.564099+0000", 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103618 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.567911+0000", 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038117020000000001 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.613096+0000", 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045184728 2024-09-16T01:52:32.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.613165+0000", 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9016999999999993e-05 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.613205+0000", 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0246000000000001e-05 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4578 3.7 3:e298cd56:::100000001f8.000003df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.507757+0000", 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.0624253360000004, 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.112995548, 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4578 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.697 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.507757+0000", 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.507757+0000", 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.507759+0000", 2024-09-16T01:52:32.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3769999999999999e-06 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594324+0000", 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086564957999999997 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594339+0000", 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5053e-05 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:32.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594347+0000", 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0190000000000006e-06 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594458+0000", 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011088500000000001 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594547+0000", 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9177000000000004e-05 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594669+0000", 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012209300000000001 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.598868+0000", 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004199075 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.620660+0000", 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.021792169 2024-09-16T01:52:32.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.620700+0000", 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9552999999999999e-05 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.620753+0000", 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3189000000000002e-05 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:32.702 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:32.703 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:32.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:52:32.704 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:32.705 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:32.703+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:32.804 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:32.803+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:32.823 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:32.823 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:32.823 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:32.823 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:32.833 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:32.905 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:32.904+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:32.941 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:32.941 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:32.941 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:32.941 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:32.942 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:32.952 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:32.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.965+0000 7fd8b4800640 1 Processor -- start 2024-09-16T01:52:32.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.966+0000 7fd8b4800640 1 -- start start 2024-09-16T01:52:32.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b4800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac15aab0 0x7fd8ac0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b4800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac15af00 con 0x7fd8ac0cb2f0 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b4800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ac15b1d0 con 0x7fd8ac0cc0c0 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b4800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ac0cbd00 con 0x7fd8ac15aab0 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b1e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd8ac0cc0c0 0x7fd8ac15e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36258/0 (socket says 172.21.15.146:36258) 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b1e00640 1 -- 172.21.15.146:0/85029678 learned_addr learned my addr 172.21.15.146:0/85029678 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b2800640 1 --2- 172.21.15.146:0/85029678 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac15aab0 0x7fd8ac0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b0a00640 1 -- 172.21.15.146:0/85029678 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968474873 0 0) 0x7fd8ac15b1d0 con 0x7fd8ac0cc0c0 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.968+0000 7fd8b0a00640 1 -- 172.21.15.146:0/85029678 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd894003520 con 0x7fd8ac0cc0c0 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.969+0000 7fd8b2800640 1 -- 172.21.15.146:0/85029678 >> v1:172.21.15.161:6789/0 conn(0x7fd8ac0cc0c0 legacy=0x7fd8ac15e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:32.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.969+0000 7fd8b2800640 1 -- 172.21.15.146:0/85029678 >> v1:172.21.15.146:6789/0 conn(0x7fd8ac0cb2f0 legacy=0x7fd8ac163420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.969+0000 7fd8b2800640 1 -- 172.21.15.146:0/85029678 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac15b4b0 con 0x7fd8ac15aab0 2024-09-16T01:52:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.970+0000 7fd8b2800640 1 --2- 172.21.15.146:0/85029678 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac15aab0 0x7fd8ac0cc4a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009dc0 tx=0x7fd8a802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4270e2f5f8c6c7dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:32.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.970+0000 7fd8b0a00640 1 -- 172.21.15.146:0/85029678 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8007740 con 0x7fd8ac15aab0 2024-09-16T01:52:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.970+0000 7fd8b0a00640 1 -- 172.21.15.146:0/85029678 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a80078e0 con 0x7fd8ac15aab0 2024-09-16T01:52:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.971+0000 7fd8b0a00640 1 -- 172.21.15.146:0/85029678 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8007c90 con 0x7fd8ac15aab0 2024-09-16T01:52:32.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.971+0000 7fd8b4800640 1 -- 172.21.15.146:0/85029678 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac15aab0 msgr2=0x7fd8ac0cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.971+0000 7fd8b4800640 1 --2- 172.21.15.146:0/85029678 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac15aab0 0x7fd8ac0cc4a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009dc0 tx=0x7fd8a802a6d0 comp rx=0 tx=0).stop 2024-09-16T01:52:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.972+0000 7fd8b4800640 1 -- 172.21.15.146:0/85029678 shutdown_connections 2024-09-16T01:52:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.972+0000 7fd8b4800640 1 --2- 172.21.15.146:0/85029678 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac15aab0 0x7fd8ac0cc4a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.972+0000 7fd8b4800640 1 -- 172.21.15.146:0/85029678 >> 172.21.15.146:0/85029678 conn(0x7fd8ac06d560 msgr2=0x7fd8ac06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.972+0000 7fd8b4800640 1 -- 172.21.15.146:0/85029678 shutdown_connections 2024-09-16T01:52:32.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.972+0000 7fd8b4800640 1 -- 172.21.15.146:0/85029678 wait complete. 2024-09-16T01:52:32.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.973+0000 7fd8b4800640 1 Processor -- start 2024-09-16T01:52:32.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 -- start start 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ac0cb2f0 0x7fd8ac0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 0x7fd8ac0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8ac15aab0 0x7fd8ac14e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac15e540 con 0x7fd8ac0cb2f0 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ac15ba70 con 0x7fd8ac15aab0 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.974+0000 7fd8b4800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ac15e1d0 con 0x7fd8ac0cc0c0 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 0x7fd8ac0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b2800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8ac15aab0 0x7fd8ac14e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:32.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 0x7fd8ac0b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42868/0 (socket says 172.21.15.146:42868) 2024-09-16T01:52:32.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b1400640 1 -- 172.21.15.146:0/965300081 learned_addr learned my addr 172.21.15.146:0/965300081 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:32.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b1e00640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ac0cb2f0 0x7fd8ac0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:32.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b2800640 1 -- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 msgr2=0x7fd8ac0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:32.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b2800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 0x7fd8ac0b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:32.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b2800640 1 -- 172.21.15.146:0/965300081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ac0cb2f0 msgr2=0x7fd8ac0c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:32.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b2800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ac0cb2f0 0x7fd8ac0c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:32.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b2800640 1 -- 172.21.15.146:0/965300081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ac0c7f00 con 0x7fd8ac15aab0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.975+0000 7fd8b1400640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 0x7fd8ac0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.976+0000 7fd8b2800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8ac15aab0 0x7fd8ac14e2d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac06bce0 tx=0x7fd8a8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.976+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8009450 con 0x7fd8ac15aab0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.976+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a8004260 con 0x7fd8ac15aab0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.976+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ac14f650 con 0x7fd8ac15aab0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.976+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ac14fc10 con 0x7fd8ac15aab0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.976+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8004650 con 0x7fd8ac15aab0 2024-09-16T01:52:32.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.977+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd888005290 con 0x7fd8ac15aab0 2024-09-16T01:52:32.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.978+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd8a8039040 con 0x7fd8ac15aab0 2024-09-16T01:52:32.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.979+0000 7fd8a6000640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89006c2b0 0x7fd89006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:32.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.979+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5741+0+0 (secure 0 0 0) 0x7fd8a80ac9d0 con 0x7fd8ac15aab0 2024-09-16T01:52:32.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.979+0000 7fd8b1e00640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89006c2b0 0x7fd89006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:32.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.980+0000 7fd8b1e00640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89006c2b0 0x7fd89006e770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00011e0 tx=0x7fd8a0007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:32.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:32.984+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd8a8040030 con 0x7fd8ac15aab0 2024-09-16T01:52:33.006 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:33.005+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:33.091 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:33.092 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:33.092 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-16T01:52:33.092 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:52:33.092 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:52:33.092 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-16T01:52:33.107 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:33.106+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:13 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 204.25268533900001, 2024-09-16T01:52:33.115 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013380318, 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860535+0000", 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860537+0000", 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0490000000000002e-06 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860555+0000", 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7951000000000001e-05 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.117 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860573+0000", 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7779999999999999e-05 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860582+0000", 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8260000000000005e-06 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860665+0000", 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:52:33.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860805+0000", 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013978600000000001 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860969+0000", 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016432099999999999 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:33.119 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862912+0000", 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0019427680000000001 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873810+0000", 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010897676 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873874+0000", 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3857999999999997e-05 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.120 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873915+0000", 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.1436000000000001e-05 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(mds.1.9:30 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e22)", 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 204.25258683199999, 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013414977999999999, 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.121 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860633+0000", 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.122 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860634+0000", 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.16e-07 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860640+0000", 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0759999999999999e-06 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860651+0000", 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0394000000000001e-05 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.123 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.860658+0000", 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2930000000000003e-06 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861251+0000", 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059260400000000005 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861331+0000", 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9983999999999999e-05 2024-09-16T01:52:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.861450+0000", 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119474 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.862920+0000", 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0014703019999999999 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:33.125 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.873961+0000", 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011040329999999999 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874010+0000", 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9292999999999997e-05 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:49:08.874048+0000", 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8312000000000002e-05 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:17 3.21 e55/41 3:86f42aac:::10000000002.0000000e:head v 55'1, mlcod=0'0)", 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 102.736292906, 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011486753000000001, 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:52:33.127 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376927+0000", 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.376928+0000", 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0950000000000001e-06 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.128 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385558+0000", 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086297260000000008 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385576+0000", 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.806e-05 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385587+0000", 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1165999999999999e-05 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.129 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385660+0000", 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2346000000000002e-05 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.385679+0000", 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9038999999999998e-05 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388352+0000", 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026728210000000001 2024-09-16T01:52:33.130 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.388414+0000", 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2500000000000001e-05 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:20 3.21 e55/41 3:85380590:::10000000002.00000011:head v 55'2, mlcod=0'0)", 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 102.66265077200001, 2024-09-16T01:52:33.131 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011038759, 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450569+0000", 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.132 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.450571+0000", 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.111e-06 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458973+0000", 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0084022200000000002 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.458993+0000", 2024-09-16T01:52:33.133 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9945e-05 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459003+0000", 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9860000000000006e-06 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459070+0000", 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7600000000000003e-05 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.134 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.459092+0000", 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1860000000000001e-05 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.461569+0000", 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0024764679999999999 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.461608+0000", 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9569e-05 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.135 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:62 3.c e55/35 3:31d8a57c:::10000000002.0000003b:head v 55'1, mlcod=0'0)", 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 102.27410879, 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011246087, 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.136 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839111+0000", 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.839113+0000", 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.167e-06 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.137 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847457+0000", 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0083445559999999995 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847465+0000", 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7719999999999994e-06 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847469+0000", 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0820000000000001e-06 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.138 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847509+0000", 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0423999999999998e-05 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.847524+0000", 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.484e-05 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.850313+0000", 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027888370000000002 2024-09-16T01:52:33.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.850358+0000", 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4409000000000001e-05 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:772 3.21 e68/67 3:848ebb0b:::100000001f7.00000001:head v 68'21, mlcod=65'19)", 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 72.841825120999999, 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.025350340999999998, 2024-09-16T01:52:33.140 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271395+0000", 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.141 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.271396+0000", 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.144e-06 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293800+0000", 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022404103000000002 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293826+0000", 2024-09-16T01:52:33.142 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5809999999999999e-05 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293845+0000", 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8394e-05 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293958+0000", 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011299899999999999 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.143 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.293987+0000", 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8952999999999999e-05 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.296701+0000", 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027147180000000001 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:20.296745+0000", 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.422e-05 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.144 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1085 3.c e76/67 3:33b8095f:::100000001f8.0000003a:head v 76'25, mlcod=74'23)", 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 54.237244549000003, 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013510061, 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.145 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.875976+0000", 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8799999999999999e-07 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.146 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885660+0000", 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0096844449999999999 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885680+0000", 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9292000000000001e-05 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885689+0000", 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7920000000000004e-06 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.147 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885767+0000", 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7769999999999996e-05 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.885790+0000", 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2697999999999999e-05 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.889443+0000", 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036530579999999998 2024-09-16T01:52:33.148 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:38.889486+0000", 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2818000000000002e-05 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1134 3.21 e76/67 3:87440b43:::100000001f8.0000006b:head v 76'37, mlcod=74'35)", 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.687953809, 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016188002, 2024-09-16T01:52:33.149 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425266+0000", 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.150 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.425267+0000", 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0200000000000001e-07 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438526+0000", 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013259675 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438543+0000", 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7163000000000002e-05 2024-09-16T01:52:33.151 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438555+0000", 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1928e-05 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438633+0000", 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7935999999999994e-05 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.438653+0000", 2024-09-16T01:52:33.152 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9695999999999999e-05 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.441400+0000", 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0027465649999999999 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.441454+0000", 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4836999999999997e-05 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.153 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1678 3.21 e81/67 3:8795a5a4:::100000001f8.0000028b:head v 81'47, mlcod=81'45)", 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 47.104838592999997, 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011676199, 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.154 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008382+0000", 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.008383+0000", 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6500000000000008e-07 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015309+0000", 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0069268530000000002 2024-09-16T01:52:33.155 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015320+0000", 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0733999999999999e-05 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015326+0000", 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3279999999999996e-06 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.156 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015402+0000", 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6369e-05 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.015426+0000", 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3966000000000001e-05 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.020011+0000", 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045850730000000003 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.157 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:46.020058+0000", 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6910999999999999e-05 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1927 3.8 e84/83 3:13b44a54:::100000001f8.00000384:head v 84'56, mlcod=0'0)", 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.942424244999998, 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011956664000000001, 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.158 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.159 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.170796+0000", 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.85e-07 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179466+0000", 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086694140000000003 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179476+0000", 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0341e-05 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.160 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179482+0000", 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.353e-06 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179532+0000", 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9956000000000001e-05 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.179556+0000", 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3456e-05 2024-09-16T01:52:33.161 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.182704+0000", 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031482120000000001 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.182753+0000", 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8746999999999999e-05 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.162 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:1947 3.2c e84/77 3:3557c0c2:::100000001f8.00000398:head v 84'41, mlcod=81'39)", 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 43.825593353000002, 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016887692999999999, 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.163 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287627+0000", 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.287629+0000", 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.164 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300794+0000", 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013164334999999999 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300803+0000", 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2520000000000005e-06 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300809+0000", 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2790000000000004e-06 2024-09-16T01:52:33.165 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300867+0000", 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7485000000000003e-05 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.166 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.300890+0000", 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.372e-05 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.304472+0000", 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035818379999999999 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.304515+0000", 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2311999999999997e-05 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.167 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2105 3.21 e86/67 3:84901bec:::100000001f8.00000436:head v 86'54, mlcod=83'52)", 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 41.617917687000002, 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016103329, 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.168 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.495303+0000", 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6400000000000001e-07 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.169 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507953+0000", 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012649781000000001 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507969+0000", 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6367000000000001e-05 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.507977+0000", 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8250000000000005e-06 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.170 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.508038+0000", 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0900000000000003e-05 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.508061+0000", 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2453999999999999e-05 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.511370+0000", 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003309181 2024-09-16T01:52:33.171 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.511406+0000", 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6056999999999997e-05 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2381 3.34 e88/83 3:2c202722:::100000001f8.0000054a:head v 88'56, mlcod=87'54)", 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 38.346286974999998, 2024-09-16T01:52:33.172 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012640500000000001, 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766933+0000", 2024-09-16T01:52:33.173 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.766935+0000", 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.606e-06 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774938+0000", 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0080026449999999992 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.174 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774955+0000", 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7736000000000001e-05 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.774965+0000", 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4930000000000004e-06 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.775023+0000", 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8697999999999999e-05 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.175 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.775047+0000", 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3889000000000001e-05 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.779505+0000", 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044578710000000004 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:54.779574+0000", 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8561999999999998e-05 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.176 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2520 3.2e e88/67 3:75f1801c:::100000001f8.000005d5:head v 88'45, mlcod=88'43)", 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.324696+0000", 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.788524099999997, 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012355377000000001, 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.177 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.324696+0000", 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.324696+0000", 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.178 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.324697+0000", 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2500000000000004e-07 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333359+0000", 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086616520000000006 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333376+0000", 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7377999999999999e-05 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.179 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333385+0000", 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2229999999999996e-06 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333455+0000", 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9479000000000003e-05 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.333480+0000", 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5079000000000002e-05 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.180 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.336995+0000", 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035154869999999999 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.337052+0000", 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6153999999999998e-05 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.181 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2575 3.34 e88/83 3:2f1ca3fd:::100000001f8.0000060c:head v 88'61, mlcod=88'59)", 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:56.920492+0000", 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.192728668999997, 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.016208686999999999, 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.182 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.920492+0000", 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.920492+0000", 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.920493+0000", 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.879999999999999e-07 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.183 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933099+0000", 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012606640000000001 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933114+0000", 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.438e-05 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.184 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.185 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.185 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933120+0000", 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8990000000000004e-06 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933181+0000", 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0646999999999997e-05 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.933205+0000", 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.351e-05 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.186 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.936629+0000", 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034247420000000002 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:56.936700+0000", 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0881000000000001e-05 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.187 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:2856 3.c e89/67 3:333b8194:::100000001f8.00000725:head v 89'51, mlcod=89'49)", 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:59.956289+0000", 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 33.156930965999997, 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012073321, 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.188 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.956289+0000", 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.956289+0000", 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.956291+0000", 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4890000000000001e-06 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.189 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965386+0000", 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0090953500000000003 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965411+0000", 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4992999999999999e-05 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965422+0000", 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1317e-05 2024-09-16T01:52:33.190 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965488+0000", 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.5575999999999997e-05 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.965507+0000", 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9029e-05 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.191 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.968322+0000", 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028148380000000001 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:59.968363+0000", 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0729000000000003e-05 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:3010 3.2e e90/67 3:7418dfc9:::100000001f8.000007bf:head v 90'53, mlcod=90'51)", 2024-09-16T01:52:33.192 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 31.518063210000001, 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012532477, 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.193 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.595157+0000", 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4700000000000002e-07 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603459+0000", 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0083016960000000008 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.194 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603480+0000", 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0397000000000001e-05 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603493+0000", 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3293e-05 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603570+0000", 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7446000000000004e-05 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.195 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.603598+0000", 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7463000000000001e-05 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.607657+0000", 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004059287 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.607689+0000", 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2447999999999998e-05 2024-09-16T01:52:33.196 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:3379 3.34 e92/83 3:2d9351da:::100000001f8.00000930:head v 92'71, mlcod=92'69)", 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:06.013315+0000", 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 27.099904764000001, 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.011711634, 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.197 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.013315+0000", 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.013315+0000", 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.013317+0000", 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2470000000000001e-06 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.198 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.019884+0000", 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065672760000000004 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.019900+0000", 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6124999999999999e-05 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.019907+0000", 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.109e-06 2024-09-16T01:52:33.199 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.019957+0000", 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9888000000000001e-05 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.019976+0000", 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9381e-05 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.024985+0000", 2024-09-16T01:52:33.200 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050081659999999997 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.025027+0000", 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2441999999999998e-05 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.201 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:3507 3.2c e93/77 3:37473aed:::100000001f8.000009b0:head v 93'68, mlcod=92'66)", 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:07.529669+0000", 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 25.583551013000001, 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017911284, 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.529669+0000", 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.202 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.529669+0000", 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.529671+0000", 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4219999999999999e-06 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.544322+0000", 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014651004 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.203 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.544338+0000", 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6843000000000001e-05 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.544347+0000", 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7560000000000005e-06 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.544413+0000", 2024-09-16T01:52:33.204 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6012000000000004e-05 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.544433+0000", 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9585999999999998e-05 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.547531+0000", 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003098572 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.205 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.547580+0000", 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9088999999999997e-05 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_repop(client.4619.0:3758 3.34 e97/94 3:2fb335cc:::100000001f8.000000ab:head v 97'75, mlcod=93'73)", 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:16.908232+0000", 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.204987990999999, 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013062775, 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:33.206 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.908232+0000", 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.908232+0000", 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.908233+0000", 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4300000000000001e-07 2024-09-16T01:52:33.207 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.915516+0000", 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0072832460000000002 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.915548+0000", 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1139999999999997e-05 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:33.208 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.915562+0000", 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4262e-05 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.915645+0000", 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2925999999999999e-05 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.915673+0000", 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7914000000000001e-05 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.209 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.921256+0000", 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055837930000000001 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.921295+0000", 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8550999999999997e-05 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:33.210 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:33.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:33.212 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:33.211+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:33.312 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:33.312+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:33.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.386+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fd888005580 con 0x7fd8ac15aab0 2024-09-16T01:52:33.390 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:33.390 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T01:52:33.390 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:33.451 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:33.450+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:33.504 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:33.504 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:33.504 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:33.504 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:33.514 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:33.513+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:33.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:33.615 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:33.614+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:33.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.660+0000 7fd8a6000640 1 -- 172.21.15.146:0/965300081 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v104) ==== 97+0+0 (secure 0 0 0) 0x7fd8a807aa30 con 0x7fd8ac15aab0 2024-09-16T01:52:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:52:33.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89006c2b0 msgr2=0x7fd89006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89006c2b0 0x7fd89006e770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00011e0 tx=0x7fd8a0007d30 comp rx=0 tx=0).stop 2024-09-16T01:52:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8ac15aab0 msgr2=0x7fd8ac14e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8ac15aab0 0x7fd8ac14e2d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac06bce0 tx=0x7fd8a8004a40 comp rx=0 tx=0).stop 2024-09-16T01:52:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 shutdown_connections 2024-09-16T01:52:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd89006c2b0 0x7fd89006e770 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8ac15aab0 0x7fd8ac14e2d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8ac0cc0c0 0x7fd8ac0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 --2- 172.21.15.146:0/965300081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ac0cb2f0 0x7fd8ac0c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 >> 172.21.15.146:0/965300081 conn(0x7fd8ac06d560 msgr2=0x7fd8ac158920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.664+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 shutdown_connections 2024-09-16T01:52:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.665+0000 7fd8b4800640 1 -- 172.21.15.146:0/965300081 wait complete. 2024-09-16T01:52:33.715 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:33.714+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:33.816 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:33.815+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:33.896 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:33.896 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:33.896 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:33.896 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:33.896 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:33.918 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:33.917+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:33.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:33.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.954+0000 7f75b9a00640 1 Processor -- start 2024-09-16T01:52:33.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.955+0000 7f75b9a00640 1 -- start start 2024-09-16T01:52:33.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.957+0000 7f75b9a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75b4154bc0 0x7f75b40ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.957+0000 7f75b9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b40702a0 con 0x7f75b4070eb0 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.957+0000 7f75b9a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b4070580 con 0x7f75b40c5280 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.957+0000 7f75b9a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f75b4070ad0 con 0x7f75b4154bc0 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.957+0000 7f75b3400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f75b40c5280 0x7f75b4165180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36260/0 (socket says 172.21.15.146:36260) 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.957+0000 7f75b3400640 1 -- 172.21.15.146:0/558372296 learned_addr learned my addr 172.21.15.146:0/558372296 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b2a00640 1 --2- 172.21.15.146:0/558372296 >> v2:172.21.15.146:3300/0 conn(0x7f75b4154bc0 0x7f75b40ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869537684 0 0) 0x7f75b4070580 con 0x7f75b40c5280 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7598003520 con 0x7f75b40c5280 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255038530 0 0) 0x7f75b40702a0 con 0x7f75b4070eb0 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75b4070580 con 0x7f75b4070eb0 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1441781832 0 0) 0x7f7598003520 con 0x7f75b40c5280 2024-09-16T01:52:33.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.958+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 >> v2:172.21.15.146:3300/0 conn(0x7f75b4154bc0 msgr2=0x7f75b40ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 --2- 172.21.15.146:0/558372296 >> v2:172.21.15.146:3300/0 conn(0x7f75b4154bc0 0x7f75b40ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 >> v1:172.21.15.146:6789/0 conn(0x7f75b4070eb0 legacy=0x7f75b40c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b40ca6c0 con 0x7f75b40c5280 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f75ac0019f0 con 0x7f75b40c5280 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 >> v1:172.21.15.161:6789/0 conn(0x7f75b40c5280 legacy=0x7f75b4165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7598003b40 0x7f7598003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7598004800 0x7f7598004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.959+0000 7f75b1600640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75980051c0 0x7f759800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f759800c0c0 con 0x7f75980051c0 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f759800c270 con 0x7f7598003b40 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f759800c470 con 0x7f7598004800 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b2a00640 1 --2- 172.21.15.146:0/558372296 >> v2:172.21.15.146:3300/0 conn(0x7f75b4154bc0 0x7f75b40ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b2000640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7598004800 0x7f7598004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b3400640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75980051c0 0x7f759800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b2a00640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7598003b40 0x7f7598003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b3400640 1 -- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7598004800 msgr2=0x7f7598004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b3400640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7598004800 0x7f7598004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.960+0000 7f75b3400640 1 -- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7598003b40 msgr2=0x7f7598003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.961+0000 7f75b3400640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7598003b40 0x7f7598003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.961+0000 7f75b3400640 1 -- 172.21.15.146:0/558372296 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7598003520 con 0x7f75980051c0 2024-09-16T01:52:33.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.961+0000 7f75b2a00640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7598003b40 0x7f7598003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:33.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.962+0000 7f75b3400640 1 -- 172.21.15.146:0/558372296 reap_dead start 2024-09-16T01:52:33.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.962+0000 7f75b3400640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75980051c0 0x7f759800b9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00a730 tx=0x7f7598005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f717eaeeaa3f68e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:33.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.963+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00a350 con 0x7f75980051c0 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.963+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ac02f070 con 0x7f75980051c0 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.963+0000 7f75b1600640 1 -- 172.21.15.146:0/558372296 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac02bb90 con 0x7f75980051c0 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.963+0000 7f75b9a00640 1 -- 172.21.15.146:0/558372296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75980051c0 msgr2=0x7f759800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.963+0000 7f75b9a00640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75980051c0 0x7f759800b9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00a730 tx=0x7f7598005830 comp rx=0 tx=0).stop 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.964+0000 7f75b9a00640 1 -- 172.21.15.146:0/558372296 shutdown_connections 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.964+0000 7f75b9a00640 1 --2- 172.21.15.146:0/558372296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75980051c0 0x7f759800b9c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.964+0000 7f75b9a00640 1 -- 172.21.15.146:0/558372296 >> 172.21.15.146:0/558372296 conn(0x7f75b406c2a0 msgr2=0x7f75b4067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.964+0000 7f75b9a00640 1 -- 172.21.15.146:0/558372296 shutdown_connections 2024-09-16T01:52:33.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.964+0000 7f75b9a00640 1 -- 172.21.15.146:0/558372296 wait complete. 2024-09-16T01:52:33.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.966+0000 7f75b9a00640 1 Processor -- start 2024-09-16T01:52:33.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.966+0000 7f75b9a00640 1 -- start start 2024-09-16T01:52:33.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b9a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 0x7f75b4148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b40cd9f0 0x7f75b414ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b9a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b4150460 0x7f75b4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f75b40ca470 con 0x7f75b40cd9f0 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b9a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75b40c8fe0 con 0x7f75b4150460 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b9a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75b40caa30 con 0x7f75980051c0 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b2a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 0x7f75b4148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b2a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 0x7f75b4148600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42888/0 (socket says 172.21.15.146:42888) 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b2a00640 1 -- 172.21.15.146:0/2293098591 learned_addr learned my addr 172.21.15.146:0/2293098591 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b40cd9f0 0x7f75b414ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.967+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b4150460 0x7f75b4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b2a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b4150460 msgr2=0x7f75b4159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b4150460 0x7f75b4159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b2a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b40cd9f0 msgr2=0x7f75b414ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75b40cd9f0 0x7f75b414ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b2a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b414ee60 con 0x7f75980051c0 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 0x7f75b4148600 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f75a40108c0 tx=0x7f75a4010d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a401a070 con 0x7f75980051c0 2024-09-16T01:52:33.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a400cb40 con 0x7f75980051c0 2024-09-16T01:52:33.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a400f040 con 0x7f75980051c0 2024-09-16T01:52:33.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.968+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b4147740 con 0x7f75980051c0 2024-09-16T01:52:33.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.969+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b4151b20 con 0x7f75980051c0 2024-09-16T01:52:33.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.970+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f75b4151930 con 0x7f75980051c0 2024-09-16T01:52:33.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.972+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f75a401d070 con 0x7f75980051c0 2024-09-16T01:52:33.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.973+0000 7f75aaa00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759406c2e0 0x7f759406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.973+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 5741+0+0 (secure 0 0 0) 0x7f75a408b8a0 con 0x7f75980051c0 2024-09-16T01:52:33.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.973+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759406c2e0 0x7f759406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.974+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v104) ==== 54+0+16 (secure 0 0 0) 0x7f75a408bbc0 con 0x7f75980051c0 2024-09-16T01:52:33.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.974+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b414e9d0 0x7f75b4159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:33.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.974+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75b40cae20 con 0x7f75b414e9d0 2024-09-16T01:52:33.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.974+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b414e9d0 0x7f75b4159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:33.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.974+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759406c2e0 0x7f759406e7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f75b4148f60 tx=0x7f75a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:33.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.975+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b414e9d0 0x7f75b4159a70 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:33.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:33.976+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b40cae20 con 0x7f75b414e9d0 2024-09-16T01:52:34.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.001+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b4148b40 con 0x7f75b414e9d0 2024-09-16T01:52:34.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.006+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b4148b40 con 0x7f75b414e9d0 2024-09-16T01:52:34.008 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:34.008 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:34.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.007+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b4151cf0 0x7f75b41448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.007+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f75b414f4b0 con 0x7f75b4151cf0 2024-09-16T01:52:34.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.008+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b4151cf0 0x7f75b41448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.010+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b4151cf0 0x7f75b41448e0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.013+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b414f4b0 con 0x7f75b4151cf0 2024-09-16T01:52:34.018 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:34.017+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:34.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.039+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b40ce0a0 con 0x7f75b4151cf0 2024-09-16T01:52:34.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.044+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b40ce0a0 con 0x7f75b4151cf0 2024-09-16T01:52:34.049 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:34.050 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:34.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.045+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b4062630 0x7f75b4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.045+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f75b41456f0 con 0x7f75b4062630 2024-09-16T01:52:34.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.049+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b4062630 0x7f75b4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.052+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b4062630 0x7f75b4062a30 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.060+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b41456f0 con 0x7f75b4062630 2024-09-16T01:52:34.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.086+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b40631c0 con 0x7f75b4062630 2024-09-16T01:52:34.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.089+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b40631c0 con 0x7f75b4062630 2024-09-16T01:52:34.090 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:34.091 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:34.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.090+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b40cc600 0x7f75b40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.090+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f75b40ccf60 con 0x7f75b40cc600 2024-09-16T01:52:34.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.091+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b40cc600 0x7f75b40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.092+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b40cc600 0x7f75b40cca20 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.093+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b40ccf60 con 0x7f75b40cc600 2024-09-16T01:52:34.119 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:34.118+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:34.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.122+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b402c910 con 0x7f75b40cc600 2024-09-16T01:52:34.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.128+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b402c910 con 0x7f75b40cc600 2024-09-16T01:52:34.131 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:34.131 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:34.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.130+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b402cf90 0x7f75b402d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.130+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f75b402d930 con 0x7f75b402cf90 2024-09-16T01:52:34.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.130+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b402cf90 0x7f75b402d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.131+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b402cf90 0x7f75b402d3f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.132+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b402d930 con 0x7f75b402cf90 2024-09-16T01:52:34.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.160+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b402d930 con 0x7f75b402cf90 2024-09-16T01:52:34.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.162+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b402d930 con 0x7f75b402cf90 2024-09-16T01:52:34.163 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:34.163 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:34.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.162+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b4204cd0 0x7f75b42070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.162+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f75b402daa0 con 0x7f75b4204cd0 2024-09-16T01:52:34.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.162+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b4204cd0 0x7f75b42070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.163+0000 7f75b2000640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b4204cd0 0x7f75b42070c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.164+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b402daa0 con 0x7f75b4204cd0 2024-09-16T01:52:34.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.191+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b402daa0 con 0x7f75b4204cd0 2024-09-16T01:52:34.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.195+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b402daa0 con 0x7f75b4204cd0 2024-09-16T01:52:34.199 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:34.199 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:34.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.199+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b4207b10 0x7f75b4209f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.199+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f75b420ceb0 con 0x7f75b4207b10 2024-09-16T01:52:34.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.201+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b4207b10 0x7f75b4209f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.201+0000 7f75b3400640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b4207b10 0x7f75b4209f00 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.202+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b420ceb0 con 0x7f75b4207b10 2024-09-16T01:52:34.220 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:34.219+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:34.223 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:34.223 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:34.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.231+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b420ce80 con 0x7f75b4207b10 2024-09-16T01:52:34.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.233+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b420ce80 con 0x7f75b4207b10 2024-09-16T01:52:34.236 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:34.237 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:34.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.236+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b420c0c0 0x7f75b420c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:34.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.236+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f75b4210280 con 0x7f75b420c0c0 2024-09-16T01:52:34.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.236+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b420c0c0 0x7f75b420c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:34.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.237+0000 7f75b2a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b420c0c0 0x7f75b420c500 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:34.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.238+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75b4210280 con 0x7f75b420c0c0 2024-09-16T01:52:34.244 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:34.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.272+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f75b420a470 con 0x7f75b420c0c0 2024-09-16T01:52:34.283 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.277+0000 7f75aaa00640 1 -- 172.21.15.146:0/2293098591 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f75b420a470 con 0x7f75b420c0c0 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b414e9d0 msgr2=0x7f75b4159a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f75b414e9d0 0x7f75b4159a70 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b4151cf0 msgr2=0x7f75b41448e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f75b4151cf0 0x7f75b41448e0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b4062630 msgr2=0x7f75b4062a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f75b4062630 0x7f75b4062a30 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b40cc600 msgr2=0x7f75b40cca20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f75b40cc600 0x7f75b40cca20 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b402cf90 msgr2=0x7f75b402d3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f75b402cf90 0x7f75b402d3f0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b4204cd0 msgr2=0x7f75b42070c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f75b4204cd0 0x7f75b42070c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b4207b10 msgr2=0x7f75b4209f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f75b4207b10 0x7f75b4209f00 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b420c0c0 msgr2=0x7f75b420c500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f75b420c0c0 0x7f75b420c500 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b3400640 1 -- 172.21.15.146:0/2293098591 reap_dead start 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b3400640 1 -- 172.21.15.146:0/2293098591 reap_dead start 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b3400640 1 -- 172.21.15.146:0/2293098591 reap_dead start 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b3400640 1 -- 172.21.15.146:0/2293098591 reap_dead start 2024-09-16T01:52:34.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.278+0000 7f75b3400640 1 -- 172.21.15.146:0/2293098591 reap_dead start 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759406c2e0 msgr2=0x7f759406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759406c2e0 0x7f759406e7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f75b4148f60 tx=0x7f75a0009040 comp rx=0 tx=0).stop 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 msgr2=0x7f75b4148600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 0x7f75b4148600 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f75a40108c0 tx=0x7f75a4010d80 comp rx=0 tx=0).stop 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 shutdown_connections 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f759406c2e0 0x7f759406e7a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 --2- 172.21.15.146:0/2293098591 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75980051c0 0x7f75b4148600 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 >> 172.21.15.146:0/2293098591 conn(0x7f75b406c2a0 msgr2=0x7f75b40c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 shutdown_connections 2024-09-16T01:52:34.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:34.279+0000 7f75b9a00640 1 -- 172.21.15.146:0/2293098591 wait complete. 2024-09-16T01:52:34.322 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:34.321+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:34.422 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:34.421+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:34.523 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:34.522+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:34.540 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:34.541 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:34.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:34.649 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:34.648+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:34.726 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:34.725+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:34.827 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:34.826+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:34.864 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:34.864 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:34.881 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:34.927 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:34.926+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:35.030 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:35.029+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:35.129 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:35.128+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:35.210 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:35.210 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:35.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:35.266 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:35.265+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:35.331 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:35.330+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:35.436 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:35.435+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:35.525 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:35.525 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:35.535 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:35.532+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:35.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:35.636 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:35.635+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:35.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:52:35.750 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:35.749+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:35.837 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:35.836+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:35.871 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:35.872 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:35.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:35.947 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:35.946+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:35.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.976+0000 7fe4cde00640 1 Processor -- start 2024-09-16T01:52:35.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.977+0000 7fe4cde00640 1 -- start start 2024-09-16T01:52:35.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.978+0000 7fe4cde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4c80cb2f0 0x7fe4c815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:35.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.978+0000 7fe4cde00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c815b1d0 con 0x7fe4c80cc4a0 2024-09-16T01:52:35.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.978+0000 7fe4cde00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c815b4b0 con 0x7fe4c80cc0c0 2024-09-16T01:52:35.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.978+0000 7fe4cde00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c80cbd00 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.978+0000 7fe4c6a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe4c80cc0c0 0x7fe4c815e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36264/0 (socket says 172.21.15.146:36264) 2024-09-16T01:52:35.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.978+0000 7fe4c6a00640 1 -- 172.21.15.146:0/1784839201 learned_addr learned my addr 172.21.15.146:0/1784839201 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:35.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.979+0000 7fe4c6000640 1 --2- 172.21.15.146:0/1784839201 >> v2:172.21.15.146:3300/0 conn(0x7fe4c80cb2f0 0x7fe4c815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.980+0000 7fe4c6000640 1 -- 172.21.15.146:0/1784839201 >> v1:172.21.15.161:6789/0 conn(0x7fe4c80cc0c0 legacy=0x7fe4c815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.980+0000 7fe4c6000640 1 -- 172.21.15.146:0/1784839201 >> v1:172.21.15.146:6789/0 conn(0x7fe4c80cc4a0 legacy=0x7fe4c8165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.980+0000 7fe4c6000640 1 -- 172.21.15.146:0/1784839201 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c815b790 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.981+0000 7fe4c6000640 1 --2- 172.21.15.146:0/1784839201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c815aab0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009980 tx=0x7fe4b802a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c938a8a228af03b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.982+0000 7fe4c5600640 1 -- 172.21.15.146:0/1784839201 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8037070 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.982+0000 7fe4c5600640 1 -- 172.21.15.146:0/1784839201 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b802ae20 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.982+0000 7fe4c5600640 1 -- 172.21.15.146:0/1784839201 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8032660 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.983+0000 7fe4cde00640 1 -- 172.21.15.146:0/1784839201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cb2f0 msgr2=0x7fe4c815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:35.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.983+0000 7fe4cde00640 1 --2- 172.21.15.146:0/1784839201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c815aab0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009980 tx=0x7fe4b802a320 comp rx=0 tx=0).stop 2024-09-16T01:52:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.984+0000 7fe4cde00640 1 -- 172.21.15.146:0/1784839201 shutdown_connections 2024-09-16T01:52:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.984+0000 7fe4cde00640 1 --2- 172.21.15.146:0/1784839201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c815aab0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:35.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.984+0000 7fe4cde00640 1 -- 172.21.15.146:0/1784839201 >> 172.21.15.146:0/1784839201 conn(0x7fe4c806d560 msgr2=0x7fe4c806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:35.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.986+0000 7fe4cde00640 1 -- 172.21.15.146:0/1784839201 shutdown_connections 2024-09-16T01:52:35.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.987+0000 7fe4cde00640 1 -- 172.21.15.146:0/1784839201 wait complete. 2024-09-16T01:52:35.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.988+0000 7fe4cde00640 1 Processor -- start 2024-09-16T01:52:35.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.988+0000 7fe4cde00640 1 -- start start 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.989+0000 7fe4cde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c8146b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.989+0000 7fe4cde00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4c80cc0c0 0x7fe4c815d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.989+0000 7fe4cde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 0x7fe4c81f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.989+0000 7fe4cde00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c815e540 con 0x7fe4c80cc4a0 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.989+0000 7fe4cde00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c815ba70 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.989+0000 7fe4cde00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c80cc8b0 con 0x7fe4c80cc0c0 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 0x7fe4c81f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:35.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 0x7fe4c81f8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39946/0 (socket says 172.21.15.146:39946) 2024-09-16T01:52:35.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c7400640 1 -- 172.21.15.146:0/57380835 learned_addr learned my addr 172.21.15.146:0/57380835 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:35.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c6a00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c8146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:35.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c6a00640 1 -- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4c80cc0c0 msgr2=0x7fe4c815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:35.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c6a00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4c80cc0c0 0x7fe4c815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c6a00640 1 -- 172.21.15.146:0/57380835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 msgr2=0x7fe4c81f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c6a00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 0x7fe4c81f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.990+0000 7fe4c6a00640 1 -- 172.21.15.146:0/57380835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c80c9ec0 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.991+0000 7fe4c7400640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 0x7fe4c81f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.991+0000 7fe4c6a00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c8146b10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000c520 tx=0x7fe4b000c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.991+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0018070 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.991+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b00074e0 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.992+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0004a00 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.993+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c8146550 con 0x7fe4c80cb2f0 2024-09-16T01:52:35.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.993+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c80ca130 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.994+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c80d0180 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.996+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe4b0004ba0 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.998+0000 7fe4bf400640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe4a406c2e0 0x7fe4a406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:35.998+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5741+0+0 (secure 0 0 0) 0x7fe4b008a590 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.001+0000 7fe4c6000640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe4a406c2e0 0x7fe4a406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.006+0000 7fe4c6000640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe4a406c2e0 0x7fe4a406e7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8034000 tx=0x7fe4b802c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.006+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe4c80d0180 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.038 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:36.036+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:36.139 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:36.138+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:36.210 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:36.210 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:36.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:36.240 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:36.239+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:36.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:36.364 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:36.362+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.416+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fe4c80c5560 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.441 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:36.440+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:36.531 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:36.531 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:36.544 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:36.540+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:36.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:36.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.572+0000 7f8a1b600640 1 Processor -- start 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.572+0000 7f8a1b600640 1 -- start start 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.574+0000 7f8a1b600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8a1406ece0 0x7f8a140c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.574+0000 7f8a1b600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8a141507f0 0x7f8a14150c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.574+0000 7f8a1b600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a141575b0 con 0x7f8a14071c70 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.574+0000 7f8a1b600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8a14157890 con 0x7f8a141507f0 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.574+0000 7f8a1b600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8a1406fad0 con 0x7f8a1406ece0 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.576+0000 7f8a13e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8a14071c70 0x7f8a1406f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36280/0 (socket says 172.21.15.146:36280) 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.576+0000 7f8a18c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8a1406ece0 0x7f8a140c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.576+0000 7f8a19600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8a141507f0 0x7f8a14150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.576+0000 7f8a18c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8a1406ece0 0x7f8a140c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:32940/0 (socket says 172.21.15.146:32940) 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.576+0000 7f8a18c00640 1 -- 172.21.15.146:0/3933718178 learned_addr learned my addr 172.21.15.146:0/3933718178 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.577+0000 7f8a18c00640 1 -- 172.21.15.146:0/3933718178 >> v2:172.21.15.146:3300/0 conn(0x7f8a141507f0 msgr2=0x7f8a14150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.577+0000 7f8a18c00640 1 --2- 172.21.15.146:0/3933718178 >> v2:172.21.15.146:3300/0 conn(0x7f8a141507f0 0x7f8a14150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.577+0000 7f8a18c00640 1 -- 172.21.15.146:0/3933718178 >> v1:172.21.15.161:6789/0 conn(0x7f8a14071c70 legacy=0x7f8a1406f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.578+0000 7f8a18c00640 1 -- 172.21.15.146:0/3933718178 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1406fdb0 con 0x7f8a1406ece0 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.580+0000 7f8a18c00640 1 --2- 172.21.15.146:0/3933718178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c85a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8a140c9850 tx=0x7f8a0c02a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644a41e1fd2f6072 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.581+0000 7f8a19600640 1 --2- 172.21.15.146:0/3933718178 >> v2:172.21.15.146:3300/0 conn(0x7f8a141507f0 0x7f8a14150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.582+0000 7f8a13400640 1 -- 172.21.15.146:0/3933718178 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c0325f0 con 0x7f8a1406ece0 2024-09-16T01:52:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.582+0000 7f8a13400640 1 -- 172.21.15.146:0/3933718178 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0c032790 con 0x7f8a1406ece0 2024-09-16T01:52:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.582+0000 7f8a13400640 1 -- 172.21.15.146:0/3933718178 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c032b40 con 0x7f8a1406ece0 2024-09-16T01:52:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.583+0000 7f8a1b600640 1 -- 172.21.15.146:0/3933718178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 msgr2=0x7f8a140c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.583+0000 7f8a1b600640 1 --2- 172.21.15.146:0/3933718178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c85a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8a140c9850 tx=0x7f8a0c02a780 comp rx=0 tx=0).stop 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.583+0000 7f8a1b600640 1 -- 172.21.15.146:0/3933718178 shutdown_connections 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.583+0000 7f8a1b600640 1 --2- 172.21.15.146:0/3933718178 >> v2:172.21.15.146:3300/0 conn(0x7f8a141507f0 0x7f8a14150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.583+0000 7f8a1b600640 1 --2- 172.21.15.146:0/3933718178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c85a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.583+0000 7f8a1b600640 1 -- 172.21.15.146:0/3933718178 >> 172.21.15.146:0/3933718178 conn(0x7f8a1406bc10 msgr2=0x7f8a1406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.584+0000 7f8a1b600640 1 -- 172.21.15.146:0/3933718178 shutdown_connections 2024-09-16T01:52:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.584+0000 7f8a1b600640 1 -- 172.21.15.146:0/3933718178 wait complete. 2024-09-16T01:52:36.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.586+0000 7f8a1b600640 1 Processor -- start 2024-09-16T01:52:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.587+0000 7f8a1b600640 1 -- start start 2024-09-16T01:52:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a1b600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a1b600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a14071c70 0x7f8a1414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a1b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a141507f0 0x7f8a140cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a1b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1406c880 con 0x7f8a141507f0 2024-09-16T01:52:36.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a1b600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1406c700 con 0x7f8a1406ece0 2024-09-16T01:52:36.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a1b600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8a14072a30 con 0x7f8a14071c70 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.588+0000 7f8a18c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a13e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a14071c70 0x7f8a1414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a18c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:32956/0 (socket says 172.21.15.146:32956) 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a19600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a141507f0 0x7f8a140cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a18c00640 1 -- 172.21.15.146:0/574036329 learned_addr learned my addr 172.21.15.146:0/574036329 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a14071c70 msgr2=0x7f8a1414bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a14071c70 0x7f8a1414bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 msgr2=0x7f8a140c16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a1406ece0 0x7f8a140c16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.589+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a141513a0 con 0x7f8a141507f0 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.590+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a141507f0 0x7f8a140cab50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400d560 tx=0x7f8a0400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.590+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04004230 con 0x7f8a141507f0 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.590+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a040043d0 con 0x7f8a141507f0 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.590+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a04004230 con 0x7f8a141507f0 2024-09-16T01:52:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.591+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a140c7860 con 0x7f8a141507f0 2024-09-16T01:52:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.591+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a14152860 con 0x7f8a141507f0 2024-09-16T01:52:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.592+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8a140c85a0 con 0x7f8a141507f0 2024-09-16T01:52:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.593+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8a04012070 con 0x7f8a141507f0 2024-09-16T01:52:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.596+0000 7f8a10c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f89f406c4b0 0x7f89f406e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.596+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f89f406c4b0 0x7f89f406e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.596+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5741+0+0 (secure 0 0 0) 0x7f8a0408af00 con 0x7f8a141507f0 2024-09-16T01:52:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.597+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f89f406c4b0 0x7f89f406e970 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c02a7c0 tx=0x7f8a0c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.597+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v104) ==== 54+0+16 (secure 0 0 0) 0x7f8a0408d580 con 0x7f8a141507f0 2024-09-16T01:52:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.598+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f8a140c9fb0 0x7f8a141f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.598+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a1406c4e0 con 0x7f8a140c9fb0 2024-09-16T01:52:36.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.599+0000 7f8a13e00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f8a140c9fb0 0x7f8a141f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.600+0000 7f8a13e00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f8a140c9fb0 0x7f8a141f82f0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.602+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a1406c4e0 con 0x7f8a140c9fb0 2024-09-16T01:52:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.630+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a141543d0 con 0x7f8a140c9fb0 2024-09-16T01:52:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.633+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a141543d0 con 0x7f8a140c9fb0 2024-09-16T01:52:36.635 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.634+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f8a14152a60 0x7f8a141f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.634+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8a14156420 con 0x7f8a14152a60 2024-09-16T01:52:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.634+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f8a14152a60 0x7f8a141f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.635+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f8a14152a60 0x7f8a141f8d40 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.643 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:36.642+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.649+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a14156420 con 0x7f8a14152a60 2024-09-16T01:52:36.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.695+0000 7fe4bf400640 1 -- 172.21.15.146:0/57380835 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v105) ==== 91+0+0 (secure 0 0 0) 0x7fe4b0058690 con 0x7fe4c80cb2f0 2024-09-16T01:52:36.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.696+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a140c7a30 con 0x7f8a14152a60 2024-09-16T01:52:36.699 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:52:36.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.700+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe4a406c2e0 msgr2=0x7fe4a406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.700+0000 7fe4cde00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe4a406c2e0 0x7fe4a406e7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8034000 tx=0x7fe4b802c000 comp rx=0 tx=0).stop 2024-09-16T01:52:36.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.700+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c80cb2f0 msgr2=0x7fe4c8146b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.700+0000 7fe4cde00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c8146b10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000c520 tx=0x7fe4b000c9e0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.702+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a140c7a30 con 0x7f8a14152a60 2024-09-16T01:52:36.705 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:36.705 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 shutdown_connections 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe4a406c2e0 0x7fe4a406e7a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4c80cc4a0 0x7fe4c81f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4c80cc0c0 0x7fe4c815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 --2- 172.21.15.146:0/57380835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c80cb2f0 0x7fe4c8146b10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 >> 172.21.15.146:0/57380835 conn(0x7fe4c806d560 msgr2=0x7fe4c80cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 shutdown_connections 2024-09-16T01:52:36.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7fe4cde00640 1 -- 172.21.15.146:0/57380835 wait complete. 2024-09-16T01:52:36.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f8a14153060 0x7f8a141448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.703+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8a140cb200 con 0x7f8a14153060 2024-09-16T01:52:36.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.704+0000 7f8a13e00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f8a14153060 0x7f8a141448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.705+0000 7f8a13e00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f8a14153060 0x7f8a141448f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.706+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a140cb200 con 0x7f8a14153060 2024-09-16T01:52:36.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.734+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a14145080 con 0x7f8a14153060 2024-09-16T01:52:36.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.736+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a14145080 con 0x7f8a14153060 2024-09-16T01:52:36.738 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:36.739 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:36.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.737+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8a14062f00 0x7f8a1414e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.737+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8a14145080 con 0x7f8a14062f00 2024-09-16T01:52:36.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.737+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8a14062f00 0x7f8a1414e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.738+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8a14062f00 0x7f8a1414e280 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.740+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a14145080 con 0x7f8a14062f00 2024-09-16T01:52:36.744 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:36.742+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:36.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.767+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a14145080 con 0x7f8a14062f00 2024-09-16T01:52:36.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.770+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a14145080 con 0x7f8a14062f00 2024-09-16T01:52:36.772 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:36.772 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:36.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.770+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8a1414ecd0 0x7f8a1402c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.770+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8a14145080 con 0x7f8a1414ecd0 2024-09-16T01:52:36.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.771+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8a1414ecd0 0x7f8a1402c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.771+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8a1414ecd0 0x7f8a1402c640 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.772+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a14145080 con 0x7f8a1414ecd0 2024-09-16T01:52:36.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.793+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a1414f130 con 0x7f8a1414ecd0 2024-09-16T01:52:36.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.795+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a1414f130 con 0x7f8a1414ecd0 2024-09-16T01:52:36.798 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:36.798 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:36.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.796+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f8a1402d090 0x7f8a1402d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.796+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8a1402da50 con 0x7f8a1402d090 2024-09-16T01:52:36.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.797+0000 7f8a13e00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f8a1402d090 0x7f8a1402d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.798+0000 7f8a13e00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f8a1402d090 0x7f8a1402d510 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.799+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a1402da50 con 0x7f8a1402d090 2024-09-16T01:52:36.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.821+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a1402da50 con 0x7f8a1402d090 2024-09-16T01:52:36.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.823+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a1402da50 con 0x7f8a1402d090 2024-09-16T01:52:36.826 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:36.826 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:36.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.824+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8a1420bda0 0x7f8a1420e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.824+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8a14145080 con 0x7f8a1420bda0 2024-09-16T01:52:36.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.824+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8a1420bda0 0x7f8a1420e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.825+0000 7f8a19600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8a1420bda0 0x7f8a1420e190 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.826+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a14145080 con 0x7f8a1420bda0 2024-09-16T01:52:36.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.842+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a14145080 con 0x7f8a1420bda0 2024-09-16T01:52:36.845 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:36.843+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:36.846 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:36.846 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:36.846 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:36.847 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:36.847 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:36.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.844+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a14145080 con 0x7f8a1420bda0 2024-09-16T01:52:36.848 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:36.849 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:36.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.845+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f8a1420ebe0 0x7f8a14210fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:36.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.845+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8a14213f80 con 0x7f8a1420ebe0 2024-09-16T01:52:36.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.845+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f8a1420ebe0 0x7f8a14210fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:36.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.846+0000 7f8a18c00640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f8a1420ebe0 0x7f8a14210fd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:36.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.847+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a14213f80 con 0x7f8a1420ebe0 2024-09-16T01:52:36.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:36.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.867+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8a14212ca0 con 0x7f8a1420ebe0 2024-09-16T01:52:36.874 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a10c00640 1 -- 172.21.15.146:0/574036329 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8a14212ca0 con 0x7f8a1420ebe0 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f8a140c9fb0 msgr2=0x7f8a141f82f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f8a140c9fb0 0x7f8a141f82f0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f8a14152a60 msgr2=0x7f8a141f8d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f8a14152a60 0x7f8a141f8d40 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f8a14153060 msgr2=0x7f8a141448f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f8a14153060 0x7f8a141448f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8a14062f00 msgr2=0x7f8a1414e280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8a14062f00 0x7f8a1414e280 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8a1414ecd0 msgr2=0x7f8a1402c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f8a1414ecd0 0x7f8a1402c640 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f8a1402d090 msgr2=0x7f8a1402d510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f8a1402d090 0x7f8a1402d510 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8a1420bda0 msgr2=0x7f8a1420e190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8a1420bda0 0x7f8a1420e190 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f8a1420ebe0 msgr2=0x7f8a14210fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.869+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f8a1420ebe0 0x7f8a14210fd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f89f406c4b0 msgr2=0x7f89f406e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f89f406c4b0 0x7f89f406e970 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c02a7c0 tx=0x7f8a0c02e000 comp rx=0 tx=0).stop 2024-09-16T01:52:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a141507f0 msgr2=0x7f8a140cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 reap_dead start 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a1b600640 1 --2- 172.21.15.146:0/574036329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a141507f0 0x7f8a140cab50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a0400d560 tx=0x7f8a0400da20 comp rx=0 tx=0).stop 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 reap_dead start 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 reap_dead start 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 reap_dead start 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.870+0000 7f8a19600640 1 -- 172.21.15.146:0/574036329 reap_dead start 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.872+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 shutdown_connections 2024-09-16T01:52:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.872+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 >> 172.21.15.146:0/574036329 conn(0x7f8a1406bc10 msgr2=0x7f8a14154e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.874+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 shutdown_connections 2024-09-16T01:52:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:36.874+0000 7f8a1b600640 1 -- 172.21.15.146:0/574036329 wait complete. 2024-09-16T01:52:36.946 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:36.944+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:37.049 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:37.047+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:37.085 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:37.085 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:37.085 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:37.085 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.232610437999995, 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.086 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.087 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:37.088 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.088 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.088 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.088 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.089 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.090 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.091 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.092 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:37.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.094 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:37.095 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:37.096 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 97.090150773000005, 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:37.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:37.099 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:37.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.101 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.102 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.103 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.104 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:37.105 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.742684550000007, 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.106 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.107 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.108 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:37.109 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.110 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:37.111 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:37.112 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.113 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 76.858393186000001, 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.114 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.115 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.116 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:37.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:37.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:37.120 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.820653683000003, 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.121 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:37.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:37.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.127 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.961666682999997, 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.128 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:37.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:37.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:37.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:37.136 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.094627709000001, 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:37.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:37.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:37.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:37.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.898675488000002, 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.144 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:37.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.148 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:37.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.857673544999997, 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.151 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:37.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:37.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.633445985000002, 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:37.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:37.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.676664432000003, 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:37.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:37.170 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.765684082999996, 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.171 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:37.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.177 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.329153100999999, 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.178 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:52:37.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.305999183000001, 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.185 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:52:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:52:37.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:52:37.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:52:37.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:52:37.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.193 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.295063339999999, 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:52:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.264868916000001, 2024-09-16T01:52:37.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:52:37.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:52:37.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.223463463000002, 2024-09-16T01:52:37.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:37.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:52:37.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:52:37.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3680 3.2f 3:f55c0805:::100000001f8.0000005d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:37.211 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.880456+0000", 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.200891678000001, 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107100212, 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3680 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880456+0000", 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880456+0000", 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880458+0000", 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1629999999999999e-06 2024-09-16T01:52:37.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.909010+0000", 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028551679999999999 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.909034+0000", 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.459e-05 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.909046+0000", 2024-09-16T01:52:37.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1806e-05 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.916790+0000", 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0077436229999999998 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.916900+0000", 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110333 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.917071+0000", 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017061600000000001 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972447+0000", 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055376085999999998 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.987402+0000", 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014954771 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.987506+0000", 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010464200000000001 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.987556+0000", 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9901999999999998e-05 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.217 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4502 3.d 3:b11edeb1:::100000001f8.00000393:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.496261+0000", 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.585086423, 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108009744, 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4502 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.496261+0000", 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.496261+0000", 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.496264+0000", 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.486e-06 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.521540+0000", 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025275928 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.521566+0000", 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6489999999999999e-05 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.521589+0000", 2024-09-16T01:52:37.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2971000000000001e-05 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.526217+0000", 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046282830000000004 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.526752+0000", 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00053502499999999995 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.527565+0000", 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00081244300000000004 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.575356+0000", 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047790780999999997 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.604141+0000", 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028785628000000001 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.604216+0000", 2024-09-16T01:52:37.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4543999999999996e-05 2024-09-16T01:52:37.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.604271+0000", 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5164999999999998e-05 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5039 3.35 3:afdde0ae:::100000001f8.000005ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:33.007654+0000", 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.0736933369999999, 2024-09-16T01:52:37.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10917603400000001, 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5039 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:37.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.007654+0000", 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.007654+0000", 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.007656+0000", 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080126+0000", 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072470100999999995 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080142+0000", 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5739999999999998e-05 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080151+0000", 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5710000000000004e-06 2024-09-16T01:52:37.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080231+0000", 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9308999999999996e-05 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080331+0000", 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9874e-05 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:37.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080484+0000", 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015373399999999999 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.110967+0000", 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030483135000000001 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.115628+0000", 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046609260000000001 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.116770+0000", 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011416099999999999 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.116830+0000", 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0331000000000002e-05 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:37.230 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:37.232 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:37.233 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:37.230+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:37.334 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:37.331+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:37.353 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:37.354 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:37.354 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:37.354 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:37.366 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:37.434 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:37.433+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:37.488 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:37.489 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:37.489 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:37.489 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:37.489 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:37.500 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:37.545 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:37.543+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:37.637 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:37.636+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:37.668 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:37.668 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:37.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:37.668 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:37.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.668 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 106.809247341, 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:52:37.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:37.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:52:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 105.897669029, 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.675 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.677 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:52:37.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:52:37.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 105.85340879899999, 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.681 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:52:37.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.685 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:52:37.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 105.78135345699999, 2024-09-16T01:52:37.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:52:37.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.694 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 97.524347035999995, 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:52:37.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:52:37.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.702 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 97.489550737000002, 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:52:37.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:52:37.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:52:37.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 97.406101784000001, 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.710 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:52:37.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:52:37.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 83.206798997000007, 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.716 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:52:37.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:52:37.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 83.061922058999997, 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:52:37.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:52:37.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 83.021277635000004, 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:52:37.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 82.863334800999993, 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:52:37.734 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:52:37.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:52:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 82.738207540000005, 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:52:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:52:37.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:37.746 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 81.701407609, 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:52:37.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:52:37.751 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:37.752 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 76.538381967000007, 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:37.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:52:37.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:52:37.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.758 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.404644177999998, 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.759 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:52:37.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:52:37.764 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 57.710254063999997, 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.765 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:52:37.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:52:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.383279498, 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.771 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:37.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:52:37.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.242369357999998, 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.777 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:52:37.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 53.189566413999998, 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.783 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.786 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:52:37.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 51.870307873999998, 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.790 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:52:37.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.792 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:52:37.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:37.795 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:37.796 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:37.797 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:37.798 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:37.795+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:37.898 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:37.896+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:37.954 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:37.954 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:37.954 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:37.954 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:37.966 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:37.999 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:37.997+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:38.089 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:38.089 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:38.089 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:38.089 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:38.089 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:38.099 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:38.099 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:38.098+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:38.200 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:38.199+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:32 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:38.258 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 107.73230540500001, 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10783086, 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 32 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:38.259 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523200+0000", 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.523201+0000", 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.125e-06 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.260 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612171+0000", 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.088969477000000005 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612191+0000", 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9905e-05 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612198+0000", 2024-09-16T01:52:38.261 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6879999999999999e-06 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612256+0000", 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7201999999999999e-05 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612402+0000", 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014645399999999999 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.262 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.612503+0000", 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100857 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.615349+0000", 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028458110000000002 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630949+0000", 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015600731 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.263 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.630985+0000", 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5879e-05 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.631031+0000", 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5731e-05 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.264 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1663 3.17 3:ea019aa2:::100000001f8.0000027c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 52.478814575000001, 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124529129, 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1663 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.265 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776691+0000", 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.776694+0000", 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.8320000000000002e-06 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.266 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866207+0000", 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089513588000000005 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866225+0000", 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7325000000000001e-05 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866232+0000", 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6149999999999999e-06 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866286+0000", 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3863000000000003e-05 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866393+0000", 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106443 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.866499+0000", 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010595299999999999 2024-09-16T01:52:38.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.869501+0000", 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030023570000000002 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901146+0000", 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031644671999999999 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901180+0000", 2024-09-16T01:52:38.269 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4277999999999999e-05 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.901220+0000", 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0203000000000001e-05 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2039 3.31 3:8f066b95:::100000001f8.000003f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 47.733208638000001, 2024-09-16T01:52:38.270 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12858615900000001, 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2039 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522297+0000", 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.522299+0000", 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0109999999999999e-06 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.272 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586678+0000", 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064378764000000005 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586695+0000", 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7159999999999998e-05 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586705+0000", 2024-09-16T01:52:38.273 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9839999999999996e-06 2024-09-16T01:52:38.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.275 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586760+0000", 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5099999999999998e-05 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.586894+0000", 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013428999999999999 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.587008+0000", 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114178 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.590811+0000", 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038024529999999999 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650793+0000", 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059982029999999999 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650843+0000", 2024-09-16T01:52:38.277 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9876999999999997e-05 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.650883+0000", 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0312000000000003e-05 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2092 3.1e 3:7aa4b9dd:::100000001f8.00000429:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e86)", 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 46.991614583999997, 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10603082799999999, 2024-09-16T01:52:38.278 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2092 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263891+0000", 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.263893+0000", 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2249999999999999e-06 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292346+0000", 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028453092999999999 2024-09-16T01:52:38.280 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292359+0000", 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2306999999999999e-05 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292364+0000", 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7409999999999998e-06 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.281 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292417+0000", 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2673000000000002e-05 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292522+0000", 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010544600000000001 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.282 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.292624+0000", 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101473 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.295728+0000", 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031039010000000001 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369848+0000", 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074120309999999995 2024-09-16T01:52:38.283 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369888+0000", 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9465000000000002e-05 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:51.369922+0000", 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4193999999999998e-05 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.284 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3029 3.17 3:ebfd80ec:::100000001f8.000007d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 36.521697336999999, 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.099718444000000003, 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.285 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.286 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.286 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3029 2024-09-16T01:52:38.286 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.286 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.286 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733808+0000", 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.287 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.733810+0000", 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1560000000000002e-06 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763618+0000", 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029807628999999999 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763642+0000", 2024-09-16T01:52:38.288 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3655999999999998e-05 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763651+0000", 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5259999999999999e-06 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763711+0000", 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9721e-05 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.289 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763830+0000", 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118792 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.763940+0000", 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110046 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.767357+0000", 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034171829999999999 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.290 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833431+0000", 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066074421999999994 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833483+0000", 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1261999999999998e-05 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:01.833527+0000", 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4051000000000002e-05 2024-09-16T01:52:38.291 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3344 3.17 3:e83df825:::100000001f8.0000090d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.696638104999998, 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096291749999999995, 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3344 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.293 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558867+0000", 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.558869+0000", 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.294 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614237+0000", 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.055368217999999997 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614250+0000", 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3259e-05 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614255+0000", 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2100000000000001e-06 2024-09-16T01:52:38.295 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614314+0000", 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8730999999999997e-05 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614411+0000", 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7116999999999999e-05 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.296 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.614507+0000", 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5575999999999994e-05 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.617642+0000", 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003135166 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655055+0000", 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.037413257999999998 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.297 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655113+0000", 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7726999999999998e-05 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.655159+0000", 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6292999999999999e-05 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3358 3.27 3:e7c6e76b:::100000001f8.0000091b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:38.298 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.521764351000002, 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096871567000000006, 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3358 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.299 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733741+0000", 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.733744+0000", 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.886e-06 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.300 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787499+0000", 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053755360000000002 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787517+0000", 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7645000000000001e-05 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787525+0000", 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8129999999999996e-06 2024-09-16T01:52:38.301 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787592+0000", 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6891999999999996e-05 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787751+0000", 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158975 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.787911+0000", 2024-09-16T01:52:38.302 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016023400000000001 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.793715+0000", 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058038480000000003 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.303 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830517+0000", 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036802435000000001 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830571+0000", 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3409000000000003e-05 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.830613+0000", 2024-09-16T01:52:38.304 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2070000000000002e-05 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3535 3.0 3:0314dbed:::100000001f8.000009cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e93)", 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:07.806791+0000", 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 30.448714815999999, 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10098130900000001, 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.305 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3535 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.806791+0000", 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.806791+0000", 2024-09-16T01:52:38.306 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.806793+0000", 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.216e-06 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889287+0000", 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.082494206 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.307 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889300+0000", 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2405999999999999e-05 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889307+0000", 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8180000000000004e-06 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889371+0000", 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3439e-05 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.308 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889478+0000", 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107463 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.889565+0000", 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.6687999999999994e-05 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.892897+0000", 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033321700000000002 2024-09-16T01:52:38.309 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.907707+0000", 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014809666000000001 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.907742+0000", 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5358999999999998e-05 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:07.907772+0000", 2024-09-16T01:52:38.310 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9878000000000001e-05 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3678 3.3d 3:bd5b01a1:::100000001f8.0000005b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:15.845258+0000", 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 22.410247836, 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.14510701300000001, 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.311 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3678 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.845258+0000", 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.845258+0000", 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.312 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.845259+0000", 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.559e-06 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874177+0000", 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028917361999999999 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874191+0000", 2024-09-16T01:52:38.313 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4112e-05 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874199+0000", 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8269999999999998e-06 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874263+0000", 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4771999999999996e-05 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.314 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874330+0000", 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6367999999999999e-05 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.874428+0000", 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7827999999999999e-05 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.878015+0000", 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035875270000000001 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.315 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.990289+0000", 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.11227349 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.990328+0000", 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9393e-05 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:15.990365+0000", 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6774999999999999e-05 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.316 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3767 3.17 3:ea5aa73c:::100000001f8.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:16.992733+0000", 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 21.262772129999998, 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098425198000000005, 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3767 2024-09-16T01:52:38.317 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.992733+0000", 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.992733+0000", 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.318 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:16.992736+0000", 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3420000000000002e-06 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020069+0000", 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027332838000000002 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020086+0000", 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7093000000000001e-05 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.319 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020093+0000", 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.605e-06 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020154+0000", 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0449999999999999e-05 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020224+0000", 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0395000000000006e-05 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.320 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.020319+0000", 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4666999999999993e-05 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.025192+0000", 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048735690000000003 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.091050+0000", 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065857268999999996 2024-09-16T01:52:38.321 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.091108+0000", 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8604000000000003e-05 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.091159+0000", 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0365999999999999e-05 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.322 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4143 3.38 3:1ff14137:::100000001f8.0000022c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:21.806808+0000", 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.448697723999999, 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.114230891, 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4143 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.323 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.806808+0000", 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.806808+0000", 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.806809+0000", 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.113e-06 2024-09-16T01:52:38.324 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855271+0000", 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048461536 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855283+0000", 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2850000000000001e-05 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855289+0000", 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.088e-06 2024-09-16T01:52:38.325 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855343+0000", 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3408000000000001e-05 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855441+0000", 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7683000000000006e-05 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.855590+0000", 2024-09-16T01:52:38.326 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014963800000000001 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.860163+0000", 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0045727429999999998 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.920937+0000", 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060774256999999998 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.327 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.920994+0000", 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6390000000000001e-05 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.921039+0000", 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5185000000000002e-05 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4149 3.3d 3:bd018e47:::100000001f8.00000232:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:21.858586+0000", 2024-09-16T01:52:38.328 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 16.396919746999998, 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.120072939, 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4149 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.858586+0000", 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.329 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.858586+0000", 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.858588+0000", 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885427+0000", 2024-09-16T01:52:38.330 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026839279000000001 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885444+0000", 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6639000000000001e-05 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885450+0000", 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7730000000000002e-06 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.331 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885501+0000", 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0745999999999998e-05 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885589+0000", 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7706000000000004e-05 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.885730+0000", 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014126199999999999 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.332 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.891308+0000", 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0055783200000000003 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.978563+0000", 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.087254524 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.978612+0000", 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9527000000000002e-05 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.333 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.978659+0000", 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6279000000000002e-05 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4514 3.31 3:8d5ad0f7:::100000001f8.0000039f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:26.668664+0000", 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.586841550999999, 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12311899900000001, 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.334 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4514 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.668664+0000", 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.335 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.668664+0000", 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.668666+0000", 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4899999999999999e-06 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714354+0000", 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045688078999999999 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.336 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714371+0000", 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7238999999999999e-05 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714379+0000", 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0330000000000008e-06 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714433+0000", 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4403000000000001e-05 2024-09-16T01:52:38.337 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714505+0000", 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1579e-05 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714619+0000", 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0001142 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.719684+0000", 2024-09-16T01:52:38.338 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0050654039999999999 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.791705+0000", 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.072020613999999997 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.791744+0000", 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8818e-05 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.339 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.791783+0000", 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9140000000000001e-05 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4516 3.1f 3:fbaaae67:::100000001f8.000003a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:26.714401+0000", 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.541104555, 2024-09-16T01:52:38.340 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10010822, 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4516 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714401+0000", 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.341 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714401+0000", 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.714402+0000", 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2389999999999999e-06 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742530+0000", 2024-09-16T01:52:38.342 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028127632 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742547+0000", 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7448999999999998e-05 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742558+0000", 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0594999999999999e-05 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.343 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742622+0000", 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3960000000000004e-05 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742697+0000", 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5136999999999996e-05 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.742787+0000", 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9806e-05 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.344 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.747912+0000", 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0051250610000000002 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.814442+0000", 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066529753999999997 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.814477+0000", 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.553e-05 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.345 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.814509+0000", 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2057000000000002e-05 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4556 3.17 3:e95f8c08:::100000001f8.000003c9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.184667+0000", 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.07083825, 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.095277206000000003, 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.346 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4556 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.184667+0000", 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.347 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.184667+0000", 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.184668+0000", 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.145e-06 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224466+0000", 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.039797470000000001 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.348 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224480+0000", 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3746e-05 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224504+0000", 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3986999999999999e-05 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224584+0000", 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0331e-05 2024-09-16T01:52:38.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224670+0000", 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5536000000000002e-05 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224795+0000", 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012539199999999999 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.229058+0000", 2024-09-16T01:52:38.350 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042626299999999999 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.279846+0000", 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050788341000000001 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.279895+0000", 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9468999999999997e-05 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.351 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.279945+0000", 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9159000000000001e-05 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4558 3.31 3:8c5060e5:::100000001f8.000003cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.224529+0000", 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.030976368999999, 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.110665689, 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.352 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4558 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224529+0000", 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224529+0000", 2024-09-16T01:52:38.353 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.224530+0000", 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8700000000000004e-07 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280539+0000", 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056008702 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280550+0000", 2024-09-16T01:52:38.354 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0834e-05 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280559+0000", 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0929999999999992e-06 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280609+0000", 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9965e-05 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.355 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280680+0000", 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1489999999999995e-05 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.280790+0000", 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0001098 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.285201+0000", 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044107510000000001 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.356 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.335088+0000", 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049887163999999998 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.335148+0000", 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9647e-05 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.335195+0000", 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7256000000000003e-05 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.357 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4579 3.3b 3:deddf43b:::100000001f8.000003e0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.536685+0000", 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 10.71882068, 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.116318755, 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4579 2024-09-16T01:52:38.358 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.536685+0000", 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.536685+0000", 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.359 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.536687+0000", 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8810000000000001e-06 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616285+0000", 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.079598133000000001 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616302+0000", 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7014e-05 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.360 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616312+0000", 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0515e-05 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616455+0000", 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014210199999999999 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616600+0000", 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014501299999999999 2024-09-16T01:52:38.361 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.616823+0000", 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022324000000000001 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.622521+0000", 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056982409999999997 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.652897+0000", 2024-09-16T01:52:38.362 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030375537000000001 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.652955+0000", 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8412999999999998e-05 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.653004+0000", 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.8665999999999998e-05 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.363 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4642 3.29 3:963f87f9:::100000001f8.0000041f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e102)", 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:28.282905+0000", 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 9.9726003130000009, 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.096186552999999994, 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4642 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.364 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.282905+0000", 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.282905+0000", 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.365 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.282907+0000", 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.784e-06 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.310989+0000", 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028081608000000001 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311013+0000", 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3951000000000002e-05 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.366 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311025+0000", 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2268e-05 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311099+0000", 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4212e-05 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311199+0000", 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100348 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.367 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.311357+0000", 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000157937 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.316147+0000", 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047894620000000004 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.378946+0000", 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062799520999999997 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.368 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.379035+0000", 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8933999999999999e-05 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:28.379092+0000", 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6527999999999998e-05 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.369 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5038 3.31 3:8e3087f4:::100000001f8.000005ab:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:32.997596+0000", 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.2579099999999999, 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.151359625, 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5038 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.997596+0000", 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.997596+0000", 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.997597+0000", 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5319999999999999e-06 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.371 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.062749+0000", 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065151595000000007 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.062773+0000", 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3887999999999999e-05 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.062785+0000", 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2045000000000001e-05 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.372 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.063010+0000", 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00022532300000000001 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.063130+0000", 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000120408 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.063250+0000", 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119716 2024-09-16T01:52:38.373 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.067118+0000", 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038675099999999998 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.148890+0000", 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.081772021 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.148924+0000", 2024-09-16T01:52:38.374 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4513e-05 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.148955+0000", 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1074000000000001e-05 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5095 3.0 3:03b4351b:::100000001f8.000005e4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:33.702862+0000", 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.5526434269999996, 2024-09-16T01:52:38.375 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.097799229000000001, 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5095 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.702862+0000", 2024-09-16T01:52:38.376 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.702862+0000", 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.702864+0000", 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.578e-06 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.751229+0000", 2024-09-16T01:52:38.377 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048365700999999997 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.751244+0000", 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5043e-05 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.751252+0000", 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4930000000000001e-06 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.378 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.751326+0000", 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.4039e-05 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.751413+0000", 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7121999999999997e-05 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.751528+0000", 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000114851 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.379 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.756666+0000", 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.005138348 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.800562+0000", 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.043895284 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.800619+0000", 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7451999999999998e-05 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.380 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.800661+0000", 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2317999999999997e-05 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.381 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:38.382 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:38.383 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:38.380+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:38.409 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:52:38.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T01:52:38.486 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:38.484+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:38.536 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:38.536 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5513 3.b 3:d170221d:::100000001f8.00000786:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:38.477376+0000", 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.058612602999999999, 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058631994, 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5513 2024-09-16T01:52:38.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.477376+0000", 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.477376+0000", 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.477378+0000", 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1659999999999999e-06 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.526172+0000", 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048794109000000002 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.526192+0000", 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0622e-05 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.526201+0000", 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3420000000000008e-06 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.526264+0000", 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3595000000000004e-05 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.526347+0000", 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2867000000000005e-05 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.526453+0000", 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010623799999999999 2024-09-16T01:52:38.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:38.530375+0000", 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0039219820000000001 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 1 2024-09-16T01:52:38.542 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:38.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:38.590 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:38.589+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:38.663 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:38.663 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:38.663 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:38.663 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:38.663 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:38.672 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:38.686 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:38.684+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:38.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.687+0000 7f2c79a00640 1 Processor -- start 2024-09-16T01:52:38.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.687+0000 7f2c79a00640 1 -- start start 2024-09-16T01:52:38.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.689+0000 7f2c79a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 0x7f2c740d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:38.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c79a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c74154ea0 con 0x7f2c74154ac0 2024-09-16T01:52:38.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c79a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c740c5340 con 0x7f2c740d0d00 2024-09-16T01:52:38.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c79a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2c740c5620 con 0x7f2c7415b190 2024-09-16T01:52:38.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c73400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 0x7f2c740d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:38.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c73400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 0x7f2c740d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42914/0 (socket says 172.21.15.146:42914) 2024-09-16T01:52:38.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c73400640 1 -- 172.21.15.146:0/1107725172 learned_addr learned my addr 172.21.15.146:0/1107725172 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:38.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c73400640 1 -- 172.21.15.146:0/1107725172 >> v1:172.21.15.161:6789/0 conn(0x7f2c740d0d00 legacy=0x7f2c7415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:38.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.690+0000 7f2c73400640 1 -- 172.21.15.146:0/1107725172 >> v1:172.21.15.146:6789/0 conn(0x7f2c74154ac0 legacy=0x7f2c7406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:38.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.691+0000 7f2c73400640 1 -- 172.21.15.146:0/1107725172 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c740c5900 con 0x7f2c7415b190 2024-09-16T01:52:38.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.693+0000 7f2c73400640 1 --2- 172.21.15.146:0/1107725172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 0x7f2c740d07c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098a0 tx=0x7f2c6002a370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3355048b599808 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:38.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.693+0000 7f2c72a00640 1 -- 172.21.15.146:0/1107725172 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60037070 con 0x7f2c7415b190 2024-09-16T01:52:38.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.693+0000 7f2c72a00640 1 -- 172.21.15.146:0/1107725172 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c60032640 con 0x7f2c7415b190 2024-09-16T01:52:38.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.693+0000 7f2c72a00640 1 -- 172.21.15.146:0/1107725172 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60032a10 con 0x7f2c7415b190 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 -- 172.21.15.146:0/1107725172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 msgr2=0x7f2c740d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 --2- 172.21.15.146:0/1107725172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 0x7f2c740d07c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098a0 tx=0x7f2c6002a370 comp rx=0 tx=0).stop 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 -- 172.21.15.146:0/1107725172 shutdown_connections 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 --2- 172.21.15.146:0/1107725172 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c7415b190 0x7f2c740d07c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 -- 172.21.15.146:0/1107725172 >> 172.21.15.146:0/1107725172 conn(0x7f2c7406e400 msgr2=0x7f2c7406e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 -- 172.21.15.146:0/1107725172 shutdown_connections 2024-09-16T01:52:38.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.695+0000 7f2c79a00640 1 -- 172.21.15.146:0/1107725172 wait complete. 2024-09-16T01:52:38.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.698+0000 7f2c79a00640 1 Processor -- start 2024-09-16T01:52:38.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.698+0000 7f2c79a00640 1 -- start start 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c740d0d00 0x7f2c740cf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 0x7f2c740cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7415b190 0x7f2c740cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2c740c61a0 con 0x7f2c7415b190 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2c740c5ec0 con 0x7f2c74154ac0 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2c740cad60 con 0x7f2c740d0d00 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 0x7f2c740cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:38.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 0x7f2c740cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:32970/0 (socket says 172.21.15.146:32970) 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 -- 172.21.15.146:0/4206870363 learned_addr learned my addr 172.21.15.146:0/4206870363 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c79000640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7415b190 0x7f2c740cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73e00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c740d0d00 0x7f2c740cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 -- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c740d0d00 msgr2=0x7f2c740cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c740d0d00 0x7f2c740cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 -- 172.21.15.146:0/4206870363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7415b190 msgr2=0x7f2c740cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7415b190 0x7f2c740cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.700+0000 7f2c73400640 1 -- 172.21.15.146:0/4206870363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c741f1680 con 0x7f2c74154ac0 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c79000640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7415b190 0x7f2c740cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c73e00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c740d0d00 0x7f2c740cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:38.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c73400640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 0x7f2c740cb820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098a0 tx=0x7f2c600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60037040 con 0x7f2c74154ac0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c60042070 con 0x7f2c74154ac0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60030bb0 con 0x7f2c74154ac0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.701+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c741e7e00 con 0x7f2c74154ac0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.702+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c740ccbf0 con 0x7f2c74154ac0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.703+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c7415b5b0 con 0x7f2c74154ac0 2024-09-16T01:52:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.704+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2c60032500 con 0x7f2c74154ac0 2024-09-16T01:52:38.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.706+0000 7f2c6be00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2c5406c300 0x7f2c5406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:38.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.707+0000 7f2c73e00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2c5406c300 0x7f2c5406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.707+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5741+0+0 (secure 0 0 0) 0x7f2c600abed0 con 0x7f2c74154ac0 2024-09-16T01:52:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.707+0000 7f2c73e00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2c5406c300 0x7f2c5406e7c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2c64001680 tx=0x7f2c64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:38.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:38.711+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2c60045360 con 0x7f2c74154ac0 2024-09-16T01:52:38.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:52:38.790 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:38.788+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:38.835 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:38.835 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:38.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:38.835 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:38.835 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:482 3.32 3:4c3c6424:::10000000002.000000df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 98.768180393999998, 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.78198721800000004, 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 482 2024-09-16T01:52:38.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064702+0000", 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.064704+0000", 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.578e-06 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102892+0000", 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038187684999999999 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102910+0000", 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8338999999999998e-05 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102918+0000", 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.259e-06 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.102991+0000", 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2638999999999994e-05 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.103003+0000", 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2269999999999999e-05 2024-09-16T01:52:38.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.816160+0000", 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.71315740100000002 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.820892+0000", 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047315539999999998 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835633+0000", 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014741232999999999 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835741+0000", 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000107768 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.835903+0000", 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016157399999999999 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.841 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846577+0000", 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010674197 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846586+0000", 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5149999999999995e-06 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846627+0000", 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0513000000000003e-05 2024-09-16T01:52:38.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846690+0000", 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2693000000000004e-05 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:487 3.33 3:ce8011cb:::10000000002.000000e4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 98.72774776, 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.80604562300000004, 2024-09-16T01:52:38.843 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 487 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105135+0000", 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.105136+0000", 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1429999999999999e-06 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131251+0000", 2024-09-16T01:52:38.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026114741 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131264+0000", 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3325e-05 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131270+0000", 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1399999999999997e-06 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131326+0000", 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.5643000000000002e-05 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.131334+0000", 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4020000000000002e-06 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.832106+0000", 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.700771794 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.833244+0000", 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0011381679999999999 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890416+0000", 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057171588000000002 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890476+0000", 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0118999999999998e-05 2024-09-16T01:52:38.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.890571+0000", 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5011000000000002e-05 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.895250+0000", 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0046789140000000002 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911079+0000", 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015829296999999999 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911119+0000", 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9489000000000001e-05 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.911181+0000", 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1848999999999997e-05 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.850 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:506 3.32 3:4c9ed383:::10000000002.000000f7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 98.549432768000003, 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.56375856099999999, 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.851 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 506 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283450+0000", 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.283453+0000", 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.9679999999999998e-06 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324488+0000", 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041035220999999997 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324503+0000", 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5167000000000001e-05 2024-09-16T01:52:38.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324512+0000", 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2019999999999993e-06 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324603+0000", 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0494999999999993e-05 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.324612+0000", 2024-09-16T01:52:38.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0769999999999997e-06 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.821133+0000", 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.496521196 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.824031+0000", 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0028972809999999998 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841705+0000", 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017674083 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841807+0000", 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102268 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 4", 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.841968+0000", 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016139600000000001 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.846962+0000", 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049941990000000004 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847135+0000", 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00017284200000000001 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847161+0000", 2024-09-16T01:52:38.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5193000000000001e-05 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.847208+0000", 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7973000000000003e-05 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:538 3.33 3:cf13e063:::100000001f7.00000017:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 84.252399803000003, 2024-09-16T01:52:38.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.109075775, 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 538 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:38.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580483+0000", 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.580485+0000", 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1449999999999998e-06 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637145+0000", 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056659689999999999 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637165+0000", 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0092e-05 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637173+0000", 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0660000000000004e-06 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637263+0000", 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0059e-05 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637365+0000", 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101922 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.637467+0000", 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000101853 2024-09-16T01:52:38.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.640930+0000", 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003463617 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689487+0000", 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048556903999999998 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689524+0000", 2024-09-16T01:52:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6776999999999997e-05 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689559+0000", 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4650000000000002e-05 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1953 3.7 3:e1d5d0f0:::100000001f8.0000039e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:38.864 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.583877432999998, 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.108386528, 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1953 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.865 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249005+0000", 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.249006+0000", 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1179999999999999e-06 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.866 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307322+0000", 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058315141000000001 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307341+0000", 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9011000000000001e-05 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307349+0000", 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.2810000000000008e-06 2024-09-16T01:52:38.867 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307416+0000", 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7377000000000002e-05 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307582+0000", 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016564800000000001 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.307687+0000", 2024-09-16T01:52:38.868 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104881 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.314441+0000", 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.006754726 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357290+0000", 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042848857999999997 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.869 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357330+0000", 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9477000000000002e-05 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.357392+0000", 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2009999999999998e-05 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1968 3.12 3:4ba9d71c:::100000001f8.000003ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:38.870 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.381629408000002, 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.10484845199999999, 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1968 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.871 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451253+0000", 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.451256+0000", 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7369999999999998e-06 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.872 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483869+0000", 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032612532 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483888+0000", 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9029e-05 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483894+0000", 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.674e-06 2024-09-16T01:52:38.873 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.483963+0000", 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8381999999999997e-05 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484065+0000", 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010199 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.484168+0000", 2024-09-16T01:52:38.874 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103727 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.489008+0000", 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048394880000000003 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556018+0000", 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.067010583999999998 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.875 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556061+0000", 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3062999999999997e-05 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.556102+0000", 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0246000000000001e-05 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1988 3.12 3:4924d375:::100000001f8.000003c1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:38.876 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 49.069428303999999, 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098484843000000002, 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1988 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.877 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763454+0000", 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.763456+0000", 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7290000000000001e-06 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.878 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819882+0000", 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.056425945999999998 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819899+0000", 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6441000000000001e-05 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819905+0000", 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8750000000000002e-06 2024-09-16T01:52:38.879 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.819953+0000", 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.7774e-05 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820053+0000", 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9702000000000002e-05 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.880 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.820148+0000", 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5147000000000002e-05 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.823215+0000", 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030665369999999998 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861864+0000", 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.038649944999999998 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.881 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861903+0000", 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8126000000000001e-05 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:49.861939+0000", 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6621000000000001e-05 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.882 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:2037 3.12 3:486bd0c2:::100000001f8.000003f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e85)", 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 48.345812594000002, 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.124209681, 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 2037 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.883 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487070+0000", 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.487072+0000", 2024-09-16T01:52:38.884 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0590000000000001e-06 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523700+0000", 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036627383999999999 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523718+0000", 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8297000000000001e-05 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.885 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523725+0000", 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523780+0000", 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4598000000000001e-05 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.523909+0000", 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012891700000000001 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.886 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.524030+0000", 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000120874 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.527427+0000", 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033977389999999999 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.887 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611203+0000", 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.083775383999999994 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611243+0000", 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0114000000000003e-05 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.888 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:50.611280+0000", 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6965999999999998e-05 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3374 3.33 3:cde6a495:::100000001f8.0000092b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:05.878092+0000", 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 32.954790275000001, 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.18611583200000001, 2024-09-16T01:52:38.889 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3374 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.878092+0000", 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.890 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.878092+0000", 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.878096+0000", 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.163e-06 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990033+0000", 2024-09-16T01:52:38.891 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.111937466 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990049+0000", 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5953000000000001e-05 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990056+0000", 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6839999999999999e-06 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.892 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990108+0000", 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.198e-05 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990227+0000", 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000119752 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.990332+0000", 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010474799999999999 2024-09-16T01:52:38.893 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:05.994827+0000", 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044950930000000004 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.064103+0000", 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.069275854999999997 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.064150+0000", 2024-09-16T01:52:38.894 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6845000000000003e-05 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:06.064208+0000", 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8292999999999999e-05 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3836 3.24 3:268062e0:::100000001f8.000000f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:17.895041+0000", 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.937841631000001, 2024-09-16T01:52:38.895 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.120890712, 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3836 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.895041+0000", 2024-09-16T01:52:38.896 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.895041+0000", 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.895042+0000", 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3500000000000005e-07 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.897 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984228+0000", 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.089185981999999997 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984239+0000", 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1443e-05 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984245+0000", 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1869999999999996e-06 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.898 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984288+0000", 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3278000000000001e-05 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984355+0000", 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.6909000000000006e-05 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.984448+0000", 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3339999999999997e-05 2024-09-16T01:52:38.899 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:17.988056+0000", 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0036076229999999999 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.015861+0000", 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027805407000000001 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.015899+0000", 2024-09-16T01:52:38.900 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7626000000000002e-05 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.015932+0000", 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2981999999999997e-05 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:3865 3.1 3:81b781a2:::100000001f8.00000116:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:18.383704+0000", 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 20.449178890999999, 2024-09-16T01:52:38.901 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.100186924, 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 3865 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.383704+0000", 2024-09-16T01:52:38.902 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.383704+0000", 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.383706+0000", 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9199999999999998e-06 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.903 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442239+0000", 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058533636 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442254+0000", 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4221e-05 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442260+0000", 2024-09-16T01:52:38.904 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7839999999999998e-06 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442360+0000", 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9674999999999997e-05 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442430+0000", 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.0436000000000001e-05 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.905 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.442530+0000", 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9454e-05 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.447395+0000", 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0048653280000000004 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.483813+0000", 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036417599000000002 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.906 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.483854+0000", 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0858000000000001e-05 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:18.483891+0000", 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7013e-05 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.907 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4141 3.1 3:8217cd07:::100000001f8.0000022a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:21.759930+0000", 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 17.072952961999999, 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.12925631700000001, 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4141 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.908 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.759930+0000", 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.759930+0000", 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.759933+0000", 2024-09-16T01:52:38.909 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.7760000000000002e-06 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826535+0000", 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.066602090000000003 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826548+0000", 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3106e-05 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.910 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826553+0000", 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6640000000000003e-06 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826599+0000", 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5812000000000001e-05 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826671+0000", 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1588999999999995e-05 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.911 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.826805+0000", 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000134358 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.835476+0000", 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0086709700000000001 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.889079+0000", 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.053602740000000003 2024-09-16T01:52:38.912 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.889140+0000", 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1122000000000002e-05 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:21.889186+0000", 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6090000000000001e-05 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.913 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4279 3.24 3:268558f9:::100000001f8.000002b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:23.725725+0000", 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 15.107157299000001, 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098008718999999994, 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4279 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.914 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.725725+0000", 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.725725+0000", 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.915 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.725727+0000", 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.978e-06 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783117+0000", 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057389759999999998 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.916 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783138+0000", 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1134000000000001e-05 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783150+0000", 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1785e-05 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783274+0000", 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012360299999999999 2024-09-16T01:52:38.917 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783398+0000", 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012403300000000001 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.783530+0000", 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013236500000000001 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.789530+0000", 2024-09-16T01:52:38.918 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0059996290000000002 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.823675+0000", 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034144865000000003 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.823707+0000", 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2746000000000001e-05 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.919 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:23.823734+0000", 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6821e-05 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4506 3.b 3:d1e869e5:::100000001f8.00000397:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:26.514914+0000", 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 12.317968537, 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.098290625000000006, 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.920 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4506 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.514914+0000", 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.921 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.514914+0000", 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.514915+0000", 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7899999999999997e-07 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563848+0000", 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.048933046000000001 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.922 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563864+0000", 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5948999999999999e-05 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563872+0000", 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3340000000000002e-06 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563924+0000", 2024-09-16T01:52:38.923 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.1907999999999998e-05 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.563995+0000", 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1198000000000005e-05 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.564099+0000", 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103618 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.924 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.567911+0000", 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038117020000000001 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.613096+0000", 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045184728 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.925 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.613165+0000", 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9016999999999993e-05 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:26.613205+0000", 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0246000000000001e-05 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.926 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:4578 3.7 3:e298cd56:::100000001f8.000003df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:27.507757+0000", 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 11.325125423999999, 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.112995548, 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.927 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 4578 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.507757+0000", 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.928 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.507757+0000", 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.507759+0000", 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3769999999999999e-06 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594324+0000", 2024-09-16T01:52:38.929 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.086564957999999997 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594339+0000", 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5053e-05 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594347+0000", 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.0190000000000006e-06 2024-09-16T01:52:38.930 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594458+0000", 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00011088500000000001 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594547+0000", 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9177000000000004e-05 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.931 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.594669+0000", 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012209300000000001 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.598868+0000", 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004199075 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.932 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.620660+0000", 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.021792169 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.620700+0000", 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9552999999999999e-05 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:27.620753+0000", 2024-09-16T01:52:38.933 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.3189000000000002e-05 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5010 3.33 3:cd109807:::100000001f8.0000058f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:32.674457+0000", 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 6.1584261009999999, 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.108762213, 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.934 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5010 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.674457+0000", 2024-09-16T01:52:38.935 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.674457+0000", 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.674462+0000", 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6949999999999997e-06 2024-09-16T01:52:38.936 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.717670+0000", 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.043208119000000003 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.717689+0000", 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8338999999999998e-05 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.937 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.717697+0000", 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4549999999999995e-06 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.717793+0000", 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6067e-05 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.717873+0000", 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9457999999999997e-05 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.717975+0000", 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102326 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.939 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.722770+0000", 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0047949150000000003 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.783145+0000", 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060375243000000002 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.940 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.783182+0000", 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6340000000000001e-05 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:32.783219+0000", 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7255999999999997e-05 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.941 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5041 3.12 3:48dc6f9d:::100000001f8.000005ae:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:33.041338+0000", 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 5.7915444029999996, 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.107591781, 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.942 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5041 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.041338+0000", 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.041338+0000", 2024-09-16T01:52:38.943 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.041339+0000", 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.113e-06 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.065424+0000", 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024084807999999999 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.944 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.065450+0000", 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.5837e-05 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.065463+0000", 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2826000000000001e-05 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.945 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.065546+0000", 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3471000000000001e-05 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.065658+0000", 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111885 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:38.946 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.065810+0000", 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015192 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.069596+0000", 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0037856560000000001 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.148845+0000", 2024-09-16T01:52:38.947 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.079249293999999998 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.148885+0000", 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0132000000000001e-05 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.148930+0000", 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4839000000000002e-05 2024-09-16T01:52:38.948 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5117 3.25 3:a7395892:::100000001f8.000005fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e104)", 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:33.954980+0000", 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.8779025059999999, 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.13598764299999999, 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.949 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5117 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.954980+0000", 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.950 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.954980+0000", 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:33.954982+0000", 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2259999999999999e-06 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.951 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.052684+0000", 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.097701378000000005 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.052699+0000", 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5305999999999999e-05 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.052706+0000", 2024-09-16T01:52:38.952 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9800000000000001e-06 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.052764+0000", 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7623e-05 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.052862+0000", 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7948000000000005e-05 2024-09-16T01:52:38.953 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.053002+0000", 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014026800000000001 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.059832+0000", 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0068296679999999997 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.954 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.090895+0000", 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031063463999999999 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.090932+0000", 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7320999999999997e-05 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.090968+0000", 2024-09-16T01:52:38.955 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5460999999999998e-05 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5125 3.33 3:cd383402:::100000001f8.00000602:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e104)", 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:34.114357+0000", 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 4.7185256600000001, 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.103871668, 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.956 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5125 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.114357+0000", 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.957 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.114357+0000", 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.114358+0000", 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.1900000000000001e-07 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.958 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.192290+0000", 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077931705000000004 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.192303+0000", 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3366e-05 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.192309+0000", 2024-09-16T01:52:38.959 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4749999999999998e-06 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.192387+0000", 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.7474999999999999e-05 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.192455+0000", 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8510999999999995e-05 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.960 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.192552+0000", 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6254000000000004e-05 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.195986+0000", 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003434501 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.961 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.218158+0000", 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022172061 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.218195+0000", 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7094000000000001e-05 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:34.218229+0000", 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.3407e-05 2024-09-16T01:52:38.962 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5352 3.7 3:e3d0cfe5:::100000001f8.000006e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e104)", 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:36.765731+0000", 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 2.067151596, 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.175995715, 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:38.963 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5352 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.765731+0000", 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.964 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.765731+0000", 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.765733+0000", 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4759999999999999e-06 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.965 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.927234+0000", 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.16150102199999999 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.927251+0000", 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7088999999999999e-05 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:38.966 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.927258+0000", 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8199999999999999e-06 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.927312+0000", 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.4029000000000001e-05 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.927381+0000", 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9592999999999996e-05 2024-09-16T01:52:38.967 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.927485+0000", 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000103947 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.931070+0000", 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0035844200000000001 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.968 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.941653+0000", 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.010583787000000001 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.941690+0000", 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.6458999999999998e-05 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:36.941727+0000", 2024-09-16T01:52:38.969 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7073000000000002e-05 2024-09-16T01:52:38.970 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.970 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.970 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.970 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:38.970 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:38.970 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:38.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:38.972 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:38.972 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:38.970+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:39.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.068+0000 7f3a8f200640 1 Processor -- start 2024-09-16T01:52:39.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.068+0000 7f3a8f200640 1 -- start start 2024-09-16T01:52:39.071 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:39.070+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:39.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.070+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a90073600 0x7f3a900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.070+0000 7f3a8f200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a900c7050 0x7f3a90070800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.070+0000 7f3a8f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a90072c10 con 0x7f3a900c6720 2024-09-16T01:52:39.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.070+0000 7f3a8f200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3a90073b50 con 0x7f3a900c7050 2024-09-16T01:52:39.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.070+0000 7f3a8f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3a90074030 con 0x7f3a90073600 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.071+0000 7f3a8d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a90073600 0x7f3a900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.071+0000 7f3a8d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a90073600 0x7f3a900702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42932/0 (socket says 172.21.15.146:42932) 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.071+0000 7f3a8d400640 1 -- 172.21.15.146:0/225731227 learned_addr learned my addr 172.21.15.146:0/225731227 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.071+0000 7f3a8e800640 1 --2- 172.21.15.146:0/225731227 >> v2:172.21.15.161:3300/0 conn(0x7f3a900c7050 0x7f3a90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.072+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744872340 0 0) 0x7f3a90072c10 con 0x7f3a900c6720 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a74003520 con 0x7f3a900c6720 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4265442294 0 0) 0x7f3a74003520 con 0x7f3a900c6720 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a90073600 msgr2=0x7f3a900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:39.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a90073600 0x7f3a900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 >> v2:172.21.15.161:3300/0 conn(0x7f3a900c7050 msgr2=0x7f3a90070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:39.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 --2- 172.21.15.146:0/225731227 >> v2:172.21.15.161:3300/0 conn(0x7f3a900c7050 0x7f3a90070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9015b650 con 0x7f3a900c6720 2024-09-16T01:52:39.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f3a80001670 con 0x7f3a900c6720 2024-09-16T01:52:39.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 >> v1:172.21.15.146:6789/0 conn(0x7f3a900c6720 legacy=0x7f3a901576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a740039b0 0x7f3a74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a740043a0 0x7f3a74004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a74004fa0 0x7f3a740145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90073b50 con 0x7f3a740043a0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90074030 con 0x7f3a74004fa0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.073+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3a740099e0 con 0x7f3a740039b0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8e800640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a74004fa0 0x7f3a740145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8de00640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a740039b0 0x7f3a74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8e800640 1 -- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a740039b0 msgr2=0x7f3a74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8e800640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a740039b0 0x7f3a74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8e800640 1 -- 172.21.15.146:0/225731227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a740043a0 msgr2=0x7f3a74004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8e800640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a740043a0 0x7f3a74004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.074+0000 7f3a8e800640 1 -- 172.21.15.146:0/225731227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9015b650 con 0x7f3a74004fa0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.076+0000 7f3a8e800640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a74004fa0 0x7f3a740145c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800bbd0 tx=0x7f3a74005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=761c1770041ed7c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.077+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8800b820 con 0x7f3a74004fa0 2024-09-16T01:52:39.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.077+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a88033630 con 0x7f3a74004fa0 2024-09-16T01:52:39.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.077+0000 7f3a8ca00640 1 -- 172.21.15.146:0/225731227 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88033a00 con 0x7f3a74004fa0 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.078+0000 7f3a8f200640 1 -- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a74004fa0 msgr2=0x7f3a740145c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.078+0000 7f3a8f200640 1 --2- 172.21.15.146:0/225731227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a74004fa0 0x7f3a740145c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800bbd0 tx=0x7f3a74005830 comp rx=0 tx=0).stop 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.078+0000 7f3a8e800640 1 -- 172.21.15.146:0/225731227 reap_dead start 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.078+0000 7f3a8f200640 1 -- 172.21.15.146:0/225731227 shutdown_connections 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.079+0000 7f3a8f200640 1 -- 172.21.15.146:0/225731227 >> 172.21.15.146:0/225731227 conn(0x7f3a90067950 msgr2=0x7f3a90067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.079+0000 7f3a8f200640 1 -- 172.21.15.146:0/225731227 shutdown_connections 2024-09-16T01:52:39.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.079+0000 7f3a8f200640 1 -- 172.21.15.146:0/225731227 wait complete. 2024-09-16T01:52:39.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.082+0000 7f3a8f200640 1 Processor -- start 2024-09-16T01:52:39.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.082+0000 7f3a8f200640 1 -- start start 2024-09-16T01:52:39.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.083+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a74004fa0 0x7f3a90156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.083+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a9014a980 0x7f3a901ed3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.083+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a9014c480 0x7f3a901efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.083+0000 7f3a8f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90073380 con 0x7f3a74004fa0 2024-09-16T01:52:39.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.083+0000 7f3a8f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a900730a0 con 0x7f3a9014a980 2024-09-16T01:52:39.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.083+0000 7f3a8f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3a90157de0 con 0x7f3a9014c480 2024-09-16T01:52:39.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.084+0000 7f3a8de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a74004fa0 0x7f3a90156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.084+0000 7f3a8e800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a9014c480 0x7f3a901efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.084+0000 7f3a8e800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a9014c480 0x7f3a901efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42944/0 (socket says 172.21.15.146:42944) 2024-09-16T01:52:39.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.084+0000 7f3a8e800640 1 -- 172.21.15.146:0/3760694555 learned_addr learned my addr 172.21.15.146:0/3760694555 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:39.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.084+0000 7f3a8d400640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a9014a980 0x7f3a901ed3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.085+0000 7f3a8de00640 1 -- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a9014c480 msgr2=0x7f3a901efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.085+0000 7f3a8de00640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a9014c480 0x7f3a901efe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.085+0000 7f3a8de00640 1 -- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a9014a980 msgr2=0x7f3a901ed3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.085+0000 7f3a8de00640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a9014a980 0x7f3a901ed3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.085+0000 7f3a8de00640 1 -- 172.21.15.146:0/3760694555 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a901553d0 con 0x7f3a74004fa0 2024-09-16T01:52:39.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.086+0000 7f3a8de00640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a74004fa0 0x7f3a90156150 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000f9b0 tx=0x7f3a8000fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.086+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a800053a0 con 0x7f3a74004fa0 2024-09-16T01:52:39.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.086+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a80018070 con 0x7f3a74004fa0 2024-09-16T01:52:39.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.086+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8000d9d0 con 0x7f3a74004fa0 2024-09-16T01:52:39.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.086+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a90155690 con 0x7f3a74004fa0 2024-09-16T01:52:39.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.087+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a90155c50 con 0x7f3a74004fa0 2024-09-16T01:52:39.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.089+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a900c6b00 con 0x7f3a74004fa0 2024-09-16T01:52:39.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.091+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3a8000db70 con 0x7f3a74004fa0 2024-09-16T01:52:39.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.092+0000 7f3a86000640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3a7006c3b0 0x7f3a7006e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.092+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5741+0+0 (secure 0 0 0) 0x7f3a8008b500 con 0x7f3a74004fa0 2024-09-16T01:52:39.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.093+0000 7f3a8d400640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3a7006c3b0 0x7f3a7006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.094+0000 7f3a8d400640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3a7006c3b0 0x7f3a7006e870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c006cc0 tx=0x7f3a7c006bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.099+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3a8000de60 con 0x7f3a74004fa0 2024-09-16T01:52:39.130 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:39.130 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:39.130 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:39.130 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:39.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:39.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.155+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f2c740cdb40 con 0x7f2c74154ac0 2024-09-16T01:52:39.182 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:39.181+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:39.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.218+0000 7f6187a00640 1 Processor -- start 2024-09-16T01:52:39.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.218+0000 7f6187a00640 1 -- start start 2024-09-16T01:52:39.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6187a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6180151820 0x7f61800c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6187a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61801585b0 con 0x7f6180071a10 2024-09-16T01:52:39.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6187a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61801582d0 con 0x7f61800c7980 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6187a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f61801580a0 con 0x7f6180151820 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6185000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6180151820 0x7f61800c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f617fe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f61800c7980 0x7f61800c8070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36290/0 (socket says 172.21.15.146:36290) 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6185000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6180151820 0x7f61800c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33000/0 (socket says 172.21.15.146:33000) 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.220+0000 7f6185000640 1 -- 172.21.15.146:0/1237435641 learned_addr learned my addr 172.21.15.146:0/1237435641 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.221+0000 7f6185000640 1 -- 172.21.15.146:0/1237435641 >> v1:172.21.15.161:6789/0 conn(0x7f61800c7980 legacy=0x7f61800c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.221+0000 7f6185000640 1 -- 172.21.15.146:0/1237435641 >> v1:172.21.15.146:6789/0 conn(0x7f6180071a10 legacy=0x7f61800c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:39.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.221+0000 7f6185000640 1 -- 172.21.15.146:0/1237435641 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61800c6bc0 con 0x7f6180151820 2024-09-16T01:52:39.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.222+0000 7f6185000640 1 --2- 172.21.15.146:0/1237435641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 0x7f61800c36b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6174009870 tx=0x7f617402a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59a0e2403c49caae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.222+0000 7f617f400640 1 -- 172.21.15.146:0/1237435641 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174037070 con 0x7f6180151820 2024-09-16T01:52:39.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.222+0000 7f617f400640 1 -- 172.21.15.146:0/1237435641 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174032640 con 0x7f6180151820 2024-09-16T01:52:39.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.222+0000 7f617f400640 1 -- 172.21.15.146:0/1237435641 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174032a10 con 0x7f6180151820 2024-09-16T01:52:39.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 -- 172.21.15.146:0/1237435641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 msgr2=0x7f61800c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 --2- 172.21.15.146:0/1237435641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 0x7f61800c36b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6174009870 tx=0x7f617402a330 comp rx=0 tx=0).stop 2024-09-16T01:52:39.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 -- 172.21.15.146:0/1237435641 shutdown_connections 2024-09-16T01:52:39.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 --2- 172.21.15.146:0/1237435641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 0x7f61800c36b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 -- 172.21.15.146:0/1237435641 >> 172.21.15.146:0/1237435641 conn(0x7f618006bc10 msgr2=0x7f618006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:39.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 -- 172.21.15.146:0/1237435641 shutdown_connections 2024-09-16T01:52:39.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.223+0000 7f6187a00640 1 -- 172.21.15.146:0/1237435641 wait complete. 2024-09-16T01:52:39.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.226+0000 7f6187a00640 1 Processor -- start 2024-09-16T01:52:39.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.226+0000 7f6187a00640 1 -- start start 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6187a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 0x7f61800733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6187a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61800c7980 0x7f6180073900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6187a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 0x7f61801f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6187a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6180071e80 con 0x7f61800c7980 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6187a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61800c6680 con 0x7f6180151820 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6187a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f61800c74f0 con 0x7f6180071a10 2024-09-16T01:52:39.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6185000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 0x7f61800733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6185000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 0x7f61800733c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42948/0 (socket says 172.21.15.146:42948) 2024-09-16T01:52:39.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6185000640 1 -- 172.21.15.146:0/3655472058 learned_addr learned my addr 172.21.15.146:0/3655472058 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:39.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 0x7f61801f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.228+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61800c7980 0x7f6180073900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6185000640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 msgr2=0x7f61801f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6180151820 0x7f61801f00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6185000640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61800c7980 msgr2=0x7f6180073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61800c7980 0x7f6180073900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6185000640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61800cb180 con 0x7f6180071a10 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61800c7980 0x7f6180073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 0x7f61800733c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6174009870 tx=0x7f61740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174041020 con 0x7f6180071a10 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174030990 con 0x7f6180071a10 2024-09-16T01:52:39.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174037040 con 0x7f6180071a10 2024-09-16T01:52:39.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.229+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61800cb460 con 0x7f6180071a10 2024-09-16T01:52:39.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.230+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61800cba20 con 0x7f6180071a10 2024-09-16T01:52:39.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.231+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f61800c7180 con 0x7f6180071a10 2024-09-16T01:52:39.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.232+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6174044020 con 0x7f6180071a10 2024-09-16T01:52:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.232+0000 7f617cc00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f616006c2b0 0x7f616006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.233+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 5741+0+0 (secure 0 0 0) 0x7f61740abee0 con 0x7f6180071a10 2024-09-16T01:52:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.233+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f616006c2b0 0x7f616006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.233+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v105) ==== 54+0+16 (secure 0 0 0) 0x7f617404b030 con 0x7f6180071a10 2024-09-16T01:52:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.233+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f61801f05e0 0x7f61800c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.233+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61800c7e30 con 0x7f61801f05e0 2024-09-16T01:52:39.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.234+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f61801f05e0 0x7f61800c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.234+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f616006c2b0 0x7f616006e770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6180147140 tx=0x7f616c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.234+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f61801f05e0 0x7f61800c5110 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.235+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61800c7e30 con 0x7f61801f05e0 2024-09-16T01:52:39.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.251+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61800cc350 con 0x7f61801f05e0 2024-09-16T01:52:39.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.253+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61800cc350 con 0x7f61801f05e0 2024-09-16T01:52:39.255 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.253+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f61801f2640 0x7f61801f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.253+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f61801f2f60 con 0x7f61801f2640 2024-09-16T01:52:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.254+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f61801f2640 0x7f61801f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.254+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f61801f2640 0x7f61801f2a20 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.255+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61801f2f60 con 0x7f61801f2640 2024-09-16T01:52:39.273 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:39.272+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:39.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.274+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6180146ad0 con 0x7f61801f2640 2024-09-16T01:52:39.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.276+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6180146ad0 con 0x7f61801f2640 2024-09-16T01:52:39.278 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.276+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f618013cce0 0x7f618013d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.276+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f618013d600 con 0x7f618013cce0 2024-09-16T01:52:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.277+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f618013cce0 0x7f618013d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.277+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f618013cce0 0x7f618013d0c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.279+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f618013d600 con 0x7f618013cce0 2024-09-16T01:52:39.297 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:39.297 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:39.297 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:39.297 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:39.297 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:39.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.305+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61801f09c0 con 0x7f618013cce0 2024-09-16T01:52:39.308 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:39.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.309+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61801f09c0 con 0x7f618013cce0 2024-09-16T01:52:39.312 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:39.312 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:39.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.309+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6180062f00 0x7f6180143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.310+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6180063360 con 0x7f6180062f00 2024-09-16T01:52:39.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.310+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6180062f00 0x7f6180143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.311+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6180062f00 0x7f6180143fb0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.312+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6180063360 con 0x7f6180062f00 2024-09-16T01:52:39.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.326+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6180063360 con 0x7f6180062f00 2024-09-16T01:52:39.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.328+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6180063360 con 0x7f6180062f00 2024-09-16T01:52:39.330 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:39.331 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:39.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.328+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6180144c50 0x7f618002c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.328+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f61800635c0 con 0x7f6180144c50 2024-09-16T01:52:39.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.329+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6180144c50 0x7f618002c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.330+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6180144c50 0x7f618002c640 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.331+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61800635c0 con 0x7f6180144c50 2024-09-16T01:52:39.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.347+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61800635c0 con 0x7f6180144c50 2024-09-16T01:52:39.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.349+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61800635c0 con 0x7f6180144c50 2024-09-16T01:52:39.352 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:39.352 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:39.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.350+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f618002d090 0x7f618002d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.350+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f618002da50 con 0x7f618002d090 2024-09-16T01:52:39.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.350+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f618002d090 0x7f618002d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.351+0000 7f617fe00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f618002d090 0x7f618002d510 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.352+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f618002da50 con 0x7f618002d090 2024-09-16T01:52:39.374 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:39.373+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:39.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.379+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f618002da50 con 0x7f618002d090 2024-09-16T01:52:39.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.381+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f618002da50 con 0x7f618002d090 2024-09-16T01:52:39.383 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:39.384 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:39.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.381+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6180205be0 0x7f6180207fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.381+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f618020ada0 con 0x7f6180205be0 2024-09-16T01:52:39.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.381+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6180205be0 0x7f6180207fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.382+0000 7f6185a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6180205be0 0x7f6180207fd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.383+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f618020ada0 con 0x7f6180205be0 2024-09-16T01:52:39.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.403+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6180208510 con 0x7f6180205be0 2024-09-16T01:52:39.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.405+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6180208510 con 0x7f6180205be0 2024-09-16T01:52:39.407 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:39.408 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:39.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.405+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6180208b90 0x7f618020af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:39.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.405+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f618020de00 con 0x7f6180208b90 2024-09-16T01:52:39.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.406+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6180208b90 0x7f618020af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:39.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.406+0000 7f6185000640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6180208b90 0x7f618020af80 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:39.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.407+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f618020de00 con 0x7f6180208b90 2024-09-16T01:52:39.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.425+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f618020b790 con 0x7f6180208b90 2024-09-16T01:52:39.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.427+0000 7f617cc00640 1 -- 172.21.15.146:0/3655472058 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f618020b790 con 0x7f6180208b90 2024-09-16T01:52:39.430 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:39.430 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:39.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f61801f05e0 msgr2=0x7f61800c5110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f61801f05e0 0x7f61800c5110 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f61801f2640 msgr2=0x7f61801f2a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f61801f2640 0x7f61801f2a20 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f618013cce0 msgr2=0x7f618013d0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f618013cce0 0x7f618013d0c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6180062f00 msgr2=0x7f6180143fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f6180062f00 0x7f6180143fb0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6180144c50 msgr2=0x7f618002c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f6180144c50 0x7f618002c640 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f618002d090 msgr2=0x7f618002d510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f618002d090 0x7f618002d510 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6180205be0 msgr2=0x7f6180207fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f6180205be0 0x7f6180207fd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6180208b90 msgr2=0x7f618020af80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.428+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f6180208b90 0x7f618020af80 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6185a00640 1 -- 172.21.15.146:0/3655472058 reap_dead start 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6185a00640 1 -- 172.21.15.146:0/3655472058 reap_dead start 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6185a00640 1 -- 172.21.15.146:0/3655472058 reap_dead start 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6185a00640 1 -- 172.21.15.146:0/3655472058 reap_dead start 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6185a00640 1 -- 172.21.15.146:0/3655472058 reap_dead start 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f616006c2b0 msgr2=0x7f616006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f616006c2b0 0x7f616006e770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6180147140 tx=0x7f616c009040 comp rx=0 tx=0).stop 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 msgr2=0x7f61800733c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.429+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 0x7f61800733c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6174009870 tx=0x7f61740046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.430+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 shutdown_connections 2024-09-16T01:52:39.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.430+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f616006c2b0 0x7f616006e770 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.430+0000 7f6187a00640 1 --2- 172.21.15.146:0/3655472058 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6180071a10 0x7f61800733c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.430+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 >> 172.21.15.146:0/3655472058 conn(0x7f618006bc10 msgr2=0x7f6180154640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:39.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.430+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 shutdown_connections 2024-09-16T01:52:39.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.430+0000 7f6187a00640 1 -- 172.21.15.146:0/3655472058 wait complete. 2024-09-16T01:52:39.435 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:39.435 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:39.443 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:39.497 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:39.495+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:39.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.491+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f3a900c07d0 con 0x7f3a74004fa0 2024-09-16T01:52:39.577 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:39.575+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:39.678 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:39.676+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:39.680 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:39.680 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.689+0000 7f3a86000640 1 -- 172.21.15.146:0/3760694555 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v106) ==== 101+0+0 (secure 0 0 0) 0x7f3a80059550 con 0x7f3a74004fa0 2024-09-16T01:52:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:52:39.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.693+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3a7006c3b0 msgr2=0x7f3a7006e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.693+0000 7f3a8f200640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3a7006c3b0 0x7f3a7006e870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c006cc0 tx=0x7f3a7c006bd0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.693+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a74004fa0 msgr2=0x7f3a90156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.693+0000 7f3a8f200640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a74004fa0 0x7f3a90156150 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000f9b0 tx=0x7f3a8000fe70 comp rx=0 tx=0).stop 2024-09-16T01:52:39.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f2c6be00640 1 -- 172.21.15.146:0/4206870363 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v106) ==== 113+0+0 (secure 0 0 0) 0x7f2c6007a690 con 0x7f2c74154ac0 2024-09-16T01:52:39.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 shutdown_connections 2024-09-16T01:52:39.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3a7006c3b0 0x7f3a7006e870 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a9014c480 0x7f3a901efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a9014a980 0x7f3a901ed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 --2- 172.21.15.146:0/3760694555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a74004fa0 0x7f3a90156150 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 >> 172.21.15.146:0/3760694555 conn(0x7f3a90067950 msgr2=0x7f3a90074ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:39.699 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T01:52:39.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 shutdown_connections 2024-09-16T01:52:39.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.694+0000 7f3a8f200640 1 -- 172.21.15.146:0/3760694555 wait complete. 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.700+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2c5406c300 msgr2=0x7f2c5406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.700+0000 7f2c79a00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2c5406c300 0x7f2c5406e7c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2c64001680 tx=0x7f2c64009040 comp rx=0 tx=0).stop 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.700+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 msgr2=0x7f2c740cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.700+0000 7f2c79a00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 0x7f2c740cb820 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2c600098a0 tx=0x7f2c600046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.702+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 shutdown_connections 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.702+0000 7f2c79a00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2c5406c300 0x7f2c5406e7c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.702+0000 7f2c79a00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7415b190 0x7f2c740cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.702+0000 7f2c79a00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2c74154ac0 0x7f2c740cb820 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.702+0000 7f2c79a00640 1 --2- 172.21.15.146:0/4206870363 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2c740d0d00 0x7f2c740cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:39.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.702+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 >> 172.21.15.146:0/4206870363 conn(0x7f2c7406e400 msgr2=0x7f2c74151520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:39.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.706+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 shutdown_connections 2024-09-16T01:52:39.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:39.707+0000 7f2c79a00640 1 -- 172.21.15.146:0/4206870363 wait complete. 2024-09-16T01:52:39.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:39.745 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T01:52:39.778 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:39.777+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:39.880 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:39.878+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:39.971 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:39.971 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:39.980 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:39.979+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:39.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:40.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.055+0000 7ffb78a00640 1 Processor -- start 2024-09-16T01:52:40.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.055+0000 7ffb78a00640 1 -- start start 2024-09-16T01:52:40.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.056+0000 7ffb78a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce5f0 0x7ffb7415ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:40.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb78a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb740c9a20 con 0x7ffb740ce9d0 2024-09-16T01:52:40.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb78a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb7406cf70 con 0x7ffb7406eaa0 2024-09-16T01:52:40.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb78a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7415ef70 con 0x7ffb740ce5f0 2024-09-16T01:52:40.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb71e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce5f0 0x7ffb7415ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:40.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb72800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ffb7406eaa0 0x7ffb74157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36300/0 (socket says 172.21.15.146:36300) 2024-09-16T01:52:40.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb71400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffb740ce9d0 0x7ffb7415b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56490/0 (socket says 172.21.15.146:56490) 2024-09-16T01:52:40.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb72800640 1 -- 172.21.15.146:0/3060030026 learned_addr learned my addr 172.21.15.146:0/3060030026 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:40.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb70a00640 1 -- 172.21.15.146:0/3060030026 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193519737 0 0) 0x7ffb7406cf70 con 0x7ffb7406eaa0 2024-09-16T01:52:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb70a00640 1 -- 172.21.15.146:0/3060030026 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb58003520 con 0x7ffb7406eaa0 2024-09-16T01:52:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb71e00640 1 -- 172.21.15.146:0/3060030026 >> v1:172.21.15.161:6789/0 conn(0x7ffb7406eaa0 legacy=0x7ffb74157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb71e00640 1 -- 172.21.15.146:0/3060030026 >> v1:172.21.15.146:6789/0 conn(0x7ffb740ce9d0 legacy=0x7ffb7415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.057+0000 7ffb71e00640 1 -- 172.21.15.146:0/3060030026 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7406d250 con 0x7ffb740ce5f0 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.059+0000 7ffb71e00640 1 --2- 172.21.15.146:0/3060030026 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce5f0 0x7ffb7415ea30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffb640097d0 tx=0x7ffb6402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab029d53c5776f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.059+0000 7ffb70a00640 1 -- 172.21.15.146:0/3060030026 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb64037070 con 0x7ffb740ce5f0 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.059+0000 7ffb70a00640 1 -- 172.21.15.146:0/3060030026 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb64032440 con 0x7ffb740ce5f0 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.059+0000 7ffb70a00640 1 -- 172.21.15.146:0/3060030026 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb64032810 con 0x7ffb740ce5f0 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.060+0000 7ffb78a00640 1 -- 172.21.15.146:0/3060030026 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce5f0 msgr2=0x7ffb7415ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.060+0000 7ffb78a00640 1 --2- 172.21.15.146:0/3060030026 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce5f0 0x7ffb7415ea30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffb640097d0 tx=0x7ffb6402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.061+0000 7ffb78a00640 1 -- 172.21.15.146:0/3060030026 shutdown_connections 2024-09-16T01:52:40.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.061+0000 7ffb78a00640 1 --2- 172.21.15.146:0/3060030026 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce5f0 0x7ffb7415ea30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.061+0000 7ffb78a00640 1 -- 172.21.15.146:0/3060030026 >> 172.21.15.146:0/3060030026 conn(0x7ffb7406c240 msgr2=0x7ffb74067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.061+0000 7ffb78a00640 1 -- 172.21.15.146:0/3060030026 shutdown_connections 2024-09-16T01:52:40.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.061+0000 7ffb78a00640 1 -- 172.21.15.146:0/3060030026 wait complete. 2024-09-16T01:52:40.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.062+0000 7ffb78a00640 1 Processor -- start 2024-09-16T01:52:40.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.063+0000 7ffb78a00640 1 -- start start 2024-09-16T01:52:40.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb78a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7406eaa0 0x7ffb740c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb78a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 0x7ffb740ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb78a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce9d0 0x7ffb741f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb78a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7415b8f0 con 0x7ffb7406eaa0 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb78a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7406cd20 con 0x7ffb740ce5f0 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb78a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7415ba70 con 0x7ffb740ce9d0 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7406eaa0 0x7ffb740c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 0x7ffb740ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 0x7ffb740ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33024/0 (socket says 172.21.15.146:33024) 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7406eaa0 0x7ffb740c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40010/0 (socket says 172.21.15.146:40010) 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 -- 172.21.15.146:0/1358327970 learned_addr learned my addr 172.21.15.146:0/1358327970 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:40.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb72800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce9d0 0x7ffb741f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 -- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce9d0 msgr2=0x7ffb741f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce9d0 0x7ffb741f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 -- 172.21.15.146:0/1358327970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7406eaa0 msgr2=0x7ffb740c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7406eaa0 0x7ffb740c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.064+0000 7ffb71400640 1 -- 172.21.15.146:0/1358327970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb740afdb0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.065+0000 7ffb71400640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 0x7ffb740ccc00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c390 tx=0x7ffb6000c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:40.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.065+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600074e0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.065+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb600047e0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.065+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb740b0090 con 0x7ffb740ce5f0 2024-09-16T01:52:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.065+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60004bd0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.065+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb740b0650 con 0x7ffb740ce5f0 2024-09-16T01:52:40.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.067+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb740c94e0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.069+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ffb6001a020 con 0x7ffb740ce5f0 2024-09-16T01:52:40.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.070+0000 7ffb6a000640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffb5406c2e0 0x7ffb5406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:40.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.070+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5741+0+0 (secure 0 0 0) 0x7ffb6008abc0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.070+0000 7ffb71e00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffb5406c2e0 0x7ffb5406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:40.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.071+0000 7ffb71e00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffb5406c2e0 0x7ffb5406e7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ffb64034000 tx=0x7ffb6402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:40.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.072+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ffb60058ca0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.082 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:40.080+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:40.183 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:40.181+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:40.271 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:40.272 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:40.283 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:40.281+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:40.291 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:40.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.381+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7ffb740ca0e0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.384 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:40.383+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:40.485 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:40.483+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:40.586 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:40.584+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:40.619 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:40.620 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:40.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:40.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.694+0000 7ffb6a000640 1 -- 172.21.15.146:0/1358327970 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v107) ==== 113+0+0 (secure 0 0 0) 0x7ffb6005cfa0 con 0x7ffb740ce5f0 2024-09-16T01:52:40.696 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T01:52:40.699 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:40.697+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:40.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.699+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffb5406c2e0 msgr2=0x7ffb5406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.699+0000 7ffb78a00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffb5406c2e0 0x7ffb5406e7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ffb64034000 tx=0x7ffb6402e000 comp rx=0 tx=0).stop 2024-09-16T01:52:40.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.699+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 msgr2=0x7ffb740ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.699+0000 7ffb78a00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 0x7ffb740ccc00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c390 tx=0x7ffb6000c790 comp rx=0 tx=0).stop 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 shutdown_connections 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ffb5406c2e0 0x7ffb5406e7a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb740ce9d0 0x7ffb741f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb740ce5f0 0x7ffb740ccc00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 --2- 172.21.15.146:0/1358327970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb7406eaa0 0x7ffb740c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 >> 172.21.15.146:0/1358327970 conn(0x7ffb7406c240 msgr2=0x7ffb7415b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:40.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 shutdown_connections 2024-09-16T01:52:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:40.702+0000 7ffb78a00640 1 -- 172.21.15.146:0/1358327970 wait complete. 2024-09-16T01:52:40.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T01:52:40.787 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:40.786+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:40.888 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:40.887+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:40.989 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:40.987+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:41.003 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:41.003 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:41.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:41.089 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:41.088+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:41.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.110+0000 7f4bc2200640 1 Processor -- start 2024-09-16T01:52:41.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.110+0000 7f4bc2200640 1 -- start start 2024-09-16T01:52:41.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.112+0000 7f4bc2200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc158fe0 0x7f4bbc0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.112+0000 7f4bc2200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4bbc15eec0 0x7f4bbc15f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.112+0000 7f4bc2200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bbc0725b0 con 0x7f4bbc071d20 2024-09-16T01:52:41.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.112+0000 7f4bc2200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4bbc072890 con 0x7f4bbc15eec0 2024-09-16T01:52:41.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.112+0000 7f4bc2200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4bbc0c8110 con 0x7f4bbc158fe0 2024-09-16T01:52:41.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.113+0000 7f4bbaa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4bbc071d20 0x7f4bbc06f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56504/0 (socket says 172.21.15.146:56504) 2024-09-16T01:52:41.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.113+0000 7f4bbaa00640 1 -- 172.21.15.146:0/1639884594 learned_addr learned my addr 172.21.15.146:0/1639884594 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:41.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.113+0000 7f4bbbe00640 1 --2- 172.21.15.146:0/1639884594 >> v2:172.21.15.146:3300/0 conn(0x7f4bbc15eec0 0x7f4bbc15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.114+0000 7f4bbbe00640 1 -- 172.21.15.146:0/1639884594 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc158fe0 msgr2=0x7f4bbc0c96a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:41.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.114+0000 7f4bbbe00640 1 --2- 172.21.15.146:0/1639884594 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc158fe0 0x7f4bbc0c96a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.114+0000 7f4bbbe00640 1 -- 172.21.15.146:0/1639884594 >> v1:172.21.15.146:6789/0 conn(0x7f4bbc071d20 legacy=0x7f4bbc06f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.114+0000 7f4bbbe00640 1 -- 172.21.15.146:0/1639884594 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bbc0c77e0 con 0x7f4bbc15eec0 2024-09-16T01:52:41.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.119+0000 7f4bbbe00640 1 --2- 172.21.15.146:0/1639884594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc15eec0 0x7f4bbc15f370 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009d90 tx=0x7f4bb402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf818697c3541547 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.119+0000 7f4bba000640 1 -- 172.21.15.146:0/1639884594 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4007740 con 0x7f4bbc15eec0 2024-09-16T01:52:41.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.120+0000 7f4bba000640 1 -- 172.21.15.146:0/1639884594 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bb40078e0 con 0x7f4bbc15eec0 2024-09-16T01:52:41.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.120+0000 7f4bba000640 1 -- 172.21.15.146:0/1639884594 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4007cb0 con 0x7f4bbc15eec0 2024-09-16T01:52:41.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.122+0000 7f4bc2200640 1 -- 172.21.15.146:0/1639884594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc15eec0 msgr2=0x7f4bbc15f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.122+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1639884594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc15eec0 0x7f4bbc15f370 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009d90 tx=0x7f4bb402a680 comp rx=0 tx=0).stop 2024-09-16T01:52:41.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.123+0000 7f4bc2200640 1 -- 172.21.15.146:0/1639884594 shutdown_connections 2024-09-16T01:52:41.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.123+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1639884594 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc15eec0 0x7f4bbc15f370 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.123+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1639884594 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc158fe0 0x7f4bbc0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.123+0000 7f4bc2200640 1 -- 172.21.15.146:0/1639884594 >> 172.21.15.146:0/1639884594 conn(0x7f4bbc06e670 msgr2=0x7f4bbc06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:41.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.124+0000 7f4bc2200640 1 -- 172.21.15.146:0/1639884594 shutdown_connections 2024-09-16T01:52:41.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.124+0000 7f4bc2200640 1 -- 172.21.15.146:0/1639884594 wait complete. 2024-09-16T01:52:41.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.126+0000 7f4bc2200640 1 Processor -- start 2024-09-16T01:52:41.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.133+0000 7f4bc2200640 1 -- start start 2024-09-16T01:52:41.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bc2200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bbc071d20 0x7f4bbc1f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bc2200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc158fe0 0x7f4bbc1f8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bc2200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc15eec0 0x7f4bbc15afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bc2200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc0c7da0 con 0x7f4bbc158fe0 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bc2200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4bbc15f8b0 con 0x7f4bbc071d20 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bc2200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4bbc0c9be0 con 0x7f4bbc15eec0 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.137+0000 7f4bbbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc15eec0 0x7f4bbc15afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.138+0000 7f4bbbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc15eec0 0x7f4bbc15afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42980/0 (socket says 172.21.15.146:42980) 2024-09-16T01:52:41.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.138+0000 7f4bbbe00640 1 -- 172.21.15.146:0/1075525543 learned_addr learned my addr 172.21.15.146:0/1075525543 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:41.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.138+0000 7f4bbb400640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bbc071d20 0x7f4bbc1f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.138+0000 7f4bbb400640 1 -- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc15eec0 msgr2=0x7f4bbc15afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.138+0000 7f4bbb400640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc15eec0 0x7f4bbc15afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.138+0000 7f4bbb400640 1 -- 172.21.15.146:0/1075525543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc158fe0 msgr2=0x7f4bbc1f8270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:52:41.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.139+0000 7f4bbaa00640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc158fe0 0x7f4bbc1f8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.140+0000 7f4bbb400640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc158fe0 0x7f4bbc1f8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.140+0000 7f4bbb400640 1 -- 172.21.15.146:0/1075525543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bbc15a830 con 0x7f4bbc071d20 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.140+0000 7f4bbaa00640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc158fe0 0x7f4bbc1f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.140+0000 7f4bbb400640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bbc071d20 0x7f4bbc1f7d30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00b410 tx=0x7f4bac00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.141+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac015020 con 0x7f4bbc071d20 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.141+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bac011830 con 0x7f4bbc071d20 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.141+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac011c40 con 0x7f4bbc071d20 2024-09-16T01:52:41.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.142+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bbc1f8900 con 0x7f4bbc071d20 2024-09-16T01:52:41.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.142+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bbc162b40 con 0x7f4bbc071d20 2024-09-16T01:52:41.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.147+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bbc1593c0 con 0x7f4bbc071d20 2024-09-16T01:52:41.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.150+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4bac018020 con 0x7f4bbc071d20 2024-09-16T01:52:41.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.151+0000 7f4bb3400640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4b9c06c300 0x7f4b9c06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.151+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5919+0+0 (secure 0 0 0) 0x7f4bac08b1b0 con 0x7f4bbc071d20 2024-09-16T01:52:41.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.152+0000 7f4bbaa00640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4b9c06c300 0x7f4b9c06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.153+0000 7f4bbaa00640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4b9c06c300 0x7f4b9c06e7c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc15ee70 tx=0x7f4ba8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.158+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4bac01f020 con 0x7f4bbc071d20 2024-09-16T01:52:41.190 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:41.189+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:41.291 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:41.290+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:41.391 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:41.391 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:41.391 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:41.391+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:41.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:41.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:41.492 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:41.491+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:41.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.578+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f4bbc155f20 con 0x7f4bbc071d20 2024-09-16T01:52:41.594 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:41.592+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:41.694 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:41.693+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:41.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.710+0000 7f4bb3400640 1 -- 172.21.15.146:0/1075525543 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v108) ==== 113+0+0 (secure 0 0 0) 0x7f4bac0172c0 con 0x7f4bbc071d20 2024-09-16T01:52:41.715 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T01:52:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.716+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4b9c06c300 msgr2=0x7f4b9c06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.716+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4b9c06c300 0x7f4b9c06e7c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc15ee70 tx=0x7f4ba8009040 comp rx=0 tx=0).stop 2024-09-16T01:52:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.716+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bbc071d20 msgr2=0x7f4bbc1f7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.716+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bbc071d20 0x7f4bbc1f7d30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00b410 tx=0x7f4bac00b8d0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 shutdown_connections 2024-09-16T01:52:41.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4b9c06c300 0x7f4b9c06e7c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4bbc15eec0 0x7f4bbc15afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bbc158fe0 0x7f4bbc1f8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 --2- 172.21.15.146:0/1075525543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bbc071d20 0x7f4bbc1f7d30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 >> 172.21.15.146:0/1075525543 conn(0x7f4bbc06e670 msgr2=0x7f4bbc154140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:41.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 shutdown_connections 2024-09-16T01:52:41.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.720+0000 7f4bc2200640 1 -- 172.21.15.146:0/1075525543 wait complete. 2024-09-16T01:52:41.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:52:41.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.758+0000 7fa591000640 1 Processor -- start 2024-09-16T01:52:41.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.758+0000 7fa591000640 1 -- start start 2024-09-16T01:52:41.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T01:52:41.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.760+0000 7fa591000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 0x7fa58c15b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.760+0000 7fa591000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa58c15e0b0 con 0x7fa58c0cafa0 2024-09-16T01:52:41.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.760+0000 7fa591000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa58c15e390 con 0x7fa58c0cca70 2024-09-16T01:52:41.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.760+0000 7fa591000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa58c15de60 con 0x7fa58c154c10 2024-09-16T01:52:41.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.761+0000 7fa589e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 0x7fa58c15b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.761+0000 7fa589e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 0x7fa58c15b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:42996/0 (socket says 172.21.15.146:42996) 2024-09-16T01:52:41.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.761+0000 7fa589e00640 1 -- 172.21.15.146:0/3689344328 learned_addr learned my addr 172.21.15.146:0/3689344328 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:41.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.762+0000 7fa589e00640 1 -- 172.21.15.146:0/3689344328 >> v1:172.21.15.161:6789/0 conn(0x7fa58c0cca70 legacy=0x7fa58c073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.762+0000 7fa589e00640 1 -- 172.21.15.146:0/3689344328 >> v1:172.21.15.146:6789/0 conn(0x7fa58c0cafa0 legacy=0x7fa58c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.762+0000 7fa589e00640 1 -- 172.21.15.146:0/3689344328 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa58c15e670 con 0x7fa58c154c10 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.765+0000 7fa589e00640 1 --2- 172.21.15.146:0/3689344328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 0x7fa58c15b370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009820 tx=0x7fa57c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d89854490f2d26c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.765+0000 7fa589400640 1 -- 172.21.15.146:0/3689344328 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c037070 con 0x7fa58c154c10 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.765+0000 7fa589400640 1 -- 172.21.15.146:0/3689344328 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa57c032440 con 0x7fa58c154c10 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.765+0000 7fa589400640 1 -- 172.21.15.146:0/3689344328 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57c032810 con 0x7fa58c154c10 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.766+0000 7fa591000640 1 -- 172.21.15.146:0/3689344328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 msgr2=0x7fa58c15b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.767+0000 7fa591000640 1 --2- 172.21.15.146:0/3689344328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 0x7fa58c15b370 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009820 tx=0x7fa57c02a180 comp rx=0 tx=0).stop 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.768+0000 7fa591000640 1 -- 172.21.15.146:0/3689344328 shutdown_connections 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.768+0000 7fa591000640 1 --2- 172.21.15.146:0/3689344328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c154c10 0x7fa58c15b370 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.768+0000 7fa591000640 1 -- 172.21.15.146:0/3689344328 >> 172.21.15.146:0/3689344328 conn(0x7fa58c06ce80 msgr2=0x7fa58c06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:41.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.769+0000 7fa591000640 1 -- 172.21.15.146:0/3689344328 shutdown_connections 2024-09-16T01:52:41.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.769+0000 7fa591000640 1 -- 172.21.15.146:0/3689344328 wait complete. 2024-09-16T01:52:41.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.772+0000 7fa591000640 1 Processor -- start 2024-09-16T01:52:41.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.772+0000 7fa591000640 1 -- start start 2024-09-16T01:52:41.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.774+0000 7fa591000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c0cafa0 0x7fa58c157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.774+0000 7fa591000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa58c0cca70 0x7fa58c14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.774+0000 7fa591000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa58c154c10 0x7fa58c14df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.775+0000 7fa591000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa58c154ff0 con 0x7fa58c154c10 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.775+0000 7fa591000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa58c15ec30 con 0x7fa58c0cca70 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.775+0000 7fa591000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa58c073a50 con 0x7fa58c0cafa0 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.777+0000 7fa589e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c0cafa0 0x7fa58c157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.777+0000 7fa581400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa58c0cca70 0x7fa58c14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.777+0000 7fa581400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa58c0cca70 0x7fa58c14d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33036/0 (socket says 172.21.15.146:33036) 2024-09-16T01:52:41.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.777+0000 7fa581400640 1 -- 172.21.15.146:0/816134810 learned_addr learned my addr 172.21.15.146:0/816134810 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:41.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.777+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa58c154c10 0x7fa58c14df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.778+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c0cafa0 msgr2=0x7fa58c157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.778+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa58c0cafa0 0x7fa58c157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.779+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa58c0cca70 msgr2=0x7fa58c14d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:41.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.779+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa58c0cca70 0x7fa58c14d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:41.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.779+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa58c0cfe10 con 0x7fa58c154c10 2024-09-16T01:52:41.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.779+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa58c154c10 0x7fa58c14df00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c760 tx=0x7fa58400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.780+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584016020 con 0x7fa58c154c10 2024-09-16T01:52:41.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.780+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa584006d50 con 0x7fa58c154c10 2024-09-16T01:52:41.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.780+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584003b70 con 0x7fa58c154c10 2024-09-16T01:52:41.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.781+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa58c0d00f0 con 0x7fa58c154c10 2024-09-16T01:52:41.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.781+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa58c1f4720 con 0x7fa58c154c10 2024-09-16T01:52:41.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.784+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa58c15b370 con 0x7fa58c154c10 2024-09-16T01:52:41.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.788+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa584003d10 con 0x7fa58c154c10 2024-09-16T01:52:41.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.790+0000 7fa582a00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa56806c330 0x7fa56806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.790+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6023+0+0 (secure 0 0 0) 0x7fa58408bf20 con 0x7fa58c154c10 2024-09-16T01:52:41.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.790+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v108) ==== 54+0+16 (secure 0 0 0) 0x7fa584020020 con 0x7fa58c154c10 2024-09-16T01:52:41.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.792+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa58c158030 0x7fa58c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.793+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa58c156fb0 con 0x7fa58c158030 2024-09-16T01:52:41.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.794+0000 7fa581400640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa58c158030 0x7fa58c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.795+0000 7fa581400640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa58c158030 0x7fa58c0b5040 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.805 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:41.799+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:41.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.811+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c156fb0 con 0x7fa58c158030 2024-09-16T01:52:41.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.811+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa56806c330 0x7fa56806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.830 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:41.830 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:41.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.827+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa56806c330 0x7fa56806e7f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa57c034000 tx=0x7fa57c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:41.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.877+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c15dc20 con 0x7fa58c158030 2024-09-16T01:52:41.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.881+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c15dc20 con 0x7fa58c158030 2024-09-16T01:52:41.896 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:41.894+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:41.899 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:41.900 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:41.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.883+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa58c155520 0x7fa58c155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.883+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa58c1521d0 con 0x7fa58c155520 2024-09-16T01:52:41.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.883+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa58c155520 0x7fa58c155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.885+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa58c155520 0x7fa58c155900 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.887+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c1521d0 con 0x7fa58c155520 2024-09-16T01:52:41.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.915+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c0736c0 con 0x7fa58c155520 2024-09-16T01:52:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.928+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c0736c0 con 0x7fa58c155520 2024-09-16T01:52:41.932 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.930+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa58c15bf60 0x7fa58c15c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.930+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa58c0ce3a0 con 0x7fa58c15bf60 2024-09-16T01:52:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.931+0000 7fa581400640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa58c15bf60 0x7fa58c15c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.932+0000 7fa581400640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa58c15bf60 0x7fa58c15c3c0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.937+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c0ce3a0 con 0x7fa58c15bf60 2024-09-16T01:52:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.962+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c0ce7a0 con 0x7fa58c15bf60 2024-09-16T01:52:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.971+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c0ce7a0 con 0x7fa58c15bf60 2024-09-16T01:52:41.973 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.971+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa58c062630 0x7fa58c062ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.971+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa58c0cf280 con 0x7fa58c062630 2024-09-16T01:52:41.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.982+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa58c062630 0x7fa58c062ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:41.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.990+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa58c062630 0x7fa58c062ab0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:41.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:41.993+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c0cf280 con 0x7fa58c062630 2024-09-16T01:52:42.007 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:42.005+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:42.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.019+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c063240 con 0x7fa58c062630 2024-09-16T01:52:42.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.023+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c063240 con 0x7fa58c062630 2024-09-16T01:52:42.028 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.026+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa58c02cb50 0x7fa58c02cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.026+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa58c063160 con 0x7fa58c02cb50 2024-09-16T01:52:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.027+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa58c02cb50 0x7fa58c02cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.032+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa58c02cb50 0x7fa58c02cfb0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.033+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c063160 con 0x7fa58c02cb50 2024-09-16T01:52:42.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.063+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c063450 con 0x7fa58c02cb50 2024-09-16T01:52:42.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.064+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c063450 con 0x7fa58c02cb50 2024-09-16T01:52:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:42.071 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:42.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.069+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa58c204c70 0x7fa58c207060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.069+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa58c063520 con 0x7fa58c204c70 2024-09-16T01:52:42.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.069+0000 7fa581400640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa58c204c70 0x7fa58c207060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.070+0000 7fa581400640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa58c204c70 0x7fa58c207060 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.071+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c063520 con 0x7fa58c204c70 2024-09-16T01:52:42.098 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:42.095+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:42.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.098+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c063520 con 0x7fa58c204c70 2024-09-16T01:52:42.103 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:42.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.100+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c063520 con 0x7fa58c204c70 2024-09-16T01:52:42.104 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:42.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.101+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa58c207ab0 0x7fa58c209ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.101+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa58c02db30 con 0x7fa58c207ab0 2024-09-16T01:52:42.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.101+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa58c207ab0 0x7fa58c209ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.102+0000 7fa58a800640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa58c207ab0 0x7fa58c209ea0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.103+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c02db30 con 0x7fa58c207ab0 2024-09-16T01:52:42.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.108+0000 7f734c400640 1 Processor -- start 2024-09-16T01:52:42.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.108+0000 7f734c400640 1 -- start start 2024-09-16T01:52:42.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.110+0000 7f734c400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f733c0d8830 0x7f733c049910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.110+0000 7f734c400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f733c0dee00 0x7f733c049e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.110+0000 7f734c400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f733c050810 con 0x7f733c050290 2024-09-16T01:52:42.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.110+0000 7f734c400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f733c0e1c20 con 0x7f733c0d8830 2024-09-16T01:52:42.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.110+0000 7f734c400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f733c0e1f00 con 0x7f733c0dee00 2024-09-16T01:52:42.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f733c0dee00 0x7f733c049e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f733c0dee00 0x7f733c049e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33046/0 (socket says 172.21.15.146:33046) 2024-09-16T01:52:42.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 -- 172.21.15.146:0/3758381251 learned_addr learned my addr 172.21.15.146:0/3758381251 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:42.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 -- 172.21.15.146:0/3758381251 >> v2:172.21.15.146:3300/0 conn(0x7f733c0d8830 msgr2=0x7f733c049910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:42.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 --2- 172.21.15.146:0/3758381251 >> v2:172.21.15.146:3300/0 conn(0x7f733c0d8830 0x7f733c049910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 -- 172.21.15.146:0/3758381251 >> v1:172.21.15.161:6789/0 conn(0x7f733c050290 legacy=0x7f733c0de6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:42.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.112+0000 7f734a400640 1 -- 172.21.15.146:0/3758381251 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c0e21e0 con 0x7f733c0dee00 2024-09-16T01:52:42.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.113+0000 7f734a400640 1 --2- 172.21.15.146:0/3758381251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0dee00 0x7f733c049e50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f734406b460 tx=0x7f73440a6ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d031cf1eab65e05c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.113+0000 7f7343e00640 1 -- 172.21.15.146:0/3758381251 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73440a8980 con 0x7f733c0dee00 2024-09-16T01:52:42.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.113+0000 7f7343e00640 1 -- 172.21.15.146:0/3758381251 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73440a8b20 con 0x7f733c0dee00 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.113+0000 7f7343e00640 1 -- 172.21.15.146:0/3758381251 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73440b0420 con 0x7f733c0dee00 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.114+0000 7f734c400640 1 -- 172.21.15.146:0/3758381251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0dee00 msgr2=0x7f733c049e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.114+0000 7f734c400640 1 --2- 172.21.15.146:0/3758381251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0dee00 0x7f733c049e50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f734406b460 tx=0x7f73440a6ba0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.115+0000 7f734c400640 1 -- 172.21.15.146:0/3758381251 shutdown_connections 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.115+0000 7f734c400640 1 --2- 172.21.15.146:0/3758381251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0dee00 0x7f733c049e50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.115+0000 7f734c400640 1 --2- 172.21.15.146:0/3758381251 >> v2:172.21.15.146:3300/0 conn(0x7f733c0d8830 0x7f733c049910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.116+0000 7fc632600640 1 Processor -- start 2024-09-16T01:52:42.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.115+0000 7f734c400640 1 -- 172.21.15.146:0/3758381251 >> 172.21.15.146:0/3758381251 conn(0x7f733c054410 msgr2=0x7f733c054820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:42.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.116+0000 7f734c400640 1 -- 172.21.15.146:0/3758381251 shutdown_connections 2024-09-16T01:52:42.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.116+0000 7f734c400640 1 -- 172.21.15.146:0/3758381251 wait complete. 2024-09-16T01:52:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.116+0000 7fc632600640 1 -- start start 2024-09-16T01:52:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.118+0000 7fc632600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6240091b0 0x7fc62404e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.118+0000 7fc632600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc6240e6b50 0x7fc624051df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.118+0000 7fc632600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc62404df00 con 0x7fc6240dccc0 2024-09-16T01:52:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.119+0000 7fc632600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc6240e6fa0 con 0x7fc6240091b0 2024-09-16T01:52:42.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.119+0000 7fc632600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc6240e60b0 con 0x7fc6240e6b50 2024-09-16T01:52:42.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.119+0000 7fc631200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc6240dccc0 0x7fc62404cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:56530/0 (socket says 172.21.15.146:56530) 2024-09-16T01:52:42.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.119+0000 7fc631200640 1 -- 172.21.15.146:0/459424339 learned_addr learned my addr 172.21.15.146:0/459424339 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:42.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.120+0000 7fc631c00640 1 --2- 172.21.15.146:0/459424339 >> v2:172.21.15.161:3300/0 conn(0x7fc6240e6b50 0x7fc624051df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.120+0000 7fc631c00640 1 -- 172.21.15.146:0/459424339 >> v2:172.21.15.146:3300/0 conn(0x7fc6240091b0 msgr2=0x7fc62404e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:42.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.120+0000 7fc631c00640 1 --2- 172.21.15.146:0/459424339 >> v2:172.21.15.146:3300/0 conn(0x7fc6240091b0 0x7fc62404e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.120+0000 7fc631c00640 1 -- 172.21.15.146:0/459424339 >> v1:172.21.15.146:6789/0 conn(0x7fc6240dccc0 legacy=0x7fc62404cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:42.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.120+0000 7fc631c00640 1 -- 172.21.15.146:0/459424339 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc624046e50 con 0x7fc6240e6b50 2024-09-16T01:52:42.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.123+0000 7f734c400640 1 Processor -- start 2024-09-16T01:52:42.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.124+0000 7f734c400640 1 -- start start 2024-09-16T01:52:42.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.125+0000 7fc631c00640 1 --2- 172.21.15.146:0/459424339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc624051df0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc62c02dd50 tx=0x7fc62c0a6ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68c569074e0b5e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c050290 0x7f733c052cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734c400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0d8830 0x7f733c0d4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734c400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 0x7f733c0d7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f733c0d8c10 con 0x7f733c050290 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734c400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f733c00f6e0 con 0x7f733c0d8830 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734c400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f733c0d8d90 con 0x7f733c0dee00 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 0x7f733c0d7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734a400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 0x7f733c0d7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43012/0 (socket says 172.21.15.146:43012) 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734a400640 1 -- 172.21.15.146:0/1631115234 learned_addr learned my addr 172.21.15.146:0/1631115234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734a400640 1 -- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0d8830 msgr2=0x7f733c0d4100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734a400640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0d8830 0x7f733c0d4100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7f734a400640 1 -- 172.21.15.146:0/1631115234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c050290 msgr2=0x7f733c052cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7fc62b400640 1 -- 172.21.15.146:0/459424339 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0aa910 con 0x7fc6240e6b50 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7fc62b400640 1 -- 172.21.15.146:0/459424339 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c0aaab0 con 0x7fc6240e6b50 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.126+0000 7fc62b400640 1 -- 172.21.15.146:0/459424339 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0b2400 con 0x7fc6240e6b50 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.127+0000 7fc632600640 1 -- 172.21.15.146:0/459424339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 msgr2=0x7fc624051df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.127+0000 7fc632600640 1 --2- 172.21.15.146:0/459424339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc624051df0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc62c02dd50 tx=0x7fc62c0a6ad0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.127+0000 7fc632600640 1 -- 172.21.15.146:0/459424339 shutdown_connections 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.127+0000 7fc632600640 1 --2- 172.21.15.146:0/459424339 >> v2:172.21.15.146:3300/0 conn(0x7fc6240091b0 0x7fc62404e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.127+0000 7fc632600640 1 --2- 172.21.15.146:0/459424339 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc624051df0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.127+0000 7fc632600640 1 -- 172.21.15.146:0/459424339 >> 172.21.15.146:0/459424339 conn(0x7fc62400db90 msgr2=0x7fc62400df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:42.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.128+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c02db90 con 0x7fa58c207ab0 2024-09-16T01:52:42.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.128+0000 7f7349a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c050290 0x7f733c052cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.128+0000 7f734a400640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c050290 0x7f733c052cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.128+0000 7f734a400640 1 -- 172.21.15.146:0/1631115234 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c053480 con 0x7f733c0dee00 2024-09-16T01:52:42.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.128+0000 7f7349a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c050290 0x7f733c052cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:42.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.129+0000 7fc632600640 1 -- 172.21.15.146:0/459424339 shutdown_connections 2024-09-16T01:52:42.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.130+0000 7fc632600640 1 -- 172.21.15.146:0/459424339 wait complete. 2024-09-16T01:52:42.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.129+0000 7f734a400640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 0x7f733c0d7170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7344051790 tx=0x7f734406b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.130+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73440bf070 con 0x7f733c0dee00 2024-09-16T01:52:42.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.130+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73440b0b60 con 0x7f733c0dee00 2024-09-16T01:52:42.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.130+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73440bb440 con 0x7f733c0dee00 2024-09-16T01:52:42.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.131+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58405e1f0 con 0x7fa58c207ab0 2024-09-16T01:52:42.134 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:42.134 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:42.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.131+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa58c20c270 0x7fa58c20c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.131+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa58c20b7a0 con 0x7fa58c20c270 2024-09-16T01:52:42.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.133+0000 7fc632600640 1 Processor -- start 2024-09-16T01:52:42.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.133+0000 7fc632600640 1 -- start start 2024-09-16T01:52:42.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.134+0000 7fc632600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 0x7fc62404b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.134+0000 7fc632600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6240dccc0 0x7fc624036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.134+0000 7fc632600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc6240372f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.130+0000 7f734c400640 1 -- 172.21.15.146:0/1631115234 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f733c053740 con 0x7f733c0dee00 2024-09-16T01:52:42.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.130+0000 7f734c400640 1 -- 172.21.15.146:0/1631115234 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f733c0d64d0 con 0x7f733c0dee00 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7fc632600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc6240e6630 con 0x7fc6240091b0 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7fc632600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc6240e6350 con 0x7fc6240e6b50 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7fc632600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc6240473f0 con 0x7fc6240dccc0 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7fc631200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 0x7fc62404b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7fc631200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 0x7fc62404b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40068/0 (socket says 172.21.15.146:40068) 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7fc631200640 1 -- 172.21.15.146:0/1927223462 learned_addr learned my addr 172.21.15.146:0/1927223462 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:42.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.132+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa58c20c270 0x7fa58c20c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc62be00640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6240dccc0 0x7fc624036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc631c00640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc6240372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.135+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f73440c2070 con 0x7f733c0dee00 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc631200640 1 -- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6240dccc0 msgr2=0x7fc624036db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc631200640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6240dccc0 0x7fc624036db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc631200640 1 -- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 msgr2=0x7fc6240372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc631200640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc6240372f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.136+0000 7fc631200640 1 -- 172.21.15.146:0/1927223462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc62404bd40 con 0x7fc6240091b0 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.137+0000 7fc631200640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 0x7fc62404b2a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc62000b430 tx=0x7fc62000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.137+0000 7fa589e00640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa58c20c270 0x7fa58c20c6b0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.137+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7318005290 con 0x7f733c0dee00 2024-09-16T01:52:42.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.138+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620015020 con 0x7fc6240091b0 2024-09-16T01:52:42.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.138+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa58c20b7a0 con 0x7fa58c20c270 2024-09-16T01:52:42.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.138+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6240dc550 con 0x7fc6240091b0 2024-09-16T01:52:42.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.138+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc620011830 con 0x7fc6240091b0 2024-09-16T01:52:42.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.138+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620011c20 con 0x7fc6240091b0 2024-09-16T01:52:42.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.139+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62404c290 con 0x7fc6240091b0 2024-09-16T01:52:42.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.142+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc620018020 con 0x7fc6240091b0 2024-09-16T01:52:42.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.142+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6240dd0a0 con 0x7fc6240091b0 2024-09-16T01:52:42.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.143+0000 7fc628c00640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc60c06c2e0 0x7fc60c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.144+0000 7fc62be00640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc60c06c2e0 0x7fc60c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.145+0000 7f7341600640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f732406c380 0x7f732406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:42.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.146+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6023+0+0 (secure 0 0 0) 0x7fc620088d10 con 0x7fc6240091b0 2024-09-16T01:52:42.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.146+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(108..108 src has 1..108) ==== 6023+0+0 (secure 0 0 0) 0x7f7344129160 con 0x7f733c0dee00 2024-09-16T01:52:42.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.146+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f734412d480 con 0x7f733c0dee00 2024-09-16T01:52:42.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.149+0000 7fc62be00640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc60c06c2e0 0x7fc60c06e7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc6240e1910 tx=0x7fc618008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.151+0000 7f7349a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f732406c380 0x7f732406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:42.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.152+0000 7f7349a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f732406c380 0x7f732406e840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f733000a870 tx=0x7f7330008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:42.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.158+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc620058ed0 con 0x7fc6240091b0 2024-09-16T01:52:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.188+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa58c20b7a0 con 0x7fa58c20c270 2024-09-16T01:52:42.193 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:42.193 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:42.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.190+0000 7fa582a00640 1 -- 172.21.15.146:0/816134810 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa58c20b7a0 con 0x7fa58c20c270 2024-09-16T01:52:42.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa58c158030 msgr2=0x7fa58c0b5040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa58c158030 0x7fa58c0b5040 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa58c155520 msgr2=0x7fa58c155900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa58c155520 0x7fa58c155900 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa58c15bf60 msgr2=0x7fa58c15c3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa58c15bf60 0x7fa58c15c3c0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa58c062630 msgr2=0x7fa58c062ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa58c062630 0x7fa58c062ab0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa58c02cb50 msgr2=0x7fa58c02cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa58c02cb50 0x7fa58c02cfb0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa58c204c70 msgr2=0x7fa58c207060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa58c204c70 0x7fa58c207060 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa58c207ab0 msgr2=0x7fa58c209ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa58c207ab0 0x7fa58c209ea0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa58c20c270 msgr2=0x7fa58c20c6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa58c20c270 0x7fa58c20c6b0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 reap_dead start 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 reap_dead start 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 reap_dead start 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 reap_dead start 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.192+0000 7fa58a800640 1 -- 172.21.15.146:0/816134810 reap_dead start 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa56806c330 msgr2=0x7fa56806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa56806c330 0x7fa56806e7f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa57c034000 tx=0x7fa57c02e000 comp rx=0 tx=0).stop 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa58c154c10 msgr2=0x7fa58c14df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa58c154c10 0x7fa58c14df00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c760 tx=0x7fa58400cc20 comp rx=0 tx=0).stop 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 shutdown_connections 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa56806c330 0x7fa56806e7f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 --2- 172.21.15.146:0/816134810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa58c154c10 0x7fa58c14df00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.193+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 >> 172.21.15.146:0/816134810 conn(0x7fa58c06ce80 msgr2=0x7fa58c0cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:42.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.194+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 shutdown_connections 2024-09-16T01:52:42.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.194+0000 7fa591000640 1 -- 172.21.15.146:0/816134810 wait complete. 2024-09-16T01:52:42.203 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:42.199+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:42.217 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:42.217 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:42.217 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:42.217 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:42.217 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:42.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:42.339 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:42.337+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:42.401 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:42.400+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:42.503 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:42.501+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:42.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.548+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f7318005580 con 0x7f733c0dee00 2024-09-16T01:52:42.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.572+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fc62404d790 con 0x7fc6240091b0 2024-09-16T01:52:42.603 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:42.602+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:42.612 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:42.612 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:42.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:42.612 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:42.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.612 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.317682496000003, 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.613 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:42.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:42.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:42.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:42.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:42.620 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.288521641000003, 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:42.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:42.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.256170073, 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:42.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.240222807000002, 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:42.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:42.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.693147642, 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:42.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:42.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.508218286000002, 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.645 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:42.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:42.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:42.652 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.505680478000002, 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:42.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1966 3.3c 3:3c832e37:::100000001f8.000003ab:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:42.658 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.152645737, 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10596815800000001, 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1966 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:42.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456219+0000", 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.456221+0000", 2024-09-16T01:52:42.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.125e-06 2024-09-16T01:52:42.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520630+0000", 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064408699 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520660+0000", 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0122000000000001e-05 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520672+0000", 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2179e-05 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520772+0000", 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.993e-05 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.520970+0000", 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198097 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.521122+0000", 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151065 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.535969+0000", 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014847075 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562086+0000", 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026117475000000001 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562144+0000", 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7806000000000002e-05 2024-09-16T01:52:42.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.562187+0000", 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3584999999999997e-05 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.896269246000003, 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:42.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:42.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:42.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3377 3.26 3:6604db39:::100000001f8.0000092e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:42.671 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.682878678999998, 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.104633035, 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3377 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925986+0000", 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.925988+0000", 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0549999999999998e-06 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983358+0000", 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057369575999999999 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983375+0000", 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7356000000000001e-05 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983387+0000", 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1277000000000001e-05 2024-09-16T01:52:42.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983509+0000", 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012268399999999999 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983659+0000", 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014946499999999999 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983799+0000", 2024-09-16T01:52:42.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000139787 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.000668+0000", 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016869359 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030498+0000", 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029829855999999998 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030562+0000", 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4199e-05 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.030619+0000", 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7420999999999998e-05 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3378 3.e 3:709bbee6:::100000001f8.0000092f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e92)", 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:42.677 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.625454863999998, 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108434431, 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3378 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:42.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983410+0000", 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:05.983412+0000", 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.528e-06 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.059475+0000", 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076063617 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060086+0000", 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00061040300000000003 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060097+0000", 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0842999999999999e-05 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060152+0000", 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5025999999999998e-05 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060248+0000", 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6351e-05 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.060334+0000", 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.6216000000000003e-05 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.077660+0000", 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017325871 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091714+0000", 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014053566 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091794+0000", 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0434999999999998e-05 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:06.091845+0000", 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0575000000000003e-05 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.858012811000002, 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.683 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:52:42.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:52:42.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.782941076, 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.689 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:52:42.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.693 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.694 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3682 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.894687+0000", 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.714177903, 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10534299599999999, 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3682 2024-09-16T01:52:42.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.894687+0000", 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.894687+0000", 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.894689+0000", 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.902e-06 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919371+0000", 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.024682101000000001 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919396+0000", 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4736000000000002e-05 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919410+0000", 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3859000000000001e-05 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.921989+0000", 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0025788249999999999 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.922099+0000", 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000110003 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.922251+0000", 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015253900000000001 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.951255+0000", 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029004268999999999 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.999912+0000", 2024-09-16T01:52:42.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048656491000000003 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.999980+0000", 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7875999999999999e-05 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:16.000030+0000", 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0395000000000001e-05 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4142 3.16 3:6ba609e9:::100000001f8.0000022b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.821844521999999, 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108516207, 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4142 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.701 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787022+0000", 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.61e-06 2024-09-16T01:52:42.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842519+0000", 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055496399000000002 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842539+0000", 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0820000000000001e-05 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842550+0000", 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0414999999999999e-05 2024-09-16T01:52:42.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842641+0000", 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1657999999999996e-05 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842742+0000", 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010075 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842888+0000", 2024-09-16T01:52:42.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014623699999999999 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.892001+0000", 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112211000000003 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895434+0000", 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034334869999999998 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895499+0000", 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5313000000000003e-05 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895537+0000", 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7307e-05 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4246 3.34 3:2f44359e:::100000001f8.00000293:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.279231960000001, 2024-09-16T01:52:42.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.142262732, 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4246 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329635+0000", 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9180000000000001e-06 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410104+0000", 2024-09-16T01:52:42.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080469499 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410128+0000", 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3243000000000001e-05 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410141+0000", 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3067999999999999e-05 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419779+0000", 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096380589999999992 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419889+0000", 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.420057+0000", 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016837699999999999 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456784+0000", 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036727237000000003 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471769+0000", 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014985112 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471839+0000", 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9275000000000003e-05 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471896+0000", 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7259000000000002e-05 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4247 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.198676852999998, 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120085209, 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.712 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4247 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410190+0000", 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451530+0000", 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041340783999999998 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451572+0000", 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1690000000000002e-05 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451620+0000", 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8180999999999998e-05 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452310+0000", 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068996900000000002 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452413+0000", 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010248599999999999 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452588+0000", 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017574800000000001 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.518054+0000", 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065465819999999994 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530168+0000", 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012113413999999999 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530225+0000", 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7284999999999998e-05 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530273+0000", 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8381999999999999e-05 2024-09-16T01:52:42.717 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4503 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.097121498, 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13319203700000001, 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.718 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4503 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511746+0000", 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0250000000000001e-06 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576368+0000", 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064622151000000003 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576386+0000", 2024-09-16T01:52:42.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7878999999999999e-05 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576398+0000", 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578343+0000", 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001945397 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578445+0000", 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102001 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578601+0000", 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015608399999999999 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.643878+0000", 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065277247999999996 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644839+0000", 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096038899999999997 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644895+0000", 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6484000000000003e-05 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644936+0000", 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0324999999999998e-05 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.723 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4504 3.f e101/94 3:f3dac423:::100000001f8.00000395:head v 101'76, mlcod=100'74)", 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 16.014195381, 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14827673899999999, 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:42.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594672+0000", 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3309999999999998e-06 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629004+0000", 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034332342000000002 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629020+0000", 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6121e-05 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629029+0000", 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7479999999999999e-06 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635523+0000", 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064935909999999999 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635549+0000", 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5808000000000001e-05 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742856+0000", 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107307556 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742946+0000", 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0241999999999994e-05 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4549 3.2 3:428d9500:::100000001f8.000003c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.439231934, 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11639424900000001, 2024-09-16T01:52:42.728 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4549 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169636+0000", 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8660000000000002e-06 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212280+0000", 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042644350999999997 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212312+0000", 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2088999999999997e-05 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212327+0000", 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521e-05 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212426+0000", 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9245000000000003e-05 2024-09-16T01:52:42.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212539+0000", 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112711 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212725+0000", 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018583400000000001 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285881+0000", 2024-09-16T01:52:42.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073156321999999996 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285914+0000", 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2929999999999998e-05 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285979+0000", 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5048000000000005e-05 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:42.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.286027+0000", 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8331999999999997e-05 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:42.734 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:42.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:42.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.704+0000 7fc628c00640 1 -- 172.21.15.146:0/1927223462 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v109) ==== 87+0+0 (secure 0 0 0) 0x7fc6200587d0 con 0x7fc6240091b0 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.711+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc60c06c2e0 msgr2=0x7fc60c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.711+0000 7fc632600640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc60c06c2e0 0x7fc60c06e7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc6240e1910 tx=0x7fc618008040 comp rx=0 tx=0).stop 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.711+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 msgr2=0x7fc62404b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.711+0000 7fc632600640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 0x7fc62404b2a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc62000b430 tx=0x7fc62000b8f0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.712+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 shutdown_connections 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.712+0000 7fc632600640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc60c06c2e0 0x7fc60c06e7a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.711+0000 7f7341600640 1 -- 172.21.15.146:0/1631115234 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v109) ==== 113+0+0 (secure 0 0 0) 0x7f73440f77d0 con 0x7f733c0dee00 2024-09-16T01:52:42.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.712+0000 7fc632600640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc6240e6b50 0x7fc6240372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.712+0000 7fc632600640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc6240dccc0 0x7fc624036db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.712+0000 7fc632600640 1 --2- 172.21.15.146:0/1927223462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6240091b0 0x7fc62404b2a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.712+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 >> 172.21.15.146:0/1927223462 conn(0x7fc62400db90 msgr2=0x7fc6240515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:42.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.713+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 shutdown_connections 2024-09-16T01:52:42.738 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.713+0000 7fc632600640 1 -- 172.21.15.146:0/1927223462 wait complete. 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.719+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f732406c380 msgr2=0x7f732406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.719+0000 7f7336a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f732406c380 0x7f732406e840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f733000a870 tx=0x7f7330008040 comp rx=0 tx=0).stop 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.719+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 msgr2=0x7f733c0d7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.719+0000 7f7336a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 0x7f733c0d7170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7344051790 tx=0x7f734406b5d0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.720+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 shutdown_connections 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.720+0000 7f7336a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f732406c380 0x7f732406e840 secure :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f733000a870 tx=0x7f7330008040 comp rx=0 tx=0).stop 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.720+0000 7f7336a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f733c0dee00 0x7f733c0d7170 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.720+0000 7f7336a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f733c0d8830 0x7f733c0d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.720+0000 7f7336a00640 1 --2- 172.21.15.146:0/1631115234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c050290 0x7f733c052cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:42.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.720+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 >> 172.21.15.146:0/1631115234 conn(0x7f733c054410 msgr2=0x7f733c0d52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:42.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.721+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 shutdown_connections 2024-09-16T01:52:42.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:42.721+0000 7f7336a00640 1 -- 172.21.15.146:0/1631115234 wait complete. 2024-09-16T01:52:42.747 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:42.744+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:42.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T01:52:42.836 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:42.835+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:42.936 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:42.935+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:43.041 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:43.039+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:43.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.085+0000 7f501f000640 1 Processor -- start 2024-09-16T01:52:43.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.086+0000 7f501f000640 1 -- start start 2024-09-16T01:52:43.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f501f000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f501815a790 0x7f501806d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f501f000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5018157e30 0x7f50181582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f501f000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50180c4820 con 0x7f50180c7200 2024-09-16T01:52:43.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f501f000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f50180c69b0 con 0x7f501815a790 2024-09-16T01:52:43.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f501f000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5018157950 con 0x7f5018157e30 2024-09-16T01:52:43.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f5017e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f50180c7200 0x7f501806e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33538/0 (socket says 172.21.15.146:33538) 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.087+0000 7f5017e00640 1 -- 172.21.15.146:0/2827856456 learned_addr learned my addr 172.21.15.146:0/2827856456 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.088+0000 7f501d000640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5018157e30 0x7f50181582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.088+0000 7f5017400640 1 --2- 172.21.15.146:0/2827856456 >> v2:172.21.15.161:3300/0 conn(0x7f501815a790 0x7f501806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569400058 0 0) 0x7f50180c4820 con 0x7f50180c7200 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ffc003520 con 0x7f50180c7200 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3282178647 0 0) 0x7f4ffc003520 con 0x7f50180c7200 2024-09-16T01:52:43.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5018157e30 msgr2=0x7f50181582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:43.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5018157e30 0x7f50181582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 >> v2:172.21.15.161:3300/0 conn(0x7f501815a790 msgr2=0x7f501806d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:43.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 --2- 172.21.15.146:0/2827856456 >> v2:172.21.15.161:3300/0 conn(0x7f501815a790 0x7f501806d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50180c6110 con 0x7f50180c7200 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f50100019f0 con 0x7f50180c7200 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 >> v1:172.21.15.146:6789/0 conn(0x7f50180c7200 legacy=0x7f501806e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ffc0039b0 0x7f4ffc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc0043a0 0x7f4ffc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ffc005040 0x7f4ffc00fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f50180c69b0 con 0x7f4ffc0043a0 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5018157950 con 0x7f4ffc005040 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.089+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4ffc0103f0 con 0x7f4ffc0039b0 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.090+0000 7f5017e00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ffc0039b0 0x7f4ffc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.090+0000 7f501d000640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5018157e30 0x7f50181582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.090+0000 7f5017e00640 1 -- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ffc005040 msgr2=0x7f4ffc00fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.090+0000 7f5017e00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ffc005040 0x7f4ffc00fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.090+0000 7f5017e00640 1 -- 172.21.15.146:0/2827856456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc0043a0 msgr2=0x7f4ffc004880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:43.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.091+0000 7f5017400640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc0043a0 0x7f4ffc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.091+0000 7f5017e00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc0043a0 0x7f4ffc004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.091+0000 7f5017e00640 1 -- 172.21.15.146:0/2827856456 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50180c6110 con 0x7f4ffc0039b0 2024-09-16T01:52:43.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.091+0000 7f5017400640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc0043a0 0x7f4ffc004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:43.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.092+0000 7f5017e00640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ffc0039b0 0x7f4ffc003dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5010002230 tx=0x7f4ffc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=624242f0cf6a180d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:43.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.092+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501000a500 con 0x7f4ffc0039b0 2024-09-16T01:52:43.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.092+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501002f070 con 0x7f4ffc0039b0 2024-09-16T01:52:43.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.092+0000 7f5016a00640 1 -- 172.21.15.146:0/2827856456 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501002bcf0 con 0x7f4ffc0039b0 2024-09-16T01:52:43.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.093+0000 7f501f000640 1 -- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ffc0039b0 msgr2=0x7f4ffc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:43.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.093+0000 7f501f000640 1 --2- 172.21.15.146:0/2827856456 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ffc0039b0 0x7f4ffc003dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5010002230 tx=0x7f4ffc005830 comp rx=0 tx=0).stop 2024-09-16T01:52:43.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.093+0000 7f501d000640 1 -- 172.21.15.146:0/2827856456 reap_dead start 2024-09-16T01:52:43.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.094+0000 7f501f000640 1 -- 172.21.15.146:0/2827856456 shutdown_connections 2024-09-16T01:52:43.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.094+0000 7f501f000640 1 -- 172.21.15.146:0/2827856456 >> 172.21.15.146:0/2827856456 conn(0x7f50180728b0 msgr2=0x7f5018072c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:43.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.097+0000 7f501f000640 1 -- 172.21.15.146:0/2827856456 shutdown_connections 2024-09-16T01:52:43.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.097+0000 7f501f000640 1 -- 172.21.15.146:0/2827856456 wait complete. 2024-09-16T01:52:43.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.100+0000 7f501f000640 1 Processor -- start 2024-09-16T01:52:43.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.100+0000 7f501f000640 1 -- start start 2024-09-16T01:52:43.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f501f000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50180c21b0 0x7f50181e0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f501f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 0x7f5018159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f501f000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50181df8a0 0x7f50180c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f501f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f50180c44e0 con 0x7f50180c2590 2024-09-16T01:52:43.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f501f000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50180c66d0 con 0x7f50181df8a0 2024-09-16T01:52:43.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f501f000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f50180c78e0 con 0x7f50180c21b0 2024-09-16T01:52:43.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f5017400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 0x7f5018159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f5017400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 0x7f5018159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45626/0 (socket says 172.21.15.146:45626) 2024-09-16T01:52:43.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.103+0000 7f5017400640 1 -- 172.21.15.146:0/575412983 learned_addr learned my addr 172.21.15.146:0/575412983 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:43.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017e00640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50180c21b0 0x7f50181e0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f501d000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50181df8a0 0x7f50180c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017e00640 1 -- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50181df8a0 msgr2=0x7f50180c31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017e00640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50181df8a0 0x7f50180c31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017e00640 1 -- 172.21.15.146:0/575412983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 msgr2=0x7f5018159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017e00640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 0x7f5018159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017e00640 1 -- 172.21.15.146:0/575412983 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50181f5920 con 0x7f50180c21b0 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.104+0000 7f5017400640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 0x7f5018159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:43.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.105+0000 7f5017e00640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50180c21b0 0x7f50181e0720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5010030000 tx=0x7f5010001cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.105+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010034670 con 0x7f50180c21b0 2024-09-16T01:52:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.105+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5010034810 con 0x7f50180c21b0 2024-09-16T01:52:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.105+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010034bc0 con 0x7f50180c21b0 2024-09-16T01:52:43.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.105+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50180c1020 con 0x7f50180c21b0 2024-09-16T01:52:43.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.107+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50181f55e0 con 0x7f50180c21b0 2024-09-16T01:52:43.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.108+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50180c6db0 con 0x7f50180c21b0 2024-09-16T01:52:43.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.111+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5010046060 con 0x7f50180c21b0 2024-09-16T01:52:43.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.112+0000 7f500fe00640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4ff806c3b0 0x7f4ff806e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:43.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.112+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(109..109 src has 1..109) ==== 6023+0+0 (secure 0 0 0) 0x7f50100ab080 con 0x7f50180c21b0 2024-09-16T01:52:43.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.117+0000 7f5017400640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4ff806c3b0 0x7f4ff806e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:43.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.118+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5010079720 con 0x7f50180c21b0 2024-09-16T01:52:43.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.119+0000 7f5017400640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4ff806c3b0 0x7f4ff806e870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f50180c9d90 tx=0x7f5008008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:43.133 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5850 3.1d 3:ba9c4a52:::100000001f8.000008d7:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e109)", 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:43.073916+0000", 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.055488441999999999, 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055520259000000002, 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.134 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5850 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.073916+0000", 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.073916+0000", 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.073918+0000", 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3140000000000002e-06 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.107213+0000", 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033294773 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.107244+0000", 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1337999999999997e-05 2024-09-16T01:52:43.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.107258+0000", 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3764999999999999e-05 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.107348+0000", 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0216999999999994e-05 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.107447+0000", 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8876000000000007e-05 2024-09-16T01:52:43.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:43.107632+0000", 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018479999999999999 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:52:43.138 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:43.139 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:43.138+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:43.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:43.240 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:43.239+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:43.341 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:43.340+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:43.442 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:43.441+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:43.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.487+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f5018072ee0 con 0x7f50180c21b0 2024-09-16T01:52:43.501 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:43.501 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:43.501 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:43.501 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:43.501 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:43.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:43.543 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:43.542+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:43.644 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:43.643+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.735+0000 7f500fe00640 1 -- 172.21.15.146:0/575412983 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v110) ==== 113+0+0 (secure 0 0 0) 0x7f5010079020 con 0x7f50180c21b0 2024-09-16T01:52:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T01:52:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.740+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4ff806c3b0 msgr2=0x7f4ff806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.740+0000 7f501f000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4ff806c3b0 0x7f4ff806e870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f50180c9d90 tx=0x7f5008008040 comp rx=0 tx=0).stop 2024-09-16T01:52:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.740+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50180c21b0 msgr2=0x7f50181e0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.740+0000 7f501f000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50180c21b0 0x7f50181e0720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5010030000 tx=0x7f5010001cf0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 shutdown_connections 2024-09-16T01:52:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4ff806c3b0 0x7f4ff806e870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50181df8a0 0x7f50180c31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50180c2590 0x7f5018159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 --2- 172.21.15.146:0/575412983 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50180c21b0 0x7f50181e0720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 >> 172.21.15.146:0/575412983 conn(0x7f50180728b0 msgr2=0x7f501806c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:43.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 shutdown_connections 2024-09-16T01:52:43.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:43.741+0000 7f501f000640 1 -- 172.21.15.146:0/575412983 wait complete. 2024-09-16T01:52:43.746 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:43.744+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:43.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:43.807 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.952853424, 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:43.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:43.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.81039375899999, 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:43.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:43.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.823 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 103.462927536, 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.824 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:43.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:43.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.578636172000003, 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.832 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:43.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:43.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:43.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:43.838 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.540896669000006, 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:43.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:43.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:43.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.844 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.681909668999999, 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.845 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:43.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:43.851 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.814870695000003, 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:43.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:43.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:43.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:43.858 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.618918473999997, 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:43.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:43.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.577916531, 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:43.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:43.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:43.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 54.353688970999997, 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:43.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:43.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:43.877 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 48.396907417999998, 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:43.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.485927068999999, 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:43.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.049396087000002, 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:52:43.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:52:43.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:43.896 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.026242169, 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:43.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:52:43.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 28.015306326000001, 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.901 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.905 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:43.906 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.985111902, 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:52:43.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:43.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:43.912 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 27.943706449, 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.913 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:52:43.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:43.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.918 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5039 3.35 3:afdde0ae:::100000001f8.000005ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e103)", 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:33.007654+0000", 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.793936323, 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.10917603400000001, 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5039 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.007654+0000", 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.007654+0000", 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.007656+0000", 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080126+0000", 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.072470100999999995 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080142+0000", 2024-09-16T01:52:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5739999999999998e-05 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080151+0000", 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5710000000000004e-06 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080231+0000", 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9308999999999996e-05 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080331+0000", 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9874e-05 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.080484+0000", 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015373399999999999 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.110967+0000", 2024-09-16T01:52:43.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030483135000000001 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.115628+0000", 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0046609260000000001 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.116770+0000", 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011416099999999999 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:33.116830+0000", 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0331000000000002e-05 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5551 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.9628993589999997, 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13476244500000001, 2024-09-16T01:52:43.925 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5551 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838694+0000", 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.306e-06 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866535+0000", 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027841175999999999 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866552+0000", 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7387000000000001e-05 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866563+0000", 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0447e-05 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866665+0000", 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101894 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866788+0000", 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123194 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866953+0000", 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165365 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.918808+0000", 2024-09-16T01:52:43.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051855367999999999 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973333+0000", 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054524250000000003 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973402+0000", 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9648999999999997e-05 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973454+0000", 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5812 3.a 3:51094797:::100000001f8.000008b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 1.2692002439999999, 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14779089000000001, 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:43.933 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5812 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532392+0000", 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8679999999999999e-06 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653045+0000", 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120652599 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653062+0000", 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6929e-05 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653073+0000", 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0925e-05 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656653+0000", 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035800049999999998 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656738+0000", 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5599000000000005e-05 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:43.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656890+0000", 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151874 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.676041+0000", 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019150358999999999 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680071+0000", 2024-09-16T01:52:43.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004030624 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680131+0000", 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9704000000000003e-05 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680181+0000", 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0404000000000001e-05 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.939 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:43.940 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:43.940 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:43.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:43.941 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:43.928+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:44.030 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:44.029+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:44.072 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:44.072 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:44.072 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:44.073 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:44.086 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:44.171 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:44.170+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:44.232 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:44.230+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.233+0000 7f971ec00640 1 Processor -- start 2024-09-16T01:52:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.233+0000 7f971ec00640 1 -- start start 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.234+0000 7f971ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f971806bc10 0x7f97180717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.234+0000 7f971ec00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f971806bff0 0x7f9718157630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.234+0000 7f971ec00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971815a250 con 0x7f9718071ec0 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.234+0000 7f971ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9718157b70 con 0x7f971806bc10 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.234+0000 7f971ec00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f971815a4c0 con 0x7f971806bff0 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.234+0000 7f971d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f971806bc10 0x7f97180717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f971806bc10 0x7f97180717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45638/0 (socket says 172.21.15.146:45638) 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971d800640 1 -- 172.21.15.146:0/1391099267 learned_addr learned my addr 172.21.15.146:0/1391099267 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:44.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971ce00640 1 --2- 172.21.15.146:0/1391099267 >> v2:172.21.15.161:3300/0 conn(0x7f971806bff0 0x7f9718157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971ce00640 1 -- 172.21.15.146:0/1391099267 >> v2:172.21.15.146:3300/0 conn(0x7f971806bc10 msgr2=0x7f97180717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971ce00640 1 --2- 172.21.15.146:0/1391099267 >> v2:172.21.15.146:3300/0 conn(0x7f971806bc10 0x7f97180717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971ce00640 1 -- 172.21.15.146:0/1391099267 >> v1:172.21.15.161:6789/0 conn(0x7f9718071ec0 legacy=0x7f97180722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.235+0000 7f971ce00640 1 -- 172.21.15.146:0/1391099267 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971815a7a0 con 0x7f971806bff0 2024-09-16T01:52:44.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.237+0000 7f971ce00640 1 --2- 172.21.15.146:0/1391099267 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f971806bff0 0x7f9718157630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f971806e2c0 tx=0x7f970402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1353a11f61dc8c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.237+0000 7f9717e00640 1 -- 172.21.15.146:0/1391099267 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704037070 con 0x7f971806bff0 2024-09-16T01:52:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.237+0000 7f9717e00640 1 -- 172.21.15.146:0/1391099267 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704032440 con 0x7f971806bff0 2024-09-16T01:52:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.237+0000 7f9717e00640 1 -- 172.21.15.146:0/1391099267 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704032810 con 0x7f971806bff0 2024-09-16T01:52:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.238+0000 7f971ec00640 1 -- 172.21.15.146:0/1391099267 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f971806bff0 msgr2=0x7f9718157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.238+0000 7f971ec00640 1 --2- 172.21.15.146:0/1391099267 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f971806bff0 0x7f9718157630 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f971806e2c0 tx=0x7f970402a250 comp rx=0 tx=0).stop 2024-09-16T01:52:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.238+0000 7f971ec00640 1 -- 172.21.15.146:0/1391099267 shutdown_connections 2024-09-16T01:52:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.238+0000 7f971ec00640 1 --2- 172.21.15.146:0/1391099267 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f971806bff0 0x7f9718157630 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.238+0000 7f971ec00640 1 --2- 172.21.15.146:0/1391099267 >> v2:172.21.15.146:3300/0 conn(0x7f971806bc10 0x7f97180717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.238+0000 7f971ec00640 1 -- 172.21.15.146:0/1391099267 >> 172.21.15.146:0/1391099267 conn(0x7f971806f710 msgr2=0x7f971806fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.239+0000 7f971ec00640 1 -- 172.21.15.146:0/1391099267 shutdown_connections 2024-09-16T01:52:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.239+0000 7f971ec00640 1 -- 172.21.15.146:0/1391099267 wait complete. 2024-09-16T01:52:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.241+0000 7f971ec00640 1 Processor -- start 2024-09-16T01:52:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.241+0000 7f971ec00640 1 -- start start 2024-09-16T01:52:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.241+0000 7f971ec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f971806bc10 0x7f97181f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.241+0000 7f971ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 0x7f9718159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9718071ec0 0x7f9718158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f971806c460 con 0x7f971806bff0 2024-09-16T01:52:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f971815ad60 con 0x7f9718071ec0 2024-09-16T01:52:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f97180c6ce0 con 0x7f971806bc10 2024-09-16T01:52:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 0x7f9718159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 0x7f9718159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45648/0 (socket says 172.21.15.146:45648) 2024-09-16T01:52:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 -- 172.21.15.146:0/720693176 learned_addr learned my addr 172.21.15.146:0/720693176 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971d800640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f971806bc10 0x7f97181f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 -- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f971806bc10 msgr2=0x7f97181f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f971806bc10 0x7f97181f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 -- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9718071ec0 msgr2=0x7f9718158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9718071ec0 0x7f9718158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971ce00640 1 -- 172.21.15.146:0/720693176 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97181f3b80 con 0x7f971806bff0 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.242+0000 7f971d800640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f971806bc10 0x7f97181f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.243+0000 7f971ce00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 0x7f9718159050 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9704000c00 tx=0x7f97040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.243+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704037040 con 0x7f971806bff0 2024-09-16T01:52:44.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.243+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704042070 con 0x7f971806bff0 2024-09-16T01:52:44.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.243+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704030bd0 con 0x7f971806bff0 2024-09-16T01:52:44.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.243+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97180ca7a0 con 0x7f971806bff0 2024-09-16T01:52:44.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.244+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97181f4100 con 0x7f971806bff0 2024-09-16T01:52:44.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.245+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97181511e0 con 0x7f971806bff0 2024-09-16T01:52:44.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.246+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9704047460 con 0x7f971806bff0 2024-09-16T01:52:44.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.250+0000 7f9715600640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f96f806c300 0x7f96f806e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.250+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5973+0+0 (secure 0 0 0) 0x7f97040ac040 con 0x7f971806bff0 2024-09-16T01:52:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.251+0000 7f971d800640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f96f806c300 0x7f96f806e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.253+0000 7f971d800640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f96f806c300 0x7f96f806e7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f970c009970 tx=0x7f970c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.258+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9704032e20 con 0x7f971806bff0 2024-09-16T01:52:44.263 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:44.263 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:44.263 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:44.263 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:44.263 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:44.274 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:44.358 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:44.357+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:44.434 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:44.432+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:44.446 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:44.446 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:52:44.446 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:52:44.446 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:44.446 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.446 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 113.58791436200001, 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.447 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.448 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:52:44.449 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.451 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:52:44.452 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:44.453 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 112.67633605, 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:44.454 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.455 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:52:44.456 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.457 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.458 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.459 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 112.63207582, 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.460 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.461 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.462 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:52:44.463 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.464 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:52:44.465 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:44.466 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 112.560020478, 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:44.467 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.468 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.469 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.470 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:52:44.471 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 104.303014057, 2024-09-16T01:52:44.472 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:52:44.474 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.475 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.476 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:52:44.477 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:52:44.478 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:52:44.479 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:44.480 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 104.26821775800001, 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.481 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.483 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.484 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:52:44.485 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.486 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:52:44.487 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:52:44.488 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:44.489 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 104.184768805, 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:52:44.490 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.491 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.492 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:52:44.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:52:44.494 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 89.985466017999997, 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:52:44.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.497 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:52:44.498 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.499 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:52:44.501 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 89.840589080000001, 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.502 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.504 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.505 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:52:44.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 89.799944655999994, 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:52:44.508 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:52:44.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:52:44.511 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.512 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.513 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 89.642001821999997, 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:52:44.514 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.515 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:52:44.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:52:44.517 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.518 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:52:44.519 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:52:44.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.521 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 89.516874560999995, 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.522 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.523 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.524 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:52:44.525 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:52:44.526 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.527 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.528 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 88.480074630000004, 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.529 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.530 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:52:44.531 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.532 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.533 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:52:44.534 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 83.317048987999996, 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.535 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.536 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:52:44.537 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:52:44.538 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.539 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:52:44.540 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 65.183311199000002, 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.541 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.542 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:52:44.543 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.544 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.545 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.546 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 64.488921085000001, 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.547 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.548 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:52:44.549 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.550 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.551 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:52:44.552 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 60.161946518999997, 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.553 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.554 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.555 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.556 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.557 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:52:44.558 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 60.021036379000002, 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:52:44.559 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.560 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:52:44.561 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:52:44.562 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:52:44.563 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:52:44.564 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 59.968233435000002, 2024-09-16T01:52:44.565 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.566 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.567 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.568 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:52:44.569 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:52:44.570 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 58.648974895000002, 2024-09-16T01:52:44.571 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.572 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:52:44.573 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.574 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.575 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.576 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.577 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:44.579 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:44.580 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:44.577+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:44.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.581+0000 7f1260a00640 1 Processor -- start 2024-09-16T01:52:44.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.582+0000 7f1260a00640 1 -- start start 2024-09-16T01:52:44.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1260a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f125c154c50 0x7f125c0c8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1260a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f125c0c6d40 0x7f125c0c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1260a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 0x7f125c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1260a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f125c15ac60 con 0x7f125c0c6d40 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1260a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f125c15af40 con 0x7f125c154c50 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1260a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f125c15b220 con 0x7f125c0c7390 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1259400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f125c0c6d40 0x7f125c0c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1259400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f125c0c6d40 0x7f125c0c90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45662/0 (socket says 172.21.15.146:45662) 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.585+0000 7f1259400640 1 -- 172.21.15.146:0/3918703503 learned_addr learned my addr 172.21.15.146:0/3918703503 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.586+0000 7f1259400640 1 -- 172.21.15.146:0/3918703503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 msgr2=0x7f125c0c9620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.586+0000 7f1259400640 1 --2- 172.21.15.146:0/3918703503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 0x7f125c0c9620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.586+0000 7f1259400640 1 -- 172.21.15.146:0/3918703503 >> v2:172.21.15.161:3300/0 conn(0x7f125c154c50 msgr2=0x7f125c0c8590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.586+0000 7f1259400640 1 --2- 172.21.15.146:0/3918703503 >> v2:172.21.15.161:3300/0 conn(0x7f125c154c50 0x7f125c0c8590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.586+0000 7f1259400640 1 -- 172.21.15.146:0/3918703503 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c15b500 con 0x7f125c0c6d40 2024-09-16T01:52:44.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.588+0000 7f1259400640 1 --2- 172.21.15.146:0/3918703503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c0c6d40 0x7f125c0c90e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f125c06cbf0 tx=0x7f125402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f2297f4d3f0634 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.589+0000 7f1258a00640 1 -- 172.21.15.146:0/3918703503 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254036070 con 0x7f125c0c6d40 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.590+0000 7f1258a00640 1 -- 172.21.15.146:0/3918703503 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1254031600 con 0x7f125c0c6d40 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.590+0000 7f1258a00640 1 -- 172.21.15.146:0/3918703503 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12540319d0 con 0x7f125c0c6d40 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.591+0000 7f1260a00640 1 -- 172.21.15.146:0/3918703503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c0c6d40 msgr2=0x7f125c0c90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.591+0000 7f1260a00640 1 --2- 172.21.15.146:0/3918703503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c0c6d40 0x7f125c0c90e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f125c06cbf0 tx=0x7f125402a6d0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 -- 172.21.15.146:0/3918703503 shutdown_connections 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 --2- 172.21.15.146:0/3918703503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 0x7f125c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 --2- 172.21.15.146:0/3918703503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c0c6d40 0x7f125c0c90e0 secure :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f125c06cbf0 tx=0x7f125402a6d0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 --2- 172.21.15.146:0/3918703503 >> v2:172.21.15.161:3300/0 conn(0x7f125c154c50 0x7f125c0c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 -- 172.21.15.146:0/3918703503 >> 172.21.15.146:0/3918703503 conn(0x7f125c06bc10 msgr2=0x7f125c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:44.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 -- 172.21.15.146:0/3918703503 shutdown_connections 2024-09-16T01:52:44.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.594+0000 7f1260a00640 1 -- 172.21.15.146:0/3918703503 wait complete. 2024-09-16T01:52:44.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.604+0000 7f1260a00640 1 Processor -- start 2024-09-16T01:52:44.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.605+0000 7f1260a00640 1 -- start start 2024-09-16T01:52:44.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.606+0000 7f1260a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 0x7f125c0c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.607+0000 7f1260a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c154c50 0x7f125c0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.607+0000 7f1260a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f125c1f3960 0x7f125c0ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.607+0000 7f1260a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f125c1551b0 con 0x7f125c154c50 2024-09-16T01:52:44.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.607+0000 7f1260a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f125c155030 con 0x7f125c1f3960 2024-09-16T01:52:44.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.607+0000 7f1260a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f125c06ee50 con 0x7f125c0c7390 2024-09-16T01:52:44.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.610+0000 7f125a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f125c1f3960 0x7f125c0ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.610+0000 7f125a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f125c1f3960 0x7f125c0ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:47588/0 (socket says 172.21.15.146:47588) 2024-09-16T01:52:44.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.610+0000 7f125a800640 1 -- 172.21.15.146:0/4184799397 learned_addr learned my addr 172.21.15.146:0/4184799397 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:44.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.610+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 0x7f125c0c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.611+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c154c50 0x7f125c0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.611+0000 7f1259400640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 msgr2=0x7f125c0c3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.611+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f125c0c7390 0x7f125c0c3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.611+0000 7f1259400640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f125c1f3960 msgr2=0x7f125c0ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.611+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f125c1f3960 0x7f125c0ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.611+0000 7f1259400640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c1fdbd0 con 0x7f125c154c50 2024-09-16T01:52:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.612+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c154c50 0x7f125c0cc230 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1254030000 tx=0x7f1254004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.613+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254041020 con 0x7f125c154c50 2024-09-16T01:52:44.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.613+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1254037040 con 0x7f125c154c50 2024-09-16T01:52:44.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.613+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254036070 con 0x7f125c154c50 2024-09-16T01:52:44.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.613+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125c0ca050 con 0x7f125c154c50 2024-09-16T01:52:44.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.613+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c0cba30 con 0x7f125c154c50 2024-09-16T01:52:44.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.616+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1234005270 con 0x7f125c154c50 2024-09-16T01:52:44.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.618+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1254044020 con 0x7f125c154c50 2024-09-16T01:52:44.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.619+0000 7f1252000640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f123c06c3b0 0x7f123c06e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.620+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5919+0+0 (secure 0 0 0) 0x7f12540abfe0 con 0x7f125c154c50 2024-09-16T01:52:44.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.621+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v111) ==== 54+0+16 (secure 0 0 0) 0x7f125404b030 con 0x7f125c154c50 2024-09-16T01:52:44.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.621+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f123c06c3b0 0x7f123c06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.622+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1234000fa0 0x7f12340045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.622+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1234004b20 con 0x7f1234000fa0 2024-09-16T01:52:44.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.622+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f123c06c3b0 0x7f123c06e870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1248006fd0 tx=0x7f1248008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.630+0000 7f125a800640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1234000fa0 0x7f12340045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.631+0000 7f125a800640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1234000fa0 0x7f12340045e0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.633+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1234004b20 con 0x7f1234000fa0 2024-09-16T01:52:44.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.663+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1234005090 con 0x7f1234000fa0 2024-09-16T01:52:44.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.666+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1234005090 con 0x7f1234000fa0 2024-09-16T01:52:44.669 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:44.670 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:44.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.667+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1234001c70 0x7f1234002990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.667+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f12340022b0 con 0x7f1234001c70 2024-09-16T01:52:44.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.667+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1234001c70 0x7f1234002990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.668+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7f97180c75f0 con 0x7f971806bff0 2024-09-16T01:52:44.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.668+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1234001c70 0x7f1234002990 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.669+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12340022b0 con 0x7f1234001c70 2024-09-16T01:52:44.680 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:44.678+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:44.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.712+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1234003200 con 0x7f1234001c70 2024-09-16T01:52:44.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.715+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1234003200 con 0x7f1234001c70 2024-09-16T01:52:44.718 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:44.719 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:44.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.716+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1234009cd0 0x7f123400c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.716+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f123400ee90 con 0x7f1234009cd0 2024-09-16T01:52:44.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.717+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1234009cd0 0x7f123400c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.717+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1234009cd0 0x7f123400c0c0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.718+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f123400ee90 con 0x7f1234009cd0 2024-09-16T01:52:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.750+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f123400c8d0 con 0x7f1234009cd0 2024-09-16T01:52:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.757+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f123400c8d0 con 0x7f1234009cd0 2024-09-16T01:52:44.760 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.758+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f123400c850 0x7f123400cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.758+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f123400d220 con 0x7f123400c850 2024-09-16T01:52:44.761 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:44.761 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:52:44.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.761 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:5966 3.38 3:1f0bad46:::100000001f8.0000094b:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e111)", 2024-09-16T01:52:44.761 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:52:44.724575+0000", 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 0.035971019, 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036011792000000001, 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 5966 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:52:44.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.724575+0000", 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.724575+0000", 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.724576+0000", 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.186e-06 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.745517+0000", 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.020940328000000001 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.745542+0000", 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4771999999999999e-05 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.745553+0000", 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1226999999999999e-05 2024-09-16T01:52:44.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.745638+0000", 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.4988999999999996e-05 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.745804+0000", 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000165871 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.746107+0000", 2024-09-16T01:52:44.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000302894 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:52:44.751783+0000", 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0056761299999999997 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: ], 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 1 2024-09-16T01:52:44.766 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.759+0000 7f125a800640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f123400c850 0x7f123400cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.762+0000 7f125a800640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f123400c850 0x7f123400cce0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.764+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f123400d220 con 0x7f123400c850 2024-09-16T01:52:44.772 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:44.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:52:44.789 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:44.788+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.789+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f123400d730 con 0x7f123400c850 2024-09-16T01:52:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.792+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f123400d730 con 0x7f123400c850 2024-09-16T01:52:44.794 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:44.795 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:44.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.793+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1234011050 0x7f1234013440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.793+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f123400de50 con 0x7f1234011050 2024-09-16T01:52:44.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.794+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1234011050 0x7f1234013440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.795+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1234011050 0x7f1234013440 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.797+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f123400de50 con 0x7f1234011050 2024-09-16T01:52:44.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.823+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f123400de50 con 0x7f1234011050 2024-09-16T01:52:44.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.827+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f123400de50 con 0x7f1234011050 2024-09-16T01:52:44.832 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:44.833 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:44.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.829+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1234013e90 0x7f1234016280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.829+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1234019120 con 0x7f1234013e90 2024-09-16T01:52:44.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.830+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1234013e90 0x7f1234016280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.835+0000 7f1259400640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1234013e90 0x7f1234016280 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.836+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1234019120 con 0x7f1234013e90 2024-09-16T01:52:44.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.866+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1234000950 con 0x7f1234013e90 2024-09-16T01:52:44.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.868+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1234000950 con 0x7f1234013e90 2024-09-16T01:52:44.872 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:44.872 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.870+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1234016cd0 0x7f1234017110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.870+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f123401c560 con 0x7f1234016cd0 2024-09-16T01:52:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.871+0000 7f125a800640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1234016cd0 0x7f1234017110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.872+0000 7f125a800640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1234016cd0 0x7f1234017110 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.874+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f123401c560 con 0x7f1234016cd0 2024-09-16T01:52:44.887 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:44.884+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:44.911 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:44.911 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:44.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.915+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1234017710 con 0x7f1234016cd0 2024-09-16T01:52:44.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.918+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1234017710 con 0x7f1234016cd0 2024-09-16T01:52:44.921 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:44.922 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:44.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.919+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1234017f40 0x7f1234018380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:44.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.919+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1234018b00 con 0x7f1234017f40 2024-09-16T01:52:44.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.920+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1234017f40 0x7f1234018380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:44.923 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:44.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.922+0000 7f1259e00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1234017f40 0x7f1234018380 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:44.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.923+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1234018b00 con 0x7f1234017f40 2024-09-16T01:52:44.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.946+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1234018a80 con 0x7f1234017f40 2024-09-16T01:52:44.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.948+0000 7f1252000640 1 -- 172.21.15.146:0/4184799397 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1234018a80 con 0x7f1234017f40 2024-09-16T01:52:44.950 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1234000fa0 msgr2=0x7f12340045e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f1234000fa0 0x7f12340045e0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1234001c70 msgr2=0x7f1234002990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f1234001c70 0x7f1234002990 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1234009cd0 msgr2=0x7f123400c0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1234009cd0 0x7f123400c0c0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f123400c850 msgr2=0x7f123400cce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f123400c850 0x7f123400cce0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1234011050 msgr2=0x7f1234013440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f1234011050 0x7f1234013440 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1234013e90 msgr2=0x7f1234016280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f1234013e90 0x7f1234016280 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1234016cd0 msgr2=0x7f1234017110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f1234016cd0 0x7f1234017110 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1234017f40 msgr2=0x7f1234018380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f1234017f40 0x7f1234018380 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f125a800640 1 -- 172.21.15.146:0/4184799397 reap_dead start 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f125a800640 1 -- 172.21.15.146:0/4184799397 reap_dead start 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f125a800640 1 -- 172.21.15.146:0/4184799397 reap_dead start 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f125a800640 1 -- 172.21.15.146:0/4184799397 reap_dead start 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f125a800640 1 -- 172.21.15.146:0/4184799397 reap_dead start 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f123c06c3b0 msgr2=0x7f123c06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f123c06c3b0 0x7f123c06e870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1248006fd0 tx=0x7f1248008040 comp rx=0 tx=0).stop 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c154c50 msgr2=0x7f125c0cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:44.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.949+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c154c50 0x7f125c0cc230 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1254030000 tx=0x7f1254004a40 comp rx=0 tx=0).stop 2024-09-16T01:52:44.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.951+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 shutdown_connections 2024-09-16T01:52:44.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.951+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f123c06c3b0 0x7f123c06e870 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.951+0000 7f1260a00640 1 --2- 172.21.15.146:0/4184799397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f125c154c50 0x7f125c0cc230 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:44.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.951+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 >> 172.21.15.146:0/4184799397 conn(0x7f125c06bc10 msgr2=0x7f125c0c9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:44.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.951+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 shutdown_connections 2024-09-16T01:52:44.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:44.951+0000 7f1260a00640 1 -- 172.21.15.146:0/4184799397 wait complete. 2024-09-16T01:52:44.982 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:44.981+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:45.054 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:45.054 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:45.066 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:45.084 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:45.082+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:45.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.093+0000 7f23a0e00640 1 Processor -- start 2024-09-16T01:52:45.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.094+0000 7f23a0e00640 1 -- start start 2024-09-16T01:52:45.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.095+0000 7f23a0e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f239c0cc0c0 0x7f239c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.095+0000 7f23a0e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cb2f0 0x7f239c0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.095+0000 7f23a0e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c15b1d0 con 0x7f239c15e9d0 2024-09-16T01:52:45.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.095+0000 7f23a0e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f239c0cbd00 con 0x7f239c0cc0c0 2024-09-16T01:52:45.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.095+0000 7f23a0e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f239c15ba70 con 0x7f239c0cb2f0 2024-09-16T01:52:45.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f239c0cc0c0 0x7f239c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f239c0cc0c0 0x7f239c15aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45674/0 (socket says 172.21.15.146:45674) 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 -- 172.21.15.146:0/630991066 learned_addr learned my addr 172.21.15.146:0/630991066 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399400640 1 --2- 172.21.15.146:0/630991066 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cb2f0 0x7f239c0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 -- 172.21.15.146:0/630991066 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cb2f0 msgr2=0x7f239c0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 --2- 172.21.15.146:0/630991066 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cb2f0 0x7f239c0cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 -- 172.21.15.146:0/630991066 >> v1:172.21.15.161:6789/0 conn(0x7f239c15e9d0 legacy=0x7f239c0cd0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.096+0000 7f2399e00640 1 -- 172.21.15.146:0/630991066 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c15b4b0 con 0x7f239c0cc0c0 2024-09-16T01:52:45.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.097+0000 7f2399e00640 1 --2- 172.21.15.146:0/630991066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cc0c0 0x7f239c15aab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2388009870 tx=0x7f238802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16069674bce5b511 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:45.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.098+0000 7f2398a00640 1 -- 172.21.15.146:0/630991066 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388037070 con 0x7f239c0cc0c0 2024-09-16T01:52:45.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.098+0000 7f2398a00640 1 -- 172.21.15.146:0/630991066 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2388032440 con 0x7f239c0cc0c0 2024-09-16T01:52:45.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.098+0000 7f2398a00640 1 -- 172.21.15.146:0/630991066 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388032810 con 0x7f239c0cc0c0 2024-09-16T01:52:45.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.099+0000 7f23a0e00640 1 -- 172.21.15.146:0/630991066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cc0c0 msgr2=0x7f239c15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.099+0000 7f23a0e00640 1 --2- 172.21.15.146:0/630991066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cc0c0 0x7f239c15aab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2388009870 tx=0x7f238802a210 comp rx=0 tx=0).stop 2024-09-16T01:52:45.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.100+0000 7f23a0e00640 1 -- 172.21.15.146:0/630991066 shutdown_connections 2024-09-16T01:52:45.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.100+0000 7f23a0e00640 1 --2- 172.21.15.146:0/630991066 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cb2f0 0x7f239c0cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.100+0000 7f23a0e00640 1 --2- 172.21.15.146:0/630991066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cc0c0 0x7f239c15aab0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.100+0000 7f23a0e00640 1 -- 172.21.15.146:0/630991066 >> 172.21.15.146:0/630991066 conn(0x7f239c06d560 msgr2=0x7f239c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:45.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.100+0000 7f23a0e00640 1 -- 172.21.15.146:0/630991066 shutdown_connections 2024-09-16T01:52:45.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.100+0000 7f23a0e00640 1 -- 172.21.15.146:0/630991066 wait complete. 2024-09-16T01:52:45.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.103+0000 7f23a0e00640 1 Processor -- start 2024-09-16T01:52:45.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.103+0000 7f23a0e00640 1 -- start start 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f23a0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 0x7f239c14e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f23a0e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cc0c0 0x7f239c0c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f23a0e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f239c15e9d0 0x7f239c14bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f23a0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f239c15e1d0 con 0x7f239c0cb2f0 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f23a0e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f239c15e540 con 0x7f239c15e9d0 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f23a0e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f239c0c5560 con 0x7f239c0cc0c0 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f2399e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 0x7f239c14e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f2399e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 0x7f239c14e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45686/0 (socket says 172.21.15.146:45686) 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f2399e00640 1 -- 172.21.15.146:0/2145433376 learned_addr learned my addr 172.21.15.146:0/2145433376 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:45.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.105+0000 7f2399400640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cc0c0 0x7f239c0c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f2399e00640 1 -- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cc0c0 msgr2=0x7f239c0c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f2399e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cc0c0 0x7f239c0c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f2399e00640 1 -- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f239c15e9d0 msgr2=0x7f239c14bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f239a800640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f239c15e9d0 0x7f239c14bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f2399e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f239c15e9d0 0x7f239c14bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f2399e00640 1 -- 172.21.15.146:0/2145433376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c14ccd0 con 0x7f239c0cb2f0 2024-09-16T01:52:45.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f239a800640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f239c15e9d0 0x7f239c14bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.106+0000 7f2399e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 0x7f239c14e010 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2388009990 tx=0x7f23880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.107+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388037040 con 0x7f239c0cb2f0 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.107+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2388042070 con 0x7f239c0cb2f0 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.107+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239c14cf70 con 0x7f239c0cb2f0 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.107+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23880309c0 con 0x7f239c0cb2f0 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.107+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239c14d530 con 0x7f239c0cb2f0 2024-09-16T01:52:45.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.108+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239c0cbea0 con 0x7f239c0cb2f0 2024-09-16T01:52:45.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.110+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2388030b60 con 0x7f239c0cb2f0 2024-09-16T01:52:45.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.112+0000 7f2392000640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f237c06c2b0 0x7f237c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.112+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5919+0+0 (secure 0 0 0) 0x7f23880abe50 con 0x7f239c0cb2f0 2024-09-16T01:52:45.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.112+0000 7f2399400640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f237c06c2b0 0x7f237c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.113+0000 7f2399400640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f237c06c2b0 0x7f237c06e770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f239c1f4be0 tx=0x7f238c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:45.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.116+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f238807a460 con 0x7f239c0cb2f0 2024-09-16T01:52:45.185 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:45.184+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:45.235 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:45.236 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:45.248 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:45.288 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:45.287+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:45.387 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:45.385+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:45.416 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:45.416 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:45.429 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:45.491 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:45.487+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:45.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.526+0000 7f9715600640 1 -- 172.21.15.146:0/720693176 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v112) ==== 113+0+0 (secure 0 0 0) 0x7f970407a6f0 con 0x7f971806bff0 2024-09-16T01:52:45.533 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T01:52:45.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f96f806c300 msgr2=0x7f96f806e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f96f806c300 0x7f96f806e7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f970c009970 tx=0x7f970c008040 comp rx=0 tx=0).stop 2024-09-16T01:52:45.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 msgr2=0x7f9718159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 0x7f9718159050 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9704000c00 tx=0x7f97040046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 shutdown_connections 2024-09-16T01:52:45.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f96f806c300 0x7f96f806e7c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9718071ec0 0x7f9718158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f971806bff0 0x7f9718159050 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 --2- 172.21.15.146:0/720693176 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f971806bc10 0x7f97181f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.532+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 >> 172.21.15.146:0/720693176 conn(0x7f971806f710 msgr2=0x7f97180c4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:45.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.533+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 shutdown_connections 2024-09-16T01:52:45.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.527+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f239c153610 con 0x7f239c0cb2f0 2024-09-16T01:52:45.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.534+0000 7f971ec00640 1 -- 172.21.15.146:0/720693176 wait complete. 2024-09-16T01:52:45.569 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:45.570 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:45.584 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T01:52:45.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:45.587 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:45.589 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:45.588+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:45.690 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:45.688+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:45.719 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:45.719 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:45.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:45.792 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:45.790+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:45.888 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:45.889 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:45.892 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:45.891+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:45.899 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:45.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.919+0000 7f5bf7000640 1 Processor -- start 2024-09-16T01:52:45.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.920+0000 7f5bf7000640 1 -- start start 2024-09-16T01:52:45.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.922+0000 7f5bf7000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5bf00cc0c0 0x7f5bf015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.922+0000 7f5bf7000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf00cb2f0 0x7f5bf00cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.922+0000 7f5bf7000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bf015b1d0 con 0x7f5bf015e9d0 2024-09-16T01:52:45.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.922+0000 7f5bf7000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf00cbd00 con 0x7f5bf00cb2f0 2024-09-16T01:52:45.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.922+0000 7f5bf7000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5bf015ba70 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.923+0000 7f5bef400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf00cb2f0 0x7f5bf00cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.923+0000 7f5bef400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf00cb2f0 0x7f5bf00cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45694/0 (socket says 172.21.15.146:45694) 2024-09-16T01:52:45.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.923+0000 7f5bef400640 1 -- 172.21.15.146:0/3025621329 learned_addr learned my addr 172.21.15.146:0/3025621329 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:45.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.924+0000 7f5bef400640 1 -- 172.21.15.146:0/3025621329 >> v2:172.21.15.161:3300/0 conn(0x7f5bf00cc0c0 msgr2=0x7f5bf015aab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:45.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.924+0000 7f5bef400640 1 --2- 172.21.15.146:0/3025621329 >> v2:172.21.15.161:3300/0 conn(0x7f5bf00cc0c0 0x7f5bf015aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.924+0000 7f5bef400640 1 -- 172.21.15.146:0/3025621329 >> v1:172.21.15.161:6789/0 conn(0x7f5bf015e9d0 legacy=0x7f5bf00cd0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:45.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.924+0000 7f5bef400640 1 -- 172.21.15.146:0/3025621329 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf015b4b0 con 0x7f5bf00cb2f0 2024-09-16T01:52:45.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.926+0000 7f5bef400640 1 --2- 172.21.15.146:0/3025621329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf00cc4a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00ce150 tx=0x7f5be002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a7201bfdbdf2ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:45.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.927+0000 7f5beea00640 1 -- 172.21.15.146:0/3025621329 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0037070 con 0x7f5bf00cb2f0 2024-09-16T01:52:45.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.927+0000 7f5beea00640 1 -- 172.21.15.146:0/3025621329 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be0032440 con 0x7f5bf00cb2f0 2024-09-16T01:52:45.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.928+0000 7f5beea00640 1 -- 172.21.15.146:0/3025621329 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0032810 con 0x7f5bf00cb2f0 2024-09-16T01:52:45.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.929+0000 7f5bf7000640 1 -- 172.21.15.146:0/3025621329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00cb2f0 msgr2=0x7f5bf00cc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.929+0000 7f5bf7000640 1 --2- 172.21.15.146:0/3025621329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf00cc4a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00ce150 tx=0x7f5be002a250 comp rx=0 tx=0).stop 2024-09-16T01:52:45.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.930+0000 7f5bf7000640 1 -- 172.21.15.146:0/3025621329 shutdown_connections 2024-09-16T01:52:45.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.930+0000 7f5bf7000640 1 --2- 172.21.15.146:0/3025621329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf00cc4a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.930+0000 7f5bf7000640 1 --2- 172.21.15.146:0/3025621329 >> v2:172.21.15.161:3300/0 conn(0x7f5bf00cc0c0 0x7f5bf015aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.930+0000 7f5bf7000640 1 -- 172.21.15.146:0/3025621329 >> 172.21.15.146:0/3025621329 conn(0x7f5bf006d560 msgr2=0x7f5bf006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:45.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.932+0000 7f5bf7000640 1 -- 172.21.15.146:0/3025621329 shutdown_connections 2024-09-16T01:52:45.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.932+0000 7f5bf7000640 1 -- 172.21.15.146:0/3025621329 wait complete. 2024-09-16T01:52:45.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.934+0000 7f5bf7000640 1 Processor -- start 2024-09-16T01:52:45.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.935+0000 7f5bf7000640 1 -- start start 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.936+0000 7f5bf7000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.936+0000 7f5bf7000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 0x7f5bf01f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.936+0000 7f5bf7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015e9d0 0x7f5bf01f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.936+0000 7f5bf7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf015e1d0 con 0x7f5bf015e9d0 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.936+0000 7f5bf7000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf015e540 con 0x7f5bf00cb2f0 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.936+0000 7f5bf7000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf00c5560 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.937+0000 7f5bef400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 0x7f5bf01f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.937+0000 7f5bef400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 0x7f5bf01f0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38174/0 (socket says 172.21.15.146:38174) 2024-09-16T01:52:45.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.937+0000 7f5bef400640 1 -- 172.21.15.146:0/1300468207 learned_addr learned my addr 172.21.15.146:0/1300468207 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:45.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.937+0000 7f5bf5000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015e9d0 0x7f5bf01f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.937+0000 7f5befe00640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bef400640 1 -- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf00cb2f0 msgr2=0x7f5bf015d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bef400640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf015d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bef400640 1 -- 172.21.15.146:0/1300468207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015e9d0 msgr2=0x7f5bf01f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:45.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bef400640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015e9d0 0x7f5bf01f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:45.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bef400640 1 -- 172.21.15.146:0/1300468207 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf01f4960 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bf5000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015e9d0 0x7f5bf01f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.938+0000 7f5bef400640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 0x7f5bf01f0b60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009960 tx=0x7f5be00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.939+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0041070 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.939+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5be003c950 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.939+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be0037040 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.939+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf01f4bd0 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.939+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf015a6f0 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.940+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf00d0ba0 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.942+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5be0044070 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.944+0000 7f5be7e00640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bd006c2e0 0x7f5bd006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:45.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.944+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 5919+0+0 (secure 0 0 0) 0x7f5be00ac080 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.944+0000 7f5befe00640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bd006c2e0 0x7f5bd006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:45.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.945+0000 7f5befe00640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bd006c2e0 0x7f5bd006e7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009fd0 tx=0x7f5be800d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:45.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:45.948+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5bf00d0ba0 con 0x7f5bf00cc0c0 2024-09-16T01:52:45.994 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:45.992+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:46.062 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:46.062 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:46.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:46.093 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:46.092+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:46.194 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:46.193+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:46.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.279+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5bf00cbea0 con 0x7f5bd006c2e0 2024-09-16T01:52:46.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.287+0000 7f5be7e00640 1 -- 172.21.15.146:0/1300468207 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254760 (secure 0 0 0) 0x7f5bf00cbea0 con 0x7f5bd006c2e0 2024-09-16T01:52:46.292 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:46.293 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:46.295 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:46.294+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:46.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bd006c2e0 msgr2=0x7f5bd006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bd006c2e0 0x7f5bd006e7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009fd0 tx=0x7f5be800d040 comp rx=0 tx=0).stop 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 msgr2=0x7f5bf01f0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 0x7f5bf01f0b60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009960 tx=0x7f5be00046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 shutdown_connections 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5bd006c2e0 0x7f5bd006e7a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf015e9d0 0x7f5bf01f10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5bf00cc0c0 0x7f5bf01f0b60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 --2- 172.21.15.146:0/1300468207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf00cb2f0 0x7f5bf015d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.292+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 >> 172.21.15.146:0/1300468207 conn(0x7f5bf006d560 msgr2=0x7f5bf00cd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.295+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 shutdown_connections 2024-09-16T01:52:46.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.295+0000 7f5bf7000640 1 -- 172.21.15.146:0/1300468207 wait complete. 2024-09-16T01:52:46.324 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-16T01:52:45.526478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":4,"num_objects_missing":7,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":6767,"num_write_kb":23366568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2340,"num_bytes_recovered":7008928941,"num_keys_recovered":115,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7208,"ondisk_log_size":7208,"up":194,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":197,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1538724,"kb_used_data":1322224,"kb_used_omap":9,"kb_used_meta":216118,"kb_avail":564692316,"statfs":{"total":579820584960,"available":578244931584,"internally_reserved":0,"allocated":1353957376,"data_stored":21532727038,"data_compressed":1065851591,"data_compressed_allocated":1345589248,"data_compressed_original":21528645632,"omap_allocated":9562,"internal_metadata":221305510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.820712"},"pg_stats":[{"pgid":"3.3f","version":"105'98","reported_seq":360,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712176+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:40.712176+0000","last_peered":"2024-09-16T01:52:40.712176+0000","last_clean":"2024-09-16T01:52:40.712176+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:40.712176+0000","last_undegraded":"2024-09-16T01:52:40.712176+0000","last_fullsized":"2024-09-16T01:52:40.712176+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"105'86","reported_seq":267,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720915+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:39.720915+0000","last_peered":"2024-09-16T01:52:39.720915+0000","last_clean":"2024-09-16T01:52:39.720915+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:39.720915+0000","last_undegraded":"2024-09-16T01:52:39.720915+0000","last_fullsized":"2024-09-16T01:52:39.720915+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"106'105","reported_seq":431,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.784965+0000","last_change":"2024-09-16T01:52:05.835179+0000","last_active":"2024-09-16T01:52:39.784965+0000","last_peered":"2024-09-16T01:52:39.784965+0000","last_clean":"2024-09-16T01:52:39.784965+0000","last_became_active":"2024-09-16T01:52:03.309937+0000","last_became_peered":"2024-09-16T01:52:03.309937+0000","last_unstale":"2024-09-16T01:52:39.784965+0000","last_undegraded":"2024-09-16T01:52:39.784965+0000","last_fullsized":"2024-09-16T01:52:39.784965+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":209715200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"107'120","reported_seq":308,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702041+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:40.702041+0000","last_peered":"2024-09-16T01:52:40.702041+0000","last_clean":"2024-09-16T01:52:40.702041+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:40.702041+0000","last_undegraded":"2024-09-16T01:52:40.702041+0000","last_fullsized":"2024-09-16T01:52:40.702041+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"106'103","reported_seq":256,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720610+0000","last_change":"2024-09-16T01:52:14.104327+0000","last_active":"2024-09-16T01:52:39.720610+0000","last_peered":"2024-09-16T01:52:39.720610+0000","last_clean":"2024-09-16T01:52:39.720610+0000","last_became_active":"2024-09-16T01:52:09.424479+0000","last_became_peered":"2024-09-16T01:52:09.424479+0000","last_unstale":"2024-09-16T01:52:39.720610+0000","last_undegraded":"2024-09-16T01:52:39.720610+0000","last_fullsized":"2024-09-16T01:52:39.720610+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"106'96","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702170+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:40.702170+0000","last_peered":"2024-09-16T01:52:40.702170+0000","last_clean":"2024-09-16T01:52:40.702170+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:40.702170+0000","last_undegraded":"2024-09-16T01:52:40.702170+0000","last_fullsized":"2024-09-16T01:52:40.702170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"107'120","reported_seq":477,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.115504+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:41.115504+0000","last_peered":"2024-09-16T01:52:41.115504+0000","last_clean":"2024-09-16T01:52:41.115504+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:41.115504+0000","last_undegraded":"2024-09-16T01:52:41.115504+0000","last_fullsized":"2024-09-16T01:52:41.115504+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"106'115","reported_seq":421,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.100907+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:40.100907+0000","last_peered":"2024-09-16T01:52:40.100907+0000","last_clean":"2024-09-16T01:52:40.100907+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:40.100907+0000","last_undegraded":"2024-09-16T01:52:40.100907+0000","last_fullsized":"2024-09-16T01:52:40.100907+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"107'108","reported_seq":291,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.840917+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:40.840917+0000","last_peered":"2024-09-16T01:52:40.840917+0000","last_clean":"2024-09-16T01:52:40.840917+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:40.840917+0000","last_undegraded":"2024-09-16T01:52:40.840917+0000","last_fullsized":"2024-09-16T01:52:40.840917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"107'108","reported_seq":395,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.979932+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:40.979932+0000","last_peered":"2024-09-16T01:52:40.979932+0000","last_clean":"2024-09-16T01:52:40.979932+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:40.979932+0000","last_undegraded":"2024-09-16T01:52:40.979932+0000","last_fullsized":"2024-09-16T01:52:40.979932+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"107'130","reported_seq":442,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714130+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:40.714130+0000","last_peered":"2024-09-16T01:52:40.714130+0000","last_clean":"2024-09-16T01:52:40.714130+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:40.714130+0000","last_undegraded":"2024-09-16T01:52:40.714130+0000","last_fullsized":"2024-09-16T01:52:40.714130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"107'101","reported_seq":92,"reported_epoch":107,"state":"active+recovering","last_fresh":"2024-09-16T01:52:41.593897+0000","last_change":"2024-09-16T01:52:40.740404+0000","last_active":"2024-09-16T01:52:41.593897+0000","last_peered":"2024-09-16T01:52:41.593897+0000","last_clean":"2024-09-16T01:52:36.904016+0000","last_became_active":"2024-09-16T01:52:40.724710+0000","last_became_peered":"2024-09-16T01:52:40.724710+0000","last_unstale":"2024-09-16T01:52:41.593897+0000","last_undegraded":"2024-09-16T01:52:41.593897+0000","last_fullsized":"2024-09-16T01:52:41.593897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":452984832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"1,2","objects":28},{"shards":"1,2,6","objects":2},{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"106'109","reported_seq":413,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.695082+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:39.695082+0000","last_peered":"2024-09-16T01:52:39.695082+0000","last_clean":"2024-09-16T01:52:39.695082+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:39.695082+0000","last_undegraded":"2024-09-16T01:52:39.695082+0000","last_fullsized":"2024-09-16T01:52:39.695082+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"106'100","reported_seq":331,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.694733+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:39.694733+0000","last_peered":"2024-09-16T01:52:39.694733+0000","last_clean":"2024-09-16T01:52:39.694733+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:39.694733+0000","last_undegraded":"2024-09-16T01:52:39.694733+0000","last_fullsized":"2024-09-16T01:52:39.694733+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"105'107","reported_seq":391,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720358+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:39.720358+0000","last_peered":"2024-09-16T01:52:39.720358+0000","last_clean":"2024-09-16T01:52:39.720358+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:39.720358+0000","last_undegraded":"2024-09-16T01:52:39.720358+0000","last_fullsized":"2024-09-16T01:52:39.720358+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"107'102","reported_seq":284,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708519+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:40.708519+0000","last_peered":"2024-09-16T01:52:40.708519+0000","last_clean":"2024-09-16T01:52:40.708519+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:40.708519+0000","last_undegraded":"2024-09-16T01:52:40.708519+0000","last_fullsized":"2024-09-16T01:52:40.708519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"106'98","reported_seq":377,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712476+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:40.712476+0000","last_peered":"2024-09-16T01:52:40.712476+0000","last_clean":"2024-09-16T01:52:40.712476+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:40.712476+0000","last_undegraded":"2024-09-16T01:52:40.712476+0000","last_fullsized":"2024-09-16T01:52:40.712476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"107'103","reported_seq":371,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.454439+0000","last_change":"2024-09-16T01:52:05.041021+0000","last_active":"2024-09-16T01:52:41.454439+0000","last_peered":"2024-09-16T01:52:41.454439+0000","last_clean":"2024-09-16T01:52:41.454439+0000","last_became_active":"2024-09-16T01:52:03.349420+0000","last_became_peered":"2024-09-16T01:52:03.349420+0000","last_unstale":"2024-09-16T01:52:41.454439+0000","last_undegraded":"2024-09-16T01:52:41.454439+0000","last_fullsized":"2024-09-16T01:52:41.454439+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:31.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"104'115","reported_seq":352,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T01:52:35.045658+0000","last_change":"2024-09-16T01:51:50.466227+0000","last_active":"2024-09-16T01:52:35.045658+0000","last_peered":"2024-09-16T01:52:35.045658+0000","last_clean":"2024-09-16T01:52:35.045658+0000","last_became_active":"2024-09-16T01:51:49.177542+0000","last_became_peered":"2024-09-16T01:51:49.177542+0000","last_unstale":"2024-09-16T01:52:35.045658+0000","last_undegraded":"2024-09-16T01:52:35.045658+0000","last_fullsized":"2024-09-16T01:52:35.045658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:14.445720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":88080384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"107'108","reported_seq":434,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.043648+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:41.043648+0000","last_peered":"2024-09-16T01:52:41.043648+0000","last_clean":"2024-09-16T01:52:41.043648+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:41.043648+0000","last_undegraded":"2024-09-16T01:52:41.043648+0000","last_fullsized":"2024-09-16T01:52:41.043648+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"106'83","reported_seq":303,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705785+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:40.705785+0000","last_peered":"2024-09-16T01:52:40.705785+0000","last_clean":"2024-09-16T01:52:40.705785+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:40.705785+0000","last_undegraded":"2024-09-16T01:52:40.705785+0000","last_fullsized":"2024-09-16T01:52:40.705785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"105'120","reported_seq":432,"reported_epoch":105,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:39.297428+0000","last_change":"2024-09-16T01:52:40.701785+0000","last_active":"2024-09-16T01:52:40.701785+0000","last_peered":"2024-09-16T01:52:39.297428+0000","last_clean":"2024-09-16T01:52:39.297428+0000","last_became_active":"2024-09-16T01:52:09.425675+0000","last_became_peered":"2024-09-16T01:52:09.425675+0000","last_unstale":"2024-09-16T01:52:39.297428+0000","last_undegraded":"2024-09-16T01:52:39.297428+0000","last_fullsized":"2024-09-16T01:52:39.297428+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":197132288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"105'91","reported_seq":272,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720565+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:39.720565+0000","last_peered":"2024-09-16T01:52:39.720565+0000","last_clean":"2024-09-16T01:52:39.720565+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:39.720565+0000","last_undegraded":"2024-09-16T01:52:39.720565+0000","last_fullsized":"2024-09-16T01:52:39.720565+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"107'127","reported_seq":309,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708637+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:40.708637+0000","last_peered":"2024-09-16T01:52:40.708637+0000","last_clean":"2024-09-16T01:52:40.708637+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:40.708637+0000","last_undegraded":"2024-09-16T01:52:40.708637+0000","last_fullsized":"2024-09-16T01:52:40.708637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":454656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"106'92","reported_seq":273,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720968+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:39.720968+0000","last_peered":"2024-09-16T01:52:39.720968+0000","last_clean":"2024-09-16T01:52:39.720968+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:39.720968+0000","last_undegraded":"2024-09-16T01:52:39.720968+0000","last_fullsized":"2024-09-16T01:52:39.720968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"107'133","reported_seq":316,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.872103+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:40.872103+0000","last_peered":"2024-09-16T01:52:40.872103+0000","last_clean":"2024-09-16T01:52:40.872103+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:40.872103+0000","last_undegraded":"2024-09-16T01:52:40.872103+0000","last_fullsized":"2024-09-16T01:52:40.872103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"106'100","reported_seq":410,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.880720+0000","last_change":"2024-09-16T01:52:04.750394+0000","last_active":"2024-09-16T01:52:39.880720+0000","last_peered":"2024-09-16T01:52:39.880720+0000","last_clean":"2024-09-16T01:52:39.880720+0000","last_became_active":"2024-09-16T01:52:03.375641+0000","last_became_peered":"2024-09-16T01:52:03.375641+0000","last_unstale":"2024-09-16T01:52:39.880720+0000","last_undegraded":"2024-09-16T01:52:39.880720+0000","last_fullsized":"2024-09-16T01:52:39.880720+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"106'95","reported_seq":369,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.695354+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:39.695354+0000","last_peered":"2024-09-16T01:52:39.695354+0000","last_clean":"2024-09-16T01:52:39.695354+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:39.695354+0000","last_undegraded":"2024-09-16T01:52:39.695354+0000","last_fullsized":"2024-09-16T01:52:39.695354+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"106'83","reported_seq":466,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.543508+0000","last_change":"2024-09-16T01:52:10.737249+0000","last_active":"2024-09-16T01:52:40.543508+0000","last_peered":"2024-09-16T01:52:40.543508+0000","last_clean":"2024-09-16T01:52:40.543508+0000","last_became_active":"2024-09-16T01:52:09.378210+0000","last_became_peered":"2024-09-16T01:52:09.378210+0000","last_unstale":"2024-09-16T01:52:40.543508+0000","last_undegraded":"2024-09-16T01:52:40.543508+0000","last_fullsized":"2024-09-16T01:52:40.543508+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:32.800278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"106'94","reported_seq":276,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709371+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:40.709371+0000","last_peered":"2024-09-16T01:52:40.709371+0000","last_clean":"2024-09-16T01:52:40.709371+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:40.709371+0000","last_undegraded":"2024-09-16T01:52:40.709371+0000","last_fullsized":"2024-09-16T01:52:40.709371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"106'96","reported_seq":382,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.704088+0000","last_change":"2024-09-16T01:52:22.869283+0000","last_active":"2024-09-16T01:52:40.704088+0000","last_peered":"2024-09-16T01:52:40.704088+0000","last_clean":"2024-09-16T01:52:40.704088+0000","last_became_active":"2024-09-16T01:52:21.489999+0000","last_became_peered":"2024-09-16T01:52:21.489999+0000","last_unstale":"2024-09-16T01:52:40.704088+0000","last_undegraded":"2024-09-16T01:52:40.704088+0000","last_fullsized":"2024-09-16T01:52:40.704088+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"104'102","reported_seq":289,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T01:52:35.726439+0000","last_change":"2024-09-16T01:50:49.729820+0000","last_active":"2024-09-16T01:52:35.726439+0000","last_peered":"2024-09-16T01:52:35.726439+0000","last_clean":"2024-09-16T01:52:35.726439+0000","last_became_active":"2024-09-16T01:50:49.729453+0000","last_became_peered":"2024-09-16T01:50:49.729453+0000","last_unstale":"2024-09-16T01:52:35.726439+0000","last_undegraded":"2024-09-16T01:52:35.726439+0000","last_fullsized":"2024-09-16T01:52:35.726439+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:45.406298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"107'93","reported_seq":276,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705218+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:40.705218+0000","last_peered":"2024-09-16T01:52:40.705218+0000","last_clean":"2024-09-16T01:52:40.705218+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:40.705218+0000","last_undegraded":"2024-09-16T01:52:40.705218+0000","last_fullsized":"2024-09-16T01:52:40.705218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":188,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720646+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:39.720646+0000","last_peered":"2024-09-16T01:52:39.720646+0000","last_clean":"2024-09-16T01:52:39.720646+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:39.720646+0000","last_undegraded":"2024-09-16T01:52:39.720646+0000","last_fullsized":"2024-09-16T01:52:39.720646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"106'99","reported_seq":519,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715559+0000","last_change":"2024-09-16T01:52:22.369179+0000","last_active":"2024-09-16T01:52:40.715559+0000","last_peered":"2024-09-16T01:52:40.715559+0000","last_clean":"2024-09-16T01:52:40.715559+0000","last_became_active":"2024-09-16T01:52:21.480658+0000","last_became_peered":"2024-09-16T01:52:21.480658+0000","last_unstale":"2024-09-16T01:52:40.715559+0000","last_undegraded":"2024-09-16T01:52:40.715559+0000","last_fullsized":"2024-09-16T01:52:40.715559+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":204,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709632+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:40.709632+0000","last_peered":"2024-09-16T01:52:40.709632+0000","last_clean":"2024-09-16T01:52:40.709632+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:40.709632+0000","last_undegraded":"2024-09-16T01:52:40.709632+0000","last_fullsized":"2024-09-16T01:52:40.709632+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":197,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.694607+0000","last_change":"2024-09-16T01:52:09.445036+0000","last_active":"2024-09-16T01:52:39.694607+0000","last_peered":"2024-09-16T01:52:39.694607+0000","last_clean":"2024-09-16T01:52:39.694607+0000","last_became_active":"2024-09-16T01:52:09.417952+0000","last_became_peered":"2024-09-16T01:52:09.417952+0000","last_unstale":"2024-09-16T01:52:39.694607+0000","last_undegraded":"2024-09-16T01:52:39.694607+0000","last_fullsized":"2024-09-16T01:52:39.694607+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:55.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"106'115","reported_seq":296,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.322386+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:40.322386+0000","last_peered":"2024-09-16T01:52:40.322386+0000","last_clean":"2024-09-16T01:52:40.322386+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:40.322386+0000","last_undegraded":"2024-09-16T01:52:40.322386+0000","last_fullsized":"2024-09-16T01:52:40.322386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":187,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708659+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:40.708659+0000","last_peered":"2024-09-16T01:52:40.708659+0000","last_clean":"2024-09-16T01:52:40.708659+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:40.708659+0000","last_undegraded":"2024-09-16T01:52:40.708659+0000","last_fullsized":"2024-09-16T01:52:40.708659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"105'106","reported_seq":254,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.703982+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:40.703982+0000","last_peered":"2024-09-16T01:52:40.703982+0000","last_clean":"2024-09-16T01:52:40.703982+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:40.703982+0000","last_undegraded":"2024-09-16T01:52:40.703982+0000","last_fullsized":"2024-09-16T01:52:40.703982+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":216,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.722262+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:40.722262+0000","last_peered":"2024-09-16T01:52:40.722262+0000","last_clean":"2024-09-16T01:52:40.722262+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:40.722262+0000","last_undegraded":"2024-09-16T01:52:40.722262+0000","last_fullsized":"2024-09-16T01:52:40.722262+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"104'148","reported_seq":325,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T01:52:35.568006+0000","last_change":"2024-09-16T01:48:58.831807+0000","last_active":"2024-09-16T01:52:35.568006+0000","last_peered":"2024-09-16T01:52:35.568006+0000","last_clean":"2024-09-16T01:52:35.568006+0000","last_became_active":"2024-09-16T01:48:58.830684+0000","last_became_peered":"2024-09-16T01:48:58.830684+0000","last_unstale":"2024-09-16T01:52:35.568006+0000","last_undegraded":"2024-09-16T01:52:35.568006+0000","last_fullsized":"2024-09-16T01:52:35.568006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:50.038247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":520192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":67,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715962+0000","last_change":"2024-09-16T01:52:21.510976+0000","last_active":"2024-09-16T01:52:40.715962+0000","last_peered":"2024-09-16T01:52:40.715962+0000","last_clean":"2024-09-16T01:52:40.715962+0000","last_became_active":"2024-09-16T01:52:21.491849+0000","last_became_peered":"2024-09-16T01:52:21.491849+0000","last_unstale":"2024-09-16T01:52:40.715962+0000","last_undegraded":"2024-09-16T01:52:40.715962+0000","last_fullsized":"2024-09-16T01:52:40.715962+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:43.042548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"106'95","reported_seq":276,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.721067+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:39.721067+0000","last_peered":"2024-09-16T01:52:39.721067+0000","last_clean":"2024-09-16T01:52:39.721067+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:39.721067+0000","last_undegraded":"2024-09-16T01:52:39.721067+0000","last_fullsized":"2024-09-16T01:52:39.721067+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.713822+0000","last_change":"2024-09-16T01:52:21.507954+0000","last_active":"2024-09-16T01:52:40.713822+0000","last_peered":"2024-09-16T01:52:40.713822+0000","last_clean":"2024-09-16T01:52:40.713822+0000","last_became_active":"2024-09-16T01:52:21.491632+0000","last_became_peered":"2024-09-16T01:52:21.491632+0000","last_unstale":"2024-09-16T01:52:40.713822+0000","last_undegraded":"2024-09-16T01:52:40.713822+0000","last_fullsized":"2024-09-16T01:52:40.713822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:36.763063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3360,"num_keys_recovered":50,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"107'101","reported_seq":283,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.710272+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:40.710272+0000","last_peered":"2024-09-16T01:52:40.710272+0000","last_clean":"2024-09-16T01:52:40.710272+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:40.710272+0000","last_undegraded":"2024-09-16T01:52:40.710272+0000","last_fullsized":"2024-09-16T01:52:40.710272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":374,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702091+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:40.702091+0000","last_peered":"2024-09-16T01:52:40.702091+0000","last_clean":"2024-09-16T01:52:40.702091+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:40.702091+0000","last_undegraded":"2024-09-16T01:52:40.702091+0000","last_fullsized":"2024-09-16T01:52:40.702091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"106'105","reported_seq":389,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.067219+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:40.067219+0000","last_peered":"2024-09-16T01:52:40.067219+0000","last_clean":"2024-09-16T01:52:40.067219+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:40.067219+0000","last_undegraded":"2024-09-16T01:52:40.067219+0000","last_fullsized":"2024-09-16T01:52:40.067219+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":244,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715431+0000","last_change":"2024-09-16T01:52:09.471855+0000","last_active":"2024-09-16T01:52:40.715431+0000","last_peered":"2024-09-16T01:52:40.715431+0000","last_clean":"2024-09-16T01:52:40.715431+0000","last_became_active":"2024-09-16T01:52:09.419033+0000","last_became_peered":"2024-09-16T01:52:09.419033+0000","last_unstale":"2024-09-16T01:52:40.715431+0000","last_undegraded":"2024-09-16T01:52:40.715431+0000","last_fullsized":"2024-09-16T01:52:40.715431+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"106'110","reported_seq":291,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.492374+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:40.492374+0000","last_peered":"2024-09-16T01:52:40.492374+0000","last_clean":"2024-09-16T01:52:40.492374+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:40.492374+0000","last_undegraded":"2024-09-16T01:52:40.492374+0000","last_fullsized":"2024-09-16T01:52:40.492374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":254,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709719+0000","last_change":"2024-09-16T01:52:03.470007+0000","last_active":"2024-09-16T01:52:40.709719+0000","last_peered":"2024-09-16T01:52:40.709719+0000","last_clean":"2024-09-16T01:52:40.709719+0000","last_became_active":"2024-09-16T01:52:03.348785+0000","last_became_peered":"2024-09-16T01:52:03.348785+0000","last_unstale":"2024-09-16T01:52:40.709719+0000","last_undegraded":"2024-09-16T01:52:40.709719+0000","last_fullsized":"2024-09-16T01:52:40.709719+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"106'114","reported_seq":427,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715670+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:40.715670+0000","last_peered":"2024-09-16T01:52:40.715670+0000","last_clean":"2024-09-16T01:52:40.715670+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:40.715670+0000","last_undegraded":"2024-09-16T01:52:40.715670+0000","last_fullsized":"2024-09-16T01:52:40.715670+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":61,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708931+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:40.708931+0000","last_peered":"2024-09-16T01:52:40.708931+0000","last_clean":"2024-09-16T01:52:40.708931+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:40.708931+0000","last_undegraded":"2024-09-16T01:52:40.708931+0000","last_fullsized":"2024-09-16T01:52:40.708931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"106'128","reported_seq":311,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701595+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:40.701595+0000","last_peered":"2024-09-16T01:52:40.701595+0000","last_clean":"2024-09-16T01:52:40.701595+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:40.701595+0000","last_undegraded":"2024-09-16T01:52:40.701595+0000","last_fullsized":"2024-09-16T01:52:40.701595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":97,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701198+0000","last_change":"2024-09-16T01:52:21.494700+0000","last_active":"2024-09-16T01:52:40.701198+0000","last_peered":"2024-09-16T01:52:40.701198+0000","last_clean":"2024-09-16T01:52:40.701198+0000","last_became_active":"2024-09-16T01:52:21.466194+0000","last_became_peered":"2024-09-16T01:52:21.466194+0000","last_unstale":"2024-09-16T01:52:40.701198+0000","last_undegraded":"2024-09-16T01:52:40.701198+0000","last_fullsized":"2024-09-16T01:52:40.701198+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:55.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"106'139","reported_seq":321,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709029+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:40.709029+0000","last_peered":"2024-09-16T01:52:40.709029+0000","last_clean":"2024-09-16T01:52:40.709029+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:40.709029+0000","last_undegraded":"2024-09-16T01:52:40.709029+0000","last_fullsized":"2024-09-16T01:52:40.709029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":56,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720892+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:39.720892+0000","last_peered":"2024-09-16T01:52:39.720892+0000","last_clean":"2024-09-16T01:52:39.720892+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:39.720892+0000","last_undegraded":"2024-09-16T01:52:39.720892+0000","last_fullsized":"2024-09-16T01:52:39.720892+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"107'99","reported_seq":383,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.373880+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:41.373880+0000","last_peered":"2024-09-16T01:52:41.373880+0000","last_clean":"2024-09-16T01:52:41.373880+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:41.373880+0000","last_undegraded":"2024-09-16T01:52:41.373880+0000","last_fullsized":"2024-09-16T01:52:41.373880+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.721004+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:39.721004+0000","last_peered":"2024-09-16T01:52:39.721004+0000","last_clean":"2024-09-16T01:52:39.721004+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:39.721004+0000","last_undegraded":"2024-09-16T01:52:39.721004+0000","last_fullsized":"2024-09-16T01:52:39.721004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"107'96","reported_seq":375,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.258476+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:41.258476+0000","last_peered":"2024-09-16T01:52:41.258476+0000","last_clean":"2024-09-16T01:52:41.258476+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:41.258476+0000","last_undegraded":"2024-09-16T01:52:41.258476+0000","last_fullsized":"2024-09-16T01:52:41.258476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720853+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:39.720853+0000","last_peered":"2024-09-16T01:52:39.720853+0000","last_clean":"2024-09-16T01:52:39.720853+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:39.720853+0000","last_undegraded":"2024-09-16T01:52:39.720853+0000","last_fullsized":"2024-09-16T01:52:39.720853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"107'95","reported_seq":504,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.953377+0000","last_change":"2024-09-16T01:52:10.143229+0000","last_active":"2024-09-16T01:52:40.953377+0000","last_peered":"2024-09-16T01:52:40.953377+0000","last_clean":"2024-09-16T01:52:40.953377+0000","last_became_active":"2024-09-16T01:52:09.413639+0000","last_became_peered":"2024-09-16T01:52:09.413639+0000","last_unstale":"2024-09-16T01:52:40.953377+0000","last_undegraded":"2024-09-16T01:52:40.953377+0000","last_fullsized":"2024-09-16T01:52:40.953377+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":318767104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":201,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.695397+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:39.695397+0000","last_peered":"2024-09-16T01:52:39.695397+0000","last_clean":"2024-09-16T01:52:39.695397+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:39.695397+0000","last_undegraded":"2024-09-16T01:52:39.695397+0000","last_fullsized":"2024-09-16T01:52:39.695397+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"107'105","reported_seq":287,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714360+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:40.714360+0000","last_peered":"2024-09-16T01:52:40.714360+0000","last_clean":"2024-09-16T01:52:40.714360+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:40.714360+0000","last_undegraded":"2024-09-16T01:52:40.714360+0000","last_fullsized":"2024-09-16T01:52:40.714360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":341,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701702+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:40.701702+0000","last_peered":"2024-09-16T01:52:40.701702+0000","last_clean":"2024-09-16T01:52:40.701702+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:40.701702+0000","last_undegraded":"2024-09-16T01:52:40.701702+0000","last_fullsized":"2024-09-16T01:52:40.701702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":190,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715864+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:40.715864+0000","last_peered":"2024-09-16T01:52:40.715864+0000","last_clean":"2024-09-16T01:52:40.715864+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:40.715864+0000","last_undegraded":"2024-09-16T01:52:40.715864+0000","last_fullsized":"2024-09-16T01:52:40.715864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"107'106","reported_seq":429,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.741036+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:40.741036+0000","last_peered":"2024-09-16T01:52:40.741036+0000","last_clean":"2024-09-16T01:52:40.741036+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:40.741036+0000","last_undegraded":"2024-09-16T01:52:40.741036+0000","last_fullsized":"2024-09-16T01:52:40.741036+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":188,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708690+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:40.708690+0000","last_peered":"2024-09-16T01:52:40.708690+0000","last_clean":"2024-09-16T01:52:40.708690+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:40.708690+0000","last_undegraded":"2024-09-16T01:52:40.708690+0000","last_fullsized":"2024-09-16T01:52:40.708690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"105'101","reported_seq":282,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.694488+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:39.694488+0000","last_peered":"2024-09-16T01:52:39.694488+0000","last_clean":"2024-09-16T01:52:39.694488+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:39.694488+0000","last_undegraded":"2024-09-16T01:52:39.694488+0000","last_fullsized":"2024-09-16T01:52:39.694488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":229,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720421+0000","last_change":"2024-09-16T01:52:21.518958+0000","last_active":"2024-09-16T01:52:39.720421+0000","last_peered":"2024-09-16T01:52:39.720421+0000","last_clean":"2024-09-16T01:52:39.720421+0000","last_became_active":"2024-09-16T01:52:21.493366+0000","last_became_peered":"2024-09-16T01:52:21.493366+0000","last_unstale":"2024-09-16T01:52:39.720421+0000","last_undegraded":"2024-09-16T01:52:39.720421+0000","last_fullsized":"2024-09-16T01:52:39.720421+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:12.952182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"105'91","reported_seq":397,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720421+0000","last_change":"2024-09-16T01:52:07.490944+0000","last_active":"2024-09-16T01:52:39.720421+0000","last_peered":"2024-09-16T01:52:39.720421+0000","last_clean":"2024-09-16T01:52:39.720421+0000","last_became_active":"2024-09-16T01:52:03.344719+0000","last_became_peered":"2024-09-16T01:52:03.344719+0000","last_unstale":"2024-09-16T01:52:39.720421+0000","last_undegraded":"2024-09-16T01:52:39.720421+0000","last_fullsized":"2024-09-16T01:52:39.720421+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:05.418025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":150994944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":189,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709793+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:40.709793+0000","last_peered":"2024-09-16T01:52:40.709793+0000","last_clean":"2024-09-16T01:52:40.709793+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:40.709793+0000","last_undegraded":"2024-09-16T01:52:40.709793+0000","last_fullsized":"2024-09-16T01:52:40.709793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"106'107","reported_seq":389,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705561+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:40.705561+0000","last_peered":"2024-09-16T01:52:40.705561+0000","last_clean":"2024-09-16T01:52:40.705561+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:40.705561+0000","last_undegraded":"2024-09-16T01:52:40.705561+0000","last_fullsized":"2024-09-16T01:52:40.705561+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720666+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:39.720666+0000","last_peered":"2024-09-16T01:52:39.720666+0000","last_clean":"2024-09-16T01:52:39.720666+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:39.720666+0000","last_undegraded":"2024-09-16T01:52:39.720666+0000","last_fullsized":"2024-09-16T01:52:39.720666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"107'105","reported_seq":288,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708603+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:40.708603+0000","last_peered":"2024-09-16T01:52:40.708603+0000","last_clean":"2024-09-16T01:52:40.708603+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:40.708603+0000","last_undegraded":"2024-09-16T01:52:40.708603+0000","last_fullsized":"2024-09-16T01:52:40.708603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":2,"reported_epoch":107,"state":"peering","last_fresh":"2024-09-16T01:52:39.700468+0000","last_change":"2024-09-16T01:52:41.657743+0000","last_active":"2024-09-16T01:52:41.657743+0000","last_peered":"2024-09-16T01:52:39.700468+0000","last_clean":"2024-09-16T01:52:39.700468+0000","last_became_active":"2024-09-16T01:51:49.156266+0000","last_became_peered":"2024-09-16T01:51:49.156266+0000","last_unstale":"2024-09-16T01:52:39.700468+0000","last_undegraded":"2024-09-16T01:52:39.700468+0000","last_fullsized":"2024-09-16T01:52:39.700468+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"106'90","reported_seq":271,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.695443+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:39.695443+0000","last_peered":"2024-09-16T01:52:39.695443+0000","last_clean":"2024-09-16T01:52:39.695443+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:39.695443+0000","last_undegraded":"2024-09-16T01:52:39.695443+0000","last_fullsized":"2024-09-16T01:52:39.695443+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720860+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:39.720860+0000","last_peered":"2024-09-16T01:52:39.720860+0000","last_clean":"2024-09-16T01:52:39.720860+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:39.720860+0000","last_undegraded":"2024-09-16T01:52:39.720860+0000","last_fullsized":"2024-09-16T01:52:39.720860+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"105'102","reported_seq":328,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.695211+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:39.695211+0000","last_peered":"2024-09-16T01:52:39.695211+0000","last_clean":"2024-09-16T01:52:39.695211+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:39.695211+0000","last_undegraded":"2024-09-16T01:52:39.695211+0000","last_fullsized":"2024-09-16T01:52:39.695211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":206,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712573+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:40.712573+0000","last_peered":"2024-09-16T01:52:40.712573+0000","last_clean":"2024-09-16T01:52:40.712573+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:40.712573+0000","last_undegraded":"2024-09-16T01:52:40.712573+0000","last_fullsized":"2024-09-16T01:52:40.712573+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"107'112","reported_seq":272,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.137689+0000","last_change":"2024-09-16T01:52:06.711927+0000","last_active":"2024-09-16T01:52:41.137689+0000","last_peered":"2024-09-16T01:52:41.137689+0000","last_clean":"2024-09-16T01:52:41.137689+0000","last_became_active":"2024-09-16T01:52:03.364819+0000","last_became_peered":"2024-09-16T01:52:03.364819+0000","last_unstale":"2024-09-16T01:52:41.137689+0000","last_undegraded":"2024-09-16T01:52:41.137689+0000","last_fullsized":"2024-09-16T01:52:41.137689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:01.570129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":255852544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":61,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.703706+0000","last_change":"2024-09-16T01:52:21.473114+0000","last_active":"2024-09-16T01:52:40.703706+0000","last_peered":"2024-09-16T01:52:40.703706+0000","last_clean":"2024-09-16T01:52:40.703706+0000","last_became_active":"2024-09-16T01:52:21.465256+0000","last_became_peered":"2024-09-16T01:52:21.465256+0000","last_unstale":"2024-09-16T01:52:40.703706+0000","last_undegraded":"2024-09-16T01:52:40.703706+0000","last_fullsized":"2024-09-16T01:52:40.703706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:59.896936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"107'113","reported_seq":350,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.882542+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:40.882542+0000","last_peered":"2024-09-16T01:52:40.882542+0000","last_clean":"2024-09-16T01:52:40.882542+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:40.882542+0000","last_undegraded":"2024-09-16T01:52:40.882542+0000","last_fullsized":"2024-09-16T01:52:40.882542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.694477+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:39.694477+0000","last_peered":"2024-09-16T01:52:39.694477+0000","last_clean":"2024-09-16T01:52:39.694477+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:39.694477+0000","last_undegraded":"2024-09-16T01:52:39.694477+0000","last_fullsized":"2024-09-16T01:52:39.694477+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"106'106","reported_seq":338,"reported_epoch":107,"state":"unknown","last_fresh":"2024-09-16T01:52:40.700618+0000","last_change":"2024-09-16T01:52:41.595972+0000","last_active":"2024-09-16T01:52:40.700618+0000","last_peered":"2024-09-16T01:52:39.701448+0000","last_clean":"2024-09-16T01:52:39.701448+0000","last_became_active":"2024-09-16T01:52:09.394197+0000","last_became_peered":"2024-09-16T01:52:09.394197+0000","last_unstale":"2024-09-16T01:52:40.700618+0000","last_undegraded":"2024-09-16T01:52:40.700618+0000","last_fullsized":"2024-09-16T01:52:40.700618+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":352321536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.694777+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:39.694777+0000","last_peered":"2024-09-16T01:52:39.694777+0000","last_clean":"2024-09-16T01:52:39.694777+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:39.694777+0000","last_undegraded":"2024-09-16T01:52:39.694777+0000","last_fullsized":"2024-09-16T01:52:39.694777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"106'100","reported_seq":392,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.694778+0000","last_change":"2024-09-16T01:52:22.800180+0000","last_active":"2024-09-16T01:52:39.694778+0000","last_peered":"2024-09-16T01:52:39.694778+0000","last_clean":"2024-09-16T01:52:39.694778+0000","last_became_active":"2024-09-16T01:52:21.475090+0000","last_became_peered":"2024-09-16T01:52:21.475090+0000","last_unstale":"2024-09-16T01:52:39.694778+0000","last_undegraded":"2024-09-16T01:52:39.694778+0000","last_fullsized":"2024-09-16T01:52:39.694778+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:38.993027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":189,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715768+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:40.715768+0000","last_peered":"2024-09-16T01:52:40.715768+0000","last_clean":"2024-09-16T01:52:40.715768+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:40.715768+0000","last_undegraded":"2024-09-16T01:52:40.715768+0000","last_fullsized":"2024-09-16T01:52:40.715768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"107'89","reported_seq":271,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.244459+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:41.244459+0000","last_peered":"2024-09-16T01:52:41.244459+0000","last_clean":"2024-09-16T01:52:41.244459+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:41.244459+0000","last_undegraded":"2024-09-16T01:52:41.244459+0000","last_fullsized":"2024-09-16T01:52:41.244459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"107'123","reported_seq":424,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714258+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:40.714258+0000","last_peered":"2024-09-16T01:52:40.714258+0000","last_clean":"2024-09-16T01:52:40.714258+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:40.714258+0000","last_undegraded":"2024-09-16T01:52:40.714258+0000","last_fullsized":"2024-09-16T01:52:40.714258+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":58,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714247+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:40.714247+0000","last_peered":"2024-09-16T01:52:40.714247+0000","last_clean":"2024-09-16T01:52:40.714247+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:40.714247+0000","last_undegraded":"2024-09-16T01:52:40.714247+0000","last_fullsized":"2024-09-16T01:52:40.714247+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"106'101","reported_seq":379,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714356+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:40.714356+0000","last_peered":"2024-09-16T01:52:40.714356+0000","last_clean":"2024-09-16T01:52:40.714356+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:40.714356+0000","last_undegraded":"2024-09-16T01:52:40.714356+0000","last_fullsized":"2024-09-16T01:52:40.714356+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":26,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.784564+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:40.784564+0000","last_peered":"2024-09-16T01:52:40.784564+0000","last_clean":"2024-09-16T01:52:40.784564+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:40.784564+0000","last_undegraded":"2024-09-16T01:52:40.784564+0000","last_fullsized":"2024-09-16T01:52:40.784564+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"105'86","reported_seq":267,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720994+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:39.720994+0000","last_peered":"2024-09-16T01:52:39.720994+0000","last_clean":"2024-09-16T01:52:39.720994+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:39.720994+0000","last_undegraded":"2024-09-16T01:52:39.720994+0000","last_fullsized":"2024-09-16T01:52:39.720994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":307200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":210,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.721020+0000","last_change":"2024-09-16T01:52:21.474204+0000","last_active":"2024-09-16T01:52:39.721020+0000","last_peered":"2024-09-16T01:52:39.721020+0000","last_clean":"2024-09-16T01:52:39.721020+0000","last_became_active":"2024-09-16T01:52:21.473912+0000","last_became_peered":"2024-09-16T01:52:21.473912+0000","last_unstale":"2024-09-16T01:52:39.721020+0000","last_undegraded":"2024-09-16T01:52:39.721020+0000","last_fullsized":"2024-09-16T01:52:39.721020+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:49.490388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"106'103","reported_seq":284,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T01:52:39.720979+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:39.720979+0000","last_peered":"2024-09-16T01:52:39.720979+0000","last_clean":"2024-09-16T01:52:39.720979+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:39.720979+0000","last_undegraded":"2024-09-16T01:52:39.720979+0000","last_fullsized":"2024-09-16T01:52:39.720979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712358+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:40.712358+0000","last_peered":"2024-09-16T01:52:40.712358+0000","last_clean":"2024-09-16T01:52:40.712358+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:40.712358+0000","last_undegraded":"2024-09-16T01:52:40.712358+0000","last_fullsized":"2024-09-16T01:52:40.712358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":23363584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":7008681984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1409548288,"data_stored":22552772608,"data_compressed":1115355264,"data_compressed_allocated":1409548288,"data_compressed_original":22552772608,"omap_allocated":0,"internal_metadata":0},"log_size":6731,"ondisk_log_size":6731,"up":128,"acting":127,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":246957,"num_keys_recovered":115,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":996246,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509491,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368356,"kb_used_data":337172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94003484,"statfs":{"total":96636764160,"available":96259567616,"internally_reserved":0,"allocated":345264128,"data_stored":5503515173,"data_compressed":272614656,"data_compressed_allocated":343932928,"data_compressed_original":5502926848,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509491,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373816,"kb_used_data":342628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93998024,"statfs":{"total":96636764160,"available":96253976576,"internally_reserved":0,"allocated":350851072,"data_stored":5587780817,"data_compressed":276985216,"data_compressed_allocated":349175808,"data_compressed_original":5586812928,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":245456,"kb_used_data":205692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94126384,"statfs":{"total":96636764160,"available":96385417216,"internally_reserved":0,"allocated":210628608,"data_stored":3346515636,"data_compressed":165607715,"data_compressed_allocated":209149952,"data_compressed_original":3345764352,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542194,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46812,"kb_used_data":11464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325028,"statfs":{"total":96636764160,"available":96588828672,"internally_reserved":0,"allocated":11739136,"data_stored":168355868,"data_compressed":8298496,"data_compressed_allocated":10485760,"data_compressed_original":167772160,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254800,"kb_used_data":215292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94117040,"statfs":{"total":96636764160,"available":96375848960,"internally_reserved":0,"allocated":220459008,"data_stored":3507024748,"data_compressed":173207936,"data_compressed_allocated":219152384,"data_compressed_original":3506438144,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542194,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249484,"kb_used_data":209976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94122356,"statfs":{"total":96636764160,"available":96381292544,"internally_reserved":0,"allocated":215015424,"data_stored":3419534796,"data_compressed":169137572,"data_compressed_allocated":213692416,"data_compressed_original":3418931200,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574899,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":377702,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":377763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213647360,"data_stored":3418357760,"data_compressed":169361408,"data_compressed_allocated":213647360,"data_compressed_original":3418357760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349175808,"data_stored":5586812928,"data_compressed":277031808,"data_compressed_allocated":349175808,"data_compressed_original":5586812928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343932928,"data_stored":5502926848,"data_compressed":273045632,"data_compressed_allocated":343932928,"data_compressed_original":5502926848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":213647360,"data_stored":3418357760,"data_compressed":169151744,"data_compressed_allocated":213647360,"data_compressed_original":3418357760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":219152384,"data_stored":3506438144,"data_compressed":173475840,"data_compressed_allocated":219152384,"data_compressed_original":3506438144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":167772160,"data_compressed":8310144,"data_compressed_allocated":10485760,"data_compressed_original":167772160,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:46.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:46.346 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:46.346 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:46.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:46.396 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:46.395+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:46.497 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:46.496+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.523+0000 7f2392000640 1 -- 172.21.15.146:0/2145433376 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v113) ==== 97+0+0 (secure 0 0 0) 0x7f2388079d60 con 0x7f239c0cb2f0 2024-09-16T01:52:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:52:46.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.527+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f237c06c2b0 msgr2=0x7f237c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.527+0000 7f23a0e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f237c06c2b0 0x7f237c06e770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f239c1f4be0 tx=0x7f238c008040 comp rx=0 tx=0).stop 2024-09-16T01:52:46.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.527+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 msgr2=0x7f239c14e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.527+0000 7f23a0e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 0x7f239c14e010 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2388009990 tx=0x7f23880046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.529+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 shutdown_connections 2024-09-16T01:52:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.529+0000 7f23a0e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f237c06c2b0 0x7f237c06e770 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.529+0000 7f23a0e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f239c15e9d0 0x7f239c14bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.529+0000 7f23a0e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f239c0cc0c0 0x7f239c0c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.529+0000 7f23a0e00640 1 --2- 172.21.15.146:0/2145433376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0cb2f0 0x7f239c14e010 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.529+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 >> 172.21.15.146:0/2145433376 conn(0x7f239c06d560 msgr2=0x7f239c153e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.531+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 shutdown_connections 2024-09-16T01:52:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.531+0000 7f23a0e00640 1 -- 172.21.15.146:0/2145433376 wait complete. 2024-09-16T01:52:46.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.589+0000 7fa790a00640 1 Processor -- start 2024-09-16T01:52:46.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.589+0000 7fa790a00640 1 -- start start 2024-09-16T01:52:46.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.591+0000 7fa790a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa78c0c75a0 0x7fa78c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.591+0000 7fa790a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa78c06d140 con 0x7fa78c0cc040 2024-09-16T01:52:46.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.591+0000 7fa790a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa78c06d420 con 0x7fa78c0c6b70 2024-09-16T01:52:46.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.591+0000 7fa790a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa78c15b880 con 0x7fa78c0c75a0 2024-09-16T01:52:46.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.591+0000 7fa78b200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa78c0cc040 0x7fa78c165160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33544/0 (socket says 172.21.15.146:33544) 2024-09-16T01:52:46.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.591+0000 7fa78b200640 1 -- 172.21.15.146:0/137144770 learned_addr learned my addr 172.21.15.146:0/137144770 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:46.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789e00640 1 --2- 172.21.15.146:0/137144770 >> v2:172.21.15.146:3300/0 conn(0x7fa78c0c75a0 0x7fa78c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204624979 0 0) 0x7fa78c06d140 con 0x7fa78c0cc040 2024-09-16T01:52:46.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa770003520 con 0x7fa78c0cc040 2024-09-16T01:52:46.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021991150 0 0) 0x7fa78c06d420 con 0x7fa78c0c6b70 2024-09-16T01:52:46.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa78c06d140 con 0x7fa78c0c6b70 2024-09-16T01:52:46.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3710444362 0 0) 0x7fa78c06d140 con 0x7fa78c0c6b70 2024-09-16T01:52:46.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 >> v2:172.21.15.146:3300/0 conn(0x7fa78c0c75a0 msgr2=0x7fa78c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:46.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 --2- 172.21.15.146:0/137144770 >> v2:172.21.15.146:3300/0 conn(0x7fa78c0c75a0 0x7fa78c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 >> v1:172.21.15.146:6789/0 conn(0x7fa78c0cc040 legacy=0x7fa78c165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c06d700 con 0x7fa78c0c6b70 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fa77c001670 con 0x7fa78c0c6b70 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 >> v1:172.21.15.161:6789/0 conn(0x7fa78c0c6b70 legacy=0x7fa78c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa770004bf0 0x7fa770004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7700037a0 0x7fa770003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.592+0000 7fa789400640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa7700141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.593+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c15b880 con 0x7fa7700037a0 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.593+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa770014ad0 con 0x7fa770004bf0 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.593+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa770014cd0 con 0x7fa770004310 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.593+0000 7fa789e00640 1 --2- 172.21.15.146:0/137144770 >> v2:172.21.15.146:3300/0 conn(0x7fa78c0c75a0 0x7fa78c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.593+0000 7fa78a800640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa770004bf0 0x7fa770004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.593+0000 7fa78b200640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa7700141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.594+0000 7fa789e00640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7700037a0 0x7fa770003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.594+0000 7fa78b200640 1 -- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa770004bf0 msgr2=0x7fa770004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:46.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.594+0000 7fa78b200640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa770004bf0 0x7fa770004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.594+0000 7fa78b200640 1 -- 172.21.15.146:0/137144770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7700037a0 msgr2=0x7fa770003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:46.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.594+0000 7fa78b200640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7700037a0 0x7fa770003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.594+0000 7fa78b200640 1 -- 172.21.15.146:0/137144770 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c06d140 con 0x7fa770004310 2024-09-16T01:52:46.599 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:46.598+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:46.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.595+0000 7fa78b200640 1 -- 172.21.15.146:0/137144770 reap_dead start 2024-09-16T01:52:46.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.595+0000 7fa789e00640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7700037a0 0x7fa770003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:46.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.598+0000 7fa78b200640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa7700141f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa78400c2f0 tx=0x7fa770005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=253b01a174e4c6ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:46.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.599+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784035070 con 0x7fa770004310 2024-09-16T01:52:46.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.599+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa784030600 con 0x7fa770004310 2024-09-16T01:52:46.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.599+0000 7fa789400640 1 -- 172.21.15.146:0/137144770 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784002090 con 0x7fa770004310 2024-09-16T01:52:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.603+0000 7fa790a00640 1 -- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 msgr2=0x7fa7700141f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.603+0000 7fa790a00640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa7700141f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa78400c2f0 tx=0x7fa770005830 comp rx=0 tx=0).stop 2024-09-16T01:52:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.605+0000 7fa790a00640 1 -- 172.21.15.146:0/137144770 shutdown_connections 2024-09-16T01:52:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.605+0000 7fa790a00640 1 --2- 172.21.15.146:0/137144770 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa7700141f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.605+0000 7fa790a00640 1 -- 172.21.15.146:0/137144770 >> 172.21.15.146:0/137144770 conn(0x7fa78c0ccc20 msgr2=0x7fa78c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:46.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.605+0000 7fa790a00640 1 -- 172.21.15.146:0/137144770 shutdown_connections 2024-09-16T01:52:46.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.605+0000 7fa790a00640 1 -- 172.21.15.146:0/137144770 wait complete. 2024-09-16T01:52:46.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.608+0000 7fa790a00640 1 Processor -- start 2024-09-16T01:52:46.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.608+0000 7fa790a00640 1 -- start start 2024-09-16T01:52:46.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.609+0000 7fa790a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa78c15ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.609+0000 7fa790a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c15cf20 0x7fa78c153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.609+0000 7fa790a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa78c1533d0 0x7fa78c1f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.609+0000 7fa790a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0cb050 con 0x7fa78c1533d0 2024-09-16T01:52:46.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.609+0000 7fa790a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0cab20 con 0x7fa78c15cf20 2024-09-16T01:52:46.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.609+0000 7fa790a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa78c06cd20 con 0x7fa770004310 2024-09-16T01:52:46.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa78c15ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa78c15ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38190/0 (socket says 172.21.15.146:38190) 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 -- 172.21.15.146:0/644984999 learned_addr learned my addr 172.21.15.146:0/644984999 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa789e00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c15cf20 0x7fa78c153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 -- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c15cf20 msgr2=0x7fa78c153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c15cf20 0x7fa78c153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 -- 172.21.15.146:0/644984999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa78c1533d0 msgr2=0x7fa78c1f0050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78b200640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa78c1533d0 0x7fa78c1f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa78c1533d0 0x7fa78c1f0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.610+0000 7fa78a800640 1 -- 172.21.15.146:0/644984999 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c15d820 con 0x7fa770004310 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa789e00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c15cf20 0x7fa78c153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa78b200640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa78c1533d0 0x7fa78c1f0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa78a800640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa78c15ba20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00f820 tx=0x7fa77c00fce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0052e0 con 0x7fa770004310 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa77c018070 con 0x7fa770004310 2024-09-16T01:52:46.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c00d930 con 0x7fa770004310 2024-09-16T01:52:46.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78c15c0d0 con 0x7fa770004310 2024-09-16T01:52:46.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.611+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78c15dd70 con 0x7fa770004310 2024-09-16T01:52:46.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.613+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78c0cef80 con 0x7fa770004310 2024-09-16T01:52:46.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.615+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa77c00dad0 con 0x7fa770004310 2024-09-16T01:52:46.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.616+0000 7fa782a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa76c06c2b0 0x7fa76c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:46.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.616+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 6119+0+0 (secure 0 0 0) 0x7fa77c08b6f0 con 0x7fa770004310 2024-09-16T01:52:46.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.617+0000 7fa789e00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa76c06c2b0 0x7fa76c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:46.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.618+0000 7fa789e00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa76c06c2b0 0x7fa76c06e770 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0c4770 tx=0x7fa778008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:46.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.621+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa78c0cef80 con 0x7fa770004310 2024-09-16T01:52:46.677 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:46.678 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:46.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:46.739 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:46.737+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:46.801 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:46.800+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:46.902 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:46.901+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:46.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.924+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa78c0ce6e0 con 0x7fa76c06c2b0 2024-09-16T01:52:46.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.936+0000 7fa782a00640 1 -- 172.21.15.146:0/644984999 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254432 (secure 0 0 0) 0x7fa78c0ce6e0 con 0x7fa76c06c2b0 2024-09-16T01:52:46.940 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:46.944 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:46.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa76c06c2b0 msgr2=0x7fa76c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa76c06c2b0 0x7fa76c06e770 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0c4770 tx=0x7fa778008040 comp rx=0 tx=0).stop 2024-09-16T01:52:46.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 msgr2=0x7fa78c15ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:46.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa78c15ba20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00f820 tx=0x7fa77c00fce0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 shutdown_connections 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa76c06c2b0 0x7fa76c06e770 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa78c1533d0 0x7fa78c1f0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa78c15cf20 0x7fa78c153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 --2- 172.21.15.146:0/644984999 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa770004310 0x7fa78c15ba20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.944+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 >> 172.21.15.146:0/644984999 conn(0x7fa78c0ccc20 msgr2=0x7fa78c0cf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.945+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 shutdown_connections 2024-09-16T01:52:46.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:46.945+0000 7fa790a00640 1 -- 172.21.15.146:0/644984999 wait complete. 2024-09-16T01:52:46.983 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-16T01:52:46.563482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":4,"num_objects_missing":7,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":6905,"num_write_kb":23931816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2460,"num_bytes_recovered":7415776429,"num_keys_recovered":115,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7346,"ondisk_log_size":7346,"up":194,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1573404,"kb_used_data":1356644,"kb_used_omap":9,"kb_used_meta":216118,"kb_avail":564657636,"statfs":{"total":579820584960,"available":578209419264,"internally_reserved":0,"allocated":1389203456,"data_stored":22093809590,"data_compressed":1096590311,"data_compressed_allocated":1380651008,"data_compressed_original":22089633792,"omap_allocated":9562,"internal_metadata":221305510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":565248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":406847488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.853567"},"pg_stats":[{"pgid":"3.3f","version":"105'98","reported_seq":360,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712176+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:40.712176+0000","last_peered":"2024-09-16T01:52:40.712176+0000","last_clean":"2024-09-16T01:52:40.712176+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:40.712176+0000","last_undegraded":"2024-09-16T01:52:40.712176+0000","last_fullsized":"2024-09-16T01:52:40.712176+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532378+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:45.532378+0000","last_peered":"2024-09-16T01:52:45.532378+0000","last_clean":"2024-09-16T01:52:45.532378+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:45.532378+0000","last_undegraded":"2024-09-16T01:52:45.532378+0000","last_fullsized":"2024-09-16T01:52:45.532378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":446,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.091998+0000","last_change":"2024-09-16T01:52:45.534236+0000","last_active":"2024-09-16T01:52:45.534236+0000","last_peered":"2024-09-16T01:52:45.091998+0000","last_clean":"2024-09-16T01:52:45.091998+0000","last_became_active":"2024-09-16T01:52:03.309937+0000","last_became_peered":"2024-09-16T01:52:03.309937+0000","last_unstale":"2024-09-16T01:52:45.091998+0000","last_undegraded":"2024-09-16T01:52:45.091998+0000","last_fullsized":"2024-09-16T01:52:45.091998+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":209715200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"107'120","reported_seq":308,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702041+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:40.702041+0000","last_peered":"2024-09-16T01:52:40.702041+0000","last_clean":"2024-09-16T01:52:40.702041+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:40.702041+0000","last_undegraded":"2024-09-16T01:52:40.702041+0000","last_fullsized":"2024-09-16T01:52:40.702041+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"110'109","reported_seq":273,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507567+0000","last_change":"2024-09-16T01:52:45.533438+0000","last_active":"2024-09-16T01:52:45.533438+0000","last_peered":"2024-09-16T01:52:44.507567+0000","last_clean":"2024-09-16T01:52:44.507567+0000","last_became_active":"2024-09-16T01:52:09.424479+0000","last_became_peered":"2024-09-16T01:52:09.424479+0000","last_unstale":"2024-09-16T01:52:44.507567+0000","last_undegraded":"2024-09-16T01:52:44.507567+0000","last_fullsized":"2024-09-16T01:52:44.507567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"106'96","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702170+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:40.702170+0000","last_peered":"2024-09-16T01:52:40.702170+0000","last_clean":"2024-09-16T01:52:40.702170+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:40.702170+0000","last_undegraded":"2024-09-16T01:52:40.702170+0000","last_fullsized":"2024-09-16T01:52:40.702170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"107'120","reported_seq":477,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.115504+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:41.115504+0000","last_peered":"2024-09-16T01:52:41.115504+0000","last_clean":"2024-09-16T01:52:41.115504+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:41.115504+0000","last_undegraded":"2024-09-16T01:52:41.115504+0000","last_fullsized":"2024-09-16T01:52:41.115504+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"111'127","reported_seq":445,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532294+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:45.532294+0000","last_peered":"2024-09-16T01:52:45.532294+0000","last_clean":"2024-09-16T01:52:45.532294+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:45.532294+0000","last_undegraded":"2024-09-16T01:52:45.532294+0000","last_fullsized":"2024-09-16T01:52:45.532294+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":466944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"107'108","reported_seq":291,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.840917+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:40.840917+0000","last_peered":"2024-09-16T01:52:40.840917+0000","last_clean":"2024-09-16T01:52:40.840917+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:40.840917+0000","last_undegraded":"2024-09-16T01:52:40.840917+0000","last_fullsized":"2024-09-16T01:52:40.840917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"107'108","reported_seq":395,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.979932+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:40.979932+0000","last_peered":"2024-09-16T01:52:40.979932+0000","last_clean":"2024-09-16T01:52:40.979932+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:40.979932+0000","last_undegraded":"2024-09-16T01:52:40.979932+0000","last_fullsized":"2024-09-16T01:52:40.979932+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"107'130","reported_seq":442,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714130+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:40.714130+0000","last_peered":"2024-09-16T01:52:40.714130+0000","last_clean":"2024-09-16T01:52:40.714130+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:40.714130+0000","last_undegraded":"2024-09-16T01:52:40.714130+0000","last_fullsized":"2024-09-16T01:52:40.714130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"107'101","reported_seq":92,"reported_epoch":107,"state":"active+recovering","last_fresh":"2024-09-16T01:52:41.593897+0000","last_change":"2024-09-16T01:52:40.740404+0000","last_active":"2024-09-16T01:52:41.593897+0000","last_peered":"2024-09-16T01:52:41.593897+0000","last_clean":"2024-09-16T01:52:36.904016+0000","last_became_active":"2024-09-16T01:52:40.724710+0000","last_became_peered":"2024-09-16T01:52:40.724710+0000","last_unstale":"2024-09-16T01:52:41.593897+0000","last_undegraded":"2024-09-16T01:52:41.593897+0000","last_fullsized":"2024-09-16T01:52:41.593897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":452984832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"1,2","objects":28},{"shards":"1,2,6","objects":2},{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"111'111","reported_seq":427,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.529634+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:45.529634+0000","last_peered":"2024-09-16T01:52:45.529634+0000","last_clean":"2024-09-16T01:52:45.529634+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:45.529634+0000","last_undegraded":"2024-09-16T01:52:45.529634+0000","last_fullsized":"2024-09-16T01:52:45.529634+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"110'103","reported_seq":346,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527928+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:45.527928+0000","last_peered":"2024-09-16T01:52:45.527928+0000","last_clean":"2024-09-16T01:52:45.527928+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:45.527928+0000","last_undegraded":"2024-09-16T01:52:45.527928+0000","last_fullsized":"2024-09-16T01:52:45.527928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"111'108","reported_seq":404,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.531409+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:45.531409+0000","last_peered":"2024-09-16T01:52:45.531409+0000","last_clean":"2024-09-16T01:52:45.531409+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:45.531409+0000","last_undegraded":"2024-09-16T01:52:45.531409+0000","last_fullsized":"2024-09-16T01:52:45.531409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"107'102","reported_seq":284,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708519+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:40.708519+0000","last_peered":"2024-09-16T01:52:40.708519+0000","last_clean":"2024-09-16T01:52:40.708519+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:40.708519+0000","last_undegraded":"2024-09-16T01:52:40.708519+0000","last_fullsized":"2024-09-16T01:52:40.708519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"106'98","reported_seq":377,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712476+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:40.712476+0000","last_peered":"2024-09-16T01:52:40.712476+0000","last_clean":"2024-09-16T01:52:40.712476+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:40.712476+0000","last_undegraded":"2024-09-16T01:52:40.712476+0000","last_fullsized":"2024-09-16T01:52:40.712476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"107'103","reported_seq":371,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.454439+0000","last_change":"2024-09-16T01:52:05.041021+0000","last_active":"2024-09-16T01:52:41.454439+0000","last_peered":"2024-09-16T01:52:41.454439+0000","last_clean":"2024-09-16T01:52:41.454439+0000","last_became_active":"2024-09-16T01:52:03.349420+0000","last_became_peered":"2024-09-16T01:52:03.349420+0000","last_unstale":"2024-09-16T01:52:41.454439+0000","last_undegraded":"2024-09-16T01:52:41.454439+0000","last_fullsized":"2024-09-16T01:52:41.454439+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:31.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"104'115","reported_seq":352,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T01:52:35.045658+0000","last_change":"2024-09-16T01:51:50.466227+0000","last_active":"2024-09-16T01:52:35.045658+0000","last_peered":"2024-09-16T01:52:35.045658+0000","last_clean":"2024-09-16T01:52:35.045658+0000","last_became_active":"2024-09-16T01:51:49.177542+0000","last_became_peered":"2024-09-16T01:51:49.177542+0000","last_unstale":"2024-09-16T01:52:35.045658+0000","last_undegraded":"2024-09-16T01:52:35.045658+0000","last_fullsized":"2024-09-16T01:52:35.045658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:14.445720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":88080384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"107'108","reported_seq":434,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.043648+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:41.043648+0000","last_peered":"2024-09-16T01:52:41.043648+0000","last_clean":"2024-09-16T01:52:41.043648+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:41.043648+0000","last_undegraded":"2024-09-16T01:52:41.043648+0000","last_fullsized":"2024-09-16T01:52:41.043648+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"106'83","reported_seq":303,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705785+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:40.705785+0000","last_peered":"2024-09-16T01:52:40.705785+0000","last_clean":"2024-09-16T01:52:40.705785+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:40.705785+0000","last_undegraded":"2024-09-16T01:52:40.705785+0000","last_fullsized":"2024-09-16T01:52:40.705785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"105'120","reported_seq":432,"reported_epoch":105,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:39.297428+0000","last_change":"2024-09-16T01:52:40.701785+0000","last_active":"2024-09-16T01:52:40.701785+0000","last_peered":"2024-09-16T01:52:39.297428+0000","last_clean":"2024-09-16T01:52:39.297428+0000","last_became_active":"2024-09-16T01:52:09.425675+0000","last_became_peered":"2024-09-16T01:52:09.425675+0000","last_unstale":"2024-09-16T01:52:39.297428+0000","last_undegraded":"2024-09-16T01:52:39.297428+0000","last_fullsized":"2024-09-16T01:52:39.297428+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":197132288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532334+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:45.532334+0000","last_peered":"2024-09-16T01:52:45.532334+0000","last_clean":"2024-09-16T01:52:45.532334+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:45.532334+0000","last_undegraded":"2024-09-16T01:52:45.532334+0000","last_fullsized":"2024-09-16T01:52:45.532334+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"107'127","reported_seq":309,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708637+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:40.708637+0000","last_peered":"2024-09-16T01:52:40.708637+0000","last_clean":"2024-09-16T01:52:40.708637+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:40.708637+0000","last_undegraded":"2024-09-16T01:52:40.708637+0000","last_fullsized":"2024-09-16T01:52:40.708637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":454656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"111'100","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532061+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:45.532061+0000","last_peered":"2024-09-16T01:52:45.532061+0000","last_clean":"2024-09-16T01:52:45.532061+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:45.532061+0000","last_undegraded":"2024-09-16T01:52:45.532061+0000","last_fullsized":"2024-09-16T01:52:45.532061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"107'133","reported_seq":316,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.872103+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:40.872103+0000","last_peered":"2024-09-16T01:52:40.872103+0000","last_clean":"2024-09-16T01:52:40.872103+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:40.872103+0000","last_undegraded":"2024-09-16T01:52:40.872103+0000","last_fullsized":"2024-09-16T01:52:40.872103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"110'102","reported_seq":423,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.528487+0000","last_change":"2024-09-16T01:52:45.527683+0000","last_active":"2024-09-16T01:52:45.527683+0000","last_peered":"2024-09-16T01:52:44.528487+0000","last_clean":"2024-09-16T01:52:44.528487+0000","last_became_active":"2024-09-16T01:52:03.375641+0000","last_became_peered":"2024-09-16T01:52:03.375641+0000","last_unstale":"2024-09-16T01:52:44.528487+0000","last_undegraded":"2024-09-16T01:52:44.528487+0000","last_fullsized":"2024-09-16T01:52:44.528487+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"111'98","reported_seq":384,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528055+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:45.528055+0000","last_peered":"2024-09-16T01:52:45.528055+0000","last_clean":"2024-09-16T01:52:45.528055+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:45.528055+0000","last_undegraded":"2024-09-16T01:52:45.528055+0000","last_fullsized":"2024-09-16T01:52:45.528055+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"109'87","reported_seq":481,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507610+0000","last_change":"2024-09-16T01:52:45.531675+0000","last_active":"2024-09-16T01:52:45.531675+0000","last_peered":"2024-09-16T01:52:44.507610+0000","last_clean":"2024-09-16T01:52:44.507610+0000","last_became_active":"2024-09-16T01:52:09.378210+0000","last_became_peered":"2024-09-16T01:52:09.378210+0000","last_unstale":"2024-09-16T01:52:44.507610+0000","last_undegraded":"2024-09-16T01:52:44.507610+0000","last_fullsized":"2024-09-16T01:52:44.507610+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"106'94","reported_seq":276,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709371+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:40.709371+0000","last_peered":"2024-09-16T01:52:40.709371+0000","last_clean":"2024-09-16T01:52:40.709371+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:40.709371+0000","last_undegraded":"2024-09-16T01:52:40.709371+0000","last_fullsized":"2024-09-16T01:52:40.709371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"106'96","reported_seq":382,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.704088+0000","last_change":"2024-09-16T01:52:22.869283+0000","last_active":"2024-09-16T01:52:40.704088+0000","last_peered":"2024-09-16T01:52:40.704088+0000","last_clean":"2024-09-16T01:52:40.704088+0000","last_became_active":"2024-09-16T01:52:21.489999+0000","last_became_peered":"2024-09-16T01:52:21.489999+0000","last_unstale":"2024-09-16T01:52:40.704088+0000","last_undegraded":"2024-09-16T01:52:40.704088+0000","last_fullsized":"2024-09-16T01:52:40.704088+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"111'111","reported_seq":417,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527513+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:45.527513+0000","last_peered":"2024-09-16T01:52:45.527513+0000","last_clean":"2024-09-16T01:52:45.527513+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:45.527513+0000","last_undegraded":"2024-09-16T01:52:45.527513+0000","last_fullsized":"2024-09-16T01:52:45.527513+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"107'93","reported_seq":276,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705218+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:40.705218+0000","last_peered":"2024-09-16T01:52:40.705218+0000","last_clean":"2024-09-16T01:52:40.705218+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:40.705218+0000","last_undegraded":"2024-09-16T01:52:40.705218+0000","last_fullsized":"2024-09-16T01:52:40.705218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":200,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534083+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:45.534083+0000","last_peered":"2024-09-16T01:52:45.534083+0000","last_clean":"2024-09-16T01:52:45.534083+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:45.534083+0000","last_undegraded":"2024-09-16T01:52:45.534083+0000","last_fullsized":"2024-09-16T01:52:45.534083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"106'99","reported_seq":519,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715559+0000","last_change":"2024-09-16T01:52:22.369179+0000","last_active":"2024-09-16T01:52:40.715559+0000","last_peered":"2024-09-16T01:52:40.715559+0000","last_clean":"2024-09-16T01:52:40.715559+0000","last_became_active":"2024-09-16T01:52:21.480658+0000","last_became_peered":"2024-09-16T01:52:21.480658+0000","last_unstale":"2024-09-16T01:52:40.715559+0000","last_undegraded":"2024-09-16T01:52:40.715559+0000","last_fullsized":"2024-09-16T01:52:40.715559+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":204,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709632+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:40.709632+0000","last_peered":"2024-09-16T01:52:40.709632+0000","last_clean":"2024-09-16T01:52:40.709632+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:40.709632+0000","last_undegraded":"2024-09-16T01:52:40.709632+0000","last_fullsized":"2024-09-16T01:52:40.709632+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":208,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.496173+0000","last_change":"2024-09-16T01:52:45.528156+0000","last_active":"2024-09-16T01:52:45.528156+0000","last_peered":"2024-09-16T01:52:44.496173+0000","last_clean":"2024-09-16T01:52:44.496173+0000","last_became_active":"2024-09-16T01:52:09.417952+0000","last_became_peered":"2024-09-16T01:52:09.417952+0000","last_unstale":"2024-09-16T01:52:44.496173+0000","last_undegraded":"2024-09-16T01:52:44.496173+0000","last_fullsized":"2024-09-16T01:52:44.496173+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"111'121","reported_seq":314,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532429+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:45.532429+0000","last_peered":"2024-09-16T01:52:45.532429+0000","last_clean":"2024-09-16T01:52:45.532429+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:45.532429+0000","last_undegraded":"2024-09-16T01:52:45.532429+0000","last_fullsized":"2024-09-16T01:52:45.532429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":438272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":187,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708659+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:40.708659+0000","last_peered":"2024-09-16T01:52:40.708659+0000","last_clean":"2024-09-16T01:52:40.708659+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:40.708659+0000","last_undegraded":"2024-09-16T01:52:40.708659+0000","last_fullsized":"2024-09-16T01:52:40.708659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"105'106","reported_seq":254,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.703982+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:40.703982+0000","last_peered":"2024-09-16T01:52:40.703982+0000","last_clean":"2024-09-16T01:52:40.703982+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:40.703982+0000","last_undegraded":"2024-09-16T01:52:40.703982+0000","last_fullsized":"2024-09-16T01:52:40.703982+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":216,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.722262+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:40.722262+0000","last_peered":"2024-09-16T01:52:40.722262+0000","last_clean":"2024-09-16T01:52:40.722262+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:40.722262+0000","last_undegraded":"2024-09-16T01:52:40.722262+0000","last_fullsized":"2024-09-16T01:52:40.722262+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"112'163","reported_seq":367,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.567223+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:45.567223+0000","last_peered":"2024-09-16T01:52:45.567223+0000","last_clean":"2024-09-16T01:52:45.567223+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:45.567223+0000","last_undegraded":"2024-09-16T01:52:45.567223+0000","last_fullsized":"2024-09-16T01:52:45.567223+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":67,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715962+0000","last_change":"2024-09-16T01:52:21.510976+0000","last_active":"2024-09-16T01:52:40.715962+0000","last_peered":"2024-09-16T01:52:40.715962+0000","last_clean":"2024-09-16T01:52:40.715962+0000","last_became_active":"2024-09-16T01:52:21.491849+0000","last_became_peered":"2024-09-16T01:52:21.491849+0000","last_unstale":"2024-09-16T01:52:40.715962+0000","last_undegraded":"2024-09-16T01:52:40.715962+0000","last_fullsized":"2024-09-16T01:52:40.715962+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:43.042548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"110'98","reported_seq":291,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532194+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:45.532194+0000","last_peered":"2024-09-16T01:52:45.532194+0000","last_clean":"2024-09-16T01:52:45.532194+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:45.532194+0000","last_undegraded":"2024-09-16T01:52:45.532194+0000","last_fullsized":"2024-09-16T01:52:45.532194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.713822+0000","last_change":"2024-09-16T01:52:21.507954+0000","last_active":"2024-09-16T01:52:40.713822+0000","last_peered":"2024-09-16T01:52:40.713822+0000","last_clean":"2024-09-16T01:52:40.713822+0000","last_became_active":"2024-09-16T01:52:21.491632+0000","last_became_peered":"2024-09-16T01:52:21.491632+0000","last_unstale":"2024-09-16T01:52:40.713822+0000","last_undegraded":"2024-09-16T01:52:40.713822+0000","last_fullsized":"2024-09-16T01:52:40.713822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:36.763063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3360,"num_keys_recovered":50,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"107'101","reported_seq":283,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.710272+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:40.710272+0000","last_peered":"2024-09-16T01:52:40.710272+0000","last_clean":"2024-09-16T01:52:40.710272+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:40.710272+0000","last_undegraded":"2024-09-16T01:52:40.710272+0000","last_fullsized":"2024-09-16T01:52:40.710272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":374,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702091+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:40.702091+0000","last_peered":"2024-09-16T01:52:40.702091+0000","last_clean":"2024-09-16T01:52:40.702091+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:40.702091+0000","last_undegraded":"2024-09-16T01:52:40.702091+0000","last_fullsized":"2024-09-16T01:52:40.702091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":407,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.526908+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:45.526908+0000","last_peered":"2024-09-16T01:52:45.526908+0000","last_clean":"2024-09-16T01:52:45.526908+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:45.526908+0000","last_undegraded":"2024-09-16T01:52:45.526908+0000","last_fullsized":"2024-09-16T01:52:45.526908+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":244,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715431+0000","last_change":"2024-09-16T01:52:09.471855+0000","last_active":"2024-09-16T01:52:40.715431+0000","last_peered":"2024-09-16T01:52:40.715431+0000","last_clean":"2024-09-16T01:52:40.715431+0000","last_became_active":"2024-09-16T01:52:09.419033+0000","last_became_peered":"2024-09-16T01:52:09.419033+0000","last_unstale":"2024-09-16T01:52:40.715431+0000","last_undegraded":"2024-09-16T01:52:40.715431+0000","last_fullsized":"2024-09-16T01:52:40.715431+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"111'115","reported_seq":308,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528039+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:45.528039+0000","last_peered":"2024-09-16T01:52:45.528039+0000","last_clean":"2024-09-16T01:52:45.528039+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:45.528039+0000","last_undegraded":"2024-09-16T01:52:45.528039+0000","last_fullsized":"2024-09-16T01:52:45.528039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":254,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709719+0000","last_change":"2024-09-16T01:52:03.470007+0000","last_active":"2024-09-16T01:52:40.709719+0000","last_peered":"2024-09-16T01:52:40.709719+0000","last_clean":"2024-09-16T01:52:40.709719+0000","last_became_active":"2024-09-16T01:52:03.348785+0000","last_became_peered":"2024-09-16T01:52:03.348785+0000","last_unstale":"2024-09-16T01:52:40.709719+0000","last_undegraded":"2024-09-16T01:52:40.709719+0000","last_fullsized":"2024-09-16T01:52:40.709719+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"106'114","reported_seq":427,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715670+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:40.715670+0000","last_peered":"2024-09-16T01:52:40.715670+0000","last_clean":"2024-09-16T01:52:40.715670+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:40.715670+0000","last_undegraded":"2024-09-16T01:52:40.715670+0000","last_fullsized":"2024-09-16T01:52:40.715670+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":61,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708931+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:40.708931+0000","last_peered":"2024-09-16T01:52:40.708931+0000","last_clean":"2024-09-16T01:52:40.708931+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:40.708931+0000","last_undegraded":"2024-09-16T01:52:40.708931+0000","last_fullsized":"2024-09-16T01:52:40.708931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"106'128","reported_seq":311,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701595+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:40.701595+0000","last_peered":"2024-09-16T01:52:40.701595+0000","last_clean":"2024-09-16T01:52:40.701595+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:40.701595+0000","last_undegraded":"2024-09-16T01:52:40.701595+0000","last_fullsized":"2024-09-16T01:52:40.701595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":97,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701198+0000","last_change":"2024-09-16T01:52:21.494700+0000","last_active":"2024-09-16T01:52:40.701198+0000","last_peered":"2024-09-16T01:52:40.701198+0000","last_clean":"2024-09-16T01:52:40.701198+0000","last_became_active":"2024-09-16T01:52:21.466194+0000","last_became_peered":"2024-09-16T01:52:21.466194+0000","last_unstale":"2024-09-16T01:52:40.701198+0000","last_undegraded":"2024-09-16T01:52:40.701198+0000","last_fullsized":"2024-09-16T01:52:40.701198+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:55.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"106'139","reported_seq":321,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709029+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:40.709029+0000","last_peered":"2024-09-16T01:52:40.709029+0000","last_clean":"2024-09-16T01:52:40.709029+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:40.709029+0000","last_undegraded":"2024-09-16T01:52:40.709029+0000","last_fullsized":"2024-09-16T01:52:40.709029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":68,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532060+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:45.532060+0000","last_peered":"2024-09-16T01:52:45.532060+0000","last_clean":"2024-09-16T01:52:45.532060+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:45.532060+0000","last_undegraded":"2024-09-16T01:52:45.532060+0000","last_fullsized":"2024-09-16T01:52:45.532060+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"107'99","reported_seq":383,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.373880+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:41.373880+0000","last_peered":"2024-09-16T01:52:41.373880+0000","last_clean":"2024-09-16T01:52:41.373880+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:41.373880+0000","last_undegraded":"2024-09-16T01:52:41.373880+0000","last_fullsized":"2024-09-16T01:52:41.373880+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532518+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:45.532518+0000","last_peered":"2024-09-16T01:52:45.532518+0000","last_clean":"2024-09-16T01:52:45.532518+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:45.532518+0000","last_undegraded":"2024-09-16T01:52:45.532518+0000","last_fullsized":"2024-09-16T01:52:45.532518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"107'96","reported_seq":375,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.258476+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:41.258476+0000","last_peered":"2024-09-16T01:52:41.258476+0000","last_clean":"2024-09-16T01:52:41.258476+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:41.258476+0000","last_undegraded":"2024-09-16T01:52:41.258476+0000","last_fullsized":"2024-09-16T01:52:41.258476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":198,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534027+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:45.534027+0000","last_peered":"2024-09-16T01:52:45.534027+0000","last_clean":"2024-09-16T01:52:45.534027+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:45.534027+0000","last_undegraded":"2024-09-16T01:52:45.534027+0000","last_fullsized":"2024-09-16T01:52:45.534027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"107'95","reported_seq":504,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.953377+0000","last_change":"2024-09-16T01:52:10.143229+0000","last_active":"2024-09-16T01:52:40.953377+0000","last_peered":"2024-09-16T01:52:40.953377+0000","last_clean":"2024-09-16T01:52:40.953377+0000","last_became_active":"2024-09-16T01:52:09.413639+0000","last_became_peered":"2024-09-16T01:52:09.413639+0000","last_unstale":"2024-09-16T01:52:40.953377+0000","last_undegraded":"2024-09-16T01:52:40.953377+0000","last_fullsized":"2024-09-16T01:52:40.953377+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":318767104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":213,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528534+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:45.528534+0000","last_peered":"2024-09-16T01:52:45.528534+0000","last_clean":"2024-09-16T01:52:45.528534+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:45.528534+0000","last_undegraded":"2024-09-16T01:52:45.528534+0000","last_fullsized":"2024-09-16T01:52:45.528534+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"107'105","reported_seq":287,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714360+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:40.714360+0000","last_peered":"2024-09-16T01:52:40.714360+0000","last_clean":"2024-09-16T01:52:40.714360+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:40.714360+0000","last_undegraded":"2024-09-16T01:52:40.714360+0000","last_fullsized":"2024-09-16T01:52:40.714360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":341,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701702+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:40.701702+0000","last_peered":"2024-09-16T01:52:40.701702+0000","last_clean":"2024-09-16T01:52:40.701702+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:40.701702+0000","last_undegraded":"2024-09-16T01:52:40.701702+0000","last_fullsized":"2024-09-16T01:52:40.701702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":190,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715864+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:40.715864+0000","last_peered":"2024-09-16T01:52:40.715864+0000","last_clean":"2024-09-16T01:52:40.715864+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:40.715864+0000","last_undegraded":"2024-09-16T01:52:40.715864+0000","last_fullsized":"2024-09-16T01:52:40.715864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"107'106","reported_seq":429,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.741036+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:40.741036+0000","last_peered":"2024-09-16T01:52:40.741036+0000","last_clean":"2024-09-16T01:52:40.741036+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:40.741036+0000","last_undegraded":"2024-09-16T01:52:40.741036+0000","last_fullsized":"2024-09-16T01:52:40.741036+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":188,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708690+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:40.708690+0000","last_peered":"2024-09-16T01:52:40.708690+0000","last_clean":"2024-09-16T01:52:40.708690+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:40.708690+0000","last_undegraded":"2024-09-16T01:52:40.708690+0000","last_fullsized":"2024-09-16T01:52:40.708690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"110'104","reported_seq":297,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527497+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:45.527497+0000","last_peered":"2024-09-16T01:52:45.527497+0000","last_clean":"2024-09-16T01:52:45.527497+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:45.527497+0000","last_undegraded":"2024-09-16T01:52:45.527497+0000","last_fullsized":"2024-09-16T01:52:45.527497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":240,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507474+0000","last_change":"2024-09-16T01:52:45.531950+0000","last_active":"2024-09-16T01:52:45.531950+0000","last_peered":"2024-09-16T01:52:44.507474+0000","last_clean":"2024-09-16T01:52:44.507474+0000","last_became_active":"2024-09-16T01:52:21.493366+0000","last_became_peered":"2024-09-16T01:52:21.493366+0000","last_unstale":"2024-09-16T01:52:44.507474+0000","last_undegraded":"2024-09-16T01:52:44.507474+0000","last_fullsized":"2024-09-16T01:52:44.507474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"111'93","reported_seq":410,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.298724+0000","last_change":"2024-09-16T01:52:45.533111+0000","last_active":"2024-09-16T01:52:45.533111+0000","last_peered":"2024-09-16T01:52:45.298724+0000","last_clean":"2024-09-16T01:52:45.298724+0000","last_became_active":"2024-09-16T01:52:03.344719+0000","last_became_peered":"2024-09-16T01:52:03.344719+0000","last_unstale":"2024-09-16T01:52:45.298724+0000","last_undegraded":"2024-09-16T01:52:45.298724+0000","last_fullsized":"2024-09-16T01:52:45.298724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":150994944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":189,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709793+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:40.709793+0000","last_peered":"2024-09-16T01:52:40.709793+0000","last_clean":"2024-09-16T01:52:40.709793+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:40.709793+0000","last_undegraded":"2024-09-16T01:52:40.709793+0000","last_fullsized":"2024-09-16T01:52:40.709793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"106'107","reported_seq":389,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705561+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:40.705561+0000","last_peered":"2024-09-16T01:52:40.705561+0000","last_clean":"2024-09-16T01:52:40.705561+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:40.705561+0000","last_undegraded":"2024-09-16T01:52:40.705561+0000","last_fullsized":"2024-09-16T01:52:40.705561+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":196,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.533427+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:45.533427+0000","last_peered":"2024-09-16T01:52:45.533427+0000","last_clean":"2024-09-16T01:52:45.533427+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:45.533427+0000","last_undegraded":"2024-09-16T01:52:45.533427+0000","last_fullsized":"2024-09-16T01:52:45.533427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"107'105","reported_seq":288,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708603+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:40.708603+0000","last_peered":"2024-09-16T01:52:40.708603+0000","last_clean":"2024-09-16T01:52:40.708603+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:40.708603+0000","last_undegraded":"2024-09-16T01:52:40.708603+0000","last_fullsized":"2024-09-16T01:52:40.708603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":2,"reported_epoch":107,"state":"peering","last_fresh":"2024-09-16T01:52:39.700468+0000","last_change":"2024-09-16T01:52:41.657743+0000","last_active":"2024-09-16T01:52:41.657743+0000","last_peered":"2024-09-16T01:52:39.700468+0000","last_clean":"2024-09-16T01:52:39.700468+0000","last_became_active":"2024-09-16T01:51:49.156266+0000","last_became_peered":"2024-09-16T01:51:49.156266+0000","last_unstale":"2024-09-16T01:52:39.700468+0000","last_undegraded":"2024-09-16T01:52:39.700468+0000","last_fullsized":"2024-09-16T01:52:39.700468+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"111'99","reported_seq":292,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528157+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:45.528157+0000","last_peered":"2024-09-16T01:52:45.528157+0000","last_clean":"2024-09-16T01:52:45.528157+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:45.528157+0000","last_undegraded":"2024-09-16T01:52:45.528157+0000","last_fullsized":"2024-09-16T01:52:45.528157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532417+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:45.532417+0000","last_peered":"2024-09-16T01:52:45.532417+0000","last_clean":"2024-09-16T01:52:45.532417+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:45.532417+0000","last_undegraded":"2024-09-16T01:52:45.532417+0000","last_fullsized":"2024-09-16T01:52:45.532417+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"111'110","reported_seq":348,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528266+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:45.528266+0000","last_peered":"2024-09-16T01:52:45.528266+0000","last_clean":"2024-09-16T01:52:45.528266+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:45.528266+0000","last_undegraded":"2024-09-16T01:52:45.528266+0000","last_fullsized":"2024-09-16T01:52:45.528266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":206,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712573+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:40.712573+0000","last_peered":"2024-09-16T01:52:40.712573+0000","last_clean":"2024-09-16T01:52:40.712573+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:40.712573+0000","last_undegraded":"2024-09-16T01:52:40.712573+0000","last_fullsized":"2024-09-16T01:52:40.712573+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"107'112","reported_seq":272,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.137689+0000","last_change":"2024-09-16T01:52:06.711927+0000","last_active":"2024-09-16T01:52:41.137689+0000","last_peered":"2024-09-16T01:52:41.137689+0000","last_clean":"2024-09-16T01:52:41.137689+0000","last_became_active":"2024-09-16T01:52:03.364819+0000","last_became_peered":"2024-09-16T01:52:03.364819+0000","last_unstale":"2024-09-16T01:52:41.137689+0000","last_undegraded":"2024-09-16T01:52:41.137689+0000","last_fullsized":"2024-09-16T01:52:41.137689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:01.570129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":255852544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":61,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.703706+0000","last_change":"2024-09-16T01:52:21.473114+0000","last_active":"2024-09-16T01:52:40.703706+0000","last_peered":"2024-09-16T01:52:40.703706+0000","last_clean":"2024-09-16T01:52:40.703706+0000","last_became_active":"2024-09-16T01:52:21.465256+0000","last_became_peered":"2024-09-16T01:52:21.465256+0000","last_unstale":"2024-09-16T01:52:40.703706+0000","last_undegraded":"2024-09-16T01:52:40.703706+0000","last_fullsized":"2024-09-16T01:52:40.703706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:59.896936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"107'113","reported_seq":350,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.882542+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:40.882542+0000","last_peered":"2024-09-16T01:52:40.882542+0000","last_clean":"2024-09-16T01:52:40.882542+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:40.882542+0000","last_undegraded":"2024-09-16T01:52:40.882542+0000","last_fullsized":"2024-09-16T01:52:40.882542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":75,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527355+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:45.527355+0000","last_peered":"2024-09-16T01:52:45.527355+0000","last_clean":"2024-09-16T01:52:45.527355+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:45.527355+0000","last_undegraded":"2024-09-16T01:52:45.527355+0000","last_fullsized":"2024-09-16T01:52:45.527355+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"106'106","reported_seq":338,"reported_epoch":107,"state":"unknown","last_fresh":"2024-09-16T01:52:40.700618+0000","last_change":"2024-09-16T01:52:41.595972+0000","last_active":"2024-09-16T01:52:40.700618+0000","last_peered":"2024-09-16T01:52:39.701448+0000","last_clean":"2024-09-16T01:52:39.701448+0000","last_became_active":"2024-09-16T01:52:09.394197+0000","last_became_peered":"2024-09-16T01:52:09.394197+0000","last_unstale":"2024-09-16T01:52:40.700618+0000","last_undegraded":"2024-09-16T01:52:40.700618+0000","last_fullsized":"2024-09-16T01:52:40.700618+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":352321536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":198,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528141+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:45.528141+0000","last_peered":"2024-09-16T01:52:45.528141+0000","last_clean":"2024-09-16T01:52:45.528141+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:45.528141+0000","last_undegraded":"2024-09-16T01:52:45.528141+0000","last_fullsized":"2024-09-16T01:52:45.528141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"111'104","reported_seq":407,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.861424+0000","last_change":"2024-09-16T01:52:45.530072+0000","last_active":"2024-09-16T01:52:45.530072+0000","last_peered":"2024-09-16T01:52:44.861424+0000","last_clean":"2024-09-16T01:52:44.861424+0000","last_became_active":"2024-09-16T01:52:21.475090+0000","last_became_peered":"2024-09-16T01:52:21.475090+0000","last_unstale":"2024-09-16T01:52:44.861424+0000","last_undegraded":"2024-09-16T01:52:44.861424+0000","last_fullsized":"2024-09-16T01:52:44.861424+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":189,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715768+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:40.715768+0000","last_peered":"2024-09-16T01:52:40.715768+0000","last_clean":"2024-09-16T01:52:40.715768+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:40.715768+0000","last_undegraded":"2024-09-16T01:52:40.715768+0000","last_fullsized":"2024-09-16T01:52:40.715768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"107'89","reported_seq":271,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.244459+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:41.244459+0000","last_peered":"2024-09-16T01:52:41.244459+0000","last_clean":"2024-09-16T01:52:41.244459+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:41.244459+0000","last_undegraded":"2024-09-16T01:52:41.244459+0000","last_fullsized":"2024-09-16T01:52:41.244459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"107'123","reported_seq":424,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714258+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:40.714258+0000","last_peered":"2024-09-16T01:52:40.714258+0000","last_clean":"2024-09-16T01:52:40.714258+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:40.714258+0000","last_undegraded":"2024-09-16T01:52:40.714258+0000","last_fullsized":"2024-09-16T01:52:40.714258+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":58,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714247+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:40.714247+0000","last_peered":"2024-09-16T01:52:40.714247+0000","last_clean":"2024-09-16T01:52:40.714247+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:40.714247+0000","last_undegraded":"2024-09-16T01:52:40.714247+0000","last_fullsized":"2024-09-16T01:52:40.714247+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"106'101","reported_seq":379,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714356+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:40.714356+0000","last_peered":"2024-09-16T01:52:40.714356+0000","last_clean":"2024-09-16T01:52:40.714356+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:40.714356+0000","last_undegraded":"2024-09-16T01:52:40.714356+0000","last_fullsized":"2024-09-16T01:52:40.714356+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":26,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.784564+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:40.784564+0000","last_peered":"2024-09-16T01:52:40.784564+0000","last_clean":"2024-09-16T01:52:40.784564+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:40.784564+0000","last_undegraded":"2024-09-16T01:52:40.784564+0000","last_fullsized":"2024-09-16T01:52:40.784564+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"111'91","reported_seq":284,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532243+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:45.532243+0000","last_peered":"2024-09-16T01:52:45.532243+0000","last_clean":"2024-09-16T01:52:45.532243+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:45.532243+0000","last_undegraded":"2024-09-16T01:52:45.532243+0000","last_fullsized":"2024-09-16T01:52:45.532243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":221,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.508113+0000","last_change":"2024-09-16T01:52:45.532987+0000","last_active":"2024-09-16T01:52:45.532987+0000","last_peered":"2024-09-16T01:52:44.508113+0000","last_clean":"2024-09-16T01:52:44.508113+0000","last_became_active":"2024-09-16T01:52:21.473912+0000","last_became_peered":"2024-09-16T01:52:21.473912+0000","last_unstale":"2024-09-16T01:52:44.508113+0000","last_undegraded":"2024-09-16T01:52:44.508113+0000","last_fullsized":"2024-09-16T01:52:44.508113+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"111'112","reported_seq":305,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532474+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:45.532474+0000","last_peered":"2024-09-16T01:52:45.532474+0000","last_clean":"2024-09-16T01:52:45.532474+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:45.532474+0000","last_undegraded":"2024-09-16T01:52:45.532474+0000","last_fullsized":"2024-09-16T01:52:45.532474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712358+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:40.712358+0000","last_peered":"2024-09-16T01:52:40.712358+0000","last_clean":"2024-09-16T01:52:40.712358+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:40.712358+0000","last_undegraded":"2024-09-16T01:52:40.712358+0000","last_fullsized":"2024-09-16T01:52:40.712358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6354,"num_write_kb":23928832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":7415529472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1459617792,"data_stored":23353884672,"data_compressed":1158023040,"data_compressed_allocated":1459617792,"data_compressed_original":23353884672,"omap_allocated":0,"internal_metadata":0},"log_size":6869,"ondisk_log_size":6869,"up":128,"acting":127,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":246957,"num_keys_recovered":115,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":996246,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509491,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368356,"kb_used_data":337172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94003484,"statfs":{"total":96636764160,"available":96259567616,"internally_reserved":0,"allocated":345264128,"data_stored":5503515173,"data_compressed":272614656,"data_compressed_allocated":343932928,"data_compressed_original":5502926848,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509491,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373816,"kb_used_data":342628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93998024,"statfs":{"total":96636764160,"available":96253976576,"internally_reserved":0,"allocated":350851072,"data_stored":5587780817,"data_compressed":276985216,"data_compressed_allocated":349175808,"data_compressed_original":5586812928,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":245456,"kb_used_data":205692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94126384,"statfs":{"total":96636764160,"available":96385417216,"internally_reserved":0,"allocated":210628608,"data_stored":3346515636,"data_compressed":165607715,"data_compressed_allocated":209149952,"data_compressed_original":3345764352,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542195,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69908,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94301932,"statfs":{"total":96636764160,"available":96565178368,"internally_reserved":0,"allocated":35393536,"data_stored":545874692,"data_compressed":27121536,"data_compressed_allocated":34078720,"data_compressed_original":545259520,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542195,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":250452,"kb_used_data":210680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94121388,"statfs":{"total":96636764160,"available":96380301312,"internally_reserved":0,"allocated":215736320,"data_stored":3430510164,"data_compressed":170823584,"data_compressed_allocated":214368256,"data_compressed_original":3429892096,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265416,"kb_used_data":225908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94106424,"statfs":{"total":96636764160,"available":96364978176,"internally_reserved":0,"allocated":231329792,"data_stored":3679613108,"data_compressed":183437604,"data_compressed_allocated":229945344,"data_compressed_original":3678978048,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574899,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":377702,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":377763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213647360,"data_stored":3418357760,"data_compressed":169361408,"data_compressed_allocated":213647360,"data_compressed_original":3418357760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349175808,"data_stored":5586812928,"data_compressed":277031808,"data_compressed_allocated":349175808,"data_compressed_original":5586812928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343932928,"data_stored":5502926848,"data_compressed":273045632,"data_compressed_allocated":343932928,"data_compressed_original":5502926848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":3678404608,"data_compressed":183463424,"data_compressed_allocated":229900288,"data_compressed_original":3678404608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":222298112,"data_stored":3556769792,"data_compressed":177356160,"data_compressed_allocated":222298112,"data_compressed_original":3556769792,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41156608,"data_stored":658505728,"data_compressed":32785920,"data_compressed_allocated":41156608,"data_compressed_original":658505728,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:46.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:46.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:46.988 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:46.989 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:47.003 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:47.002+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:47.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:47.103 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:47.101+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:47.204 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:47.203+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:47.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.218+0000 7fa548c00640 1 Processor -- start 2024-09-16T01:52:47.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.218+0000 7fa548c00640 1 -- start start 2024-09-16T01:52:47.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.218+0000 7fb46a800640 1 Processor -- start 2024-09-16T01:52:47.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.218+0000 7fb46a800640 1 -- start start 2024-09-16T01:52:47.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.219+0000 7fa548c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5440d0490 0x7fa5440c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.219+0000 7fa548c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54415e220 con 0x7fa5440d0870 2024-09-16T01:52:47.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.219+0000 7fa548c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54415e500 con 0x7fa544154da0 2024-09-16T01:52:47.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.219+0000 7fa548c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa5440c6160 con 0x7fa5440d0490 2024-09-16T01:52:47.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.219+0000 7fa543e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa544154da0 0x7fa5440c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50032/0 (socket says 172.21.15.146:50032) 2024-09-16T01:52:47.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fa543e00640 1 -- 172.21.15.146:0/3927657632 learned_addr learned my addr 172.21.15.146:0/3927657632 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb46a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb4640cc0c0 0x7fb46415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb46a800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb4640cb2f0 0x7fb4640cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb46a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb46415b1d0 con 0x7fb46415e9d0 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb46a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb4640cbd00 con 0x7fb4640cc0c0 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb46a800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb46415ba70 con 0x7fb4640cb2f0 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb463400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb4640cc0c0 0x7fb46415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fa543400640 1 --2- 172.21.15.146:0/3927657632 >> v2:172.21.15.161:3300/0 conn(0x7fa5440d0490 0x7fa5440c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb463400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb4640cc0c0 0x7fb46415aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45740/0 (socket says 172.21.15.146:45740) 2024-09-16T01:52:47.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb463400640 1 -- 172.21.15.146:0/1647816223 learned_addr learned my addr 172.21.15.146:0/1647816223 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:47.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fb462a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb4640cb2f0 0x7fb4640cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1638819603 0 0) 0x7fa54415e500 con 0x7fa544154da0 2024-09-16T01:52:47.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.220+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa528003520 con 0x7fa544154da0 2024-09-16T01:52:47.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fb463400640 1 -- 172.21.15.146:0/1647816223 >> v2:172.21.15.161:3300/0 conn(0x7fb4640cb2f0 msgr2=0x7fb4640cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:47.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fb463400640 1 --2- 172.21.15.146:0/1647816223 >> v2:172.21.15.161:3300/0 conn(0x7fb4640cb2f0 0x7fb4640cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fb463400640 1 -- 172.21.15.146:0/1647816223 >> v1:172.21.15.146:6789/0 conn(0x7fb46415e9d0 legacy=0x7fb4640cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fb463400640 1 -- 172.21.15.146:0/1647816223 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46415b4b0 con 0x7fb4640cc0c0 2024-09-16T01:52:47.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4035827459 0 0) 0x7fa528003520 con 0x7fa544154da0 2024-09-16T01:52:47.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 >> v2:172.21.15.161:3300/0 conn(0x7fa5440d0490 msgr2=0x7fa5440c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:47.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 --2- 172.21.15.146:0/3927657632 >> v2:172.21.15.161:3300/0 conn(0x7fa5440d0490 0x7fa5440c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 >> v1:172.21.15.146:6789/0 conn(0x7fa5440d0870 legacy=0x7fa544154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54415e7e0 con 0x7fa544154da0 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fa53c0019f0 con 0x7fa544154da0 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa543400640 1 --2- 172.21.15.146:0/3927657632 >> v2:172.21.15.161:3300/0 conn(0x7fa5440d0490 0x7fa5440c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 >> v1:172.21.15.161:6789/0 conn(0x7fa544154da0 legacy=0x7fa5440c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa528003a00 0x7fa528003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528004800 0x7fa528004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.221+0000 7fa542000640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5280051c0 0x7fa52800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa5440c6160 con 0x7fa5280051c0 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5280148b0 con 0x7fa528004800 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa528014ab0 con 0x7fa528003a00 2024-09-16T01:52:47.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543400640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa528003a00 0x7fa528003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fb463400640 1 --2- 172.21.15.146:0/1647816223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb46415aab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb4640ce570 tx=0x7fb45402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29a024fefb65f84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fb462000640 1 -- 172.21.15.146:0/1647816223 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454037070 con 0x7fb4640cc0c0 2024-09-16T01:52:47.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543400640 1 -- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528004800 msgr2=0x7fa528004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:52:47.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543400640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528004800 0x7fa528004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543400640 1 -- 172.21.15.146:0/3927657632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5280051c0 msgr2=0x7fa52800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:47.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fb462000640 1 -- 172.21.15.146:0/1647816223 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb454032440 con 0x7fb4640cc0c0 2024-09-16T01:52:47.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fb462000640 1 -- 172.21.15.146:0/1647816223 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454032810 con 0x7fb4640cc0c0 2024-09-16T01:52:47.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543400640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5280051c0 0x7fa52800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543400640 1 -- 172.21.15.146:0/3927657632 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa528003520 con 0x7fa528003a00 2024-09-16T01:52:47.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.222+0000 7fa543e00640 1 -- 172.21.15.146:0/3927657632 reap_dead start 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.223+0000 7fb46a800640 1 -- 172.21.15.146:0/1647816223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 msgr2=0x7fb46415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.223+0000 7fb46a800640 1 --2- 172.21.15.146:0/1647816223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb46415aab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb4640ce570 tx=0x7fb45402a270 comp rx=0 tx=0).stop 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.223+0000 7fb46a800640 1 -- 172.21.15.146:0/1647816223 shutdown_connections 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.223+0000 7fb46a800640 1 --2- 172.21.15.146:0/1647816223 >> v2:172.21.15.161:3300/0 conn(0x7fb4640cb2f0 0x7fb4640cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.223+0000 7fb46a800640 1 --2- 172.21.15.146:0/1647816223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb46415aab0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.223+0000 7fb46a800640 1 -- 172.21.15.146:0/1647816223 >> 172.21.15.146:0/1647816223 conn(0x7fb46406d560 msgr2=0x7fb46406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.224+0000 7fb46a800640 1 -- 172.21.15.146:0/1647816223 shutdown_connections 2024-09-16T01:52:47.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.224+0000 7fb46a800640 1 -- 172.21.15.146:0/1647816223 wait complete. 2024-09-16T01:52:47.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.224+0000 7fa543400640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa528003a00 0x7fa528003e20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa534009c90 tx=0x7fa528005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=739947c8da2748 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.224+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534036070 con 0x7fa528003a00 2024-09-16T01:52:47.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.224+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa534031400 con 0x7fa528003a00 2024-09-16T01:52:47.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.224+0000 7fa542000640 1 -- 172.21.15.146:0/3927657632 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534009860 con 0x7fa528003a00 2024-09-16T01:52:47.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.225+0000 7fb46a800640 1 Processor -- start 2024-09-16T01:52:47.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.225+0000 7fa548c00640 1 -- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa528003a00 msgr2=0x7fa528003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.225+0000 7fa548c00640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa528003a00 0x7fa528003e20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa534009c90 tx=0x7fa528005830 comp rx=0 tx=0).stop 2024-09-16T01:52:47.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fb46a800640 1 -- start start 2024-09-16T01:52:47.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fa548c00640 1 -- 172.21.15.146:0/3927657632 shutdown_connections 2024-09-16T01:52:47.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fa548c00640 1 --2- 172.21.15.146:0/3927657632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa528003a00 0x7fa528003e20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fa548c00640 1 -- 172.21.15.146:0/3927657632 >> 172.21.15.146:0/3927657632 conn(0x7fa54406d6f0 msgr2=0x7fa54406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:47.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fa548c00640 1 -- 172.21.15.146:0/3927657632 shutdown_connections 2024-09-16T01:52:47.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fa548c00640 1 -- 172.21.15.146:0/3927657632 wait complete. 2024-09-16T01:52:47.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fb46a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4640cb2f0 0x7fb464152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fb46a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb4641f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.226+0000 7fb46a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb46415e9d0 0x7fb46414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb46a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb46415e1d0 con 0x7fb4640cc0c0 2024-09-16T01:52:47.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb46a800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb46415e540 con 0x7fb46415e9d0 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb46a800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4640c5560 con 0x7fb4640cb2f0 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb462a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb4641f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb462a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb4641f0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45748/0 (socket says 172.21.15.146:45748) 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb462a00640 1 -- 172.21.15.146:0/3884541229 learned_addr learned my addr 172.21.15.146:0/3884541229 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463400640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4640cb2f0 0x7fb464152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463e00640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb46415e9d0 0x7fb46414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463400640 1 -- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb46415e9d0 msgr2=0x7fb46414b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463400640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb46415e9d0 0x7fb46414b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463400640 1 -- 172.21.15.146:0/3884541229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 msgr2=0x7fb4641f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463400640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb4641f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb463400640 1 -- 172.21.15.146:0/3884541229 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4641506f0 con 0x7fb4640cb2f0 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.227+0000 7fb462a00640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb4641f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:47.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.228+0000 7fb463400640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4640cb2f0 0x7fb464152a60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb4640ce1d0 tx=0x7fb4540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.228+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454041030 con 0x7fb4640cb2f0 2024-09-16T01:52:47.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.228+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb454030500 con 0x7fb4640cb2f0 2024-09-16T01:52:47.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.228+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb464150990 con 0x7fb4640cb2f0 2024-09-16T01:52:47.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.228+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454037040 con 0x7fb4640cb2f0 2024-09-16T01:52:47.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.228+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4641f3a50 con 0x7fb4640cb2f0 2024-09-16T01:52:47.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.229+0000 7fa548c00640 1 Processor -- start 2024-09-16T01:52:47.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.229+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4640c7b70 con 0x7fb4640cb2f0 2024-09-16T01:52:47.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.230+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb454044070 con 0x7fb4640cb2f0 2024-09-16T01:52:47.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.230+0000 7fa548c00640 1 -- start start 2024-09-16T01:52:47.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fb45b400640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb44406c2b0 0x7fb44406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 6119+0+0 (secure 0 0 0) 0x7fb4540aba30 con 0x7fb4640cb2f0 2024-09-16T01:52:47.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa548c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528003a00 0x7fa544157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa548c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 0x7fa54415d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa548c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa544156700 0x7fa5440b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa548c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa54415dec0 con 0x7fa544156700 2024-09-16T01:52:47.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa548c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa54415eda0 con 0x7fa528003a00 2024-09-16T01:52:47.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa548c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa544154690 con 0x7fa5440b5ad0 2024-09-16T01:52:47.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa542a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 0x7fa54415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa542a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 0x7fa54415d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38218/0 (socket says 172.21.15.146:38218) 2024-09-16T01:52:47.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.231+0000 7fa542a00640 1 -- 172.21.15.146:0/3949919236 learned_addr learned my addr 172.21.15.146:0/3949919236 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:47.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fb462a00640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb44406c2b0 0x7fb44406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528003a00 0x7fa544157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa544156700 0x7fa5440b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa542a00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528003a00 msgr2=0x7fa544157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa528003a00 0x7fa544157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa542a00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa544156700 msgr2=0x7fa5440b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa544156700 0x7fa5440b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa542a00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5441581f0 con 0x7fa5440b5ad0 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa544156700 0x7fa5440b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:52:47.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.232+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 0x7fa54415d260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa53000c430 tx=0x7fa53000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.233+0000 7fb462a00640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb44406c2b0 0x7fb44406e770 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb4640b2eb0 tx=0x7fb450009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.235+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530017070 con 0x7fa5440b5ad0 2024-09-16T01:52:47.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.236+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5300052f0 con 0x7fa5440b5ad0 2024-09-16T01:52:47.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.236+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530003830 con 0x7fa5440b5ad0 2024-09-16T01:52:47.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.236+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa544155320 con 0x7fa5440b5ad0 2024-09-16T01:52:47.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.236+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5441558e0 con 0x7fa5440b5ad0 2024-09-16T01:52:47.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.236+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb4540799a0 con 0x7fb4640cb2f0 2024-09-16T01:52:47.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.237+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa544154a20 con 0x7fa5440b5ad0 2024-09-16T01:52:47.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.238+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa53001a070 con 0x7fa5440b5ad0 2024-09-16T01:52:47.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.239+0000 7fa53b400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa52406c2e0 0x7fa52406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.239+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 6119+0+0 (secure 0 0 0) 0x7fa53008a890 con 0x7fa5440b5ad0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.240+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v113) ==== 54+0+16 (secure 0 0 0) 0x7fa53008abb0 con 0x7fa5440b5ad0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.240+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa52406c2e0 0x7fa52406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.240+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa544155ab0 0x7fa5440c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.240+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5440cc3f0 con 0x7fa544155ab0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.241+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa544155ab0 0x7fa5440c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.241+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa52406c2e0 0x7fa52406e7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa534033000 tx=0x7fa534008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.241+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa544155ab0 0x7fa5440c51f0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.242+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5440cc3f0 con 0x7fa544155ab0 2024-09-16T01:52:47.268 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:47.268 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:47.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.272+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5440008d0 con 0x7fa544155ab0 2024-09-16T01:52:47.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.275+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5440008d0 con 0x7fa544155ab0 2024-09-16T01:52:47.278 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:47.279 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:47.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.276+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa544200840 0x7fa544200c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.276+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa544201180 con 0x7fa544200840 2024-09-16T01:52:47.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.276+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa544200840 0x7fa544200c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.277+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa544200840 0x7fa544200c40 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.278+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa53403e040 con 0x7fa544200840 2024-09-16T01:52:47.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:47.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.306+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5441f3000 con 0x7fa544200840 2024-09-16T01:52:47.308 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:47.306+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:47.312 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:47.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.308+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5441f3000 con 0x7fa544200840 2024-09-16T01:52:47.312 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:47.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.308+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa544062630 0x7fa544062a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.308+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa5441f3ae0 con 0x7fa544062630 2024-09-16T01:52:47.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.309+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa544062630 0x7fa544062a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.309+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa544062630 0x7fa544062a90 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.310+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5441f3ae0 con 0x7fa544062630 2024-09-16T01:52:47.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.326+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa544062fd0 con 0x7fa544062630 2024-09-16T01:52:47.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.328+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa544062fd0 con 0x7fa544062630 2024-09-16T01:52:47.330 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:47.331 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:47.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.329+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa544152630 0x7fa544152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.329+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa5440635a0 con 0x7fa544152630 2024-09-16T01:52:47.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.329+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa544152630 0x7fa544152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.330+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa544152630 0x7fa544152a90 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.331+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5440635a0 con 0x7fa544152630 2024-09-16T01:52:47.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.350+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa54402c640 con 0x7fa544152630 2024-09-16T01:52:47.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.352+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa54402c640 con 0x7fa544152630 2024-09-16T01:52:47.354 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:47.355 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:47.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.352+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa54402cd60 0x7fa54402d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.352+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa54402c640 con 0x7fa54402cd60 2024-09-16T01:52:47.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.353+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa54402cd60 0x7fa54402d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.353+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa54402cd60 0x7fa54402d1e0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.354+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa54402c640 con 0x7fa54402cd60 2024-09-16T01:52:47.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.370+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa54402c640 con 0x7fa54402cd60 2024-09-16T01:52:47.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.372+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa54402c640 con 0x7fa54402cd60 2024-09-16T01:52:47.375 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:47.375 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:47.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.373+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa544211930 0x7fa544213d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.373+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa544216af0 con 0x7fa544211930 2024-09-16T01:52:47.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.373+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa544211930 0x7fa544213d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.374+0000 7fa542a00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa544211930 0x7fa544213d20 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.375+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa544216af0 con 0x7fa544211930 2024-09-16T01:52:47.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.390+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5442142e0 con 0x7fa544211930 2024-09-16T01:52:47.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.393+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5442142e0 con 0x7fa544211930 2024-09-16T01:52:47.395 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:47.395 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:47.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.393+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa544214a00 0x7fa544216df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.393+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa544219dc0 con 0x7fa544214a00 2024-09-16T01:52:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.393+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa544214a00 0x7fa544216df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.394+0000 7fa543e00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa544214a00 0x7fa544216df0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.395+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa544219dc0 con 0x7fa544214a00 2024-09-16T01:52:47.406 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:47.405+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:47.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.414+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa544219d90 con 0x7fa544214a00 2024-09-16T01:52:47.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.416+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa544219d90 con 0x7fa544214a00 2024-09-16T01:52:47.418 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:47.418 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:47.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.416+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa544219050 0x7fa544219490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:47.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.416+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa54421d190 con 0x7fa544219050 2024-09-16T01:52:47.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.417+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa544219050 0x7fa544219490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:47.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.417+0000 7fa543400640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa544219050 0x7fa544219490 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:47.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.418+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa54421d190 con 0x7fa544219050 2024-09-16T01:52:47.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.437+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa544217400 con 0x7fa544219050 2024-09-16T01:52:47.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa53b400640 1 -- 172.21.15.146:0/3949919236 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa544217400 con 0x7fa544219050 2024-09-16T01:52:47.441 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:47.442 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:47.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa544155ab0 msgr2=0x7fa5440c51f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fa544155ab0 0x7fa5440c51f0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa544200840 msgr2=0x7fa544200c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fa544200840 0x7fa544200c40 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa544062630 msgr2=0x7fa544062a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fa544062630 0x7fa544062a90 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa544152630 msgr2=0x7fa544152a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fa544152630 0x7fa544152a90 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa54402cd60 msgr2=0x7fa54402d1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fa54402cd60 0x7fa54402d1e0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa544211930 msgr2=0x7fa544213d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fa544211930 0x7fa544213d20 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa544214a00 msgr2=0x7fa544216df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa544214a00 0x7fa544216df0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa544219050 msgr2=0x7fa544219490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fa544219050 0x7fa544219490 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa543e00640 1 -- 172.21.15.146:0/3949919236 reap_dead start 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa543e00640 1 -- 172.21.15.146:0/3949919236 reap_dead start 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa543e00640 1 -- 172.21.15.146:0/3949919236 reap_dead start 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa543e00640 1 -- 172.21.15.146:0/3949919236 reap_dead start 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa543e00640 1 -- 172.21.15.146:0/3949919236 reap_dead start 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa52406c2e0 msgr2=0x7fa52406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.439+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa52406c2e0 0x7fa52406e7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa534033000 tx=0x7fa534008030 comp rx=0 tx=0).stop 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 msgr2=0x7fa54415d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 0x7fa54415d260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa53000c430 tx=0x7fa53000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 shutdown_connections 2024-09-16T01:52:47.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa52406c2e0 0x7fa52406e7a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 --2- 172.21.15.146:0/3949919236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5440b5ad0 0x7fa54415d260 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 >> 172.21.15.146:0/3949919236 conn(0x7fa54406d6f0 msgr2=0x7fa5440cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 shutdown_connections 2024-09-16T01:52:47.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.440+0000 7fa548c00640 1 -- 172.21.15.146:0/3949919236 wait complete. 2024-09-16T01:52:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.469+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb4640cbea0 con 0x7fb44406c2b0 2024-09-16T01:52:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.477+0000 7fb45b400640 1 -- 172.21.15.146:0/3884541229 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254432 (secure 0 0 0) 0x7fb4640cbea0 con 0x7fb44406c2b0 2024-09-16T01:52:47.482 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:47.484 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:47.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.482+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb44406c2b0 msgr2=0x7fb44406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.482+0000 7fb46a800640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb44406c2b0 0x7fb44406e770 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb4640b2eb0 tx=0x7fb450009190 comp rx=0 tx=0).stop 2024-09-16T01:52:47.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.482+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4640cb2f0 msgr2=0x7fb464152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:47.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.482+0000 7fb46a800640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4640cb2f0 0x7fb464152a60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb4640ce1d0 tx=0x7fb4540046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 shutdown_connections 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb44406c2b0 0x7fb44406e770 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb46415e9d0 0x7fb46414b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4640cc0c0 0x7fb4641f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 --2- 172.21.15.146:0/3884541229 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4640cb2f0 0x7fb464152a60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 >> 172.21.15.146:0/3884541229 conn(0x7fb46406d560 msgr2=0x7fb4640cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 shutdown_connections 2024-09-16T01:52:47.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:47.483+0000 7fb46a800640 1 -- 172.21.15.146:0/3884541229 wait complete. 2024-09-16T01:52:47.493 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:47.493 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:47.493 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:47.493 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:47.493 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:47.504 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-16T01:52:46.563482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":4,"num_objects_missing":7,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":6905,"num_write_kb":23931816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2460,"num_bytes_recovered":7415776429,"num_keys_recovered":115,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7346,"ondisk_log_size":7346,"up":194,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1573404,"kb_used_data":1356644,"kb_used_omap":9,"kb_used_meta":216118,"kb_avail":564657636,"statfs":{"total":579820584960,"available":578209419264,"internally_reserved":0,"allocated":1389203456,"data_stored":22093809590,"data_compressed":1096590311,"data_compressed_allocated":1380651008,"data_compressed_original":22089633792,"omap_allocated":9562,"internal_metadata":221305510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":565248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":406847488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.853567"},"pg_stats":[{"pgid":"3.3f","version":"105'98","reported_seq":360,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712176+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:40.712176+0000","last_peered":"2024-09-16T01:52:40.712176+0000","last_clean":"2024-09-16T01:52:40.712176+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:40.712176+0000","last_undegraded":"2024-09-16T01:52:40.712176+0000","last_fullsized":"2024-09-16T01:52:40.712176+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532378+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:45.532378+0000","last_peered":"2024-09-16T01:52:45.532378+0000","last_clean":"2024-09-16T01:52:45.532378+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:45.532378+0000","last_undegraded":"2024-09-16T01:52:45.532378+0000","last_fullsized":"2024-09-16T01:52:45.532378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":446,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.091998+0000","last_change":"2024-09-16T01:52:45.534236+0000","last_active":"2024-09-16T01:52:45.534236+0000","last_peered":"2024-09-16T01:52:45.091998+0000","last_clean":"2024-09-16T01:52:45.091998+0000","last_became_active":"2024-09-16T01:52:03.309937+0000","last_became_peered":"2024-09-16T01:52:03.309937+0000","last_unstale":"2024-09-16T01:52:45.091998+0000","last_undegraded":"2024-09-16T01:52:45.091998+0000","last_fullsized":"2024-09-16T01:52:45.091998+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":209715200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"107'120","reported_seq":308,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702041+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:40.702041+0000","last_peered":"2024-09-16T01:52:40.702041+0000","last_clean":"2024-09-16T01:52:40.702041+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:40.702041+0000","last_undegraded":"2024-09-16T01:52:40.702041+0000","last_fullsized":"2024-09-16T01:52:40.702041+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"110'109","reported_seq":273,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507567+0000","last_change":"2024-09-16T01:52:45.533438+0000","last_active":"2024-09-16T01:52:45.533438+0000","last_peered":"2024-09-16T01:52:44.507567+0000","last_clean":"2024-09-16T01:52:44.507567+0000","last_became_active":"2024-09-16T01:52:09.424479+0000","last_became_peered":"2024-09-16T01:52:09.424479+0000","last_unstale":"2024-09-16T01:52:44.507567+0000","last_undegraded":"2024-09-16T01:52:44.507567+0000","last_fullsized":"2024-09-16T01:52:44.507567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"106'96","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702170+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:40.702170+0000","last_peered":"2024-09-16T01:52:40.702170+0000","last_clean":"2024-09-16T01:52:40.702170+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:40.702170+0000","last_undegraded":"2024-09-16T01:52:40.702170+0000","last_fullsized":"2024-09-16T01:52:40.702170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"107'120","reported_seq":477,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.115504+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:41.115504+0000","last_peered":"2024-09-16T01:52:41.115504+0000","last_clean":"2024-09-16T01:52:41.115504+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:41.115504+0000","last_undegraded":"2024-09-16T01:52:41.115504+0000","last_fullsized":"2024-09-16T01:52:41.115504+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"111'127","reported_seq":445,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532294+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:45.532294+0000","last_peered":"2024-09-16T01:52:45.532294+0000","last_clean":"2024-09-16T01:52:45.532294+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:45.532294+0000","last_undegraded":"2024-09-16T01:52:45.532294+0000","last_fullsized":"2024-09-16T01:52:45.532294+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":466944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"107'108","reported_seq":291,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.840917+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:40.840917+0000","last_peered":"2024-09-16T01:52:40.840917+0000","last_clean":"2024-09-16T01:52:40.840917+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:40.840917+0000","last_undegraded":"2024-09-16T01:52:40.840917+0000","last_fullsized":"2024-09-16T01:52:40.840917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"107'108","reported_seq":395,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.979932+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:40.979932+0000","last_peered":"2024-09-16T01:52:40.979932+0000","last_clean":"2024-09-16T01:52:40.979932+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:40.979932+0000","last_undegraded":"2024-09-16T01:52:40.979932+0000","last_fullsized":"2024-09-16T01:52:40.979932+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"107'130","reported_seq":442,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714130+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:40.714130+0000","last_peered":"2024-09-16T01:52:40.714130+0000","last_clean":"2024-09-16T01:52:40.714130+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:40.714130+0000","last_undegraded":"2024-09-16T01:52:40.714130+0000","last_fullsized":"2024-09-16T01:52:40.714130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":479232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"107'101","reported_seq":92,"reported_epoch":107,"state":"active+recovering","last_fresh":"2024-09-16T01:52:41.593897+0000","last_change":"2024-09-16T01:52:40.740404+0000","last_active":"2024-09-16T01:52:41.593897+0000","last_peered":"2024-09-16T01:52:41.593897+0000","last_clean":"2024-09-16T01:52:36.904016+0000","last_became_active":"2024-09-16T01:52:40.724710+0000","last_became_peered":"2024-09-16T01:52:40.724710+0000","last_unstale":"2024-09-16T01:52:41.593897+0000","last_undegraded":"2024-09-16T01:52:41.593897+0000","last_fullsized":"2024-09-16T01:52:41.593897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":452984832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"1,2","objects":28},{"shards":"1,2,6","objects":2},{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"111'111","reported_seq":427,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.529634+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:45.529634+0000","last_peered":"2024-09-16T01:52:45.529634+0000","last_clean":"2024-09-16T01:52:45.529634+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:45.529634+0000","last_undegraded":"2024-09-16T01:52:45.529634+0000","last_fullsized":"2024-09-16T01:52:45.529634+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"110'103","reported_seq":346,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527928+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:45.527928+0000","last_peered":"2024-09-16T01:52:45.527928+0000","last_clean":"2024-09-16T01:52:45.527928+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:45.527928+0000","last_undegraded":"2024-09-16T01:52:45.527928+0000","last_fullsized":"2024-09-16T01:52:45.527928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"111'108","reported_seq":404,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.531409+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:45.531409+0000","last_peered":"2024-09-16T01:52:45.531409+0000","last_clean":"2024-09-16T01:52:45.531409+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:45.531409+0000","last_undegraded":"2024-09-16T01:52:45.531409+0000","last_fullsized":"2024-09-16T01:52:45.531409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"107'102","reported_seq":284,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708519+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:40.708519+0000","last_peered":"2024-09-16T01:52:40.708519+0000","last_clean":"2024-09-16T01:52:40.708519+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:40.708519+0000","last_undegraded":"2024-09-16T01:52:40.708519+0000","last_fullsized":"2024-09-16T01:52:40.708519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"106'98","reported_seq":377,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712476+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:40.712476+0000","last_peered":"2024-09-16T01:52:40.712476+0000","last_clean":"2024-09-16T01:52:40.712476+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:40.712476+0000","last_undegraded":"2024-09-16T01:52:40.712476+0000","last_fullsized":"2024-09-16T01:52:40.712476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"107'103","reported_seq":371,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.454439+0000","last_change":"2024-09-16T01:52:05.041021+0000","last_active":"2024-09-16T01:52:41.454439+0000","last_peered":"2024-09-16T01:52:41.454439+0000","last_clean":"2024-09-16T01:52:41.454439+0000","last_became_active":"2024-09-16T01:52:03.349420+0000","last_became_peered":"2024-09-16T01:52:03.349420+0000","last_unstale":"2024-09-16T01:52:41.454439+0000","last_undegraded":"2024-09-16T01:52:41.454439+0000","last_fullsized":"2024-09-16T01:52:41.454439+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:31.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"104'115","reported_seq":352,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T01:52:35.045658+0000","last_change":"2024-09-16T01:51:50.466227+0000","last_active":"2024-09-16T01:52:35.045658+0000","last_peered":"2024-09-16T01:52:35.045658+0000","last_clean":"2024-09-16T01:52:35.045658+0000","last_became_active":"2024-09-16T01:51:49.177542+0000","last_became_peered":"2024-09-16T01:51:49.177542+0000","last_unstale":"2024-09-16T01:52:35.045658+0000","last_undegraded":"2024-09-16T01:52:35.045658+0000","last_fullsized":"2024-09-16T01:52:35.045658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:14.445720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":88080384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"107'108","reported_seq":434,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.043648+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:41.043648+0000","last_peered":"2024-09-16T01:52:41.043648+0000","last_clean":"2024-09-16T01:52:41.043648+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:41.043648+0000","last_undegraded":"2024-09-16T01:52:41.043648+0000","last_fullsized":"2024-09-16T01:52:41.043648+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"106'83","reported_seq":303,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705785+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:40.705785+0000","last_peered":"2024-09-16T01:52:40.705785+0000","last_clean":"2024-09-16T01:52:40.705785+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:40.705785+0000","last_undegraded":"2024-09-16T01:52:40.705785+0000","last_fullsized":"2024-09-16T01:52:40.705785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"105'120","reported_seq":432,"reported_epoch":105,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:39.297428+0000","last_change":"2024-09-16T01:52:40.701785+0000","last_active":"2024-09-16T01:52:40.701785+0000","last_peered":"2024-09-16T01:52:39.297428+0000","last_clean":"2024-09-16T01:52:39.297428+0000","last_became_active":"2024-09-16T01:52:09.425675+0000","last_became_peered":"2024-09-16T01:52:09.425675+0000","last_unstale":"2024-09-16T01:52:39.297428+0000","last_undegraded":"2024-09-16T01:52:39.297428+0000","last_fullsized":"2024-09-16T01:52:39.297428+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":197132288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532334+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:45.532334+0000","last_peered":"2024-09-16T01:52:45.532334+0000","last_clean":"2024-09-16T01:52:45.532334+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:45.532334+0000","last_undegraded":"2024-09-16T01:52:45.532334+0000","last_fullsized":"2024-09-16T01:52:45.532334+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"107'127","reported_seq":309,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708637+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:40.708637+0000","last_peered":"2024-09-16T01:52:40.708637+0000","last_clean":"2024-09-16T01:52:40.708637+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:40.708637+0000","last_undegraded":"2024-09-16T01:52:40.708637+0000","last_fullsized":"2024-09-16T01:52:40.708637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":454656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"111'100","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532061+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:45.532061+0000","last_peered":"2024-09-16T01:52:45.532061+0000","last_clean":"2024-09-16T01:52:45.532061+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:45.532061+0000","last_undegraded":"2024-09-16T01:52:45.532061+0000","last_fullsized":"2024-09-16T01:52:45.532061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"107'133","reported_seq":316,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.872103+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:40.872103+0000","last_peered":"2024-09-16T01:52:40.872103+0000","last_clean":"2024-09-16T01:52:40.872103+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:40.872103+0000","last_undegraded":"2024-09-16T01:52:40.872103+0000","last_fullsized":"2024-09-16T01:52:40.872103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"110'102","reported_seq":423,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.528487+0000","last_change":"2024-09-16T01:52:45.527683+0000","last_active":"2024-09-16T01:52:45.527683+0000","last_peered":"2024-09-16T01:52:44.528487+0000","last_clean":"2024-09-16T01:52:44.528487+0000","last_became_active":"2024-09-16T01:52:03.375641+0000","last_became_peered":"2024-09-16T01:52:03.375641+0000","last_unstale":"2024-09-16T01:52:44.528487+0000","last_undegraded":"2024-09-16T01:52:44.528487+0000","last_fullsized":"2024-09-16T01:52:44.528487+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"111'98","reported_seq":384,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528055+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:45.528055+0000","last_peered":"2024-09-16T01:52:45.528055+0000","last_clean":"2024-09-16T01:52:45.528055+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:45.528055+0000","last_undegraded":"2024-09-16T01:52:45.528055+0000","last_fullsized":"2024-09-16T01:52:45.528055+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"109'87","reported_seq":481,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507610+0000","last_change":"2024-09-16T01:52:45.531675+0000","last_active":"2024-09-16T01:52:45.531675+0000","last_peered":"2024-09-16T01:52:44.507610+0000","last_clean":"2024-09-16T01:52:44.507610+0000","last_became_active":"2024-09-16T01:52:09.378210+0000","last_became_peered":"2024-09-16T01:52:09.378210+0000","last_unstale":"2024-09-16T01:52:44.507610+0000","last_undegraded":"2024-09-16T01:52:44.507610+0000","last_fullsized":"2024-09-16T01:52:44.507610+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"106'94","reported_seq":276,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709371+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:40.709371+0000","last_peered":"2024-09-16T01:52:40.709371+0000","last_clean":"2024-09-16T01:52:40.709371+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:40.709371+0000","last_undegraded":"2024-09-16T01:52:40.709371+0000","last_fullsized":"2024-09-16T01:52:40.709371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":319488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"106'96","reported_seq":382,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.704088+0000","last_change":"2024-09-16T01:52:22.869283+0000","last_active":"2024-09-16T01:52:40.704088+0000","last_peered":"2024-09-16T01:52:40.704088+0000","last_clean":"2024-09-16T01:52:40.704088+0000","last_became_active":"2024-09-16T01:52:21.489999+0000","last_became_peered":"2024-09-16T01:52:21.489999+0000","last_unstale":"2024-09-16T01:52:40.704088+0000","last_undegraded":"2024-09-16T01:52:40.704088+0000","last_fullsized":"2024-09-16T01:52:40.704088+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"111'111","reported_seq":417,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527513+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:45.527513+0000","last_peered":"2024-09-16T01:52:45.527513+0000","last_clean":"2024-09-16T01:52:45.527513+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:45.527513+0000","last_undegraded":"2024-09-16T01:52:45.527513+0000","last_fullsized":"2024-09-16T01:52:45.527513+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"107'93","reported_seq":276,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705218+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:40.705218+0000","last_peered":"2024-09-16T01:52:40.705218+0000","last_clean":"2024-09-16T01:52:40.705218+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:40.705218+0000","last_undegraded":"2024-09-16T01:52:40.705218+0000","last_fullsized":"2024-09-16T01:52:40.705218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":200,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534083+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:45.534083+0000","last_peered":"2024-09-16T01:52:45.534083+0000","last_clean":"2024-09-16T01:52:45.534083+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:45.534083+0000","last_undegraded":"2024-09-16T01:52:45.534083+0000","last_fullsized":"2024-09-16T01:52:45.534083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"106'99","reported_seq":519,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715559+0000","last_change":"2024-09-16T01:52:22.369179+0000","last_active":"2024-09-16T01:52:40.715559+0000","last_peered":"2024-09-16T01:52:40.715559+0000","last_clean":"2024-09-16T01:52:40.715559+0000","last_became_active":"2024-09-16T01:52:21.480658+0000","last_became_peered":"2024-09-16T01:52:21.480658+0000","last_unstale":"2024-09-16T01:52:40.715559+0000","last_undegraded":"2024-09-16T01:52:40.715559+0000","last_fullsized":"2024-09-16T01:52:40.715559+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":204,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709632+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:40.709632+0000","last_peered":"2024-09-16T01:52:40.709632+0000","last_clean":"2024-09-16T01:52:40.709632+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:40.709632+0000","last_undegraded":"2024-09-16T01:52:40.709632+0000","last_fullsized":"2024-09-16T01:52:40.709632+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":208,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.496173+0000","last_change":"2024-09-16T01:52:45.528156+0000","last_active":"2024-09-16T01:52:45.528156+0000","last_peered":"2024-09-16T01:52:44.496173+0000","last_clean":"2024-09-16T01:52:44.496173+0000","last_became_active":"2024-09-16T01:52:09.417952+0000","last_became_peered":"2024-09-16T01:52:09.417952+0000","last_unstale":"2024-09-16T01:52:44.496173+0000","last_undegraded":"2024-09-16T01:52:44.496173+0000","last_fullsized":"2024-09-16T01:52:44.496173+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"111'121","reported_seq":314,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532429+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:45.532429+0000","last_peered":"2024-09-16T01:52:45.532429+0000","last_clean":"2024-09-16T01:52:45.532429+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:45.532429+0000","last_undegraded":"2024-09-16T01:52:45.532429+0000","last_fullsized":"2024-09-16T01:52:45.532429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":438272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":187,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708659+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:40.708659+0000","last_peered":"2024-09-16T01:52:40.708659+0000","last_clean":"2024-09-16T01:52:40.708659+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:40.708659+0000","last_undegraded":"2024-09-16T01:52:40.708659+0000","last_fullsized":"2024-09-16T01:52:40.708659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"105'106","reported_seq":254,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.703982+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:40.703982+0000","last_peered":"2024-09-16T01:52:40.703982+0000","last_clean":"2024-09-16T01:52:40.703982+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:40.703982+0000","last_undegraded":"2024-09-16T01:52:40.703982+0000","last_fullsized":"2024-09-16T01:52:40.703982+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":216,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.722262+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:40.722262+0000","last_peered":"2024-09-16T01:52:40.722262+0000","last_clean":"2024-09-16T01:52:40.722262+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:40.722262+0000","last_undegraded":"2024-09-16T01:52:40.722262+0000","last_fullsized":"2024-09-16T01:52:40.722262+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"112'163","reported_seq":367,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.567223+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:45.567223+0000","last_peered":"2024-09-16T01:52:45.567223+0000","last_clean":"2024-09-16T01:52:45.567223+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:45.567223+0000","last_undegraded":"2024-09-16T01:52:45.567223+0000","last_fullsized":"2024-09-16T01:52:45.567223+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":67,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715962+0000","last_change":"2024-09-16T01:52:21.510976+0000","last_active":"2024-09-16T01:52:40.715962+0000","last_peered":"2024-09-16T01:52:40.715962+0000","last_clean":"2024-09-16T01:52:40.715962+0000","last_became_active":"2024-09-16T01:52:21.491849+0000","last_became_peered":"2024-09-16T01:52:21.491849+0000","last_unstale":"2024-09-16T01:52:40.715962+0000","last_undegraded":"2024-09-16T01:52:40.715962+0000","last_fullsized":"2024-09-16T01:52:40.715962+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:43.042548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"110'98","reported_seq":291,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532194+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:45.532194+0000","last_peered":"2024-09-16T01:52:45.532194+0000","last_clean":"2024-09-16T01:52:45.532194+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:45.532194+0000","last_undegraded":"2024-09-16T01:52:45.532194+0000","last_fullsized":"2024-09-16T01:52:45.532194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":81,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.713822+0000","last_change":"2024-09-16T01:52:21.507954+0000","last_active":"2024-09-16T01:52:40.713822+0000","last_peered":"2024-09-16T01:52:40.713822+0000","last_clean":"2024-09-16T01:52:40.713822+0000","last_became_active":"2024-09-16T01:52:21.491632+0000","last_became_peered":"2024-09-16T01:52:21.491632+0000","last_unstale":"2024-09-16T01:52:40.713822+0000","last_undegraded":"2024-09-16T01:52:40.713822+0000","last_fullsized":"2024-09-16T01:52:40.713822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:36.763063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3360,"num_keys_recovered":50,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"107'101","reported_seq":283,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.710272+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:40.710272+0000","last_peered":"2024-09-16T01:52:40.710272+0000","last_clean":"2024-09-16T01:52:40.710272+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:40.710272+0000","last_undegraded":"2024-09-16T01:52:40.710272+0000","last_fullsized":"2024-09-16T01:52:40.710272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":374,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.702091+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:40.702091+0000","last_peered":"2024-09-16T01:52:40.702091+0000","last_clean":"2024-09-16T01:52:40.702091+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:40.702091+0000","last_undegraded":"2024-09-16T01:52:40.702091+0000","last_fullsized":"2024-09-16T01:52:40.702091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":407,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.526908+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:45.526908+0000","last_peered":"2024-09-16T01:52:45.526908+0000","last_clean":"2024-09-16T01:52:45.526908+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:45.526908+0000","last_undegraded":"2024-09-16T01:52:45.526908+0000","last_fullsized":"2024-09-16T01:52:45.526908+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":244,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715431+0000","last_change":"2024-09-16T01:52:09.471855+0000","last_active":"2024-09-16T01:52:40.715431+0000","last_peered":"2024-09-16T01:52:40.715431+0000","last_clean":"2024-09-16T01:52:40.715431+0000","last_became_active":"2024-09-16T01:52:09.419033+0000","last_became_peered":"2024-09-16T01:52:09.419033+0000","last_unstale":"2024-09-16T01:52:40.715431+0000","last_undegraded":"2024-09-16T01:52:40.715431+0000","last_fullsized":"2024-09-16T01:52:40.715431+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"111'115","reported_seq":308,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528039+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:45.528039+0000","last_peered":"2024-09-16T01:52:45.528039+0000","last_clean":"2024-09-16T01:52:45.528039+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:45.528039+0000","last_undegraded":"2024-09-16T01:52:45.528039+0000","last_fullsized":"2024-09-16T01:52:45.528039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":254,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709719+0000","last_change":"2024-09-16T01:52:03.470007+0000","last_active":"2024-09-16T01:52:40.709719+0000","last_peered":"2024-09-16T01:52:40.709719+0000","last_clean":"2024-09-16T01:52:40.709719+0000","last_became_active":"2024-09-16T01:52:03.348785+0000","last_became_peered":"2024-09-16T01:52:03.348785+0000","last_unstale":"2024-09-16T01:52:40.709719+0000","last_undegraded":"2024-09-16T01:52:40.709719+0000","last_fullsized":"2024-09-16T01:52:40.709719+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"106'114","reported_seq":427,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715670+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:40.715670+0000","last_peered":"2024-09-16T01:52:40.715670+0000","last_clean":"2024-09-16T01:52:40.715670+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:40.715670+0000","last_undegraded":"2024-09-16T01:52:40.715670+0000","last_fullsized":"2024-09-16T01:52:40.715670+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":397312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":61,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708931+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:40.708931+0000","last_peered":"2024-09-16T01:52:40.708931+0000","last_clean":"2024-09-16T01:52:40.708931+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:40.708931+0000","last_undegraded":"2024-09-16T01:52:40.708931+0000","last_fullsized":"2024-09-16T01:52:40.708931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"106'128","reported_seq":311,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701595+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:40.701595+0000","last_peered":"2024-09-16T01:52:40.701595+0000","last_clean":"2024-09-16T01:52:40.701595+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:40.701595+0000","last_undegraded":"2024-09-16T01:52:40.701595+0000","last_fullsized":"2024-09-16T01:52:40.701595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":97,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701198+0000","last_change":"2024-09-16T01:52:21.494700+0000","last_active":"2024-09-16T01:52:40.701198+0000","last_peered":"2024-09-16T01:52:40.701198+0000","last_clean":"2024-09-16T01:52:40.701198+0000","last_became_active":"2024-09-16T01:52:21.466194+0000","last_became_peered":"2024-09-16T01:52:21.466194+0000","last_unstale":"2024-09-16T01:52:40.701198+0000","last_undegraded":"2024-09-16T01:52:40.701198+0000","last_fullsized":"2024-09-16T01:52:40.701198+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:55.051289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"106'139","reported_seq":321,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709029+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:40.709029+0000","last_peered":"2024-09-16T01:52:40.709029+0000","last_clean":"2024-09-16T01:52:40.709029+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:40.709029+0000","last_undegraded":"2024-09-16T01:52:40.709029+0000","last_fullsized":"2024-09-16T01:52:40.709029+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":68,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532060+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:45.532060+0000","last_peered":"2024-09-16T01:52:45.532060+0000","last_clean":"2024-09-16T01:52:45.532060+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:45.532060+0000","last_undegraded":"2024-09-16T01:52:45.532060+0000","last_fullsized":"2024-09-16T01:52:45.532060+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"107'99","reported_seq":383,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.373880+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:41.373880+0000","last_peered":"2024-09-16T01:52:41.373880+0000","last_clean":"2024-09-16T01:52:41.373880+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:41.373880+0000","last_undegraded":"2024-09-16T01:52:41.373880+0000","last_fullsized":"2024-09-16T01:52:41.373880+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532518+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:45.532518+0000","last_peered":"2024-09-16T01:52:45.532518+0000","last_clean":"2024-09-16T01:52:45.532518+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:45.532518+0000","last_undegraded":"2024-09-16T01:52:45.532518+0000","last_fullsized":"2024-09-16T01:52:45.532518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"107'96","reported_seq":375,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.258476+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:41.258476+0000","last_peered":"2024-09-16T01:52:41.258476+0000","last_clean":"2024-09-16T01:52:41.258476+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:41.258476+0000","last_undegraded":"2024-09-16T01:52:41.258476+0000","last_fullsized":"2024-09-16T01:52:41.258476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":198,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534027+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:45.534027+0000","last_peered":"2024-09-16T01:52:45.534027+0000","last_clean":"2024-09-16T01:52:45.534027+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:45.534027+0000","last_undegraded":"2024-09-16T01:52:45.534027+0000","last_fullsized":"2024-09-16T01:52:45.534027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"107'95","reported_seq":504,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.953377+0000","last_change":"2024-09-16T01:52:10.143229+0000","last_active":"2024-09-16T01:52:40.953377+0000","last_peered":"2024-09-16T01:52:40.953377+0000","last_clean":"2024-09-16T01:52:40.953377+0000","last_became_active":"2024-09-16T01:52:09.413639+0000","last_became_peered":"2024-09-16T01:52:09.413639+0000","last_unstale":"2024-09-16T01:52:40.953377+0000","last_undegraded":"2024-09-16T01:52:40.953377+0000","last_fullsized":"2024-09-16T01:52:40.953377+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":318767104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":213,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528534+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:45.528534+0000","last_peered":"2024-09-16T01:52:45.528534+0000","last_clean":"2024-09-16T01:52:45.528534+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:45.528534+0000","last_undegraded":"2024-09-16T01:52:45.528534+0000","last_fullsized":"2024-09-16T01:52:45.528534+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"107'105","reported_seq":287,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714360+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:40.714360+0000","last_peered":"2024-09-16T01:52:40.714360+0000","last_clean":"2024-09-16T01:52:40.714360+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:40.714360+0000","last_undegraded":"2024-09-16T01:52:40.714360+0000","last_fullsized":"2024-09-16T01:52:40.714360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":341,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.701702+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:40.701702+0000","last_peered":"2024-09-16T01:52:40.701702+0000","last_clean":"2024-09-16T01:52:40.701702+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:40.701702+0000","last_undegraded":"2024-09-16T01:52:40.701702+0000","last_fullsized":"2024-09-16T01:52:40.701702+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":190,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715864+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:40.715864+0000","last_peered":"2024-09-16T01:52:40.715864+0000","last_clean":"2024-09-16T01:52:40.715864+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:40.715864+0000","last_undegraded":"2024-09-16T01:52:40.715864+0000","last_fullsized":"2024-09-16T01:52:40.715864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"107'106","reported_seq":429,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.741036+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:40.741036+0000","last_peered":"2024-09-16T01:52:40.741036+0000","last_clean":"2024-09-16T01:52:40.741036+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:40.741036+0000","last_undegraded":"2024-09-16T01:52:40.741036+0000","last_fullsized":"2024-09-16T01:52:40.741036+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":188,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708690+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:40.708690+0000","last_peered":"2024-09-16T01:52:40.708690+0000","last_clean":"2024-09-16T01:52:40.708690+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:40.708690+0000","last_undegraded":"2024-09-16T01:52:40.708690+0000","last_fullsized":"2024-09-16T01:52:40.708690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"110'104","reported_seq":297,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527497+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:45.527497+0000","last_peered":"2024-09-16T01:52:45.527497+0000","last_clean":"2024-09-16T01:52:45.527497+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:45.527497+0000","last_undegraded":"2024-09-16T01:52:45.527497+0000","last_fullsized":"2024-09-16T01:52:45.527497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":331776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":240,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507474+0000","last_change":"2024-09-16T01:52:45.531950+0000","last_active":"2024-09-16T01:52:45.531950+0000","last_peered":"2024-09-16T01:52:44.507474+0000","last_clean":"2024-09-16T01:52:44.507474+0000","last_became_active":"2024-09-16T01:52:21.493366+0000","last_became_peered":"2024-09-16T01:52:21.493366+0000","last_unstale":"2024-09-16T01:52:44.507474+0000","last_undegraded":"2024-09-16T01:52:44.507474+0000","last_fullsized":"2024-09-16T01:52:44.507474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"111'93","reported_seq":410,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.298724+0000","last_change":"2024-09-16T01:52:45.533111+0000","last_active":"2024-09-16T01:52:45.533111+0000","last_peered":"2024-09-16T01:52:45.298724+0000","last_clean":"2024-09-16T01:52:45.298724+0000","last_became_active":"2024-09-16T01:52:03.344719+0000","last_became_peered":"2024-09-16T01:52:03.344719+0000","last_unstale":"2024-09-16T01:52:45.298724+0000","last_undegraded":"2024-09-16T01:52:45.298724+0000","last_fullsized":"2024-09-16T01:52:45.298724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":150994944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":189,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.709793+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:40.709793+0000","last_peered":"2024-09-16T01:52:40.709793+0000","last_clean":"2024-09-16T01:52:40.709793+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:40.709793+0000","last_undegraded":"2024-09-16T01:52:40.709793+0000","last_fullsized":"2024-09-16T01:52:40.709793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"106'107","reported_seq":389,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.705561+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:40.705561+0000","last_peered":"2024-09-16T01:52:40.705561+0000","last_clean":"2024-09-16T01:52:40.705561+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:40.705561+0000","last_undegraded":"2024-09-16T01:52:40.705561+0000","last_fullsized":"2024-09-16T01:52:40.705561+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":196,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.533427+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:45.533427+0000","last_peered":"2024-09-16T01:52:45.533427+0000","last_clean":"2024-09-16T01:52:45.533427+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:45.533427+0000","last_undegraded":"2024-09-16T01:52:45.533427+0000","last_fullsized":"2024-09-16T01:52:45.533427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"107'105","reported_seq":288,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.708603+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:40.708603+0000","last_peered":"2024-09-16T01:52:40.708603+0000","last_clean":"2024-09-16T01:52:40.708603+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:40.708603+0000","last_undegraded":"2024-09-16T01:52:40.708603+0000","last_fullsized":"2024-09-16T01:52:40.708603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":2,"reported_epoch":107,"state":"peering","last_fresh":"2024-09-16T01:52:39.700468+0000","last_change":"2024-09-16T01:52:41.657743+0000","last_active":"2024-09-16T01:52:41.657743+0000","last_peered":"2024-09-16T01:52:39.700468+0000","last_clean":"2024-09-16T01:52:39.700468+0000","last_became_active":"2024-09-16T01:51:49.156266+0000","last_became_peered":"2024-09-16T01:51:49.156266+0000","last_unstale":"2024-09-16T01:52:39.700468+0000","last_undegraded":"2024-09-16T01:52:39.700468+0000","last_fullsized":"2024-09-16T01:52:39.700468+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"111'99","reported_seq":292,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528157+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:45.528157+0000","last_peered":"2024-09-16T01:52:45.528157+0000","last_clean":"2024-09-16T01:52:45.528157+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:45.528157+0000","last_undegraded":"2024-09-16T01:52:45.528157+0000","last_fullsized":"2024-09-16T01:52:45.528157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532417+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:45.532417+0000","last_peered":"2024-09-16T01:52:45.532417+0000","last_clean":"2024-09-16T01:52:45.532417+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:45.532417+0000","last_undegraded":"2024-09-16T01:52:45.532417+0000","last_fullsized":"2024-09-16T01:52:45.532417+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"111'110","reported_seq":348,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528266+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:45.528266+0000","last_peered":"2024-09-16T01:52:45.528266+0000","last_clean":"2024-09-16T01:52:45.528266+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:45.528266+0000","last_undegraded":"2024-09-16T01:52:45.528266+0000","last_fullsized":"2024-09-16T01:52:45.528266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":206,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712573+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:40.712573+0000","last_peered":"2024-09-16T01:52:40.712573+0000","last_clean":"2024-09-16T01:52:40.712573+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:40.712573+0000","last_undegraded":"2024-09-16T01:52:40.712573+0000","last_fullsized":"2024-09-16T01:52:40.712573+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"107'112","reported_seq":272,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.137689+0000","last_change":"2024-09-16T01:52:06.711927+0000","last_active":"2024-09-16T01:52:41.137689+0000","last_peered":"2024-09-16T01:52:41.137689+0000","last_clean":"2024-09-16T01:52:41.137689+0000","last_became_active":"2024-09-16T01:52:03.364819+0000","last_became_peered":"2024-09-16T01:52:03.364819+0000","last_unstale":"2024-09-16T01:52:41.137689+0000","last_undegraded":"2024-09-16T01:52:41.137689+0000","last_fullsized":"2024-09-16T01:52:41.137689+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:01.570129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":255852544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":61,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.703706+0000","last_change":"2024-09-16T01:52:21.473114+0000","last_active":"2024-09-16T01:52:40.703706+0000","last_peered":"2024-09-16T01:52:40.703706+0000","last_clean":"2024-09-16T01:52:40.703706+0000","last_became_active":"2024-09-16T01:52:21.465256+0000","last_became_peered":"2024-09-16T01:52:21.465256+0000","last_unstale":"2024-09-16T01:52:40.703706+0000","last_undegraded":"2024-09-16T01:52:40.703706+0000","last_fullsized":"2024-09-16T01:52:40.703706+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:07:59.896936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"107'113","reported_seq":350,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.882542+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:40.882542+0000","last_peered":"2024-09-16T01:52:40.882542+0000","last_clean":"2024-09-16T01:52:40.882542+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:40.882542+0000","last_undegraded":"2024-09-16T01:52:40.882542+0000","last_fullsized":"2024-09-16T01:52:40.882542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":75,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.527355+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:45.527355+0000","last_peered":"2024-09-16T01:52:45.527355+0000","last_clean":"2024-09-16T01:52:45.527355+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:45.527355+0000","last_undegraded":"2024-09-16T01:52:45.527355+0000","last_fullsized":"2024-09-16T01:52:45.527355+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"106'106","reported_seq":338,"reported_epoch":107,"state":"unknown","last_fresh":"2024-09-16T01:52:40.700618+0000","last_change":"2024-09-16T01:52:41.595972+0000","last_active":"2024-09-16T01:52:40.700618+0000","last_peered":"2024-09-16T01:52:39.701448+0000","last_clean":"2024-09-16T01:52:39.701448+0000","last_became_active":"2024-09-16T01:52:09.394197+0000","last_became_peered":"2024-09-16T01:52:09.394197+0000","last_unstale":"2024-09-16T01:52:40.700618+0000","last_undegraded":"2024-09-16T01:52:40.700618+0000","last_fullsized":"2024-09-16T01:52:40.700618+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":352321536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":198,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.528141+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:45.528141+0000","last_peered":"2024-09-16T01:52:45.528141+0000","last_clean":"2024-09-16T01:52:45.528141+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:45.528141+0000","last_undegraded":"2024-09-16T01:52:45.528141+0000","last_fullsized":"2024-09-16T01:52:45.528141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"111'104","reported_seq":407,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.861424+0000","last_change":"2024-09-16T01:52:45.530072+0000","last_active":"2024-09-16T01:52:45.530072+0000","last_peered":"2024-09-16T01:52:44.861424+0000","last_clean":"2024-09-16T01:52:44.861424+0000","last_became_active":"2024-09-16T01:52:21.475090+0000","last_became_peered":"2024-09-16T01:52:21.475090+0000","last_unstale":"2024-09-16T01:52:44.861424+0000","last_undegraded":"2024-09-16T01:52:44.861424+0000","last_fullsized":"2024-09-16T01:52:44.861424+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":364544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":189,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.715768+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:40.715768+0000","last_peered":"2024-09-16T01:52:40.715768+0000","last_clean":"2024-09-16T01:52:40.715768+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:40.715768+0000","last_undegraded":"2024-09-16T01:52:40.715768+0000","last_fullsized":"2024-09-16T01:52:40.715768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"107'89","reported_seq":271,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:41.244459+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:41.244459+0000","last_peered":"2024-09-16T01:52:41.244459+0000","last_clean":"2024-09-16T01:52:41.244459+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:41.244459+0000","last_undegraded":"2024-09-16T01:52:41.244459+0000","last_fullsized":"2024-09-16T01:52:41.244459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"107'123","reported_seq":424,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714258+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:40.714258+0000","last_peered":"2024-09-16T01:52:40.714258+0000","last_clean":"2024-09-16T01:52:40.714258+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:40.714258+0000","last_undegraded":"2024-09-16T01:52:40.714258+0000","last_fullsized":"2024-09-16T01:52:40.714258+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":58,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714247+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:40.714247+0000","last_peered":"2024-09-16T01:52:40.714247+0000","last_clean":"2024-09-16T01:52:40.714247+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:40.714247+0000","last_undegraded":"2024-09-16T01:52:40.714247+0000","last_fullsized":"2024-09-16T01:52:40.714247+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"106'101","reported_seq":379,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.714356+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:40.714356+0000","last_peered":"2024-09-16T01:52:40.714356+0000","last_clean":"2024-09-16T01:52:40.714356+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:40.714356+0000","last_undegraded":"2024-09-16T01:52:40.714356+0000","last_fullsized":"2024-09-16T01:52:40.714356+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":26,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.784564+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:40.784564+0000","last_peered":"2024-09-16T01:52:40.784564+0000","last_clean":"2024-09-16T01:52:40.784564+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:40.784564+0000","last_undegraded":"2024-09-16T01:52:40.784564+0000","last_fullsized":"2024-09-16T01:52:40.784564+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"111'91","reported_seq":284,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532243+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:45.532243+0000","last_peered":"2024-09-16T01:52:45.532243+0000","last_clean":"2024-09-16T01:52:45.532243+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:45.532243+0000","last_undegraded":"2024-09-16T01:52:45.532243+0000","last_fullsized":"2024-09-16T01:52:45.532243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":221,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.508113+0000","last_change":"2024-09-16T01:52:45.532987+0000","last_active":"2024-09-16T01:52:45.532987+0000","last_peered":"2024-09-16T01:52:44.508113+0000","last_clean":"2024-09-16T01:52:44.508113+0000","last_became_active":"2024-09-16T01:52:21.473912+0000","last_became_peered":"2024-09-16T01:52:21.473912+0000","last_unstale":"2024-09-16T01:52:44.508113+0000","last_undegraded":"2024-09-16T01:52:44.508113+0000","last_fullsized":"2024-09-16T01:52:44.508113+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"111'112","reported_seq":305,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532474+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:45.532474+0000","last_peered":"2024-09-16T01:52:45.532474+0000","last_clean":"2024-09-16T01:52:45.532474+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:45.532474+0000","last_undegraded":"2024-09-16T01:52:45.532474+0000","last_fullsized":"2024-09-16T01:52:45.532474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":95,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T01:52:40.712358+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:40.712358+0000","last_peered":"2024-09-16T01:52:40.712358+0000","last_clean":"2024-09-16T01:52:40.712358+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:40.712358+0000","last_undegraded":"2024-09-16T01:52:40.712358+0000","last_fullsized":"2024-09-16T01:52:40.712358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":4,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6354,"num_write_kb":23928832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2275,"num_bytes_recovered":7415529472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1459617792,"data_stored":23353884672,"data_compressed":1158023040,"data_compressed_allocated":1459617792,"data_compressed_original":23353884672,"omap_allocated":0,"internal_metadata":0},"log_size":6869,"ondisk_log_size":6869,"up":128,"acting":127,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":246957,"num_keys_recovered":115,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":996246,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509491,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368356,"kb_used_data":337172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94003484,"statfs":{"total":96636764160,"available":96259567616,"internally_reserved":0,"allocated":345264128,"data_stored":5503515173,"data_compressed":272614656,"data_compressed_allocated":343932928,"data_compressed_original":5502926848,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509491,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373816,"kb_used_data":342628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93998024,"statfs":{"total":96636764160,"available":96253976576,"internally_reserved":0,"allocated":350851072,"data_stored":5587780817,"data_compressed":276985216,"data_compressed_allocated":349175808,"data_compressed_original":5586812928,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":245456,"kb_used_data":205692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94126384,"statfs":{"total":96636764160,"available":96385417216,"internally_reserved":0,"allocated":210628608,"data_stored":3346515636,"data_compressed":165607715,"data_compressed_allocated":209149952,"data_compressed_original":3345764352,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542195,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69908,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94301932,"statfs":{"total":96636764160,"available":96565178368,"internally_reserved":0,"allocated":35393536,"data_stored":545874692,"data_compressed":27121536,"data_compressed_allocated":34078720,"data_compressed_original":545259520,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542195,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":250452,"kb_used_data":210680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94121388,"statfs":{"total":96636764160,"available":96380301312,"internally_reserved":0,"allocated":215736320,"data_stored":3430510164,"data_compressed":170823584,"data_compressed_allocated":214368256,"data_compressed_original":3429892096,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265416,"kb_used_data":225908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94106424,"statfs":{"total":96636764160,"available":96364978176,"internally_reserved":0,"allocated":231329792,"data_stored":3679613108,"data_compressed":183437604,"data_compressed_allocated":229945344,"data_compressed_original":3678978048,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574899,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":377702,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":377763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":2896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213647360,"data_stored":3418357760,"data_compressed":169361408,"data_compressed_allocated":213647360,"data_compressed_original":3418357760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349175808,"data_stored":5586812928,"data_compressed":277031808,"data_compressed_allocated":349175808,"data_compressed_original":5586812928,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343932928,"data_stored":5502926848,"data_compressed":273045632,"data_compressed_allocated":343932928,"data_compressed_original":5502926848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":3678404608,"data_compressed":183463424,"data_compressed_allocated":229900288,"data_compressed_original":3678404608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":222298112,"data_stored":3556769792,"data_compressed":177356160,"data_compressed_allocated":222298112,"data_compressed_original":3556769792,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41156608,"data_stored":658505728,"data_compressed":32785920,"data_compressed_allocated":41156608,"data_compressed_original":658505728,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:47.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:47.511 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:47.508+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:47.594 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:52:47.595 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 rewriters = 302527.16 kB/sec 2024-09-16T01:52:47.595 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 rewriters = 302517.61 kB/sec 2024-09-16T01:52:47.595 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 302527.16 kB/sec 2024-09-16T01:52:47.595 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 302527.16 kB/sec 2024-09-16T01:52:47.595 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 302527.16 kB/sec 2024-09-16T01:52:47.595 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 10485760.00 kB 2024-09-16T01:52:47.611 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:47.610+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:47.712 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:47.710+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:47.733 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 239.96253260399999, 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.734 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.736 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.737 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:52:47.737 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.738 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.739 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.740 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:52:47.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:47.742 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:47.743 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 107.345080264, 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.744 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:52:47.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:52:47.746 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:52:47.747 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.748 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.749 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.750 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:47.751 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.553063315000003, 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.752 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.754 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.755 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:52:47.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:52:47.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.517530691999994, 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:52:47.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:52:47.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:52:47.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.151441754999993, 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:52:47.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:52:47.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.587230216999998, 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.772 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:52:47.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.525442914999999, 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:52:47.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:52:47.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.783 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 57.683660469000003, 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:52:47.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:52:47.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:47.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:47.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.940221716000003, 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.790 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:52:47.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:47.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:47.795 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.896000770000001, 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:47.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:52:47.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:47.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.887584283999999, 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:52:47.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.807 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4107 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e99)", 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.454946797000002, 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31541509600000001, 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4107 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275941+0000", 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6959999999999999e-06 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303032+0000", 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027090797 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303053+0000", 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1019999999999999e-05 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303063+0000", 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6479999999999998e-06 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303176+0000", 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011292 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303185+0000", 2024-09-16T01:52:47.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.491825+0000", 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.18864050500000001 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.498314+0000", 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064884210000000003 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551379+0000", 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053065042999999999 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551446+0000", 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7477999999999993e-05 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551537+0000", 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1055000000000001e-05 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.576787+0000", 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025250156999999999 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591154+0000", 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01436693 2024-09-16T01:52:47.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591228+0000", 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4103000000000005e-05 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591355+0000", 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126367 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4250 3.10 3:0a05e26b:::100000001f8.00000297:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.420727259, 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.149078401, 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4250 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.816 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310162+0000", 2024-09-16T01:52:47.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6599999999999999e-06 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407037+0000", 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096875366000000004 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407073+0000", 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5969000000000001e-05 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407088+0000", 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5034999999999999e-05 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407218+0000", 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129462 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407332+0000", 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114183 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407516+0000", 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018366500000000001 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.446008+0000", 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038492551999999999 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456958+0000", 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01095023 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.458896+0000", 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019376650000000001 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.459238+0000", 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034161400000000001 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4504 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.201100062999998, 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22042661699999999, 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4504 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529789+0000", 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5409999999999999e-06 2024-09-16T01:52:47.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577075+0000", 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047285897 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577104+0000", 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9173999999999999e-05 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577115+0000", 2024-09-16T01:52:47.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0895e-05 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577223+0000", 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108194 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577330+0000", 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107089 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577485+0000", 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154532 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.599121+0000", 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021636636000000001 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750098+0000", 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15097622499999999 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750164+0000", 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6172000000000003e-05 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750213+0000", 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9261999999999997e-05 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.827 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4512 3.19 3:987f889e:::100000001f8.0000039d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.098906406000001, 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12513411299999999, 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4512 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631984+0000", 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8999999999999999e-06 2024-09-16T01:52:47.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679187+0000", 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047202698000000001 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679218+0000", 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1775e-05 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679233+0000", 2024-09-16T01:52:47.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4182000000000001e-05 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682406+0000", 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031734889999999998 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682504+0000", 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7374999999999995e-05 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.683089+0000", 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058535 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.705462+0000", 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022372670000000001 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757013+0000", 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051551620999999999 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757066+0000", 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3044999999999998e-05 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757114+0000", 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8007999999999998e-05 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.833 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4553 3.6 3:62a9ab61:::100000001f8.000003c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.564246974, 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.118770112, 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4553 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166642+0000", 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0470000000000001e-06 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206911+0000", 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040269395999999999 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206944+0000", 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3185000000000002e-05 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206957+0000", 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2894999999999999e-05 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207157+0000", 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199908 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207262+0000", 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105509 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207423+0000", 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016019999999999999 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.221084+0000", 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013661240999999999 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285289+0000", 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064205110999999995 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285358+0000", 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9015000000000002e-05 2024-09-16T01:52:47.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285410+0000", 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1604999999999998e-05 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4560 3.39 3:9e09c0a8:::100000001f8.000003cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:47.840 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.485161427000001, 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11688955500000001, 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4560 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245728+0000", 2024-09-16T01:52:47.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.548e-06 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301561+0000", 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055832859999999998 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301606+0000", 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5423000000000002e-05 2024-09-16T01:52:47.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301620+0000", 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.453e-05 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301728+0000", 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107836 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301847+0000", 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119127 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.302620+0000", 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000772461 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.328630+0000", 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026010253000000001 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362241+0000", 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033611125999999998 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362382+0000", 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014118400000000001 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362615+0000", 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023220700000000001 2024-09-16T01:52:47.846 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5800 3.8 3:139dc79b:::100000001f8.000008a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.3585044320000002, 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.57801824300000004, 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.847 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5800 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372384+0000", 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2840000000000001e-06 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387317+0000", 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014932126 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387330+0000", 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3658e-05 2024-09-16T01:52:47.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387338+0000", 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389339+0000", 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020019460000000001 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389354+0000", 2024-09-16T01:52:47.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.472e-05 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899029+0000", 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50967473200000002 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899208+0000", 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017872499999999999 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899396+0000", 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018809 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927572+0000", 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028175951000000001 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950173+0000", 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022601711 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950246+0000", 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2533000000000005e-05 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950400+0000", 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154418 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.853 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5815 3.21 3:84e25a9c:::100000001f8.000008b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.2063272490000001, 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.133812769, 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.854 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5815 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524561+0000", 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.968e-06 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641841+0000", 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11728021 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641861+0000", 2024-09-16T01:52:47.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9939e-05 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641868+0000", 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5339999999999996e-06 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641942+0000", 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4416e-05 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642034+0000", 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1916999999999994e-05 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642172+0000", 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013773100000000001 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656243+0000", 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014070719000000001 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658259+0000", 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020165389999999999 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658323+0000", 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3876999999999998e-05 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658372+0000", 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8918999999999997e-05 2024-09-16T01:52:47.859 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5821 3.8 3:10a53e03:::100000001f8.000008ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 5.0536684899999997, 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27338652600000002, 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:47.860 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5821 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:47.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677219+0000", 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.344e-06 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702527+0000", 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025308081999999999 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702546+0000", 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8698999999999998e-05 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702555+0000", 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2609999999999999e-06 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702663+0000", 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010725 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702675+0000", 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2743e-05 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910154+0000", 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.207478311 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910331+0000", 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000177057 2024-09-16T01:52:47.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910525+0000", 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000194074 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927630+0000", 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017105187000000001 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:47.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950420+0000", 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022789713 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950473+0000", 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3115999999999998e-05 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950605+0000", 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013168899999999999 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.866 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:47.867 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:47.867 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:47.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:47.871 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:47.868+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:47.971 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:47.969+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:48.039 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:48.039 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:48.039 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:48.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:48.097 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:48.095+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:48.170 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:48.169+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:48.271 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:48.270+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:48.282 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:48.282 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:48.282 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:48.297 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:48.373 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:48.371+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:48.473 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:48.472+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:48.478 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:48.478 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:48.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:48.478 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.184361093999996, 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.479 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:48.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.484 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.155200238999996, 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.485 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:48.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:48.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:48.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.122848671, 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.492 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:48.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:48.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:48.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.106901405000002, 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.498 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:48.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.503 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.55982624, 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.504 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.510 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.374896884000002, 2024-09-16T01:52:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.513 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:48.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:48.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.372359076000002, 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.520 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:48.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.762947844000003, 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:48.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:48.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.531 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.724691409000002, 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:52:48.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 32.649619674, 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.538 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:52:48.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.540 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.541 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.542 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.543 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4142 3.16 3:6ba609e9:::100000001f8.0000022b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.688523119999999, 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108516207, 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4142 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.544 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787022+0000", 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.61e-06 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.545 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842519+0000", 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055496399000000002 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842539+0000", 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0820000000000001e-05 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842550+0000", 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0414999999999999e-05 2024-09-16T01:52:48.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842641+0000", 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1657999999999996e-05 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842742+0000", 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010075 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842888+0000", 2024-09-16T01:52:48.547 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014623699999999999 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.892001+0000", 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112211000000003 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895434+0000", 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034334869999999998 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.548 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895499+0000", 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5313000000000003e-05 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895537+0000", 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7307e-05 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4246 3.34 3:2f44359e:::100000001f8.00000293:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:48.549 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.145910558000001, 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.142262732, 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4246 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:48.550 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329635+0000", 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9180000000000001e-06 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.551 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410104+0000", 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080469499 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410128+0000", 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3243000000000001e-05 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410141+0000", 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3067999999999999e-05 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.552 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419779+0000", 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096380589999999992 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419889+0000", 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.420057+0000", 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016837699999999999 2024-09-16T01:52:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456784+0000", 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036727237000000003 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471769+0000", 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014985112 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471839+0000", 2024-09-16T01:52:48.554 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9275000000000003e-05 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471896+0000", 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7259000000000002e-05 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4247 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.065355450999999, 2024-09-16T01:52:48.555 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120085209, 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4247 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410190+0000", 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451530+0000", 2024-09-16T01:52:48.557 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041340783999999998 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451572+0000", 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1690000000000002e-05 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451620+0000", 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8180999999999998e-05 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.558 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452310+0000", 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068996900000000002 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452413+0000", 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010248599999999999 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452588+0000", 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017574800000000001 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.559 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.518054+0000", 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065465819999999994 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530168+0000", 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012113413999999999 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530225+0000", 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7284999999999998e-05 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.560 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530273+0000", 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8381999999999999e-05 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4503 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.963800096, 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13319203700000001, 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.561 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4503 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.562 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511746+0000", 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0250000000000001e-06 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576368+0000", 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064622151000000003 2024-09-16T01:52:48.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576386+0000", 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7878999999999999e-05 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576398+0000", 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.564 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578343+0000", 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001945397 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578445+0000", 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102001 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578601+0000", 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015608399999999999 2024-09-16T01:52:48.565 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.643878+0000", 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065277247999999996 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644839+0000", 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096038899999999997 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644895+0000", 2024-09-16T01:52:48.566 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6484000000000003e-05 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644936+0000", 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0324999999999998e-05 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4504 3.f e101/94 3:f3dac423:::100000001f8.00000395:head v 101'76, mlcod=100'74)", 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.880873979, 2024-09-16T01:52:48.567 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14827673899999999, 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:48.568 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594672+0000", 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3309999999999998e-06 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629004+0000", 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034332342000000002 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.569 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629020+0000", 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6121e-05 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629029+0000", 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7479999999999999e-06 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635523+0000", 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064935909999999999 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.570 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635549+0000", 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5808000000000001e-05 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742856+0000", 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107307556 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742946+0000", 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0241999999999994e-05 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.571 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4549 3.2 3:428d9500:::100000001f8.000003c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 21.305910531999999, 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11639424900000001, 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.572 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4549 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.573 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169636+0000", 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8660000000000002e-06 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212280+0000", 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042644350999999997 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212312+0000", 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2088999999999997e-05 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.574 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212327+0000", 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521e-05 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212426+0000", 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9245000000000003e-05 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.575 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212539+0000", 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112711 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212725+0000", 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018583400000000001 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285881+0000", 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073156321999999996 2024-09-16T01:52:48.576 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285914+0000", 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2929999999999998e-05 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285979+0000", 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5048000000000005e-05 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.286027+0000", 2024-09-16T01:52:48.577 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8331999999999997e-05 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5790 3.16 3:687a9a46:::100000001f8.0000089b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.2648753959999999, 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.65913515700000003, 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.578 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5790 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210670+0000", 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244952+0000", 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034282188999999998 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244970+0000", 2024-09-16T01:52:48.580 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8179999999999999e-05 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244982+0000", 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1736000000000001e-05 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245078+0000", 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5439000000000006e-05 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.581 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245090+0000", 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2177e-05 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791600+0000", 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.546509829 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791770+0000", 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017051699999999999 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.582 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791950+0000", 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018029299999999999 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829483+0000", 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037532101999999998 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867462+0000", 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037979295000000003 2024-09-16T01:52:48.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869661+0000", 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021987059999999999 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869702+0000", 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1148e-05 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869803+0000", 2024-09-16T01:52:48.584 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101696 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5793 3.16 3:68c3c174:::100000001f8.0000089e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.2041647759999998, 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.59860013000000001, 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.585 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5793 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271381+0000", 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0669999999999999e-06 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311762+0000", 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040380644 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311785+0000", 2024-09-16T01:52:48.587 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3090000000000001e-05 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311798+0000", 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3057999999999999e-05 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311911+0000", 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113429 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:48.588 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311923+0000", 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1915e-05 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829575+0000", 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51765227000000003 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829754+0000", 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017901700000000001 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.589 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829935+0000", 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018041800000000001 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867483+0000", 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037548040999999997 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869178+0000", 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016955379999999999 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.590 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869843+0000", 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000664345 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869882+0000", 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8976000000000001e-05 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869979+0000", 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7322000000000001e-05 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.591 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5945 3.2b 3:d7b2ba59:::100000001f8.00000936:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.07802431, 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11205129899999999, 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.592 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5945 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.593 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397521+0000", 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429779+0000", 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032257503999999999 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429805+0000", 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-09-16T01:52:48.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429818+0000", 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2687999999999999e-05 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429896+0000", 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7898999999999994e-05 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.595 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430008+0000", 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011241100000000001 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430198+0000", 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000189197 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.448700+0000", 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018502761999999999 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.596 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509440+0000", 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060739425999999999 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509514+0000", 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4673999999999997e-05 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509571+0000", 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6249999999999998e-05 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5948 3.2d 3:b677c6b2:::100000001f8.00000939:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.0453291690000004, 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122297751, 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:48.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5948 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.599 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430216+0000", 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453110+0000", 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022894134 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453131+0000", 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1120000000000001e-05 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453141+0000", 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5340000000000005e-06 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495855+0000", 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042714162 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495997+0000", 2024-09-16T01:52:48.601 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014160999999999999 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537228+0000", 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041231297 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537360+0000", 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264400000000001 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6,7", 2024-09-16T01:52:48.602 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537537+0000", 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017664800000000001 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548049+0000", 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511915 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548065+0000", 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5812e-05 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.603 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552281+0000", 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042165600000000003 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552355+0000", 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3712000000000002e-05 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552512+0000", 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015707300000000001 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.604 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.605 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.605 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:48.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:52:48.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:48.607 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:48.604+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:48.707 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:48.705+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:48.808 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:48.807+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "MOSDPGPushReply(3.c 115/113 [PushReplyOp(3:3076da67:::100000001f8.000004eb:head)])", 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:48.830091+0000", 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 2.0740000000000001e-05, 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4888000000000001e-05, 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:48.831 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "queued for pg", 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:48.830091+0000", 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:48.830091+0000", 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:48.830091+0000", 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.6899999999999998e-07 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:48.830094+0000", 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8059999999999999e-06 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:48.830103+0000", 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0159999999999997e-06 2024-09-16T01:52:48.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:48.830107+0000", 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.279e-06 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 1 2024-09-16T01:52:48.834 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:48.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.836+0000 7fd35dc00640 1 Processor -- start 2024-09-16T01:52:48.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.836+0000 7fd35dc00640 1 -- start start 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.837+0000 7fd35dc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd358154d40 0x7fd358067950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.837+0000 7fd35dc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd358072cc0 0x7fd3580730d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.837+0000 7fd35dc00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3580c6cf0 con 0x7fd358072170 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.837+0000 7fd35dc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd358072830 con 0x7fd358154d40 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.837+0000 7fd35dc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd35815ab60 con 0x7fd358072cc0 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd358154d40 0x7fd358067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd358154d40 0x7fd358067950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45764/0 (socket says 172.21.15.146:45764) 2024-09-16T01:52:48.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 -- 172.21.15.146:0/4273254876 learned_addr learned my addr 172.21.15.146:0/4273254876 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd357400640 1 --2- 172.21.15.146:0/4273254876 >> v2:172.21.15.161:3300/0 conn(0x7fd358072cc0 0x7fd3580730d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 -- 172.21.15.146:0/4273254876 >> v2:172.21.15.161:3300/0 conn(0x7fd358072cc0 msgr2=0x7fd3580730d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 --2- 172.21.15.146:0/4273254876 >> v2:172.21.15.161:3300/0 conn(0x7fd358072cc0 0x7fd3580730d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 -- 172.21.15.146:0/4273254876 >> v1:172.21.15.161:6789/0 conn(0x7fd358072170 legacy=0x7fd35806cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:48.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd356a00640 1 -- 172.21.15.146:0/4273254876 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35815ae00 con 0x7fd358154d40 2024-09-16T01:52:48.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.838+0000 7fd355600640 1 -- 172.21.15.146:0/4273254876 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842819638 0 0) 0x7fd3580c6cf0 con 0x7fd358072170 2024-09-16T01:52:48.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.840+0000 7fd356a00640 1 --2- 172.21.15.146:0/4273254876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358154d40 0x7fd358067950 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd3580c38b0 tx=0x7fd34802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4fd7af6076eabd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:48.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.840+0000 7fd355600640 1 -- 172.21.15.146:0/4273254876 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348037070 con 0x7fd358154d40 2024-09-16T01:52:48.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.840+0000 7fd355600640 1 -- 172.21.15.146:0/4273254876 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd348032440 con 0x7fd358154d40 2024-09-16T01:52:48.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.840+0000 7fd355600640 1 -- 172.21.15.146:0/4273254876 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348032810 con 0x7fd358154d40 2024-09-16T01:52:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 -- 172.21.15.146:0/4273254876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358154d40 msgr2=0x7fd358067950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 --2- 172.21.15.146:0/4273254876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358154d40 0x7fd358067950 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd3580c38b0 tx=0x7fd34802a270 comp rx=0 tx=0).stop 2024-09-16T01:52:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 -- 172.21.15.146:0/4273254876 shutdown_connections 2024-09-16T01:52:48.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 --2- 172.21.15.146:0/4273254876 >> v2:172.21.15.161:3300/0 conn(0x7fd358072cc0 0x7fd3580730d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:48.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 --2- 172.21.15.146:0/4273254876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358154d40 0x7fd358067950 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:48.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 -- 172.21.15.146:0/4273254876 >> 172.21.15.146:0/4273254876 conn(0x7fd35806d8e0 msgr2=0x7fd35806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:48.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 -- 172.21.15.146:0/4273254876 shutdown_connections 2024-09-16T01:52:48.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.841+0000 7fd35dc00640 1 -- 172.21.15.146:0/4273254876 wait complete. 2024-09-16T01:52:48.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.843+0000 7fd35dc00640 1 Processor -- start 2024-09-16T01:52:48.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.843+0000 7fd35dc00640 1 -- start start 2024-09-16T01:52:48.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:48.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd35dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 0x7fd358142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:48.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd35dc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd358072cc0 0x7fd3581432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:48.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd35dc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd358154d40 0x7fd3581f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd35dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd35815b6a0 con 0x7fd358072170 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd35dc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd35815b3c0 con 0x7fd358072cc0 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd35dc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd35815b980 con 0x7fd358154d40 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd356a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 0x7fd358142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd356a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 0x7fd358142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45780/0 (socket says 172.21.15.146:45780) 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.844+0000 7fd356a00640 1 -- 172.21.15.146:0/765866090 learned_addr learned my addr 172.21.15.146:0/765866090 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356000640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd358072cc0 0x7fd3581432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd357400640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd358154d40 0x7fd3581f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356a00640 1 -- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd358154d40 msgr2=0x7fd3581f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356a00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd358154d40 0x7fd3581f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356a00640 1 -- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd358072cc0 msgr2=0x7fd3581432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356a00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd358072cc0 0x7fd3581432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:48.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356a00640 1 -- 172.21.15.146:0/765866090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3580cc080 con 0x7fd358072170 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.845+0000 7fd356a00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 0x7fd358142d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd348000c00 tx=0x7fd3480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.846+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348037040 con 0x7fd358072170 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.846+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd348042070 con 0x7fd358072170 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.846+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348030880 con 0x7fd358072170 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.846+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35813e280 con 0x7fd358072170 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.846+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3580cc2f0 con 0x7fd358072170 2024-09-16T01:52:48.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.847+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd358155120 con 0x7fd358072170 2024-09-16T01:52:48.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.849+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd348030a20 con 0x7fd358072170 2024-09-16T01:52:48.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.850+0000 7fd34ea00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd33806c2e0 0x7fd33806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:48.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.850+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6069+0+0 (secure 0 0 0) 0x7fd3480abd30 con 0x7fd358072170 2024-09-16T01:52:48.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.850+0000 7fd356000640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd33806c2e0 0x7fd33806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:48.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.851+0000 7fd356000640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd33806c2e0 0x7fd33806e7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd358150f00 tx=0x7fd344009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:48.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:48.855+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd35813e280 con 0x7fd358072170 2024-09-16T01:52:48.910 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:48.908+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:49.011 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:49.010+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:49.033 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:49.033 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:49.034 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:49.034 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:49.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:49.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.078+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd35815a630 con 0x7fd358072170 2024-09-16T01:52:49.113 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:49.111+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:49.214 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:49.212+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:49.260 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:49.260 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 109.407394242, 2024-09-16T01:52:49.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.262 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:49.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:49.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:49.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:49.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:49.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:49.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:49.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:49.281 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 109.26493457700001, 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:49.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:49.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:49.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 108.91746835399999, 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.291 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:49.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:49.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:49.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:49.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:49.299 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 89.033176990000001, 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:49.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.995437487000004, 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.306 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:49.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:49.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:49.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.136450487000005, 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:49.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:49.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:49.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:49.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.323 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:49.324 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.269411513000001, 2024-09-16T01:52:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:49.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:49.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:49.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:49.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:49.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:49.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:49.340 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.073459292000003, 2024-09-16T01:52:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:49.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:49.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.347 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 60.032457348999998, 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:49.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:49.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.354 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 59.808229789000002, 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.355 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:49.356 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:49.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 53.851448236000003, 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:49.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.362 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:49.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:49.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.366 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.940467886999997, 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.367 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:49.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:49.372 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.503936905000003, 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.373 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:52:49.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:52:49.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.480782986999998, 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.379 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:52:49.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.469847143999999, 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:49.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.439652719999998, 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.388 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:52:49.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:52:49.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.398247267000002, 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.394 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:52:49.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5551 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.417440177, 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13476244500000001, 2024-09-16T01:52:49.400 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5551 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838694+0000", 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.306e-06 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866535+0000", 2024-09-16T01:52:49.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027841175999999999 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866552+0000", 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7387000000000001e-05 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866563+0000", 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0447e-05 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.403 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866665+0000", 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101894 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866788+0000", 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123194 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866953+0000", 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165365 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.918808+0000", 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051855367999999999 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973333+0000", 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054524250000000003 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973402+0000", 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9648999999999997e-05 2024-09-16T01:52:49.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973454+0000", 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5812 3.a 3:51094797:::100000001f8.000008b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 6.7237410620000002, 2024-09-16T01:52:49.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14779089000000001, 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5812 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532392+0000", 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8679999999999999e-06 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653045+0000", 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120652599 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653062+0000", 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6929e-05 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653073+0000", 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0925e-05 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656653+0000", 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035800049999999998 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656738+0000", 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5599000000000005e-05 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656890+0000", 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151874 2024-09-16T01:52:49.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.676041+0000", 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019150358999999999 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680071+0000", 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004030624 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680131+0000", 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9704000000000003e-05 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680181+0000", 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0404000000000001e-05 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5944 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:49.412 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 4.8689119180000002, 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.115163742, 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5944 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387222+0000", 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418222+0000", 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031000603000000002 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418239+0000", 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6833e-05 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418249+0000", 2024-09-16T01:52:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8179999999999999e-06 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418846+0000", 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059710099999999995 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418945+0000", 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8826000000000006e-05 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.419093+0000", 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148553 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.442472+0000", 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023378423999999998 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502282+0000", 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059810025000000003 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502341+0000", 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8922000000000003e-05 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502383+0000", 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2590999999999999e-05 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:49.418 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:49.419 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:49.420 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:49.421 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:49.419+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:49.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.486+0000 7fd34ea00640 1 -- 172.21.15.146:0/765866090 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v116) ==== 91+0+0 (secure 0 0 0) 0x7fd348079c40 con 0x7fd358072170 2024-09-16T01:52:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:52:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.489+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd33806c2e0 msgr2=0x7fd33806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.489+0000 7fd35dc00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd33806c2e0 0x7fd33806e7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd358150f00 tx=0x7fd344009040 comp rx=0 tx=0).stop 2024-09-16T01:52:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.489+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 msgr2=0x7fd358142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.489+0000 7fd35dc00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 0x7fd358142d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd348000c00 tx=0x7fd3480046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 shutdown_connections 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd33806c2e0 0x7fd33806e7a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd358154d40 0x7fd3581f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd358072cc0 0x7fd3581432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 --2- 172.21.15.146:0/765866090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd358072170 0x7fd358142d60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 >> 172.21.15.146:0/765866090 conn(0x7fd35806d8e0 msgr2=0x7fd3580730a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 shutdown_connections 2024-09-16T01:52:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.492+0000 7fd35dc00640 1 -- 172.21.15.146:0/765866090 wait complete. 2024-09-16T01:52:49.535 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:49.535 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:49.535 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:49.536 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:49.546 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:49.561 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:49.559+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:49.621 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:49.620+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.623+0000 7f167e200640 1 Processor -- start 2024-09-16T01:52:49.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.623+0000 7f167e200640 1 -- start start 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f167e200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f167806e9c0 0x7f16780ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f167e200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16780c85e0 con 0x7f167815eaa0 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f167e200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16780c8300 con 0x7f16780cd470 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f167e200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1678155300 con 0x7f167806e9c0 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f166fe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f167806e9c0 0x7f16780ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f167ce00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f167815eaa0 0x7f167806ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33584/0 (socket says 172.21.15.146:33584) 2024-09-16T01:52:49.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f167ce00640 1 -- 172.21.15.146:0/2391566394 learned_addr learned my addr 172.21.15.146:0/2391566394 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:49.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.624+0000 7f1677e00640 1 -- 172.21.15.146:0/2391566394 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443887288 0 0) 0x7f16780c8300 con 0x7f16780cd470 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.625+0000 7f1677e00640 1 -- 172.21.15.146:0/2391566394 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f165c003520 con 0x7f16780cd470 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.625+0000 7f1677e00640 1 -- 172.21.15.146:0/2391566394 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483058944 0 0) 0x7f16780c85e0 con 0x7f167815eaa0 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.625+0000 7f166fe00640 1 -- 172.21.15.146:0/2391566394 >> v1:172.21.15.161:6789/0 conn(0x7f16780cd470 legacy=0x7f16780ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.625+0000 7f166fe00640 1 -- 172.21.15.146:0/2391566394 >> v1:172.21.15.146:6789/0 conn(0x7f167815eaa0 legacy=0x7f167806ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.625+0000 7f166fe00640 1 -- 172.21.15.146:0/2391566394 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167815ac50 con 0x7f167806e9c0 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f166fe00640 1 --2- 172.21.15.146:0/2391566394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167806e9c0 0x7f16780ccf30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16600097a0 tx=0x7f166002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f98ff1b84c511af5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f1677e00640 1 -- 172.21.15.146:0/2391566394 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660037070 con 0x7f167806e9c0 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f1677e00640 1 -- 172.21.15.146:0/2391566394 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1660032440 con 0x7f167806e9c0 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f1677e00640 1 -- 172.21.15.146:0/2391566394 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16600327f0 con 0x7f167806e9c0 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f167e200640 1 -- 172.21.15.146:0/2391566394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167806e9c0 msgr2=0x7f16780ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f167e200640 1 --2- 172.21.15.146:0/2391566394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167806e9c0 0x7f16780ccf30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16600097a0 tx=0x7f166002a230 comp rx=0 tx=0).stop 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f167e200640 1 -- 172.21.15.146:0/2391566394 shutdown_connections 2024-09-16T01:52:49.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f167e200640 1 --2- 172.21.15.146:0/2391566394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167806e9c0 0x7f16780ccf30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.626+0000 7f167e200640 1 -- 172.21.15.146:0/2391566394 >> 172.21.15.146:0/2391566394 conn(0x7f1678067d40 msgr2=0x7f1678068150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:49.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.627+0000 7f167e200640 1 -- 172.21.15.146:0/2391566394 shutdown_connections 2024-09-16T01:52:49.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.627+0000 7f167e200640 1 -- 172.21.15.146:0/2391566394 wait complete. 2024-09-16T01:52:49.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.628+0000 7f167e200640 1 Processor -- start 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.628+0000 7f167e200640 1 -- start start 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167e200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f167806e9c0 0x7f167815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 0x7f16781571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167e200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167815eaa0 0x7f16781576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f167815b4f0 con 0x7f16780cd470 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167e200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f167815b210 con 0x7f167815eaa0 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167e200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f167815ba10 con 0x7f167806e9c0 2024-09-16T01:52:49.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167ce00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f167806e9c0 0x7f167815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f166fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 0x7f16781571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167d800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167815eaa0 0x7f16781576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167ce00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f167806e9c0 0x7f167815d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38244/0 (socket says 172.21.15.146:38244) 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f166fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 0x7f16781571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45794/0 (socket says 172.21.15.146:45794) 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.629+0000 7f167ce00640 1 -- 172.21.15.146:0/285895702 learned_addr learned my addr 172.21.15.146:0/285895702 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f166fe00640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f167806e9c0 msgr2=0x7f167815d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f167806e9c0 0x7f167815d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f166fe00640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167815eaa0 msgr2=0x7f16781576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f167815eaa0 0x7f16781576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f166fe00640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16781efe40 con 0x7f16780cd470 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 0x7f16781571b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f16780cbc80 tx=0x7f16600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660037040 con 0x7f16780cd470 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1660042070 con 0x7f16780cd470 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660006e10 con 0x7f16780cd470 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16781f0120 con 0x7f16780cd470 2024-09-16T01:52:49.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.630+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16781f06e0 con 0x7f16780cd470 2024-09-16T01:52:49.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.631+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f167815ee80 con 0x7f16780cd470 2024-09-16T01:52:49.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.632+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1660032830 con 0x7f16780cd470 2024-09-16T01:52:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.633+0000 7f1675600640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f165406c2b0 0x7f165406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.633+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6019+0+0 (secure 0 0 0) 0x7f16600abd10 con 0x7f16780cd470 2024-09-16T01:52:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.633+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f165406c2b0 0x7f165406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.633+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v116) ==== 54+0+16 (secure 0 0 0) 0x7f16600ae390 con 0x7f16780cd470 2024-09-16T01:52:49.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.634+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f167806f440 0x7f1678150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.634+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16780008d0 con 0x7f167806f440 2024-09-16T01:52:49.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.634+0000 7f167d800640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f167806f440 0x7f1678150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.634+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f165406c2b0 0x7f165406e770 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1668000f60 tx=0x7f1668007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.634+0000 7f167d800640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f167806f440 0x7f1678150440 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.635+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16780008d0 con 0x7f167806f440 2024-09-16T01:52:49.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.650+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16780c9e80 con 0x7f167806f440 2024-09-16T01:52:49.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.651+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16780c9e80 con 0x7f167806f440 2024-09-16T01:52:49.653 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:49.654 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:49.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.651+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16781569f0 0x7f16781f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.651+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f16781f0a90 con 0x7f16781569f0 2024-09-16T01:52:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.652+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16781569f0 0x7f16781f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.652+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16781569f0 0x7f16781f3080 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.653+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16781f0a90 con 0x7f16781569f0 2024-09-16T01:52:49.667 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:49.668 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:49.668 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:49.668 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:49.668 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:49.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.668+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f16780cb650 con 0x7f16781569f0 2024-09-16T01:52:49.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.669+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f16780cb650 con 0x7f16781569f0 2024-09-16T01:52:49.672 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:49.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.670+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1678062630 0x7f1678062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.670+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f16781f3e90 con 0x7f1678062630 2024-09-16T01:52:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.670+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1678062630 0x7f1678062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.670+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1678062630 0x7f1678062a50 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.671+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16781f3e90 con 0x7f1678062630 2024-09-16T01:52:49.679 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:49.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.684+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1678063260 con 0x7f1678062630 2024-09-16T01:52:49.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.686+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1678063260 con 0x7f1678062630 2024-09-16T01:52:49.688 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:49.689 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:49.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.686+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f167815c1b0 0x7f167815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.686+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f167815caf0 con 0x7f167815c1b0 2024-09-16T01:52:49.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.687+0000 7f167d800640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f167815c1b0 0x7f167815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.687+0000 7f167d800640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f167815c1b0 0x7f167815c5b0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.688+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f167815caf0 con 0x7f167815c1b0 2024-09-16T01:52:49.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.701+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f167815caf0 con 0x7f167815c1b0 2024-09-16T01:52:49.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.703+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f167815caf0 con 0x7f167815c1b0 2024-09-16T01:52:49.705 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:49.705 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:49.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.703+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f167802cda0 0x7f167802d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.703+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f167802d740 con 0x7f167802cda0 2024-09-16T01:52:49.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.703+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f167802cda0 0x7f167802d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.704+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f167802cda0 0x7f167802d200 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.704+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f167802d740 con 0x7f167802cda0 2024-09-16T01:52:49.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.717+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f167802d740 con 0x7f167802cda0 2024-09-16T01:52:49.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.720+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f167802d740 con 0x7f167802cda0 2024-09-16T01:52:49.722 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:49.723 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:49.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.720+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f167802d8b0 0x7f1678206920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.720+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f16782096f0 con 0x7f167802d8b0 2024-09-16T01:52:49.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.720+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f167802d8b0 0x7f1678206920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.723 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:49.720+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:49.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.721+0000 7f166fe00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f167802d8b0 0x7f1678206920 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.722+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16782096f0 con 0x7f167802d8b0 2024-09-16T01:52:49.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.735+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1678206e60 con 0x7f167802d8b0 2024-09-16T01:52:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.738+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1678206e60 con 0x7f167802d8b0 2024-09-16T01:52:49.740 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:49.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.738+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16782074e0 0x7f16782098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.738+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f167820c6a0 con 0x7f16782074e0 2024-09-16T01:52:49.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.738+0000 7f167d800640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16782074e0 0x7f16782098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.739+0000 7f167d800640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16782074e0 0x7f16782098d0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.740+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f167820c6a0 con 0x7f16782074e0 2024-09-16T01:52:49.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.753+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1678209e10 con 0x7f16782074e0 2024-09-16T01:52:49.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.755+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1678209e10 con 0x7f16782074e0 2024-09-16T01:52:49.757 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:49.757 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:49.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.755+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f167820a490 0x7f167820a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:49.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.755+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f167820fa30 con 0x7f167820a490 2024-09-16T01:52:49.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.755+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f167820a490 0x7f167820a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:49.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.756+0000 7f167ce00640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f167820a490 0x7f167820a8f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:49.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.757+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f167820fa30 con 0x7f167820a490 2024-09-16T01:52:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.770+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f167820af40 con 0x7f167820a490 2024-09-16T01:52:49.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f1675600640 1 -- 172.21.15.146:0/285895702 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f167820af40 con 0x7f167820a490 2024-09-16T01:52:49.774 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f167806f440 msgr2=0x7f1678150440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f167806f440 0x7f1678150440 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16781569f0 msgr2=0x7f16781f3080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f16781569f0 0x7f16781f3080 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1678062630 msgr2=0x7f1678062a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f1678062630 0x7f1678062a50 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f167815c1b0 msgr2=0x7f167815c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f167815c1b0 0x7f167815c5b0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f167802cda0 msgr2=0x7f167802d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f167802cda0 0x7f167802d200 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f167802d8b0 msgr2=0x7f1678206920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f167802d8b0 0x7f1678206920 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16782074e0 msgr2=0x7f16782098d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f16782074e0 0x7f16782098d0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f167820a490 msgr2=0x7f167820a8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f167820a490 0x7f167820a8f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167d800640 1 -- 172.21.15.146:0/285895702 reap_dead start 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167d800640 1 -- 172.21.15.146:0/285895702 reap_dead start 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167d800640 1 -- 172.21.15.146:0/285895702 reap_dead start 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167d800640 1 -- 172.21.15.146:0/285895702 reap_dead start 2024-09-16T01:52:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167d800640 1 -- 172.21.15.146:0/285895702 reap_dead start 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.772+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f165406c2b0 msgr2=0x7f165406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f165406c2b0 0x7f165406e770 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1668000f60 tx=0x7f1668007ea0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 msgr2=0x7f16781571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 0x7f16781571b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f16780cbc80 tx=0x7f16600046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 shutdown_connections 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f165406c2b0 0x7f165406e770 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 --2- 172.21.15.146:0/285895702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16780cd470 0x7f16781571b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 >> 172.21.15.146:0/285895702 conn(0x7f1678067d40 msgr2=0x7f16780c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 shutdown_connections 2024-09-16T01:52:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:49.773+0000 7f167e200640 1 -- 172.21.15.146:0/285895702 wait complete. 2024-09-16T01:52:49.799 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:49.799 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:49.810 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:49.853 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:49.852+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:49.924 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:49.923+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:49.928 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:49.929 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:49.941 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:50.025 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:50.024+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:50.098 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.098 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.109 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:50.126 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:50.125+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:50.226 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.226 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.227 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:50.226+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:50.237 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:50.328 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:50.326+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:50.393 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.393 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.402 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:50.446 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:50.445+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:50.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:50.519 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.519 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.529 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:50.528+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:50.530 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:50.630 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:50.629+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:50.685 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.685 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.696 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:50.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.694+0000 7f388ee00640 1 Processor -- start 2024-09-16T01:52:50.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.695+0000 7f388ee00640 1 -- start start 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388ee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38880c75a0 0x7f388815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388ee00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f38880cc040 0x7f38880cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388ee00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f388806d140 con 0x7f38880c6b70 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388ee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f388815b880 con 0x7f38880c75a0 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388ee00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f38880cab20 con 0x7f38880cc040 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388da00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f38880c6b70 0x7f38880cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:50048/0 (socket says 172.21.15.146:50048) 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388da00640 1 -- 172.21.15.146:0/3258325956 learned_addr learned my addr 172.21.15.146:0/3258325956 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388d000640 1 --2- 172.21.15.146:0/3258325956 >> v2:172.21.15.146:3300/0 conn(0x7f38880c75a0 0x7f388815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.696+0000 7f388e400640 1 --2- 172.21.15.146:0/3258325956 >> v2:172.21.15.161:3300/0 conn(0x7f38880cc040 0x7f38880cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.697+0000 7f3887e00640 1 -- 172.21.15.146:0/3258325956 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1651823559 0 0) 0x7f388806d140 con 0x7f38880c6b70 2024-09-16T01:52:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.697+0000 7f3887e00640 1 -- 172.21.15.146:0/3258325956 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f386c003520 con 0x7f38880c6b70 2024-09-16T01:52:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.697+0000 7f388e400640 1 -- 172.21.15.146:0/3258325956 >> v2:172.21.15.146:3300/0 conn(0x7f38880c75a0 msgr2=0x7f388815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.697+0000 7f388e400640 1 --2- 172.21.15.146:0/3258325956 >> v2:172.21.15.146:3300/0 conn(0x7f38880c75a0 0x7f388815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.697+0000 7f388e400640 1 -- 172.21.15.146:0/3258325956 >> v1:172.21.15.161:6789/0 conn(0x7f38880c6b70 legacy=0x7f38880cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:50.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.697+0000 7f388e400640 1 -- 172.21.15.146:0/3258325956 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388806d420 con 0x7f38880cc040 2024-09-16T01:52:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.698+0000 7f388e400640 1 --2- 172.21.15.146:0/3258325956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880cc040 0x7f38880cc4c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f388806ee20 tx=0x7f388002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=490f141ad5a6b0b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.699+0000 7f3887e00640 1 -- 172.21.15.146:0/3258325956 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880007740 con 0x7f38880cc040 2024-09-16T01:52:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.699+0000 7f3887e00640 1 -- 172.21.15.146:0/3258325956 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38800078e0 con 0x7f38880cc040 2024-09-16T01:52:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.699+0000 7f3887e00640 1 -- 172.21.15.146:0/3258325956 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880007cb0 con 0x7f38880cc040 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.699+0000 7f388ee00640 1 -- 172.21.15.146:0/3258325956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880cc040 msgr2=0x7f38880cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.699+0000 7f388ee00640 1 --2- 172.21.15.146:0/3258325956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880cc040 0x7f38880cc4c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f388806ee20 tx=0x7f388002a5f0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.700+0000 7f388ee00640 1 -- 172.21.15.146:0/3258325956 shutdown_connections 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.700+0000 7f388ee00640 1 --2- 172.21.15.146:0/3258325956 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880cc040 0x7f38880cc4c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.700+0000 7f388ee00640 1 --2- 172.21.15.146:0/3258325956 >> v2:172.21.15.146:3300/0 conn(0x7f38880c75a0 0x7f388815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.700+0000 7f388ee00640 1 -- 172.21.15.146:0/3258325956 >> 172.21.15.146:0/3258325956 conn(0x7f38880ccc20 msgr2=0x7f38880cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.700+0000 7f388ee00640 1 -- 172.21.15.146:0/3258325956 shutdown_connections 2024-09-16T01:52:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.700+0000 7f388ee00640 1 -- 172.21.15.146:0/3258325956 wait complete. 2024-09-16T01:52:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.701+0000 7f388ee00640 1 Processor -- start 2024-09-16T01:52:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.701+0000 7f388ee00640 1 -- start start 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.702+0000 7f388ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 0x7f3888157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.702+0000 7f388ee00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880c75a0 0x7f388815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.702+0000 7f388ee00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38880cc040 0x7f388815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.702+0000 7f388ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f38880cb050 con 0x7f38880c6b70 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.702+0000 7f388ee00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f38880cad70 con 0x7f38880c75a0 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.702+0000 7f388ee00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f38880cbc80 con 0x7f38880cc040 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 0x7f3888157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 0x7f3888157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45820/0 (socket says 172.21.15.146:45820) 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 -- 172.21.15.146:0/3195801661 learned_addr learned my addr 172.21.15.146:0/3195801661 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388d000640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880c75a0 0x7f388815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388e400640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38880cc040 0x7f388815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 -- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38880cc040 msgr2=0x7f388815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38880cc040 0x7f388815bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 -- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880c75a0 msgr2=0x7f388815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880c75a0 0x7f388815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 -- 172.21.15.146:0/3195801661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38881f1940 con 0x7f38880c6b70 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388d000640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880c75a0 0x7f388815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.703+0000 7f388da00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 0x7f3888157b80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f387800c3c0 tx=0x7f387800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.704+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3878017070 con 0x7f38880c6b70 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.704+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3878005ce0 con 0x7f38880c6b70 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.704+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38780072f0 con 0x7f38880c6b70 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.704+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38881f1660 con 0x7f38880c6b70 2024-09-16T01:52:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.704+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388815e330 con 0x7f38880c6b70 2024-09-16T01:52:50.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.705+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38880c66d0 con 0x7f38880c6b70 2024-09-16T01:52:50.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.706+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f387801a070 con 0x7f38880c6b70 2024-09-16T01:52:50.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.707+0000 7f3885600640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f386806c2b0 0x7f386806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:50.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.707+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5969+0+0 (secure 0 0 0) 0x7f387808a9e0 con 0x7f38880c6b70 2024-09-16T01:52:50.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.708+0000 7f388d000640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f386806c2b0 0x7f386806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:50.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.708+0000 7f388d000640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f386806c2b0 0x7f386806e770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f38880c8160 tx=0x7f3874008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:50.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.712+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f38880c66d0 con 0x7f38880c6b70 2024-09-16T01:52:50.739 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:50.738+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:50.804 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.804 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.813 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:50.832 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:50.830+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:50.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.911+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f388806cd20 con 0x7f386806c2b0 2024-09-16T01:52:50.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.917+0000 7f3885600640 1 -- 172.21.15.146:0/3195801661 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254659 (secure 0 0 0) 0x7f388806cd20 con 0x7f386806c2b0 2024-09-16T01:52:50.922 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:50.923 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f386806c2b0 msgr2=0x7f386806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f386806c2b0 0x7f386806e770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f38880c8160 tx=0x7f3874008040 comp rx=0 tx=0).stop 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 msgr2=0x7f3888157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 0x7f3888157b80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f387800c3c0 tx=0x7f387800c880 comp rx=0 tx=0).stop 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 shutdown_connections 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f386806c2b0 0x7f386806e770 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f38880cc040 0x7f388815bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38880c75a0 0x7f388815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 --2- 172.21.15.146:0/3195801661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38880c6b70 0x7f3888157b80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:50.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 >> 172.21.15.146:0/3195801661 conn(0x7f38880ccc20 msgr2=0x7f38880c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:50.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 shutdown_connections 2024-09-16T01:52:50.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:50.921+0000 7f388ee00640 1 -- 172.21.15.146:0/3195801661 wait complete. 2024-09-16T01:52:50.933 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:50.932+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:50.939 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T01:52:50.564185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":50,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":7150,"num_write_kb":24935336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2726,"num_bytes_recovered":8296580415,"num_keys_recovered":135,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7591,"ondisk_log_size":7591,"up":194,"acting":189,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":198,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1546832,"kb_used_data":1338904,"kb_used_omap":9,"kb_used_meta":207798,"kb_avail":564684208,"statfs":{"total":579820584960,"available":578236628992,"internally_reserved":0,"allocated":1371037696,"data_stored":21799293182,"data_compressed":1088107399,"data_compressed_allocated":1362235392,"data_compressed_original":21794983936,"omap_allocated":9562,"internal_metadata":212785830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":2,"num_objects_missing":-1,"num_objects_degraded":50,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":1003520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":880803986,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000702"},"pg_stats":[{"pgid":"3.3f","version":"112'104","reported_seq":378,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556452+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:46.556452+0000","last_peered":"2024-09-16T01:52:46.556452+0000","last_clean":"2024-09-16T01:52:46.556452+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:46.556452+0000","last_undegraded":"2024-09-16T01:52:46.556452+0000","last_fullsized":"2024-09-16T01:52:46.556452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532378+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:45.532378+0000","last_peered":"2024-09-16T01:52:45.532378+0000","last_clean":"2024-09-16T01:52:45.532378+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:45.532378+0000","last_undegraded":"2024-09-16T01:52:45.532378+0000","last_fullsized":"2024-09-16T01:52:45.532378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":446,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.091998+0000","last_change":"2024-09-16T01:52:45.534236+0000","last_active":"2024-09-16T01:52:45.534236+0000","last_peered":"2024-09-16T01:52:45.091998+0000","last_clean":"2024-09-16T01:52:45.091998+0000","last_became_active":"2024-09-16T01:52:03.309937+0000","last_became_peered":"2024-09-16T01:52:03.309937+0000","last_unstale":"2024-09-16T01:52:45.091998+0000","last_undegraded":"2024-09-16T01:52:45.091998+0000","last_fullsized":"2024-09-16T01:52:45.091998+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":209715200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"111'124","reported_seq":324,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.543316+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:46.543316+0000","last_peered":"2024-09-16T01:52:46.543316+0000","last_clean":"2024-09-16T01:52:46.543316+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:46.543316+0000","last_undegraded":"2024-09-16T01:52:46.543316+0000","last_fullsized":"2024-09-16T01:52:46.543316+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"110'109","reported_seq":273,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507567+0000","last_change":"2024-09-16T01:52:45.533438+0000","last_active":"2024-09-16T01:52:45.533438+0000","last_peered":"2024-09-16T01:52:44.507567+0000","last_clean":"2024-09-16T01:52:44.507567+0000","last_became_active":"2024-09-16T01:52:09.424479+0000","last_became_peered":"2024-09-16T01:52:09.424479+0000","last_unstale":"2024-09-16T01:52:44.507567+0000","last_undegraded":"2024-09-16T01:52:44.507567+0000","last_fullsized":"2024-09-16T01:52:44.507567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"112'102","reported_seq":349,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535873+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:46.535873+0000","last_peered":"2024-09-16T01:52:46.535873+0000","last_clean":"2024-09-16T01:52:46.535873+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:46.535873+0000","last_undegraded":"2024-09-16T01:52:46.535873+0000","last_fullsized":"2024-09-16T01:52:46.535873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"112'125","reported_seq":494,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.548461+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:46.548461+0000","last_peered":"2024-09-16T01:52:46.548461+0000","last_clean":"2024-09-16T01:52:46.548461+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:46.548461+0000","last_undegraded":"2024-09-16T01:52:46.548461+0000","last_fullsized":"2024-09-16T01:52:46.548461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"111'127","reported_seq":445,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532294+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:45.532294+0000","last_peered":"2024-09-16T01:52:45.532294+0000","last_clean":"2024-09-16T01:52:45.532294+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:45.532294+0000","last_undegraded":"2024-09-16T01:52:45.532294+0000","last_fullsized":"2024-09-16T01:52:45.532294+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":466944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"112'113","reported_seq":308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.539660+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:46.539660+0000","last_peered":"2024-09-16T01:52:46.539660+0000","last_clean":"2024-09-16T01:52:46.539660+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:46.539660+0000","last_undegraded":"2024-09-16T01:52:46.539660+0000","last_fullsized":"2024-09-16T01:52:46.539660+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"112'117","reported_seq":416,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.543142+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:46.543142+0000","last_peered":"2024-09-16T01:52:46.543142+0000","last_clean":"2024-09-16T01:52:46.543142+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:46.543142+0000","last_undegraded":"2024-09-16T01:52:46.543142+0000","last_fullsized":"2024-09-16T01:52:46.543142+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"112'135","reported_seq":459,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.551141+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:46.551141+0000","last_peered":"2024-09-16T01:52:46.551141+0000","last_clean":"2024-09-16T01:52:46.551141+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:46.551141+0000","last_undegraded":"2024-09-16T01:52:46.551141+0000","last_fullsized":"2024-09-16T01:52:46.551141+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"110'105","reported_seq":119,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.565780+0000","last_change":"2024-09-16T01:52:46.540152+0000","last_active":"2024-09-16T01:52:45.565780+0000","last_peered":"2024-09-16T01:52:44.499435+0000","last_clean":"2024-09-16T01:52:44.499435+0000","last_became_active":"2024-09-16T01:52:40.724710+0000","last_became_peered":"2024-09-16T01:52:40.724710+0000","last_unstale":"2024-09-16T01:52:45.565780+0000","last_undegraded":"2024-09-16T01:52:45.565780+0000","last_fullsized":"2024-09-16T01:52:45.565780+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"113'112","reported_seq":438,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493569+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:50.493569+0000","last_peered":"2024-09-16T01:52:50.493569+0000","last_clean":"2024-09-16T01:52:50.493569+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:50.493569+0000","last_undegraded":"2024-09-16T01:52:50.493569+0000","last_fullsized":"2024-09-16T01:52:50.493569+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"113'107","reported_seq":360,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492306+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:50.492306+0000","last_peered":"2024-09-16T01:52:50.492306+0000","last_clean":"2024-09-16T01:52:50.492306+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:50.492306+0000","last_undegraded":"2024-09-16T01:52:50.492306+0000","last_fullsized":"2024-09-16T01:52:50.492306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"111'108","reported_seq":404,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.531409+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:45.531409+0000","last_peered":"2024-09-16T01:52:45.531409+0000","last_clean":"2024-09-16T01:52:45.531409+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:45.531409+0000","last_undegraded":"2024-09-16T01:52:45.531409+0000","last_fullsized":"2024-09-16T01:52:45.531409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"111'107","reported_seq":301,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535166+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:46.535166+0000","last_peered":"2024-09-16T01:52:46.535166+0000","last_clean":"2024-09-16T01:52:46.535166+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:46.535166+0000","last_undegraded":"2024-09-16T01:52:46.535166+0000","last_fullsized":"2024-09-16T01:52:46.535166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"111'103","reported_seq":394,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556216+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:46.556216+0000","last_peered":"2024-09-16T01:52:46.556216+0000","last_clean":"2024-09-16T01:52:46.556216+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:46.556216+0000","last_undegraded":"2024-09-16T01:52:46.556216+0000","last_fullsized":"2024-09-16T01:52:46.556216+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"110'105","reported_seq":383,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.562116+0000","last_change":"2024-09-16T01:52:46.548473+0000","last_active":"2024-09-16T01:52:45.562116+0000","last_peered":"2024-09-16T01:52:44.522618+0000","last_clean":"2024-09-16T01:52:44.522618+0000","last_became_active":"2024-09-16T01:52:03.349420+0000","last_became_peered":"2024-09-16T01:52:03.349420+0000","last_unstale":"2024-09-16T01:52:45.562116+0000","last_undegraded":"2024-09-16T01:52:45.562116+0000","last_fullsized":"2024-09-16T01:52:45.562116+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"114'124","reported_seq":560,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492538+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:52:50.492538+0000","last_peered":"2024-09-16T01:52:50.492538+0000","last_clean":"2024-09-16T01:52:50.492538+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:52:50.492538+0000","last_undegraded":"2024-09-16T01:52:50.492538+0000","last_fullsized":"2024-09-16T01:52:50.492538+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"112'114","reported_seq":452,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.540506+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:46.540506+0000","last_peered":"2024-09-16T01:52:46.540506+0000","last_clean":"2024-09-16T01:52:46.540506+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:46.540506+0000","last_undegraded":"2024-09-16T01:52:46.540506+0000","last_fullsized":"2024-09-16T01:52:46.540506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"112'93","reported_seq":325,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.542708+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:46.542708+0000","last_peered":"2024-09-16T01:52:46.542708+0000","last_clean":"2024-09-16T01:52:46.542708+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:46.542708+0000","last_undegraded":"2024-09-16T01:52:46.542708+0000","last_fullsized":"2024-09-16T01:52:46.542708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"112'126","reported_seq":567,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.538283+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:52:46.538283+0000","last_peered":"2024-09-16T01:52:46.538283+0000","last_clean":"2024-09-16T01:52:46.538283+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:52:46.538283+0000","last_undegraded":"2024-09-16T01:52:46.538283+0000","last_fullsized":"2024-09-16T01:52:46.538283+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532334+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:45.532334+0000","last_peered":"2024-09-16T01:52:45.532334+0000","last_clean":"2024-09-16T01:52:45.532334+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:45.532334+0000","last_undegraded":"2024-09-16T01:52:45.532334+0000","last_fullsized":"2024-09-16T01:52:45.532334+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"111'132","reported_seq":326,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535263+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:46.535263+0000","last_peered":"2024-09-16T01:52:46.535263+0000","last_clean":"2024-09-16T01:52:46.535263+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:46.535263+0000","last_undegraded":"2024-09-16T01:52:46.535263+0000","last_fullsized":"2024-09-16T01:52:46.535263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"111'100","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532061+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:45.532061+0000","last_peered":"2024-09-16T01:52:45.532061+0000","last_clean":"2024-09-16T01:52:45.532061+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:45.532061+0000","last_undegraded":"2024-09-16T01:52:45.532061+0000","last_fullsized":"2024-09-16T01:52:45.532061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"112'142","reported_seq":337,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.544916+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:46.544916+0000","last_peered":"2024-09-16T01:52:46.544916+0000","last_clean":"2024-09-16T01:52:46.544916+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:46.544916+0000","last_undegraded":"2024-09-16T01:52:46.544916+0000","last_fullsized":"2024-09-16T01:52:46.544916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"114'105","reported_seq":538,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493440+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:52:50.493440+0000","last_peered":"2024-09-16T01:52:50.493440+0000","last_clean":"2024-09-16T01:52:50.493440+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:52:50.493440+0000","last_undegraded":"2024-09-16T01:52:50.493440+0000","last_fullsized":"2024-09-16T01:52:50.493440+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"113'100","reported_seq":396,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492476+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:50.492476+0000","last_peered":"2024-09-16T01:52:50.492476+0000","last_clean":"2024-09-16T01:52:50.492476+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:50.492476+0000","last_undegraded":"2024-09-16T01:52:50.492476+0000","last_fullsized":"2024-09-16T01:52:50.492476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"109'87","reported_seq":481,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507610+0000","last_change":"2024-09-16T01:52:45.531675+0000","last_active":"2024-09-16T01:52:45.531675+0000","last_peered":"2024-09-16T01:52:44.507610+0000","last_clean":"2024-09-16T01:52:44.507610+0000","last_became_active":"2024-09-16T01:52:09.378210+0000","last_became_peered":"2024-09-16T01:52:09.378210+0000","last_unstale":"2024-09-16T01:52:44.507610+0000","last_undegraded":"2024-09-16T01:52:44.507610+0000","last_fullsized":"2024-09-16T01:52:44.507610+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"112'98","reported_seq":292,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.533925+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:46.533925+0000","last_peered":"2024-09-16T01:52:46.533925+0000","last_clean":"2024-09-16T01:52:46.533925+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:46.533925+0000","last_undegraded":"2024-09-16T01:52:46.533925+0000","last_fullsized":"2024-09-16T01:52:46.533925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"111'100","reported_seq":401,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.555742+0000","last_change":"2024-09-16T01:52:46.555911+0000","last_active":"2024-09-16T01:52:46.555742+0000","last_peered":"2024-09-16T01:52:46.555742+0000","last_clean":"2024-09-16T01:52:45.020933+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:52:46.555742+0000","last_undegraded":"2024-09-16T01:52:46.537788+0000","last_fullsized":"2024-09-16T01:52:46.555742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":33}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"113'112","reported_seq":428,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492019+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:50.492019+0000","last_peered":"2024-09-16T01:52:50.492019+0000","last_clean":"2024-09-16T01:52:50.492019+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:50.492019+0000","last_undegraded":"2024-09-16T01:52:50.492019+0000","last_fullsized":"2024-09-16T01:52:50.492019+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"112'98","reported_seq":293,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.538859+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:46.538859+0000","last_peered":"2024-09-16T01:52:46.538859+0000","last_clean":"2024-09-16T01:52:46.538859+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:46.538859+0000","last_undegraded":"2024-09-16T01:52:46.538859+0000","last_fullsized":"2024-09-16T01:52:46.538859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":200,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534083+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:45.534083+0000","last_peered":"2024-09-16T01:52:45.534083+0000","last_clean":"2024-09-16T01:52:45.534083+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:45.534083+0000","last_undegraded":"2024-09-16T01:52:45.534083+0000","last_fullsized":"2024-09-16T01:52:45.534083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"111'104","reported_seq":534,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.548924+0000","last_change":"2024-09-16T01:52:46.537357+0000","last_active":"2024-09-16T01:52:45.548924+0000","last_peered":"2024-09-16T01:52:44.543722+0000","last_clean":"2024-09-16T01:52:44.543722+0000","last_became_active":"2024-09-16T01:52:21.480658+0000","last_became_peered":"2024-09-16T01:52:21.480658+0000","last_unstale":"2024-09-16T01:52:45.548924+0000","last_undegraded":"2024-09-16T01:52:45.548924+0000","last_fullsized":"2024-09-16T01:52:45.548924+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":216,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535746+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:46.535746+0000","last_peered":"2024-09-16T01:52:46.535746+0000","last_clean":"2024-09-16T01:52:46.535746+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:46.535746+0000","last_undegraded":"2024-09-16T01:52:46.535746+0000","last_fullsized":"2024-09-16T01:52:46.535746+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":230,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492167+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:52:50.492167+0000","last_peered":"2024-09-16T01:52:50.492167+0000","last_clean":"2024-09-16T01:52:50.492167+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:52:50.492167+0000","last_undegraded":"2024-09-16T01:52:50.492167+0000","last_fullsized":"2024-09-16T01:52:50.492167+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"111'121","reported_seq":314,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532429+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:45.532429+0000","last_peered":"2024-09-16T01:52:45.532429+0000","last_clean":"2024-09-16T01:52:45.532429+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:45.532429+0000","last_undegraded":"2024-09-16T01:52:45.532429+0000","last_fullsized":"2024-09-16T01:52:45.532429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":438272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":199,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535364+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:46.535364+0000","last_peered":"2024-09-16T01:52:46.535364+0000","last_clean":"2024-09-16T01:52:46.535364+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:46.535364+0000","last_undegraded":"2024-09-16T01:52:46.535364+0000","last_fullsized":"2024-09-16T01:52:46.535364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"112'114","reported_seq":274,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.538072+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:46.538072+0000","last_peered":"2024-09-16T01:52:46.538072+0000","last_clean":"2024-09-16T01:52:46.538072+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:46.538072+0000","last_undegraded":"2024-09-16T01:52:46.538072+0000","last_fullsized":"2024-09-16T01:52:46.538072+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":228,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.539769+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:46.539769+0000","last_peered":"2024-09-16T01:52:46.539769+0000","last_clean":"2024-09-16T01:52:46.539769+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:46.539769+0000","last_undegraded":"2024-09-16T01:52:46.539769+0000","last_fullsized":"2024-09-16T01:52:46.539769+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"112'163","reported_seq":367,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.567223+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:45.567223+0000","last_peered":"2024-09-16T01:52:45.567223+0000","last_clean":"2024-09-16T01:52:45.567223+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:45.567223+0000","last_undegraded":"2024-09-16T01:52:45.567223+0000","last_fullsized":"2024-09-16T01:52:45.567223+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":82,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.557984+0000","last_change":"2024-09-16T01:52:46.558524+0000","last_active":"2024-09-16T01:52:46.557984+0000","last_peered":"2024-09-16T01:52:46.557984+0000","last_clean":"2024-09-16T01:52:44.504783+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:52:46.557984+0000","last_undegraded":"2024-09-16T01:52:46.539531+0000","last_fullsized":"2024-09-16T01:52:46.557984+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"110'98","reported_seq":291,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532194+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:45.532194+0000","last_peered":"2024-09-16T01:52:45.532194+0000","last_clean":"2024-09-16T01:52:45.532194+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:45.532194+0000","last_undegraded":"2024-09-16T01:52:45.532194+0000","last_fullsized":"2024-09-16T01:52:45.532194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":96,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.562355+0000","last_change":"2024-09-16T01:52:46.562522+0000","last_active":"2024-09-16T01:52:46.562355+0000","last_peered":"2024-09-16T01:52:46.562355+0000","last_clean":"2024-09-16T01:52:44.507789+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:52:46.562355+0000","last_undegraded":"2024-09-16T01:52:46.536286+0000","last_fullsized":"2024-09-16T01:52:46.562355+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3360,"num_keys_recovered":50,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"112'107","reported_seq":301,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537869+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:46.537869+0000","last_peered":"2024-09-16T01:52:46.537869+0000","last_clean":"2024-09-16T01:52:46.537869+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:46.537869+0000","last_undegraded":"2024-09-16T01:52:46.537869+0000","last_fullsized":"2024-09-16T01:52:46.537869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":386,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.543425+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:46.543425+0000","last_peered":"2024-09-16T01:52:46.543425+0000","last_clean":"2024-09-16T01:52:46.543425+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:46.543425+0000","last_undegraded":"2024-09-16T01:52:46.543425+0000","last_fullsized":"2024-09-16T01:52:46.543425+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":417,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493346+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:50.493346+0000","last_peered":"2024-09-16T01:52:50.493346+0000","last_clean":"2024-09-16T01:52:50.493346+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:50.493346+0000","last_undegraded":"2024-09-16T01:52:50.493346+0000","last_fullsized":"2024-09-16T01:52:50.493346+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":113,"state":"active+recovering","last_fresh":"2024-09-16T01:52:46.607325+0000","last_change":"2024-09-16T01:52:46.605694+0000","last_active":"2024-09-16T01:52:46.607325+0000","last_peered":"2024-09-16T01:52:46.607325+0000","last_clean":"2024-09-16T01:52:44.506267+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:52:46.607325+0000","last_undegraded":"2024-09-16T01:52:46.607325+0000","last_fullsized":"2024-09-16T01:52:46.607325+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":306,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4},{"shards":"3,7","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"112'117","reported_seq":320,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492401+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:50.492401+0000","last_peered":"2024-09-16T01:52:50.492401+0000","last_clean":"2024-09-16T01:52:50.492401+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:50.492401+0000","last_undegraded":"2024-09-16T01:52:50.492401+0000","last_fullsized":"2024-09-16T01:52:50.492401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":269,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.562088+0000","last_change":"2024-09-16T01:52:46.562285+0000","last_active":"2024-09-16T01:52:46.562088+0000","last_peered":"2024-09-16T01:52:46.562088+0000","last_clean":"2024-09-16T01:52:44.506397+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:52:46.562088+0000","last_undegraded":"2024-09-16T01:52:46.536120+0000","last_fullsized":"2024-09-16T01:52:46.562088+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"112'121","reported_seq":446,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537496+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:46.537496+0000","last_peered":"2024-09-16T01:52:46.537496+0000","last_clean":"2024-09-16T01:52:46.537496+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:46.537496+0000","last_undegraded":"2024-09-16T01:52:46.537496+0000","last_fullsized":"2024-09-16T01:52:46.537496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":73,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.534295+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:46.534295+0000","last_peered":"2024-09-16T01:52:46.534295+0000","last_clean":"2024-09-16T01:52:46.534295+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:46.534295+0000","last_undegraded":"2024-09-16T01:52:46.534295+0000","last_fullsized":"2024-09-16T01:52:46.534295+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"112'132","reported_seq":327,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537538+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:46.537538+0000","last_peered":"2024-09-16T01:52:46.537538+0000","last_clean":"2024-09-16T01:52:46.537538+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:46.537538+0000","last_undegraded":"2024-09-16T01:52:46.537538+0000","last_fullsized":"2024-09-16T01:52:46.537538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":125,"reported_epoch":113,"state":"active+recovering","last_fresh":"2024-09-16T01:52:46.594029+0000","last_change":"2024-09-16T01:52:46.585166+0000","last_active":"2024-09-16T01:52:46.594029+0000","last_peered":"2024-09-16T01:52:46.594029+0000","last_clean":"2024-09-16T01:52:44.496597+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:52:46.594029+0000","last_undegraded":"2024-09-16T01:52:46.594029+0000","last_fullsized":"2024-09-16T01:52:46.594029+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2},{"shards":"1,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"112'144","reported_seq":338,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.534391+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:46.534391+0000","last_peered":"2024-09-16T01:52:46.534391+0000","last_clean":"2024-09-16T01:52:46.534391+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:46.534391+0000","last_undegraded":"2024-09-16T01:52:46.534391+0000","last_fullsized":"2024-09-16T01:52:46.534391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":495616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":68,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532060+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:45.532060+0000","last_peered":"2024-09-16T01:52:45.532060+0000","last_clean":"2024-09-16T01:52:45.532060+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:45.532060+0000","last_undegraded":"2024-09-16T01:52:45.532060+0000","last_fullsized":"2024-09-16T01:52:45.532060+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"112'106","reported_seq":402,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.546659+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:46.546659+0000","last_peered":"2024-09-16T01:52:46.546659+0000","last_clean":"2024-09-16T01:52:46.546659+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:46.546659+0000","last_undegraded":"2024-09-16T01:52:46.546659+0000","last_fullsized":"2024-09-16T01:52:46.546659+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532518+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:45.532518+0000","last_peered":"2024-09-16T01:52:45.532518+0000","last_clean":"2024-09-16T01:52:45.532518+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:45.532518+0000","last_undegraded":"2024-09-16T01:52:45.532518+0000","last_fullsized":"2024-09-16T01:52:45.532518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"112'103","reported_seq":394,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.539408+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:46.539408+0000","last_peered":"2024-09-16T01:52:46.539408+0000","last_clean":"2024-09-16T01:52:46.539408+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:46.539408+0000","last_undegraded":"2024-09-16T01:52:46.539408+0000","last_fullsized":"2024-09-16T01:52:46.539408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":198,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534027+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:45.534027+0000","last_peered":"2024-09-16T01:52:45.534027+0000","last_clean":"2024-09-16T01:52:45.534027+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:45.534027+0000","last_undegraded":"2024-09-16T01:52:45.534027+0000","last_fullsized":"2024-09-16T01:52:45.534027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"111'101","reported_seq":520,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.538823+0000","last_change":"2024-09-16T01:52:46.535097+0000","last_active":"2024-09-16T01:52:45.538823+0000","last_peered":"2024-09-16T01:52:44.567355+0000","last_clean":"2024-09-16T01:52:44.567355+0000","last_became_active":"2024-09-16T01:52:09.413639+0000","last_became_peered":"2024-09-16T01:52:09.413639+0000","last_unstale":"2024-09-16T01:52:45.538823+0000","last_undegraded":"2024-09-16T01:52:45.538823+0000","last_fullsized":"2024-09-16T01:52:45.538823+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":318767104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":223,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492672+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:50.492672+0000","last_peered":"2024-09-16T01:52:50.492672+0000","last_clean":"2024-09-16T01:52:50.492672+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:50.492672+0000","last_undegraded":"2024-09-16T01:52:50.492672+0000","last_fullsized":"2024-09-16T01:52:50.492672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"112'112","reported_seq":306,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.552650+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:46.552650+0000","last_peered":"2024-09-16T01:52:46.552650+0000","last_clean":"2024-09-16T01:52:46.552650+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:46.552650+0000","last_undegraded":"2024-09-16T01:52:46.552650+0000","last_fullsized":"2024-09-16T01:52:46.552650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":353,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537651+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:46.537651+0000","last_peered":"2024-09-16T01:52:46.537651+0000","last_clean":"2024-09-16T01:52:46.537651+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:46.537651+0000","last_undegraded":"2024-09-16T01:52:46.537651+0000","last_fullsized":"2024-09-16T01:52:46.537651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":202,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537695+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:46.537695+0000","last_peered":"2024-09-16T01:52:46.537695+0000","last_clean":"2024-09-16T01:52:46.537695+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:46.537695+0000","last_undegraded":"2024-09-16T01:52:46.537695+0000","last_fullsized":"2024-09-16T01:52:46.537695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"112'111","reported_seq":446,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535786+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:46.535786+0000","last_peered":"2024-09-16T01:52:46.535786+0000","last_clean":"2024-09-16T01:52:46.535786+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:46.535786+0000","last_undegraded":"2024-09-16T01:52:46.535786+0000","last_fullsized":"2024-09-16T01:52:46.535786+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":200,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535459+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:46.535459+0000","last_peered":"2024-09-16T01:52:46.535459+0000","last_clean":"2024-09-16T01:52:46.535459+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:46.535459+0000","last_undegraded":"2024-09-16T01:52:46.535459+0000","last_fullsized":"2024-09-16T01:52:46.535459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"113'107","reported_seq":310,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492106+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:50.492106+0000","last_peered":"2024-09-16T01:52:50.492106+0000","last_clean":"2024-09-16T01:52:50.492106+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:50.492106+0000","last_undegraded":"2024-09-16T01:52:50.492106+0000","last_fullsized":"2024-09-16T01:52:50.492106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":240,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507474+0000","last_change":"2024-09-16T01:52:45.531950+0000","last_active":"2024-09-16T01:52:45.531950+0000","last_peered":"2024-09-16T01:52:44.507474+0000","last_clean":"2024-09-16T01:52:44.507474+0000","last_became_active":"2024-09-16T01:52:21.493366+0000","last_became_peered":"2024-09-16T01:52:21.493366+0000","last_unstale":"2024-09-16T01:52:44.507474+0000","last_undegraded":"2024-09-16T01:52:44.507474+0000","last_fullsized":"2024-09-16T01:52:44.507474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"111'93","reported_seq":410,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.298724+0000","last_change":"2024-09-16T01:52:45.533111+0000","last_active":"2024-09-16T01:52:45.533111+0000","last_peered":"2024-09-16T01:52:45.298724+0000","last_clean":"2024-09-16T01:52:45.298724+0000","last_became_active":"2024-09-16T01:52:03.344719+0000","last_became_peered":"2024-09-16T01:52:03.344719+0000","last_unstale":"2024-09-16T01:52:45.298724+0000","last_undegraded":"2024-09-16T01:52:45.298724+0000","last_fullsized":"2024-09-16T01:52:45.298724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":150994944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":201,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.536379+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:46.536379+0000","last_peered":"2024-09-16T01:52:46.536379+0000","last_clean":"2024-09-16T01:52:46.536379+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:46.536379+0000","last_undegraded":"2024-09-16T01:52:46.536379+0000","last_fullsized":"2024-09-16T01:52:46.536379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"111'110","reported_seq":404,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.531301+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:46.531301+0000","last_peered":"2024-09-16T01:52:46.531301+0000","last_clean":"2024-09-16T01:52:46.531301+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:46.531301+0000","last_undegraded":"2024-09-16T01:52:46.531301+0000","last_fullsized":"2024-09-16T01:52:46.531301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":196,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.533427+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:45.533427+0000","last_peered":"2024-09-16T01:52:45.533427+0000","last_clean":"2024-09-16T01:52:45.533427+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:45.533427+0000","last_undegraded":"2024-09-16T01:52:45.533427+0000","last_fullsized":"2024-09-16T01:52:45.533427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"113'110","reported_seq":305,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.547100+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:46.547100+0000","last_peered":"2024-09-16T01:52:46.547100+0000","last_clean":"2024-09-16T01:52:46.547100+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:46.547100+0000","last_undegraded":"2024-09-16T01:52:46.547100+0000","last_fullsized":"2024-09-16T01:52:46.547100+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":28,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.546395+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:52:46.546395+0000","last_peered":"2024-09-16T01:52:46.546395+0000","last_clean":"2024-09-16T01:52:46.546395+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:52:46.546395+0000","last_undegraded":"2024-09-16T01:52:46.546395+0000","last_fullsized":"2024-09-16T01:52:46.546395+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"113'103","reported_seq":306,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492575+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:50.492575+0000","last_peered":"2024-09-16T01:52:50.492575+0000","last_clean":"2024-09-16T01:52:50.492575+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:50.492575+0000","last_undegraded":"2024-09-16T01:52:50.492575+0000","last_fullsized":"2024-09-16T01:52:50.492575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532417+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:45.532417+0000","last_peered":"2024-09-16T01:52:45.532417+0000","last_clean":"2024-09-16T01:52:45.532417+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:45.532417+0000","last_undegraded":"2024-09-16T01:52:45.532417+0000","last_fullsized":"2024-09-16T01:52:45.532417+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"113'114","reported_seq":362,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493268+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:50.493268+0000","last_peered":"2024-09-16T01:52:50.493268+0000","last_clean":"2024-09-16T01:52:50.493268+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:50.493268+0000","last_undegraded":"2024-09-16T01:52:50.493268+0000","last_fullsized":"2024-09-16T01:52:50.493268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":218,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556313+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:46.556313+0000","last_peered":"2024-09-16T01:52:46.556313+0000","last_clean":"2024-09-16T01:52:46.556313+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:46.556313+0000","last_undegraded":"2024-09-16T01:52:46.556313+0000","last_fullsized":"2024-09-16T01:52:46.556313+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"111'115","reported_seq":285,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.557058+0000","last_change":"2024-09-16T01:52:46.540709+0000","last_active":"2024-09-16T01:52:45.557058+0000","last_peered":"2024-09-16T01:52:44.870429+0000","last_clean":"2024-09-16T01:52:44.870429+0000","last_became_active":"2024-09-16T01:52:03.364819+0000","last_became_peered":"2024-09-16T01:52:03.364819+0000","last_unstale":"2024-09-16T01:52:45.557058+0000","last_undegraded":"2024-09-16T01:52:45.557058+0000","last_fullsized":"2024-09-16T01:52:45.557058+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":255852544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":82,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.618081+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:52:46.618081+0000","last_peered":"2024-09-16T01:52:46.618081+0000","last_clean":"2024-09-16T01:52:46.618081+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:52:46.618081+0000","last_undegraded":"2024-09-16T01:52:46.618081+0000","last_fullsized":"2024-09-16T01:52:46.618081+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"112'114","reported_seq":363,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535865+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:46.535865+0000","last_peered":"2024-09-16T01:52:46.535865+0000","last_clean":"2024-09-16T01:52:46.535865+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:46.535865+0000","last_undegraded":"2024-09-16T01:52:46.535865+0000","last_fullsized":"2024-09-16T01:52:46.535865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492054+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:50.492054+0000","last_peered":"2024-09-16T01:52:50.492054+0000","last_clean":"2024-09-16T01:52:50.492054+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:50.492054+0000","last_undegraded":"2024-09-16T01:52:50.492054+0000","last_fullsized":"2024-09-16T01:52:50.492054+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"113'115","reported_seq":349,"reported_epoch":113,"state":"active+recovering","last_fresh":"2024-09-16T01:52:46.632747+0000","last_change":"2024-09-16T01:52:46.654054+0000","last_active":"2024-09-16T01:52:46.632747+0000","last_peered":"2024-09-16T01:52:46.632747+0000","last_clean":"2024-09-16T01:52:39.701448+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:52:46.632747+0000","last_undegraded":"2024-09-16T01:52:46.532102+0000","last_fullsized":"2024-09-16T01:52:46.632747+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":503316480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,2,7","objects":1},{"shards":"1,7","objects":6},{"shards":"2,7","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":208,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492505+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:50.492505+0000","last_peered":"2024-09-16T01:52:50.492505+0000","last_clean":"2024-09-16T01:52:50.492505+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:50.492505+0000","last_undegraded":"2024-09-16T01:52:50.492505+0000","last_fullsized":"2024-09-16T01:52:50.492505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"114'105","reported_seq":512,"reported_epoch":116,"state":"peering","last_fresh":"2024-09-16T01:52:50.241765+0000","last_change":"2024-09-16T01:52:50.492781+0000","last_active":"2024-09-16T01:52:50.492781+0000","last_peered":"2024-09-16T01:52:50.241765+0000","last_clean":"2024-09-16T01:52:44.861424+0000","last_became_active":"2024-09-16T01:52:47.546951+0000","last_became_peered":"2024-09-16T01:52:47.546951+0000","last_unstale":"2024-09-16T01:52:50.241765+0000","last_undegraded":"2024-09-16T01:52:50.241765+0000","last_fullsized":"2024-09-16T01:52:47.532896+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":201,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537603+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:46.537603+0000","last_peered":"2024-09-16T01:52:46.537603+0000","last_clean":"2024-09-16T01:52:46.537603+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:46.537603+0000","last_undegraded":"2024-09-16T01:52:46.537603+0000","last_fullsized":"2024-09-16T01:52:46.537603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"111'94","reported_seq":288,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.534025+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:46.534025+0000","last_peered":"2024-09-16T01:52:46.534025+0000","last_clean":"2024-09-16T01:52:46.534025+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:46.534025+0000","last_undegraded":"2024-09-16T01:52:46.534025+0000","last_fullsized":"2024-09-16T01:52:46.534025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"112'131","reported_seq":444,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.551191+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:46.551191+0000","last_peered":"2024-09-16T01:52:46.551191+0000","last_clean":"2024-09-16T01:52:46.551191+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:46.551191+0000","last_undegraded":"2024-09-16T01:52:46.551191+0000","last_fullsized":"2024-09-16T01:52:46.551191+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":70,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.540630+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:46.540630+0000","last_peered":"2024-09-16T01:52:46.540630+0000","last_clean":"2024-09-16T01:52:46.540630+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:46.540630+0000","last_undegraded":"2024-09-16T01:52:46.540630+0000","last_fullsized":"2024-09-16T01:52:46.540630+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"111'106","reported_seq":396,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.540728+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:46.540728+0000","last_peered":"2024-09-16T01:52:46.540728+0000","last_clean":"2024-09-16T01:52:46.540728+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:46.540728+0000","last_undegraded":"2024-09-16T01:52:46.540728+0000","last_fullsized":"2024-09-16T01:52:46.540728+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":38,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.541492+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:46.541492+0000","last_peered":"2024-09-16T01:52:46.541492+0000","last_clean":"2024-09-16T01:52:46.541492+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:46.541492+0000","last_undegraded":"2024-09-16T01:52:46.541492+0000","last_fullsized":"2024-09-16T01:52:46.541492+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"111'91","reported_seq":284,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532243+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:45.532243+0000","last_peered":"2024-09-16T01:52:45.532243+0000","last_clean":"2024-09-16T01:52:45.532243+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:45.532243+0000","last_undegraded":"2024-09-16T01:52:45.532243+0000","last_fullsized":"2024-09-16T01:52:45.532243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":221,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.508113+0000","last_change":"2024-09-16T01:52:45.532987+0000","last_active":"2024-09-16T01:52:45.532987+0000","last_peered":"2024-09-16T01:52:44.508113+0000","last_clean":"2024-09-16T01:52:44.508113+0000","last_became_active":"2024-09-16T01:52:21.473912+0000","last_became_peered":"2024-09-16T01:52:21.473912+0000","last_unstale":"2024-09-16T01:52:44.508113+0000","last_undegraded":"2024-09-16T01:52:44.508113+0000","last_fullsized":"2024-09-16T01:52:44.508113+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"111'112","reported_seq":305,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532474+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:45.532474+0000","last_peered":"2024-09-16T01:52:45.532474+0000","last_clean":"2024-09-16T01:52:45.532474+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:45.532474+0000","last_undegraded":"2024-09-16T01:52:45.532474+0000","last_fullsized":"2024-09-16T01:52:45.532474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556039+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:46.556039+0000","last_peered":"2024-09-16T01:52:46.556039+0000","last_clean":"2024-09-16T01:52:46.556039+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:46.556039+0000","last_undegraded":"2024-09-16T01:52:46.556039+0000","last_fullsized":"2024-09-16T01:52:46.556039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":41,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":24932352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2527,"num_bytes_recovered":8296333312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418461184,"data_stored":22695378944,"data_compressed":1130956288,"data_compressed_allocated":1418461184,"data_compressed_original":22695378944,"omap_allocated":0,"internal_metadata":0},"log_size":7114,"ondisk_log_size":7114,"up":128,"acting":123,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":247103,"num_keys_recovered":135,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":996044,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509492,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357424,"kb_used_data":326228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94014416,"statfs":{"total":96636764160,"available":96270761984,"internally_reserved":0,"allocated":334057472,"data_stored":5323198058,"data_compressed":265608704,"data_compressed_allocated":332660736,"data_compressed_original":5322571776,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509492,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349048,"kb_used_data":317864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94022792,"statfs":{"total":96636764160,"available":96279339008,"internally_reserved":0,"allocated":325492736,"data_stored":5180965092,"data_compressed":258638848,"data_compressed_allocated":323747840,"data_compressed_original":5179965440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509491,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249288,"kb_used_data":222264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94122552,"statfs":{"total":96636764160,"available":96381493248,"internally_reserved":0,"allocated":227598336,"data_stored":3617079895,"data_compressed":180653539,"data_compressed_allocated":226058240,"data_compressed_original":3616296960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542195,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69908,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94301932,"statfs":{"total":96636764160,"available":96565178368,"internally_reserved":0,"allocated":35393536,"data_stored":545874692,"data_compressed":27121536,"data_compressed_allocated":34078720,"data_compressed_original":545259520,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542196,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255748,"kb_used_data":212076,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94116092,"statfs":{"total":96636764160,"available":96374878208,"internally_reserved":0,"allocated":217165824,"data_stored":3452562337,"data_compressed":172647168,"data_compressed_allocated":215744512,"data_compressed_original":3451912192,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265416,"kb_used_data":225908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94106424,"statfs":{"total":96636764160,"available":96364978176,"internally_reserved":0,"allocated":231329792,"data_stored":3679613108,"data_compressed":183437604,"data_compressed_allocated":229945344,"data_compressed_original":3678978048,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574900,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":377702,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":377785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":4505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225968128,"data_stored":3615490048,"data_compressed":180631040,"data_compressed_allocated":225968128,"data_compressed_original":3615490048,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323747840,"data_stored":5179965440,"data_compressed":258720384,"data_compressed_allocated":323747840,"data_compressed_original":5179965440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332660736,"data_stored":5322571776,"data_compressed":265899904,"data_compressed_allocated":332660736,"data_compressed_original":5322571776,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":3678404608,"data_compressed":183463424,"data_compressed_allocated":229900288,"data_compressed_original":3678404608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":205520896,"data_stored":3288334336,"data_compressed":164476928,"data_compressed_allocated":205520896,"data_compressed_original":3288334336,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41156608,"data_stored":658505728,"data_compressed":32785920,"data_compressed_allocated":41156608,"data_compressed_original":658505728,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:50.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:50.966 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:50.966 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:50.976 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:51.060 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:51.059+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:51.135 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:51.133+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:51.138 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:51.138 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:51.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:51.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.160+0000 7fe245c00640 1 Processor -- start 2024-09-16T01:52:51.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.160+0000 7fe245c00640 1 -- start start 2024-09-16T01:52:51.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe245c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe2400cb2f0 0x7fe24015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe245c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe240165430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe245c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe24015b1d0 con 0x7fe2400cc0c0 2024-09-16T01:52:51.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe245c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe2400cbd00 con 0x7fe2400cb2f0 2024-09-16T01:52:51.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe245c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe24015ba70 con 0x7fe2400cc4a0 2024-09-16T01:52:51.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe23ea00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe2400cc0c0 0x7fe24015e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33598/0 (socket says 172.21.15.146:33598) 2024-09-16T01:52:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe23ea00640 1 -- 172.21.15.146:0/3632832866 learned_addr learned my addr 172.21.15.146:0/3632832866 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe23f400640 1 --2- 172.21.15.146:0/3632832866 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe240165430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe237e00640 1 --2- 172.21.15.146:0/3632832866 >> v2:172.21.15.161:3300/0 conn(0x7fe2400cb2f0 0x7fe24015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe23f400640 1 -- 172.21.15.146:0/3632832866 >> v2:172.21.15.161:3300/0 conn(0x7fe2400cb2f0 msgr2=0x7fe24015aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe23f400640 1 --2- 172.21.15.146:0/3632832866 >> v2:172.21.15.161:3300/0 conn(0x7fe2400cb2f0 0x7fe24015aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.162+0000 7fe23f400640 1 -- 172.21.15.146:0/3632832866 >> v1:172.21.15.146:6789/0 conn(0x7fe2400cc0c0 legacy=0x7fe24015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.163+0000 7fe23f400640 1 -- 172.21.15.146:0/3632832866 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24015b4b0 con 0x7fe2400cc4a0 2024-09-16T01:52:51.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.163+0000 7fe237e00640 1 --2- 172.21.15.146:0/3632832866 >> v2:172.21.15.161:3300/0 conn(0x7fe2400cb2f0 0x7fe24015aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:51.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.164+0000 7fe23f400640 1 --2- 172.21.15.146:0/3632832866 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe240165430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe238009d90 tx=0x7fe23802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de6a80dccac706e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:51.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.165+0000 7fe23e000640 1 -- 172.21.15.146:0/3632832866 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238007740 con 0x7fe2400cc4a0 2024-09-16T01:52:51.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.165+0000 7fe23e000640 1 -- 172.21.15.146:0/3632832866 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2380078e0 con 0x7fe2400cc4a0 2024-09-16T01:52:51.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.165+0000 7fe23e000640 1 -- 172.21.15.146:0/3632832866 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe238007cb0 con 0x7fe2400cc4a0 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.165+0000 7fe245c00640 1 -- 172.21.15.146:0/3632832866 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 msgr2=0x7fe240165430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.165+0000 7fe245c00640 1 --2- 172.21.15.146:0/3632832866 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe240165430 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe238009d90 tx=0x7fe23802a660 comp rx=0 tx=0).stop 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.166+0000 7fe245c00640 1 -- 172.21.15.146:0/3632832866 shutdown_connections 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.166+0000 7fe245c00640 1 --2- 172.21.15.146:0/3632832866 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe240165430 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.166+0000 7fe245c00640 1 --2- 172.21.15.146:0/3632832866 >> v2:172.21.15.161:3300/0 conn(0x7fe2400cb2f0 0x7fe24015aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.166+0000 7fe245c00640 1 -- 172.21.15.146:0/3632832866 >> 172.21.15.146:0/3632832866 conn(0x7fe24006d560 msgr2=0x7fe24006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.166+0000 7fe245c00640 1 -- 172.21.15.146:0/3632832866 shutdown_connections 2024-09-16T01:52:51.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.166+0000 7fe245c00640 1 -- 172.21.15.146:0/3632832866 wait complete. 2024-09-16T01:52:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.168+0000 7fe245c00640 1 Processor -- start 2024-09-16T01:52:51.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.168+0000 7fe245c00640 1 -- start start 2024-09-16T01:52:51.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.169+0000 7fe245c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2400cb2f0 0x7fe2400b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.169+0000 7fe245c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2400cc0c0 0x7fe24015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.169+0000 7fe245c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe24015d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.169+0000 7fe245c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe24015e1d0 con 0x7fe2400cb2f0 2024-09-16T01:52:51.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.169+0000 7fe245c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe24015e540 con 0x7fe2400cc0c0 2024-09-16T01:52:51.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.169+0000 7fe245c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe2400cc8f0 con 0x7fe2400cc4a0 2024-09-16T01:52:51.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe23f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe24015d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2400cc0c0 0x7fe24015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe23f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe24015d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38276/0 (socket says 172.21.15.146:38276) 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe23f400640 1 -- 172.21.15.146:0/3870516536 learned_addr learned my addr 172.21.15.146:0/3870516536 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe23ea00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2400cb2f0 0x7fe2400b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 -- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 msgr2=0x7fe24015d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe24015d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 -- 172.21.15.146:0/3870516536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2400cb2f0 msgr2=0x7fe2400b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2400cb2f0 0x7fe2400b5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 -- 172.21.15.146:0/3870516536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe240148310 con 0x7fe2400cc0c0 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe23f400640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe24015d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:51.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.170+0000 7fe237e00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2400cc0c0 0x7fe24015bc70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe22800d620 tx=0x7fe22800dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:51.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.171+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228004030 con 0x7fe2400cc0c0 2024-09-16T01:52:51.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.171+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2280041d0 con 0x7fe2400cc0c0 2024-09-16T01:52:51.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.171+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228014620 con 0x7fe2400cc0c0 2024-09-16T01:52:51.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.171+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe240147a70 con 0x7fe2400cc0c0 2024-09-16T01:52:51.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.171+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe240148030 con 0x7fe2400cc0c0 2024-09-16T01:52:51.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.172+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2400c7350 con 0x7fe2400cc0c0 2024-09-16T01:52:51.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.173+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe2280147c0 con 0x7fe2400cc0c0 2024-09-16T01:52:51.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.175+0000 7fe236a00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe21c06c300 0x7fe21c06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.175+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5969+0+0 (secure 0 0 0) 0x7fe22808bb70 con 0x7fe2400cc0c0 2024-09-16T01:52:51.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.175+0000 7fe23ea00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe21c06c300 0x7fe21c06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.176+0000 7fe23ea00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe21c06c300 0x7fe21c06e7c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe230007fd0 tx=0x7fe230007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:51.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.179+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe2400c7350 con 0x7fe2400cc0c0 2024-09-16T01:52:51.235 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:51.234+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:51.337 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:51.334+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:51.339 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:51.339 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:51.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:51.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.376+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe24014aed0 con 0x7fe21c06c300 2024-09-16T01:52:51.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.383+0000 7fe236a00640 1 -- 172.21.15.146:0/3870516536 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254659 (secure 0 0 0) 0x7fe24014aed0 con 0x7fe21c06c300 2024-09-16T01:52:51.388 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:51.389 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:51.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe21c06c300 msgr2=0x7fe21c06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe21c06c300 0x7fe21c06e7c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe230007fd0 tx=0x7fe230007e80 comp rx=0 tx=0).stop 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2400cc0c0 msgr2=0x7fe24015bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2400cc0c0 0x7fe24015bc70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe22800d620 tx=0x7fe22800dae0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 shutdown_connections 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe21c06c300 0x7fe21c06e7c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe2400cc4a0 0x7fe24015d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2400cc0c0 0x7fe24015bc70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 --2- 172.21.15.146:0/3870516536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2400cb2f0 0x7fe2400b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 >> 172.21.15.146:0/3870516536 conn(0x7fe24006d560 msgr2=0x7fe240153e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 shutdown_connections 2024-09-16T01:52:51.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.387+0000 7fe245c00640 1 -- 172.21.15.146:0/3870516536 wait complete. 2024-09-16T01:52:51.408 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T01:52:50.564185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":50,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":7150,"num_write_kb":24935336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2726,"num_bytes_recovered":8296580415,"num_keys_recovered":135,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7591,"ondisk_log_size":7591,"up":194,"acting":189,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":198,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1546832,"kb_used_data":1338904,"kb_used_omap":9,"kb_used_meta":207798,"kb_avail":564684208,"statfs":{"total":579820584960,"available":578236628992,"internally_reserved":0,"allocated":1371037696,"data_stored":21799293182,"data_compressed":1088107399,"data_compressed_allocated":1362235392,"data_compressed_original":21794983936,"omap_allocated":9562,"internal_metadata":212785830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":2,"num_objects_missing":-1,"num_objects_degraded":50,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":1003520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":880803986,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000702"},"pg_stats":[{"pgid":"3.3f","version":"112'104","reported_seq":378,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556452+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:46.556452+0000","last_peered":"2024-09-16T01:52:46.556452+0000","last_clean":"2024-09-16T01:52:46.556452+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:46.556452+0000","last_undegraded":"2024-09-16T01:52:46.556452+0000","last_fullsized":"2024-09-16T01:52:46.556452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532378+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:45.532378+0000","last_peered":"2024-09-16T01:52:45.532378+0000","last_clean":"2024-09-16T01:52:45.532378+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:45.532378+0000","last_undegraded":"2024-09-16T01:52:45.532378+0000","last_fullsized":"2024-09-16T01:52:45.532378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":339968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":446,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.091998+0000","last_change":"2024-09-16T01:52:45.534236+0000","last_active":"2024-09-16T01:52:45.534236+0000","last_peered":"2024-09-16T01:52:45.091998+0000","last_clean":"2024-09-16T01:52:45.091998+0000","last_became_active":"2024-09-16T01:52:03.309937+0000","last_became_peered":"2024-09-16T01:52:03.309937+0000","last_unstale":"2024-09-16T01:52:45.091998+0000","last_undegraded":"2024-09-16T01:52:45.091998+0000","last_fullsized":"2024-09-16T01:52:45.091998+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":209715200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"111'124","reported_seq":324,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.543316+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:46.543316+0000","last_peered":"2024-09-16T01:52:46.543316+0000","last_clean":"2024-09-16T01:52:46.543316+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:46.543316+0000","last_undegraded":"2024-09-16T01:52:46.543316+0000","last_fullsized":"2024-09-16T01:52:46.543316+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"110'109","reported_seq":273,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507567+0000","last_change":"2024-09-16T01:52:45.533438+0000","last_active":"2024-09-16T01:52:45.533438+0000","last_peered":"2024-09-16T01:52:44.507567+0000","last_clean":"2024-09-16T01:52:44.507567+0000","last_became_active":"2024-09-16T01:52:09.424479+0000","last_became_peered":"2024-09-16T01:52:09.424479+0000","last_unstale":"2024-09-16T01:52:44.507567+0000","last_undegraded":"2024-09-16T01:52:44.507567+0000","last_fullsized":"2024-09-16T01:52:44.507567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"112'102","reported_seq":349,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535873+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:46.535873+0000","last_peered":"2024-09-16T01:52:46.535873+0000","last_clean":"2024-09-16T01:52:46.535873+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:46.535873+0000","last_undegraded":"2024-09-16T01:52:46.535873+0000","last_fullsized":"2024-09-16T01:52:46.535873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"112'125","reported_seq":494,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.548461+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:46.548461+0000","last_peered":"2024-09-16T01:52:46.548461+0000","last_clean":"2024-09-16T01:52:46.548461+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:46.548461+0000","last_undegraded":"2024-09-16T01:52:46.548461+0000","last_fullsized":"2024-09-16T01:52:46.548461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"111'127","reported_seq":445,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532294+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:45.532294+0000","last_peered":"2024-09-16T01:52:45.532294+0000","last_clean":"2024-09-16T01:52:45.532294+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:45.532294+0000","last_undegraded":"2024-09-16T01:52:45.532294+0000","last_fullsized":"2024-09-16T01:52:45.532294+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":466944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"112'113","reported_seq":308,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.539660+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:46.539660+0000","last_peered":"2024-09-16T01:52:46.539660+0000","last_clean":"2024-09-16T01:52:46.539660+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:46.539660+0000","last_undegraded":"2024-09-16T01:52:46.539660+0000","last_fullsized":"2024-09-16T01:52:46.539660+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":380928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"112'117","reported_seq":416,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.543142+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:46.543142+0000","last_peered":"2024-09-16T01:52:46.543142+0000","last_clean":"2024-09-16T01:52:46.543142+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:46.543142+0000","last_undegraded":"2024-09-16T01:52:46.543142+0000","last_fullsized":"2024-09-16T01:52:46.543142+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"112'135","reported_seq":459,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.551141+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:46.551141+0000","last_peered":"2024-09-16T01:52:46.551141+0000","last_clean":"2024-09-16T01:52:46.551141+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:46.551141+0000","last_undegraded":"2024-09-16T01:52:46.551141+0000","last_fullsized":"2024-09-16T01:52:46.551141+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"110'105","reported_seq":119,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.565780+0000","last_change":"2024-09-16T01:52:46.540152+0000","last_active":"2024-09-16T01:52:45.565780+0000","last_peered":"2024-09-16T01:52:44.499435+0000","last_clean":"2024-09-16T01:52:44.499435+0000","last_became_active":"2024-09-16T01:52:40.724710+0000","last_became_peered":"2024-09-16T01:52:40.724710+0000","last_unstale":"2024-09-16T01:52:45.565780+0000","last_undegraded":"2024-09-16T01:52:45.565780+0000","last_fullsized":"2024-09-16T01:52:45.565780+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":478150656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"113'112","reported_seq":438,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493569+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:50.493569+0000","last_peered":"2024-09-16T01:52:50.493569+0000","last_clean":"2024-09-16T01:52:50.493569+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:50.493569+0000","last_undegraded":"2024-09-16T01:52:50.493569+0000","last_fullsized":"2024-09-16T01:52:50.493569+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"113'107","reported_seq":360,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492306+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:50.492306+0000","last_peered":"2024-09-16T01:52:50.492306+0000","last_clean":"2024-09-16T01:52:50.492306+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:50.492306+0000","last_undegraded":"2024-09-16T01:52:50.492306+0000","last_fullsized":"2024-09-16T01:52:50.492306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"111'108","reported_seq":404,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.531409+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:45.531409+0000","last_peered":"2024-09-16T01:52:45.531409+0000","last_clean":"2024-09-16T01:52:45.531409+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:45.531409+0000","last_undegraded":"2024-09-16T01:52:45.531409+0000","last_fullsized":"2024-09-16T01:52:45.531409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"111'107","reported_seq":301,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535166+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:46.535166+0000","last_peered":"2024-09-16T01:52:46.535166+0000","last_clean":"2024-09-16T01:52:46.535166+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:46.535166+0000","last_undegraded":"2024-09-16T01:52:46.535166+0000","last_fullsized":"2024-09-16T01:52:46.535166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"111'103","reported_seq":394,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556216+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:46.556216+0000","last_peered":"2024-09-16T01:52:46.556216+0000","last_clean":"2024-09-16T01:52:46.556216+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:46.556216+0000","last_undegraded":"2024-09-16T01:52:46.556216+0000","last_fullsized":"2024-09-16T01:52:46.556216+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"110'105","reported_seq":383,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.562116+0000","last_change":"2024-09-16T01:52:46.548473+0000","last_active":"2024-09-16T01:52:45.562116+0000","last_peered":"2024-09-16T01:52:44.522618+0000","last_clean":"2024-09-16T01:52:44.522618+0000","last_became_active":"2024-09-16T01:52:03.349420+0000","last_became_peered":"2024-09-16T01:52:03.349420+0000","last_unstale":"2024-09-16T01:52:45.562116+0000","last_undegraded":"2024-09-16T01:52:45.562116+0000","last_fullsized":"2024-09-16T01:52:45.562116+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"114'124","reported_seq":560,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492538+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:52:50.492538+0000","last_peered":"2024-09-16T01:52:50.492538+0000","last_clean":"2024-09-16T01:52:50.492538+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:52:50.492538+0000","last_undegraded":"2024-09-16T01:52:50.492538+0000","last_fullsized":"2024-09-16T01:52:50.492538+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"112'114","reported_seq":452,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.540506+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:46.540506+0000","last_peered":"2024-09-16T01:52:46.540506+0000","last_clean":"2024-09-16T01:52:46.540506+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:46.540506+0000","last_undegraded":"2024-09-16T01:52:46.540506+0000","last_fullsized":"2024-09-16T01:52:46.540506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"112'93","reported_seq":325,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.542708+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:46.542708+0000","last_peered":"2024-09-16T01:52:46.542708+0000","last_clean":"2024-09-16T01:52:46.542708+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:46.542708+0000","last_undegraded":"2024-09-16T01:52:46.542708+0000","last_fullsized":"2024-09-16T01:52:46.542708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"112'126","reported_seq":567,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.538283+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:52:46.538283+0000","last_peered":"2024-09-16T01:52:46.538283+0000","last_clean":"2024-09-16T01:52:46.538283+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:52:46.538283+0000","last_undegraded":"2024-09-16T01:52:46.538283+0000","last_fullsized":"2024-09-16T01:52:46.538283+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":286,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532334+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:45.532334+0000","last_peered":"2024-09-16T01:52:45.532334+0000","last_clean":"2024-09-16T01:52:45.532334+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:45.532334+0000","last_undegraded":"2024-09-16T01:52:45.532334+0000","last_fullsized":"2024-09-16T01:52:45.532334+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"111'132","reported_seq":326,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535263+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:46.535263+0000","last_peered":"2024-09-16T01:52:46.535263+0000","last_clean":"2024-09-16T01:52:46.535263+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:46.535263+0000","last_undegraded":"2024-09-16T01:52:46.535263+0000","last_fullsized":"2024-09-16T01:52:46.535263+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"111'100","reported_seq":293,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532061+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:45.532061+0000","last_peered":"2024-09-16T01:52:45.532061+0000","last_clean":"2024-09-16T01:52:45.532061+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:45.532061+0000","last_undegraded":"2024-09-16T01:52:45.532061+0000","last_fullsized":"2024-09-16T01:52:45.532061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"112'142","reported_seq":337,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.544916+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:46.544916+0000","last_peered":"2024-09-16T01:52:46.544916+0000","last_clean":"2024-09-16T01:52:46.544916+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:46.544916+0000","last_undegraded":"2024-09-16T01:52:46.544916+0000","last_fullsized":"2024-09-16T01:52:46.544916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"114'105","reported_seq":538,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493440+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:52:50.493440+0000","last_peered":"2024-09-16T01:52:50.493440+0000","last_clean":"2024-09-16T01:52:50.493440+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:52:50.493440+0000","last_undegraded":"2024-09-16T01:52:50.493440+0000","last_fullsized":"2024-09-16T01:52:50.493440+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"113'100","reported_seq":396,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492476+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:50.492476+0000","last_peered":"2024-09-16T01:52:50.492476+0000","last_clean":"2024-09-16T01:52:50.492476+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:50.492476+0000","last_undegraded":"2024-09-16T01:52:50.492476+0000","last_fullsized":"2024-09-16T01:52:50.492476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"109'87","reported_seq":481,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507610+0000","last_change":"2024-09-16T01:52:45.531675+0000","last_active":"2024-09-16T01:52:45.531675+0000","last_peered":"2024-09-16T01:52:44.507610+0000","last_clean":"2024-09-16T01:52:44.507610+0000","last_became_active":"2024-09-16T01:52:09.378210+0000","last_became_peered":"2024-09-16T01:52:09.378210+0000","last_unstale":"2024-09-16T01:52:44.507610+0000","last_undegraded":"2024-09-16T01:52:44.507610+0000","last_fullsized":"2024-09-16T01:52:44.507610+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":299008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"112'98","reported_seq":292,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.533925+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:46.533925+0000","last_peered":"2024-09-16T01:52:46.533925+0000","last_clean":"2024-09-16T01:52:46.533925+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:46.533925+0000","last_undegraded":"2024-09-16T01:52:46.533925+0000","last_fullsized":"2024-09-16T01:52:46.533925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"111'100","reported_seq":401,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.555742+0000","last_change":"2024-09-16T01:52:46.555911+0000","last_active":"2024-09-16T01:52:46.555742+0000","last_peered":"2024-09-16T01:52:46.555742+0000","last_clean":"2024-09-16T01:52:45.020933+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:52:46.555742+0000","last_undegraded":"2024-09-16T01:52:46.537788+0000","last_fullsized":"2024-09-16T01:52:46.555742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":33}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"113'112","reported_seq":428,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492019+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:50.492019+0000","last_peered":"2024-09-16T01:52:50.492019+0000","last_clean":"2024-09-16T01:52:50.492019+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:50.492019+0000","last_undegraded":"2024-09-16T01:52:50.492019+0000","last_fullsized":"2024-09-16T01:52:50.492019+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"112'98","reported_seq":293,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.538859+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:46.538859+0000","last_peered":"2024-09-16T01:52:46.538859+0000","last_clean":"2024-09-16T01:52:46.538859+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:46.538859+0000","last_undegraded":"2024-09-16T01:52:46.538859+0000","last_fullsized":"2024-09-16T01:52:46.538859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":200,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534083+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:45.534083+0000","last_peered":"2024-09-16T01:52:45.534083+0000","last_clean":"2024-09-16T01:52:45.534083+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:45.534083+0000","last_undegraded":"2024-09-16T01:52:45.534083+0000","last_fullsized":"2024-09-16T01:52:45.534083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"111'104","reported_seq":534,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.548924+0000","last_change":"2024-09-16T01:52:46.537357+0000","last_active":"2024-09-16T01:52:45.548924+0000","last_peered":"2024-09-16T01:52:44.543722+0000","last_clean":"2024-09-16T01:52:44.543722+0000","last_became_active":"2024-09-16T01:52:21.480658+0000","last_became_peered":"2024-09-16T01:52:21.480658+0000","last_unstale":"2024-09-16T01:52:45.548924+0000","last_undegraded":"2024-09-16T01:52:45.548924+0000","last_fullsized":"2024-09-16T01:52:45.548924+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":216,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535746+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:46.535746+0000","last_peered":"2024-09-16T01:52:46.535746+0000","last_clean":"2024-09-16T01:52:46.535746+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:46.535746+0000","last_undegraded":"2024-09-16T01:52:46.535746+0000","last_fullsized":"2024-09-16T01:52:46.535746+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":230,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492167+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:52:50.492167+0000","last_peered":"2024-09-16T01:52:50.492167+0000","last_clean":"2024-09-16T01:52:50.492167+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:52:50.492167+0000","last_undegraded":"2024-09-16T01:52:50.492167+0000","last_fullsized":"2024-09-16T01:52:50.492167+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"111'121","reported_seq":314,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532429+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:45.532429+0000","last_peered":"2024-09-16T01:52:45.532429+0000","last_clean":"2024-09-16T01:52:45.532429+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:45.532429+0000","last_undegraded":"2024-09-16T01:52:45.532429+0000","last_fullsized":"2024-09-16T01:52:45.532429+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":438272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":199,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535364+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:46.535364+0000","last_peered":"2024-09-16T01:52:46.535364+0000","last_clean":"2024-09-16T01:52:46.535364+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:46.535364+0000","last_undegraded":"2024-09-16T01:52:46.535364+0000","last_fullsized":"2024-09-16T01:52:46.535364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"112'114","reported_seq":274,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.538072+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:46.538072+0000","last_peered":"2024-09-16T01:52:46.538072+0000","last_clean":"2024-09-16T01:52:46.538072+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:46.538072+0000","last_undegraded":"2024-09-16T01:52:46.538072+0000","last_fullsized":"2024-09-16T01:52:46.538072+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":228,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.539769+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:46.539769+0000","last_peered":"2024-09-16T01:52:46.539769+0000","last_clean":"2024-09-16T01:52:46.539769+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:46.539769+0000","last_undegraded":"2024-09-16T01:52:46.539769+0000","last_fullsized":"2024-09-16T01:52:46.539769+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"112'163","reported_seq":367,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.567223+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:45.567223+0000","last_peered":"2024-09-16T01:52:45.567223+0000","last_clean":"2024-09-16T01:52:45.567223+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:45.567223+0000","last_undegraded":"2024-09-16T01:52:45.567223+0000","last_fullsized":"2024-09-16T01:52:45.567223+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":581632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":82,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.557984+0000","last_change":"2024-09-16T01:52:46.558524+0000","last_active":"2024-09-16T01:52:46.557984+0000","last_peered":"2024-09-16T01:52:46.557984+0000","last_clean":"2024-09-16T01:52:44.504783+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:52:46.557984+0000","last_undegraded":"2024-09-16T01:52:46.539531+0000","last_fullsized":"2024-09-16T01:52:46.557984+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"110'98","reported_seq":291,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532194+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:45.532194+0000","last_peered":"2024-09-16T01:52:45.532194+0000","last_clean":"2024-09-16T01:52:45.532194+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:45.532194+0000","last_undegraded":"2024-09-16T01:52:45.532194+0000","last_fullsized":"2024-09-16T01:52:45.532194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":348160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":96,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.562355+0000","last_change":"2024-09-16T01:52:46.562522+0000","last_active":"2024-09-16T01:52:46.562355+0000","last_peered":"2024-09-16T01:52:46.562355+0000","last_clean":"2024-09-16T01:52:44.507789+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:52:46.562355+0000","last_undegraded":"2024-09-16T01:52:46.536286+0000","last_fullsized":"2024-09-16T01:52:46.562355+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3360,"num_keys_recovered":50,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"112'107","reported_seq":301,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537869+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:46.537869+0000","last_peered":"2024-09-16T01:52:46.537869+0000","last_clean":"2024-09-16T01:52:46.537869+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:46.537869+0000","last_undegraded":"2024-09-16T01:52:46.537869+0000","last_fullsized":"2024-09-16T01:52:46.537869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":386,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.543425+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:46.543425+0000","last_peered":"2024-09-16T01:52:46.543425+0000","last_clean":"2024-09-16T01:52:46.543425+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:46.543425+0000","last_undegraded":"2024-09-16T01:52:46.543425+0000","last_fullsized":"2024-09-16T01:52:46.543425+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":417,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493346+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:50.493346+0000","last_peered":"2024-09-16T01:52:50.493346+0000","last_clean":"2024-09-16T01:52:50.493346+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:50.493346+0000","last_undegraded":"2024-09-16T01:52:50.493346+0000","last_fullsized":"2024-09-16T01:52:50.493346+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":113,"state":"active+recovering","last_fresh":"2024-09-16T01:52:46.607325+0000","last_change":"2024-09-16T01:52:46.605694+0000","last_active":"2024-09-16T01:52:46.607325+0000","last_peered":"2024-09-16T01:52:46.607325+0000","last_clean":"2024-09-16T01:52:44.506267+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:52:46.607325+0000","last_undegraded":"2024-09-16T01:52:46.607325+0000","last_fullsized":"2024-09-16T01:52:46.607325+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":306,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4},{"shards":"3,7","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"112'117","reported_seq":320,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492401+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:50.492401+0000","last_peered":"2024-09-16T01:52:50.492401+0000","last_clean":"2024-09-16T01:52:50.492401+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:50.492401+0000","last_undegraded":"2024-09-16T01:52:50.492401+0000","last_fullsized":"2024-09-16T01:52:50.492401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":269,"reported_epoch":113,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T01:52:46.562088+0000","last_change":"2024-09-16T01:52:46.562285+0000","last_active":"2024-09-16T01:52:46.562088+0000","last_peered":"2024-09-16T01:52:46.562088+0000","last_clean":"2024-09-16T01:52:44.506397+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:52:46.562088+0000","last_undegraded":"2024-09-16T01:52:46.536120+0000","last_fullsized":"2024-09-16T01:52:46.562088+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"112'121","reported_seq":446,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537496+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:46.537496+0000","last_peered":"2024-09-16T01:52:46.537496+0000","last_clean":"2024-09-16T01:52:46.537496+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:46.537496+0000","last_undegraded":"2024-09-16T01:52:46.537496+0000","last_fullsized":"2024-09-16T01:52:46.537496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":73,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.534295+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:46.534295+0000","last_peered":"2024-09-16T01:52:46.534295+0000","last_clean":"2024-09-16T01:52:46.534295+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:46.534295+0000","last_undegraded":"2024-09-16T01:52:46.534295+0000","last_fullsized":"2024-09-16T01:52:46.534295+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"112'132","reported_seq":327,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537538+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:46.537538+0000","last_peered":"2024-09-16T01:52:46.537538+0000","last_clean":"2024-09-16T01:52:46.537538+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:46.537538+0000","last_undegraded":"2024-09-16T01:52:46.537538+0000","last_fullsized":"2024-09-16T01:52:46.537538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":125,"reported_epoch":113,"state":"active+recovering","last_fresh":"2024-09-16T01:52:46.594029+0000","last_change":"2024-09-16T01:52:46.585166+0000","last_active":"2024-09-16T01:52:46.594029+0000","last_peered":"2024-09-16T01:52:46.594029+0000","last_clean":"2024-09-16T01:52:44.496597+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:52:46.594029+0000","last_undegraded":"2024-09-16T01:52:46.594029+0000","last_fullsized":"2024-09-16T01:52:46.594029+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2},{"shards":"1,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"112'144","reported_seq":338,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.534391+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:46.534391+0000","last_peered":"2024-09-16T01:52:46.534391+0000","last_clean":"2024-09-16T01:52:46.534391+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:46.534391+0000","last_undegraded":"2024-09-16T01:52:46.534391+0000","last_fullsized":"2024-09-16T01:52:46.534391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":495616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":68,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532060+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:45.532060+0000","last_peered":"2024-09-16T01:52:45.532060+0000","last_clean":"2024-09-16T01:52:45.532060+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:45.532060+0000","last_undegraded":"2024-09-16T01:52:45.532060+0000","last_fullsized":"2024-09-16T01:52:45.532060+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"112'106","reported_seq":402,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.546659+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:46.546659+0000","last_peered":"2024-09-16T01:52:46.546659+0000","last_clean":"2024-09-16T01:52:46.546659+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:46.546659+0000","last_undegraded":"2024-09-16T01:52:46.546659+0000","last_fullsized":"2024-09-16T01:52:46.546659+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532518+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:45.532518+0000","last_peered":"2024-09-16T01:52:45.532518+0000","last_clean":"2024-09-16T01:52:45.532518+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:45.532518+0000","last_undegraded":"2024-09-16T01:52:45.532518+0000","last_fullsized":"2024-09-16T01:52:45.532518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"112'103","reported_seq":394,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.539408+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:46.539408+0000","last_peered":"2024-09-16T01:52:46.539408+0000","last_clean":"2024-09-16T01:52:46.539408+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:46.539408+0000","last_undegraded":"2024-09-16T01:52:46.539408+0000","last_fullsized":"2024-09-16T01:52:46.539408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":198,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.534027+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:45.534027+0000","last_peered":"2024-09-16T01:52:45.534027+0000","last_clean":"2024-09-16T01:52:45.534027+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:45.534027+0000","last_undegraded":"2024-09-16T01:52:45.534027+0000","last_fullsized":"2024-09-16T01:52:45.534027+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"111'101","reported_seq":520,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.538823+0000","last_change":"2024-09-16T01:52:46.535097+0000","last_active":"2024-09-16T01:52:45.538823+0000","last_peered":"2024-09-16T01:52:44.567355+0000","last_clean":"2024-09-16T01:52:44.567355+0000","last_became_active":"2024-09-16T01:52:09.413639+0000","last_became_peered":"2024-09-16T01:52:09.413639+0000","last_unstale":"2024-09-16T01:52:45.538823+0000","last_undegraded":"2024-09-16T01:52:45.538823+0000","last_fullsized":"2024-09-16T01:52:45.538823+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":318767104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":223,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492672+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:50.492672+0000","last_peered":"2024-09-16T01:52:50.492672+0000","last_clean":"2024-09-16T01:52:50.492672+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:50.492672+0000","last_undegraded":"2024-09-16T01:52:50.492672+0000","last_fullsized":"2024-09-16T01:52:50.492672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"112'112","reported_seq":306,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.552650+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:46.552650+0000","last_peered":"2024-09-16T01:52:46.552650+0000","last_clean":"2024-09-16T01:52:46.552650+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:46.552650+0000","last_undegraded":"2024-09-16T01:52:46.552650+0000","last_fullsized":"2024-09-16T01:52:46.552650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":353,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537651+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:46.537651+0000","last_peered":"2024-09-16T01:52:46.537651+0000","last_clean":"2024-09-16T01:52:46.537651+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:46.537651+0000","last_undegraded":"2024-09-16T01:52:46.537651+0000","last_fullsized":"2024-09-16T01:52:46.537651+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":202,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537695+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:46.537695+0000","last_peered":"2024-09-16T01:52:46.537695+0000","last_clean":"2024-09-16T01:52:46.537695+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:46.537695+0000","last_undegraded":"2024-09-16T01:52:46.537695+0000","last_fullsized":"2024-09-16T01:52:46.537695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"112'111","reported_seq":446,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535786+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:46.535786+0000","last_peered":"2024-09-16T01:52:46.535786+0000","last_clean":"2024-09-16T01:52:46.535786+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:46.535786+0000","last_undegraded":"2024-09-16T01:52:46.535786+0000","last_fullsized":"2024-09-16T01:52:46.535786+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":200,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535459+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:46.535459+0000","last_peered":"2024-09-16T01:52:46.535459+0000","last_clean":"2024-09-16T01:52:46.535459+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:46.535459+0000","last_undegraded":"2024-09-16T01:52:46.535459+0000","last_fullsized":"2024-09-16T01:52:46.535459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"113'107","reported_seq":310,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492106+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:50.492106+0000","last_peered":"2024-09-16T01:52:50.492106+0000","last_clean":"2024-09-16T01:52:50.492106+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:50.492106+0000","last_undegraded":"2024-09-16T01:52:50.492106+0000","last_fullsized":"2024-09-16T01:52:50.492106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":240,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.507474+0000","last_change":"2024-09-16T01:52:45.531950+0000","last_active":"2024-09-16T01:52:45.531950+0000","last_peered":"2024-09-16T01:52:44.507474+0000","last_clean":"2024-09-16T01:52:44.507474+0000","last_became_active":"2024-09-16T01:52:21.493366+0000","last_became_peered":"2024-09-16T01:52:21.493366+0000","last_unstale":"2024-09-16T01:52:44.507474+0000","last_undegraded":"2024-09-16T01:52:44.507474+0000","last_fullsized":"2024-09-16T01:52:44.507474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"111'93","reported_seq":410,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:45.298724+0000","last_change":"2024-09-16T01:52:45.533111+0000","last_active":"2024-09-16T01:52:45.533111+0000","last_peered":"2024-09-16T01:52:45.298724+0000","last_clean":"2024-09-16T01:52:45.298724+0000","last_became_active":"2024-09-16T01:52:03.344719+0000","last_became_peered":"2024-09-16T01:52:03.344719+0000","last_unstale":"2024-09-16T01:52:45.298724+0000","last_undegraded":"2024-09-16T01:52:45.298724+0000","last_fullsized":"2024-09-16T01:52:45.298724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":150994944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":201,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.536379+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:46.536379+0000","last_peered":"2024-09-16T01:52:46.536379+0000","last_clean":"2024-09-16T01:52:46.536379+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:46.536379+0000","last_undegraded":"2024-09-16T01:52:46.536379+0000","last_fullsized":"2024-09-16T01:52:46.536379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"111'110","reported_seq":404,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.531301+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:46.531301+0000","last_peered":"2024-09-16T01:52:46.531301+0000","last_clean":"2024-09-16T01:52:46.531301+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:46.531301+0000","last_undegraded":"2024-09-16T01:52:46.531301+0000","last_fullsized":"2024-09-16T01:52:46.531301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":196,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.533427+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:45.533427+0000","last_peered":"2024-09-16T01:52:45.533427+0000","last_clean":"2024-09-16T01:52:45.533427+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:45.533427+0000","last_undegraded":"2024-09-16T01:52:45.533427+0000","last_fullsized":"2024-09-16T01:52:45.533427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"113'110","reported_seq":305,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.547100+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:46.547100+0000","last_peered":"2024-09-16T01:52:46.547100+0000","last_clean":"2024-09-16T01:52:46.547100+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:46.547100+0000","last_undegraded":"2024-09-16T01:52:46.547100+0000","last_fullsized":"2024-09-16T01:52:46.547100+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":28,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.546395+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:52:46.546395+0000","last_peered":"2024-09-16T01:52:46.546395+0000","last_clean":"2024-09-16T01:52:46.546395+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:52:46.546395+0000","last_undegraded":"2024-09-16T01:52:46.546395+0000","last_fullsized":"2024-09-16T01:52:46.546395+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"113'103","reported_seq":306,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492575+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:50.492575+0000","last_peered":"2024-09-16T01:52:50.492575+0000","last_clean":"2024-09-16T01:52:50.492575+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:50.492575+0000","last_undegraded":"2024-09-16T01:52:50.492575+0000","last_fullsized":"2024-09-16T01:52:50.492575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532417+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:45.532417+0000","last_peered":"2024-09-16T01:52:45.532417+0000","last_clean":"2024-09-16T01:52:45.532417+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:45.532417+0000","last_undegraded":"2024-09-16T01:52:45.532417+0000","last_fullsized":"2024-09-16T01:52:45.532417+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"113'114","reported_seq":362,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493268+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:50.493268+0000","last_peered":"2024-09-16T01:52:50.493268+0000","last_clean":"2024-09-16T01:52:50.493268+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:50.493268+0000","last_undegraded":"2024-09-16T01:52:50.493268+0000","last_fullsized":"2024-09-16T01:52:50.493268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":218,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556313+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:46.556313+0000","last_peered":"2024-09-16T01:52:46.556313+0000","last_clean":"2024-09-16T01:52:46.556313+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:46.556313+0000","last_undegraded":"2024-09-16T01:52:46.556313+0000","last_fullsized":"2024-09-16T01:52:46.556313+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"111'115","reported_seq":285,"reported_epoch":112,"state":"remapped+peering","last_fresh":"2024-09-16T01:52:45.557058+0000","last_change":"2024-09-16T01:52:46.540709+0000","last_active":"2024-09-16T01:52:45.557058+0000","last_peered":"2024-09-16T01:52:44.870429+0000","last_clean":"2024-09-16T01:52:44.870429+0000","last_became_active":"2024-09-16T01:52:03.364819+0000","last_became_peered":"2024-09-16T01:52:03.364819+0000","last_unstale":"2024-09-16T01:52:45.557058+0000","last_undegraded":"2024-09-16T01:52:45.557058+0000","last_fullsized":"2024-09-16T01:52:45.557058+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":255852544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":82,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.618081+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:52:46.618081+0000","last_peered":"2024-09-16T01:52:46.618081+0000","last_clean":"2024-09-16T01:52:46.618081+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:52:46.618081+0000","last_undegraded":"2024-09-16T01:52:46.618081+0000","last_fullsized":"2024-09-16T01:52:46.618081+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"112'114","reported_seq":363,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.535865+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:46.535865+0000","last_peered":"2024-09-16T01:52:46.535865+0000","last_clean":"2024-09-16T01:52:46.535865+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:46.535865+0000","last_undegraded":"2024-09-16T01:52:46.535865+0000","last_fullsized":"2024-09-16T01:52:46.535865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492054+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:50.492054+0000","last_peered":"2024-09-16T01:52:50.492054+0000","last_clean":"2024-09-16T01:52:50.492054+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:50.492054+0000","last_undegraded":"2024-09-16T01:52:50.492054+0000","last_fullsized":"2024-09-16T01:52:50.492054+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"113'115","reported_seq":349,"reported_epoch":113,"state":"active+recovering","last_fresh":"2024-09-16T01:52:46.632747+0000","last_change":"2024-09-16T01:52:46.654054+0000","last_active":"2024-09-16T01:52:46.632747+0000","last_peered":"2024-09-16T01:52:46.632747+0000","last_clean":"2024-09-16T01:52:39.701448+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:52:46.632747+0000","last_undegraded":"2024-09-16T01:52:46.532102+0000","last_fullsized":"2024-09-16T01:52:46.632747+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":503316480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,2,7","objects":1},{"shards":"1,7","objects":6},{"shards":"2,7","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":208,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492505+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:50.492505+0000","last_peered":"2024-09-16T01:52:50.492505+0000","last_clean":"2024-09-16T01:52:50.492505+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:50.492505+0000","last_undegraded":"2024-09-16T01:52:50.492505+0000","last_fullsized":"2024-09-16T01:52:50.492505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"114'105","reported_seq":512,"reported_epoch":116,"state":"peering","last_fresh":"2024-09-16T01:52:50.241765+0000","last_change":"2024-09-16T01:52:50.492781+0000","last_active":"2024-09-16T01:52:50.492781+0000","last_peered":"2024-09-16T01:52:50.241765+0000","last_clean":"2024-09-16T01:52:44.861424+0000","last_became_active":"2024-09-16T01:52:47.546951+0000","last_became_peered":"2024-09-16T01:52:47.546951+0000","last_unstale":"2024-09-16T01:52:50.241765+0000","last_undegraded":"2024-09-16T01:52:50.241765+0000","last_fullsized":"2024-09-16T01:52:47.532896+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":201,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.537603+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:46.537603+0000","last_peered":"2024-09-16T01:52:46.537603+0000","last_clean":"2024-09-16T01:52:46.537603+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:46.537603+0000","last_undegraded":"2024-09-16T01:52:46.537603+0000","last_fullsized":"2024-09-16T01:52:46.537603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"111'94","reported_seq":288,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.534025+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:46.534025+0000","last_peered":"2024-09-16T01:52:46.534025+0000","last_clean":"2024-09-16T01:52:46.534025+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:46.534025+0000","last_undegraded":"2024-09-16T01:52:46.534025+0000","last_fullsized":"2024-09-16T01:52:46.534025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"112'131","reported_seq":444,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.551191+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:46.551191+0000","last_peered":"2024-09-16T01:52:46.551191+0000","last_clean":"2024-09-16T01:52:46.551191+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:46.551191+0000","last_undegraded":"2024-09-16T01:52:46.551191+0000","last_fullsized":"2024-09-16T01:52:46.551191+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":70,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.540630+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:46.540630+0000","last_peered":"2024-09-16T01:52:46.540630+0000","last_clean":"2024-09-16T01:52:46.540630+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:46.540630+0000","last_undegraded":"2024-09-16T01:52:46.540630+0000","last_fullsized":"2024-09-16T01:52:46.540630+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"111'106","reported_seq":396,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.540728+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:46.540728+0000","last_peered":"2024-09-16T01:52:46.540728+0000","last_clean":"2024-09-16T01:52:46.540728+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:46.540728+0000","last_undegraded":"2024-09-16T01:52:46.540728+0000","last_fullsized":"2024-09-16T01:52:46.540728+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":38,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.541492+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:46.541492+0000","last_peered":"2024-09-16T01:52:46.541492+0000","last_clean":"2024-09-16T01:52:46.541492+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:46.541492+0000","last_undegraded":"2024-09-16T01:52:46.541492+0000","last_fullsized":"2024-09-16T01:52:46.541492+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"111'91","reported_seq":284,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532243+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:45.532243+0000","last_peered":"2024-09-16T01:52:45.532243+0000","last_clean":"2024-09-16T01:52:45.532243+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:45.532243+0000","last_undegraded":"2024-09-16T01:52:45.532243+0000","last_fullsized":"2024-09-16T01:52:45.532243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":221,"reported_epoch":111,"state":"peering","last_fresh":"2024-09-16T01:52:44.508113+0000","last_change":"2024-09-16T01:52:45.532987+0000","last_active":"2024-09-16T01:52:45.532987+0000","last_peered":"2024-09-16T01:52:44.508113+0000","last_clean":"2024-09-16T01:52:44.508113+0000","last_became_active":"2024-09-16T01:52:21.473912+0000","last_became_peered":"2024-09-16T01:52:21.473912+0000","last_unstale":"2024-09-16T01:52:44.508113+0000","last_undegraded":"2024-09-16T01:52:44.508113+0000","last_fullsized":"2024-09-16T01:52:44.508113+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"111'112","reported_seq":305,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T01:52:45.532474+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:45.532474+0000","last_peered":"2024-09-16T01:52:45.532474+0000","last_clean":"2024-09-16T01:52:45.532474+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:45.532474+0000","last_undegraded":"2024-09-16T01:52:45.532474+0000","last_fullsized":"2024-09-16T01:52:45.532474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":389120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T01:52:46.556039+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:46.556039+0000","last_peered":"2024-09-16T01:52:46.556039+0000","last_clean":"2024-09-16T01:52:46.556039+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:46.556039+0000","last_undegraded":"2024-09-16T01:52:46.556039+0000","last_fullsized":"2024-09-16T01:52:46.556039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":6,"num_objects_missing":6,"num_objects_degraded":41,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":24932352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2527,"num_bytes_recovered":8296333312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418461184,"data_stored":22695378944,"data_compressed":1130956288,"data_compressed_allocated":1418461184,"data_compressed_original":22695378944,"omap_allocated":0,"internal_metadata":0},"log_size":7114,"ondisk_log_size":7114,"up":128,"acting":123,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":247103,"num_keys_recovered":135,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":996044,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509492,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357424,"kb_used_data":326228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94014416,"statfs":{"total":96636764160,"available":96270761984,"internally_reserved":0,"allocated":334057472,"data_stored":5323198058,"data_compressed":265608704,"data_compressed_allocated":332660736,"data_compressed_original":5322571776,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509492,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349048,"kb_used_data":317864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94022792,"statfs":{"total":96636764160,"available":96279339008,"internally_reserved":0,"allocated":325492736,"data_stored":5180965092,"data_compressed":258638848,"data_compressed_allocated":323747840,"data_compressed_original":5179965440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509491,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249288,"kb_used_data":222264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94122552,"statfs":{"total":96636764160,"available":96381493248,"internally_reserved":0,"allocated":227598336,"data_stored":3617079895,"data_compressed":180653539,"data_compressed_allocated":226058240,"data_compressed_original":3616296960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542195,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69908,"kb_used_data":34564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94301932,"statfs":{"total":96636764160,"available":96565178368,"internally_reserved":0,"allocated":35393536,"data_stored":545874692,"data_compressed":27121536,"data_compressed_allocated":34078720,"data_compressed_original":545259520,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542196,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255748,"kb_used_data":212076,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94116092,"statfs":{"total":96636764160,"available":96374878208,"internally_reserved":0,"allocated":217165824,"data_stored":3452562337,"data_compressed":172647168,"data_compressed_allocated":215744512,"data_compressed_original":3451912192,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265416,"kb_used_data":225908,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94106424,"statfs":{"total":96636764160,"available":96364978176,"internally_reserved":0,"allocated":231329792,"data_stored":3679613108,"data_compressed":183437604,"data_compressed_allocated":229945344,"data_compressed_original":3678978048,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574900,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":377702,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":377785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":4505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":225968128,"data_stored":3615490048,"data_compressed":180631040,"data_compressed_allocated":225968128,"data_compressed_original":3615490048,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323747840,"data_stored":5179965440,"data_compressed":258720384,"data_compressed_allocated":323747840,"data_compressed_original":5179965440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332660736,"data_stored":5322571776,"data_compressed":265899904,"data_compressed_allocated":332660736,"data_compressed_original":5322571776,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":3678404608,"data_compressed":183463424,"data_compressed_allocated":229900288,"data_compressed_original":3678404608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":205520896,"data_stored":3288334336,"data_compressed":164476928,"data_compressed_allocated":205520896,"data_compressed_original":3288334336,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41156608,"data_stored":658505728,"data_compressed":32785920,"data_compressed_allocated":41156608,"data_compressed_original":658505728,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:51.450 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:51.448+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:51.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:52:51.538 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:51.537+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:51.596 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:51.642 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:51.640+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:51.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.698+0000 7f865de00640 1 Processor -- start 2024-09-16T01:52:51.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.698+0000 7f865de00640 1 -- start start 2024-09-16T01:52:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.699+0000 7f865de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f86580d0bf0 0x7f86580c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f865de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f86580ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f865de00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8658155430 con 0x7f865815edc0 2024-09-16T01:52:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f865de00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f865815aea0 con 0x7f86580d0bf0 2024-09-16T01:52:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f865de00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f865815b180 con 0x7f86580d0fd0 2024-09-16T01:52:51.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f86580d0bf0 0x7f86580c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f86580d0bf0 0x7f86580c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45850/0 (socket says 172.21.15.146:45850) 2024-09-16T01:52:51.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 -- 172.21.15.146:0/4013423490 learned_addr learned my addr 172.21.15.146:0/4013423490 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:51.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f864e000640 1 --2- 172.21.15.146:0/4013423490 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f86580ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 -- 172.21.15.146:0/4013423490 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 msgr2=0x7f86580ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656000640 1 -- 172.21.15.146:0/4013423490 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286486627 0 0) 0x7f8658155430 con 0x7f865815edc0 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 --2- 172.21.15.146:0/4013423490 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f86580ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 -- 172.21.15.146:0/4013423490 >> v1:172.21.15.161:6789/0 conn(0x7f865815edc0 legacy=0x7f86580b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.700+0000 7f8656a00640 1 -- 172.21.15.146:0/4013423490 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865815b460 con 0x7f86580d0bf0 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.701+0000 7f8656a00640 1 --2- 172.21.15.146:0/4013423490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f86580c9fe0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8648009870 tx=0x7f864802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654cbb9d8c31842e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.702+0000 7f8656000640 1 -- 172.21.15.146:0/4013423490 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648037070 con 0x7f86580d0bf0 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.702+0000 7f8656000640 1 -- 172.21.15.146:0/4013423490 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864802ae50 con 0x7f86580d0bf0 2024-09-16T01:52:51.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.702+0000 7f8656000640 1 -- 172.21.15.146:0/4013423490 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648032860 con 0x7f86580d0bf0 2024-09-16T01:52:51.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.702+0000 7f865de00640 1 -- 172.21.15.146:0/4013423490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 msgr2=0x7f86580c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.702+0000 7f865de00640 1 --2- 172.21.15.146:0/4013423490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f86580c9fe0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8648009870 tx=0x7f864802a300 comp rx=0 tx=0).stop 2024-09-16T01:52:51.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.703+0000 7f865de00640 1 -- 172.21.15.146:0/4013423490 shutdown_connections 2024-09-16T01:52:51.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.703+0000 7f865de00640 1 --2- 172.21.15.146:0/4013423490 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f86580ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.703+0000 7f865de00640 1 --2- 172.21.15.146:0/4013423490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f86580c9fe0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.703+0000 7f865de00640 1 -- 172.21.15.146:0/4013423490 >> 172.21.15.146:0/4013423490 conn(0x7f865806c900 msgr2=0x7f865806cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:51.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.703+0000 7f865de00640 1 -- 172.21.15.146:0/4013423490 shutdown_connections 2024-09-16T01:52:51.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.703+0000 7f865de00640 1 -- 172.21.15.146:0/4013423490 wait complete. 2024-09-16T01:52:51.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.704+0000 7f865de00640 1 Processor -- start 2024-09-16T01:52:51.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 -- start start 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f8658151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f8658158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f865815edc0 0x7f8658159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f86580b3790 con 0x7f86580d0bf0 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f865815ba20 con 0x7f865815edc0 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.705+0000 7f865de00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f86580b3910 con 0x7f86580d0fd0 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f8658158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f8658158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38292/0 (socket says 172.21.15.146:38292) 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f8657400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f865815edc0 0x7f8658159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 -- 172.21.15.146:0/2417140933 learned_addr learned my addr 172.21.15.146:0/2417140933 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:51.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f8656a00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f8658151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 -- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f865815edc0 msgr2=0x7f8658159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f865815edc0 0x7f8658159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 -- 172.21.15.146:0/2417140933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 msgr2=0x7f8658151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f8658151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 -- 172.21.15.146:0/2417140933 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865815df30 con 0x7f86580d0fd0 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f8657400640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f865815edc0 0x7f8658159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.706+0000 7f864e000640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f8658158c30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f864000e870 tx=0x7f864000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.707+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640017070 con 0x7f86580d0fd0 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.707+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864000c8e0 con 0x7f86580d0fd0 2024-09-16T01:52:51.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.707+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864000ccd0 con 0x7f86580d0fd0 2024-09-16T01:52:51.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.707+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865815c360 con 0x7f86580d0fd0 2024-09-16T01:52:51.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.707+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f865815c920 con 0x7f86580d0fd0 2024-09-16T01:52:51.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.708+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86580b3080 con 0x7f86580d0fd0 2024-09-16T01:52:51.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.709+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f864001a070 con 0x7f86580d0fd0 2024-09-16T01:52:51.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.710+0000 7f864fe00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f863406c2b0 0x7f863406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:51.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.710+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(118..118 src has 1..118) ==== 5919+0+0 (secure 0 0 0) 0x7f864008b760 con 0x7f86580d0fd0 2024-09-16T01:52:51.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.710+0000 7f8656a00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f863406c2b0 0x7f863406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.711+0000 7f8656a00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f863406c2b0 0x7f863406e770 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8648034000 tx=0x7f864802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.714+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8640059710 con 0x7f86580d0fd0 2024-09-16T01:52:51.741 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:51.739+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:51.773 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:51.773 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:51.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:51.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:51.877 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:51.875+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:51.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.916+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f865806cef0 con 0x7f86580d0fd0 2024-09-16T01:52:51.942 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:51.941+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:51.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.995+0000 7f549c600640 1 Processor -- start 2024-09-16T01:52:51.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.995+0000 7f549c600640 1 -- start start 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.997+0000 7f549c600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5494151820 0x7f54940c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.997+0000 7f549c600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494071a10 0x7f54940c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.997+0000 7f549c600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54941585b0 con 0x7f54940c7980 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.997+0000 7f549c600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f54941580a0 con 0x7f5494151820 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.997+0000 7f549c600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f54940c6680 con 0x7f5494071a10 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f5499200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f54940c7980 0x7f54940c8070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33612/0 (socket says 172.21.15.146:33612) 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f5499200640 1 -- 172.21.15.146:0/2097260118 learned_addr learned my addr 172.21.15.146:0/2097260118 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f549a600640 1 --2- 172.21.15.146:0/2097260118 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494071a10 0x7f54940c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f5499c00640 1 --2- 172.21.15.146:0/2097260118 >> v2:172.21.15.146:3300/0 conn(0x7f5494151820 0x7f54940c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f5493e00640 1 -- 172.21.15.146:0/2097260118 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2089151263 0 0) 0x7f54941585b0 con 0x7f54940c7980 2024-09-16T01:52:52.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f549a600640 1 -- 172.21.15.146:0/2097260118 >> v2:172.21.15.146:3300/0 conn(0x7f5494151820 msgr2=0x7f54940c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:52.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f549a600640 1 --2- 172.21.15.146:0/2097260118 >> v2:172.21.15.146:3300/0 conn(0x7f5494151820 0x7f54940c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f549a600640 1 -- 172.21.15.146:0/2097260118 >> v1:172.21.15.146:6789/0 conn(0x7f54940c7980 legacy=0x7f54940c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f549a600640 1 -- 172.21.15.146:0/2097260118 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54941582d0 con 0x7f5494071a10 2024-09-16T01:52:52.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:51.998+0000 7f5499c00640 1 --2- 172.21.15.146:0/2097260118 >> v2:172.21.15.146:3300/0 conn(0x7f5494151820 0x7f54940c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:52.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.000+0000 7f549a600640 1 --2- 172.21.15.146:0/2097260118 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494071a10 0x7f54940c5180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f548c009d90 tx=0x7f548c02a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f2212b02f6a7c94 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.000+0000 7f5493e00640 1 -- 172.21.15.146:0/2097260118 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c007740 con 0x7f5494071a10 2024-09-16T01:52:52.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f5493e00640 1 -- 172.21.15.146:0/2097260118 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f548c0078e0 con 0x7f5494071a10 2024-09-16T01:52:52.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f5493e00640 1 -- 172.21.15.146:0/2097260118 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c007cb0 con 0x7f5494071a10 2024-09-16T01:52:52.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f549c600640 1 -- 172.21.15.146:0/2097260118 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494071a10 msgr2=0x7f54940c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f549c600640 1 --2- 172.21.15.146:0/2097260118 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494071a10 0x7f54940c5180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f548c009d90 tx=0x7f548c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f549c600640 1 -- 172.21.15.146:0/2097260118 shutdown_connections 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f549c600640 1 --2- 172.21.15.146:0/2097260118 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494071a10 0x7f54940c5180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f549c600640 1 --2- 172.21.15.146:0/2097260118 >> v2:172.21.15.146:3300/0 conn(0x7f5494151820 0x7f54940c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.001+0000 7f549c600640 1 -- 172.21.15.146:0/2097260118 >> 172.21.15.146:0/2097260118 conn(0x7f549406bc10 msgr2=0x7f549406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.002+0000 7f549c600640 1 -- 172.21.15.146:0/2097260118 shutdown_connections 2024-09-16T01:52:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.002+0000 7f549c600640 1 -- 172.21.15.146:0/2097260118 wait complete. 2024-09-16T01:52:52.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.003+0000 7f549c600640 1 Processor -- start 2024-09-16T01:52:52.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.003+0000 7f549c600640 1 -- start start 2024-09-16T01:52:52.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.004+0000 7f549c600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5494071a10 0x7f54940beed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.004+0000 7f549c600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54940c7980 0x7f54941e0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.004+0000 7f549c600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494151820 0x7f549414aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.004+0000 7f549c600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f54940c6ea0 con 0x7f5494071a10 2024-09-16T01:52:52.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.004+0000 7f549c600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f54940c74f0 con 0x7f54940c7980 2024-09-16T01:52:52.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.004+0000 7f549c600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f54940c7180 con 0x7f5494151820 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5494071a10 0x7f54940beed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f549a600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494151820 0x7f549414aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54940c7980 0x7f54941e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f549a600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494151820 0x7f549414aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:38310/0 (socket says 172.21.15.146:38310) 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f549a600640 1 -- 172.21.15.146:0/1595233989 learned_addr learned my addr 172.21.15.146:0/1595233989 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494151820 msgr2=0x7f549414aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5494151820 0x7f549414aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5494071a10 msgr2=0x7f54940beed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5494071a10 0x7f54940beed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54941e3590 con 0x7f54940c7980 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.005+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54940c7980 0x7f54941e0990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f548000c4b0 tx=0x7f548000c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.006+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480018070 con 0x7f54940c7980 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.006+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54800074e0 con 0x7f54940c7980 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.006+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5480004a20 con 0x7f54940c7980 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.006+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54941f0480 con 0x7f54940c7980 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.006+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54941f0a40 con 0x7f54940c7980 2024-09-16T01:52:52.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.007+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5494151c00 con 0x7f54940c7980 2024-09-16T01:52:52.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.008+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5480004bc0 con 0x7f54940c7980 2024-09-16T01:52:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.009+0000 7f5491600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f547406c2b0 0x7f547406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.009+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5919+0+0 (secure 0 0 0) 0x7f548008a7a0 con 0x7f54940c7980 2024-09-16T01:52:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.009+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f547406c2b0 0x7f547406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.009+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v118) ==== 54+0+16 (secure 0 0 0) 0x7f5480090170 con 0x7f54940c7980 2024-09-16T01:52:52.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.009+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f54941e2280 0x7f5494156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.009+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54941f1220 con 0x7f54941e2280 2024-09-16T01:52:52.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.010+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f547406c2b0 0x7f547406e770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f548800b800 tx=0x7f548800b3a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.010+0000 7f549a600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f54941e2280 0x7f5494156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.010+0000 7f549a600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f54941e2280 0x7f5494156970 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.011+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54941f1220 con 0x7f54941e2280 2024-09-16T01:52:52.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.026+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f54940c7e30 con 0x7f54941e2280 2024-09-16T01:52:52.030 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:52.030 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:52.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.028+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f54940c7e30 con 0x7f54941e2280 2024-09-16T01:52:52.031 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.028+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f54941e0ed0 0x7f54941f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.028+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f54940c6100 con 0x7f54941e0ed0 2024-09-16T01:52:52.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.029+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f54941e0ed0 0x7f54941f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.029+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f54941e0ed0 0x7f54941f3cf0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.030+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54940c6100 con 0x7f54941e0ed0 2024-09-16T01:52:52.043 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:52.042+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:52.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:52.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.046+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5494157d90 con 0x7f54941e0ed0 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.048+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5494157d90 con 0x7f54941e0ed0 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.048+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5494062630 0x7f5494062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.048+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f54940bf410 con 0x7f5494062630 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.048+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5494062630 0x7f5494062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.049+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5494062630 0x7f5494062a50 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.049+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54940bf410 con 0x7f5494062630 2024-09-16T01:52:52.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.064+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5494062f90 con 0x7f5494062630 2024-09-16T01:52:52.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.067+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5494062f90 con 0x7f5494062630 2024-09-16T01:52:52.069 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:52.070 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:52.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.068+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f54941e71e0 0x7f54941e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.068+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5494063560 con 0x7f54941e71e0 2024-09-16T01:52:52.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.068+0000 7f549a600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f54941e71e0 0x7f54941e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.068+0000 7f549a600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f54941e71e0 0x7f54941e7620 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.069+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5494063560 con 0x7f54941e71e0 2024-09-16T01:52:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.083+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5494063560 con 0x7f54941e71e0 2024-09-16T01:52:52.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.085+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5494063560 con 0x7f54941e71e0 2024-09-16T01:52:52.087 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:52.088 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:52.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.085+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f549402cb50 0x7f549402cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.085+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f549402d4f0 con 0x7f549402cb50 2024-09-16T01:52:52.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.086+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f549402cb50 0x7f549402cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.087+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f549402cb50 0x7f549402cfb0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.087+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f549402d4f0 con 0x7f549402cb50 2024-09-16T01:52:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.101+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f549402d4f0 con 0x7f549402cb50 2024-09-16T01:52:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.104+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f549402d4f0 con 0x7f549402cb50 2024-09-16T01:52:52.106 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.104+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5494204b60 0x7f5494206f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.104+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f5494209d20 con 0x7f5494204b60 2024-09-16T01:52:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.104+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5494204b60 0x7f5494206f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.105+0000 7f5499200640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5494204b60 0x7f5494206f50 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.106+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5494209d20 con 0x7f5494204b60 2024-09-16T01:52:52.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.120+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5494207490 con 0x7f5494204b60 2024-09-16T01:52:52.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.121+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5494207490 con 0x7f5494204b60 2024-09-16T01:52:52.123 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:52.124 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:52.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.122+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5494207b10 0x7f5494209f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.122+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f549420ccd0 con 0x7f5494207b10 2024-09-16T01:52:52.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.122+0000 7f549a600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5494207b10 0x7f5494209f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.123+0000 7f549a600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5494207b10 0x7f5494209f00 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.123+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f549420ccd0 con 0x7f5494207b10 2024-09-16T01:52:52.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.137+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f549420a440 con 0x7f5494207b10 2024-09-16T01:52:52.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.139+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f549420a440 con 0x7f5494207b10 2024-09-16T01:52:52.141 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:52.142 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:52.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.139+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f549420aac0 0x7f549420ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:52.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.140+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f549420fc80 con 0x7f549420aac0 2024-09-16T01:52:52.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.140+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f549420aac0 0x7f549420ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:52.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.140+0000 7f5499c00640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f549420aac0 0x7f549420ceb0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:52.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.141+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f549420fc80 con 0x7f549420aac0 2024-09-16T01:52:52.144 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:52.143+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:52.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.155+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f549420d3f0 con 0x7f549420aac0 2024-09-16T01:52:52.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.157+0000 7f5491600640 1 -- 172.21.15.146:0/1595233989 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f549420d3f0 con 0x7f549420aac0 2024-09-16T01:52:52.159 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f54941e2280 msgr2=0x7f5494156970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f54941e2280 0x7f5494156970 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f54941e0ed0 msgr2=0x7f54941f3cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f54941e0ed0 0x7f54941f3cf0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5494062630 msgr2=0x7f5494062a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f5494062630 0x7f5494062a50 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f54941e71e0 msgr2=0x7f54941e7620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f54941e71e0 0x7f54941e7620 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f549402cb50 msgr2=0x7f549402cfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f549402cb50 0x7f549402cfb0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5494204b60 msgr2=0x7f5494206f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f5494204b60 0x7f5494206f50 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5494207b10 msgr2=0x7f5494209f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f5494207b10 0x7f5494209f00 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f549420aac0 msgr2=0x7f549420ceb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f549420aac0 0x7f549420ceb0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549a600640 1 -- 172.21.15.146:0/1595233989 reap_dead start 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549a600640 1 -- 172.21.15.146:0/1595233989 reap_dead start 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549a600640 1 -- 172.21.15.146:0/1595233989 reap_dead start 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549a600640 1 -- 172.21.15.146:0/1595233989 reap_dead start 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549a600640 1 -- 172.21.15.146:0/1595233989 reap_dead start 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f547406c2b0 msgr2=0x7f547406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f547406c2b0 0x7f547406e770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f548800b800 tx=0x7f548800b3a0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54940c7980 msgr2=0x7f54941e0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54940c7980 0x7f54941e0990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f548000c4b0 tx=0x7f548000c970 comp rx=0 tx=0).stop 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 shutdown_connections 2024-09-16T01:52:52.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f547406c2b0 0x7f547406e770 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 --2- 172.21.15.146:0/1595233989 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54940c7980 0x7f54941e0990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 >> 172.21.15.146:0/1595233989 conn(0x7f549406bc10 msgr2=0x7f54941557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:52.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 shutdown_connections 2024-09-16T01:52:52.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.158+0000 7f549c600640 1 -- 172.21.15.146:0/1595233989 wait complete. 2024-09-16T01:52:52.220 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:52.220 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:52.221 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:52.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:52.245 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:52.244+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:52.345 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:52.344+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 244.63936439299999, 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.411 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:52:52.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:52:52.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.418 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 112.02191205299999, 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:52:52.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:52:52.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.423 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.426 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.229895103999993, 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:52:52.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:52:52.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 92.194362480999999, 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.433 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:52:52.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:52:52.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.828273543999998, 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.439 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:52:52.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:52:52.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.264062006000003, 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.446 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:52:52.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.451 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.202274703999997, 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.360492258000001, 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.456 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:52:52.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:52:52.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:52:52.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.617053505000001, 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.463 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.465 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:52:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.572832558999998, 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.469 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:52.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:52.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.472 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:52:52.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:52:52.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 36.564416072999997, 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.476 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4107 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e99)", 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.131778585999999, 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31541509600000001, 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.482 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4107 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275941+0000", 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6959999999999999e-06 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303032+0000", 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027090797 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303053+0000", 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1019999999999999e-05 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303063+0000", 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6479999999999998e-06 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303176+0000", 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011292 2024-09-16T01:52:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303185+0000", 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.491825+0000", 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.18864050500000001 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.498314+0000", 2024-09-16T01:52:52.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064884210000000003 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551379+0000", 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053065042999999999 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551446+0000", 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7477999999999993e-05 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551537+0000", 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1055000000000001e-05 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.576787+0000", 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025250156999999999 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591154+0000", 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01436693 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591228+0000", 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4103000000000005e-05 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591355+0000", 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126367 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.489 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4250 3.10 3:0a05e26b:::100000001f8.00000297:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.097559048000001, 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.149078401, 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4250 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310162+0000", 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6599999999999999e-06 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407037+0000", 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096875366000000004 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407073+0000", 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5969000000000001e-05 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407088+0000", 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5034999999999999e-05 2024-09-16T01:52:52.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407218+0000", 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129462 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407332+0000", 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114183 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:52.493 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407516+0000", 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018366500000000001 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.446008+0000", 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038492551999999999 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456958+0000", 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01095023 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.494 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.458896+0000", 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019376650000000001 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.459238+0000", 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034161400000000001 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4504 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:52.495 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.877931852, 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22042661699999999, 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4504 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:52.496 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529789+0000", 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5409999999999999e-06 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.497 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577075+0000", 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047285897 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577104+0000", 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9173999999999999e-05 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577115+0000", 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0895e-05 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.498 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577223+0000", 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108194 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577330+0000", 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107089 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577485+0000", 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154532 2024-09-16T01:52:52.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.599121+0000", 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021636636000000001 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750098+0000", 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15097622499999999 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.500 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750164+0000", 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6172000000000003e-05 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750213+0000", 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9261999999999997e-05 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4512 3.19 3:987f889e:::100000001f8.0000039d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:52.501 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.775738194999999, 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12513411299999999, 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4512 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631984+0000", 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8999999999999999e-06 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.503 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679187+0000", 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047202698000000001 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679218+0000", 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1775e-05 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679233+0000", 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4182000000000001e-05 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.504 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682406+0000", 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031734889999999998 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682504+0000", 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7374999999999995e-05 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.683089+0000", 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058535 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.505 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.705462+0000", 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022372670000000001 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757013+0000", 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051551620999999999 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757066+0000", 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3044999999999998e-05 2024-09-16T01:52:52.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757114+0000", 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8007999999999998e-05 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4553 3.6 3:62a9ab61:::100000001f8.000003c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.241078763000001, 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.118770112, 2024-09-16T01:52:52.507 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4553 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.508 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166642+0000", 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0470000000000001e-06 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206911+0000", 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040269395999999999 2024-09-16T01:52:52.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206944+0000", 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3185000000000002e-05 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206957+0000", 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2894999999999999e-05 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207157+0000", 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199908 2024-09-16T01:52:52.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207262+0000", 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105509 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207423+0000", 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016019999999999999 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.221084+0000", 2024-09-16T01:52:52.511 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013661240999999999 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285289+0000", 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064205110999999995 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285358+0000", 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9015000000000002e-05 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.512 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285410+0000", 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1604999999999998e-05 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4560 3.39 3:9e09c0a8:::100000001f8.000003cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.161993215999999, 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11688955500000001, 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4560 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.514 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245728+0000", 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.548e-06 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301561+0000", 2024-09-16T01:52:52.515 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055832859999999998 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301606+0000", 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5423000000000002e-05 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301620+0000", 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.453e-05 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.516 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301728+0000", 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107836 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301847+0000", 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119127 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.302620+0000", 2024-09-16T01:52:52.517 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000772461 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.328630+0000", 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026010253000000001 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362241+0000", 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033611125999999998 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.518 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362382+0000", 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014118400000000001 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362615+0000", 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023220700000000001 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.519 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5800 3.8 3:139dc79b:::100000001f8.000008a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.035336221, 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.57801824300000004, 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5800 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.520 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372384+0000", 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2840000000000001e-06 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.521 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387317+0000", 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014932126 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387330+0000", 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3658e-05 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387338+0000", 2024-09-16T01:52:52.522 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389339+0000", 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020019460000000001 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389354+0000", 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.472e-05 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899029+0000", 2024-09-16T01:52:52.523 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50967473200000002 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899208+0000", 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017872499999999999 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899396+0000", 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018809 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.524 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927572+0000", 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028175951000000001 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950173+0000", 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022601711 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950246+0000", 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2533000000000005e-05 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.525 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950400+0000", 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154418 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5815 3.21 3:84e25a9c:::100000001f8.000008b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.8831590380000005, 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.133812769, 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5815 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.527 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524561+0000", 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.968e-06 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641841+0000", 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11728021 2024-09-16T01:52:52.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641861+0000", 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9939e-05 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641868+0000", 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5339999999999996e-06 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.529 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641942+0000", 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4416e-05 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642034+0000", 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1916999999999994e-05 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642172+0000", 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013773100000000001 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.530 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656243+0000", 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014070719000000001 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658259+0000", 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020165389999999999 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658323+0000", 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3876999999999998e-05 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.531 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658372+0000", 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8918999999999997e-05 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5821 3.8 3:10a53e03:::100000001f8.000008ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.7305002789999993, 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27338652600000002, 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:52.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5821 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.533 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677219+0000", 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.344e-06 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702527+0000", 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025308081999999999 2024-09-16T01:52:52.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702546+0000", 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8698999999999998e-05 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702555+0000", 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2609999999999999e-06 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702663+0000", 2024-09-16T01:52:52.535 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010725 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702675+0000", 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2743e-05 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910154+0000", 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.207478311 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.536 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910331+0000", 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000177057 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910525+0000", 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000194074 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.537 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927630+0000", 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017105187000000001 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950420+0000", 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022789713 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950473+0000", 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3115999999999998e-05 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:52.538 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950605+0000", 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013168899999999999 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:52.539 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:52.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.502+0000 7f864fe00640 1 -- 172.21.15.146:0/2417140933 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v119) ==== 101+0+0 (secure 0 0 0) 0x7f864005da10 con 0x7f86580d0fd0 2024-09-16T01:52:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:52:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.506+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f863406c2b0 msgr2=0x7f863406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.506+0000 7f865de00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f863406c2b0 0x7f863406e770 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8648034000 tx=0x7f864802e000 comp rx=0 tx=0).stop 2024-09-16T01:52:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.506+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 msgr2=0x7f8658158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:52.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.506+0000 7f865de00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f8658158c30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f864000e870 tx=0x7f864000ed30 comp rx=0 tx=0).stop 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 shutdown_connections 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f863406c2b0 0x7f863406e770 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f865815edc0 0x7f8658159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86580d0fd0 0x7f8658158c30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 --2- 172.21.15.146:0/2417140933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86580d0bf0 0x7f8658151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 >> 172.21.15.146:0/2417140933 conn(0x7f865806c900 msgr2=0x7f86580c89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 shutdown_connections 2024-09-16T01:52:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:52.507+0000 7f865de00640 1 -- 172.21.15.146:0/2417140933 wait complete. 2024-09-16T01:52:52.543 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:52.539+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:52.642 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:52.640+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:52.742 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:52.740+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:52.752 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:52.752 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:52.752 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:52.752 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:52.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:52.852 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:52.850+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:52.944 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:52.943+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:52.976 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:52.977 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:52.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:53.046 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:53.044+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:53.153 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:53.150+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.910453560999997, 2024-09-16T01:52:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.881292705999996, 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.848941138000001, 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.832993872000003, 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.285918706999993, 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.100989350999996, 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.098451542999996, 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 63.489040310999997, 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.450783876000003, 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:52:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 37.375712141000001, 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:52:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:52:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4142 3.16 3:6ba609e9:::100000001f8.0000022b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.414615587, 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108516207, 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4142 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787022+0000", 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.61e-06 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842519+0000", 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055496399000000002 2024-09-16T01:52:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842539+0000", 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0820000000000001e-05 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842550+0000", 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0414999999999999e-05 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842641+0000", 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1657999999999996e-05 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842742+0000", 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010075 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842888+0000", 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014623699999999999 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.892001+0000", 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112211000000003 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895434+0000", 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034334869999999998 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895499+0000", 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5313000000000003e-05 2024-09-16T01:52:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895537+0000", 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7307e-05 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4246 3.34 3:2f44359e:::100000001f8.00000293:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.872003025000001, 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.142262732, 2024-09-16T01:52:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4246 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329635+0000", 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9180000000000001e-06 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410104+0000", 2024-09-16T01:52:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080469499 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410128+0000", 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3243000000000001e-05 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410141+0000", 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3067999999999999e-05 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419779+0000", 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096380589999999992 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419889+0000", 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.420057+0000", 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016837699999999999 2024-09-16T01:52:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456784+0000", 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036727237000000003 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471769+0000", 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014985112 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471839+0000", 2024-09-16T01:52:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9275000000000003e-05 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471896+0000", 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7259000000000002e-05 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4247 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.791447917999999, 2024-09-16T01:52:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120085209, 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4247 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410190+0000", 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451530+0000", 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041340783999999998 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451572+0000", 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1690000000000002e-05 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451620+0000", 2024-09-16T01:52:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8180999999999998e-05 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452310+0000", 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068996900000000002 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452413+0000", 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010248599999999999 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452588+0000", 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017574800000000001 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.518054+0000", 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065465819999999994 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530168+0000", 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012113413999999999 2024-09-16T01:52:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530225+0000", 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7284999999999998e-05 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530273+0000", 2024-09-16T01:52:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8381999999999999e-05 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4503 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.689892563000001, 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13319203700000001, 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4503 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511746+0000", 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0250000000000001e-06 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576368+0000", 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064622151000000003 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576386+0000", 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7878999999999999e-05 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576398+0000", 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-09-16T01:52:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578343+0000", 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001945397 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578445+0000", 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102001 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578601+0000", 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015608399999999999 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.643878+0000", 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065277247999999996 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644839+0000", 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096038899999999997 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644895+0000", 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6484000000000003e-05 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644936+0000", 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0324999999999998e-05 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4504 3.f e101/94 3:f3dac423:::100000001f8.00000395:head v 101'76, mlcod=100'74)", 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.606966446000001, 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14827673899999999, 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594672+0000", 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3309999999999998e-06 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629004+0000", 2024-09-16T01:52:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034332342000000002 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629020+0000", 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6121e-05 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629029+0000", 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7479999999999999e-06 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635523+0000", 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064935909999999999 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635549+0000", 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5808000000000001e-05 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742856+0000", 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107307556 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742946+0000", 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0241999999999994e-05 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4549 3.2 3:428d9500:::100000001f8.000003c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 26.032002998999999, 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11639424900000001, 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4549 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169636+0000", 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8660000000000002e-06 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212280+0000", 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042644350999999997 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212312+0000", 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2088999999999997e-05 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212327+0000", 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521e-05 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212426+0000", 2024-09-16T01:52:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9245000000000003e-05 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212539+0000", 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112711 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212725+0000", 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018583400000000001 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285881+0000", 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073156321999999996 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285914+0000", 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2929999999999998e-05 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285979+0000", 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5048000000000005e-05 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.286027+0000", 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8331999999999997e-05 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5790 3.16 3:687a9a46:::100000001f8.0000089b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.990967863, 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.65913515700000003, 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5790 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210670+0000", 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244952+0000", 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034282188999999998 2024-09-16T01:52:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244970+0000", 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8179999999999999e-05 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244982+0000", 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1736000000000001e-05 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245078+0000", 2024-09-16T01:52:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5439000000000006e-05 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245090+0000", 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2177e-05 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791600+0000", 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.546509829 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791770+0000", 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017051699999999999 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791950+0000", 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018029299999999999 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829483+0000", 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037532101999999998 2024-09-16T01:52:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867462+0000", 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037979295000000003 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869661+0000", 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021987059999999999 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869702+0000", 2024-09-16T01:52:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1148e-05 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869803+0000", 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101696 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5793 3.16 3:68c3c174:::100000001f8.0000089e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 10.930257243, 2024-09-16T01:52:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.59860013000000001, 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5793 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271381+0000", 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0669999999999999e-06 2024-09-16T01:52:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311762+0000", 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040380644 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311785+0000", 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3090000000000001e-05 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311798+0000", 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3057999999999999e-05 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311911+0000", 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113429 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311923+0000", 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1915e-05 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829575+0000", 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51765227000000003 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829754+0000", 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017901700000000001 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829935+0000", 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018041800000000001 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867483+0000", 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037548040999999997 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869178+0000", 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016955379999999999 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869843+0000", 2024-09-16T01:52:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000664345 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869882+0000", 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8976000000000001e-05 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869979+0000", 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7322000000000001e-05 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5945 3.2b 3:d7b2ba59:::100000001f8.00000936:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.8041167770000008, 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11205129899999999, 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5945 2024-09-16T01:52:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397521+0000", 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429779+0000", 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032257503999999999 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429805+0000", 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429818+0000", 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2687999999999999e-05 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429896+0000", 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7898999999999994e-05 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430008+0000", 2024-09-16T01:52:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011241100000000001 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430198+0000", 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000189197 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.448700+0000", 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018502761999999999 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509440+0000", 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060739425999999999 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509514+0000", 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4673999999999997e-05 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509571+0000", 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6249999999999998e-05 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5948 3.2d 3:b677c6b2:::100000001f8.00000939:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 8.7714216359999995, 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122297751, 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5948 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430216+0000", 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453110+0000", 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022894134 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453131+0000", 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1120000000000001e-05 2024-09-16T01:52:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453141+0000", 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5340000000000005e-06 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495855+0000", 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042714162 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495997+0000", 2024-09-16T01:52:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014160999999999999 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537228+0000", 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041231297 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537360+0000", 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264400000000001 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6,7", 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537537+0000", 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017664800000000001 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548049+0000", 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511915 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548065+0000", 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5812e-05 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552281+0000", 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042165600000000003 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552355+0000", 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3712000000000002e-05 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552512+0000", 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015707300000000001 2024-09-16T01:52:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.335 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:53.336 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 readers = 4760719.00 kB/sec 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 readers = 4759016.58 kB/sec 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 4760719.00 kB/sec 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 4760719.00 kB/sec 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 4760719.00 kB/sec 2024-09-16T01:52:53.337 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 10485760.00 kB 2024-09-16T01:52:53.338 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:53.335+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:53.438 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:53.436+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:53.538 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:53.537+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:53.539 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:53.539 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:53.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:53.640 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:53.639+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:53.741 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:53.740+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:53.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:53.842 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:53.841+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 114.07651050299999, 2024-09-16T01:52:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.934050838, 2024-09-16T01:52:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 113.58658461500001, 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 93.702293251, 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.664553748000003, 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.805566748000004, 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.938527773999994, 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.742575552999995, 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.701573609999997, 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 64.477346049999994, 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 58.520564497000002, 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.609584148000003, 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.173053166000003, 2024-09-16T01:52:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:52:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.149899247999997, 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:52:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:52:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.138963404999998, 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:52:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.108768980999997, 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:52:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:52:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.067363528000001, 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:52:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:52:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5551 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.086556438000001, 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13476244500000001, 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5551 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838694+0000", 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.306e-06 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866535+0000", 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027841175999999999 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866552+0000", 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7387000000000001e-05 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866563+0000", 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0447e-05 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866665+0000", 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101894 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866788+0000", 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123194 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866953+0000", 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165365 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.918808+0000", 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051855367999999999 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973333+0000", 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054524250000000003 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973402+0000", 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9648999999999997e-05 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973454+0000", 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5812 3.a 3:51094797:::100000001f8.000008b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 11.392857322999999, 2024-09-16T01:52:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14779089000000001, 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5812 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532392+0000", 2024-09-16T01:52:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8679999999999999e-06 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653045+0000", 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120652599 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653062+0000", 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6929e-05 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653073+0000", 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0925e-05 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656653+0000", 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035800049999999998 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656738+0000", 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5599000000000005e-05 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656890+0000", 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151874 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.676041+0000", 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019150358999999999 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680071+0000", 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004030624 2024-09-16T01:52:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680131+0000", 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9704000000000003e-05 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680181+0000", 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0404000000000001e-05 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5944 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 9.5380281789999994, 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.115163742, 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5944 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387222+0000", 2024-09-16T01:52:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418222+0000", 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031000603000000002 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418239+0000", 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6833e-05 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418249+0000", 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8179999999999999e-06 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418846+0000", 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059710099999999995 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418945+0000", 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8826000000000006e-05 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.419093+0000", 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148553 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.442472+0000", 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023378423999999998 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502282+0000", 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059810025000000003 2024-09-16T01:52:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502341+0000", 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8922000000000003e-05 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502383+0000", 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2590999999999999e-05 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:54.080 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:54.081 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:54.082 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:54.080+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:54.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:54.182 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:54.180+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:54.198 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:54.198 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:54.198 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:54.199 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:54.207 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:54.282 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:54.280+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:54.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.327+0000 7f812a800640 1 Processor -- start 2024-09-16T01:52:54.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.327+0000 7f812a800640 1 -- start start 2024-09-16T01:52:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f812a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8124154f10 0x7f81240c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f812a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 0x7f81240c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f812a800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f812415a9b0 con 0x7f81240c6cd0 2024-09-16T01:52:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f812a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f81240c7f20 con 0x7f8124154f10 2024-09-16T01:52:54.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f812a800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f812406ea70 con 0x7f812406f610 2024-09-16T01:52:54.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f8123400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8124154f10 0x7f81240c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f8123e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 0x7f81240c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f8123e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 0x7f81240c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56702/0 (socket says 172.21.15.146:56702) 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f8122a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f81240c6cd0 0x7f812406ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36236/0 (socket says 172.21.15.146:36236) 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.329+0000 7f8123e00640 1 -- 172.21.15.146:0/795543078 learned_addr learned my addr 172.21.15.146:0/795543078 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.330+0000 7f8123e00640 1 -- 172.21.15.146:0/795543078 >> v2:172.21.15.146:3300/0 conn(0x7f8124154f10 msgr2=0x7f81240c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.330+0000 7f8123e00640 1 --2- 172.21.15.146:0/795543078 >> v2:172.21.15.146:3300/0 conn(0x7f8124154f10 0x7f81240c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.330+0000 7f8123e00640 1 -- 172.21.15.146:0/795543078 >> v1:172.21.15.161:6789/0 conn(0x7f81240c6cd0 legacy=0x7f812406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.330+0000 7f8123e00640 1 -- 172.21.15.146:0/795543078 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812415ac70 con 0x7f812406f610 2024-09-16T01:52:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.331+0000 7f8123e00640 1 --2- 172.21.15.146:0/795543078 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 0x7f81240c9180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f811c009e50 tx=0x7f811c02a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eae34518323425e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.331+0000 7f8122000640 1 -- 172.21.15.146:0/795543078 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c036070 con 0x7f812406f610 2024-09-16T01:52:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.331+0000 7f8122000640 1 -- 172.21.15.146:0/795543078 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811c031600 con 0x7f812406f610 2024-09-16T01:52:54.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.331+0000 7f8122000640 1 -- 172.21.15.146:0/795543078 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0319d0 con 0x7f812406f610 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 -- 172.21.15.146:0/795543078 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 msgr2=0x7f81240c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 --2- 172.21.15.146:0/795543078 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 0x7f81240c9180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f811c009e50 tx=0x7f811c02a6c0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 -- 172.21.15.146:0/795543078 shutdown_connections 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 --2- 172.21.15.146:0/795543078 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f812406f610 0x7f81240c9180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 --2- 172.21.15.146:0/795543078 >> v2:172.21.15.146:3300/0 conn(0x7f8124154f10 0x7f81240c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 -- 172.21.15.146:0/795543078 >> 172.21.15.146:0/795543078 conn(0x7f812406bc10 msgr2=0x7f812406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 -- 172.21.15.146:0/795543078 shutdown_connections 2024-09-16T01:52:54.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.332+0000 7f812a800640 1 -- 172.21.15.146:0/795543078 wait complete. 2024-09-16T01:52:54.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.334+0000 7f812a800640 1 Processor -- start 2024-09-16T01:52:54.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.334+0000 7f812a800640 1 -- start start 2024-09-16T01:52:54.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f812a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 0x7f81240c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f812a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81240c6cd0 0x7f81241f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f812a800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8124154f10 0x7f81241f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f812a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f812415b510 con 0x7f812406f610 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f812a800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f812415b230 con 0x7f81240c6cd0 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f812a800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f812415b7f0 con 0x7f8124154f10 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 0x7f81240c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8122a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81240c6cd0 0x7f81241f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8124154f10 0x7f81241f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 0x7f81240c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57550/0 (socket says 172.21.15.146:57550) 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 -- 172.21.15.146:0/243801079 learned_addr learned my addr 172.21.15.146:0/243801079 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8124154f10 0x7f81241f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56712/0 (socket says 172.21.15.146:56712) 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8124154f10 msgr2=0x7f81241f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8124154f10 0x7f81241f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81240c6cd0 msgr2=0x7f81241f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.335+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81240c6cd0 0x7f81241f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f8123400640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81240c3650 con 0x7f812406f610 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 0x7f81240c6180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f811400e870 tx=0x7f811400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114017070 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811400c940 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81241e0b00 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811400cd30 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.336+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81240c38f0 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.337+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f81240c8130 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.338+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f811401a070 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.339+0000 7f811b400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f810406c2b0 0x7f810406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.339+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5894+0+0 (secure 0 0 0) 0x7f811408b480 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.339+0000 7f8122a00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f810406c2b0 0x7f810406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.339+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v120) ==== 54+0+16 (secure 0 0 0) 0x7f811408da20 con 0x7f812406f610 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.339+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f81241f2080 0x7f812414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.339+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8124073410 con 0x7f81241f2080 2024-09-16T01:52:54.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.340+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f81241f2080 0x7f812414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.340+0000 7f8122a00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f810406c2b0 0x7f810406e770 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8124072d40 tx=0x7f8110000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.340+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f81241f2080 0x7f812414fab0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.341+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8124073410 con 0x7f81241f2080 2024-09-16T01:52:54.355 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:54.355 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:54.355 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:54.355 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:54.355 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:54.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.355+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81240c7130 con 0x7f81241f2080 2024-09-16T01:52:54.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.357+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f81240c7130 con 0x7f81241f2080 2024-09-16T01:52:54.359 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:54.360 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:54.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.357+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f81241f43f0 0x7f81241f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.357+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8124150500 con 0x7f81241f43f0 2024-09-16T01:52:54.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.358+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f81241f43f0 0x7f81241f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.358+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f81241f43f0 0x7f81241f47d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.359+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8124150500 con 0x7f81241f43f0 2024-09-16T01:52:54.364 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:54.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.374+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81241e7220 con 0x7f81241f43f0 2024-09-16T01:52:54.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.375+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f81241e7220 con 0x7f81241f43f0 2024-09-16T01:52:54.377 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:54.378 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:54.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.375+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f81241f4d10 0x7f81240c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.375+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f81240c51b0 con 0x7f81241f4d10 2024-09-16T01:52:54.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.376+0000 7f8122a00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f81241f4d10 0x7f81240c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.376+0000 7f8122a00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f81241f4d10 0x7f81240c4c70 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.377+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81240c51b0 con 0x7f81241f4d10 2024-09-16T01:52:54.383 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:54.381+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:54.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.390+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8124062630 con 0x7f81241f4d10 2024-09-16T01:52:54.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.392+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8124062630 con 0x7f81241f4d10 2024-09-16T01:52:54.394 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.392+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8124063110 0x7f8124153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.392+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8124063550 con 0x7f8124063110 2024-09-16T01:52:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.393+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8124063110 0x7f8124153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.393+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8124063110 0x7f8124153170 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.394+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8124063550 con 0x7f8124063110 2024-09-16T01:52:54.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.407+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8124153980 con 0x7f8124063110 2024-09-16T01:52:54.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.409+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8124153980 con 0x7f8124063110 2024-09-16T01:52:54.411 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:54.412 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:54.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.409+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f812402c640 0x7f812402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.409+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8124154000 con 0x7f812402c640 2024-09-16T01:52:54.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.409+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f812402c640 0x7f812402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.410+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f812402c640 0x7f812402ca60 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.411+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8124154000 con 0x7f812402c640 2024-09-16T01:52:54.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:54.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.424+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8124154000 con 0x7f812402c640 2024-09-16T01:52:54.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.426+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8124154000 con 0x7f812402c640 2024-09-16T01:52:54.429 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:54.430 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:54.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.426+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f812402d4b0 0x7f8124207570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.427+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f812402d910 con 0x7f812402d4b0 2024-09-16T01:52:54.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.427+0000 7f8122a00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f812402d4b0 0x7f8124207570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.427+0000 7f8122a00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f812402d4b0 0x7f8124207570 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.428+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f812402d910 con 0x7f812402d4b0 2024-09-16T01:52:54.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.443+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f812402d910 con 0x7f812402d4b0 2024-09-16T01:52:54.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.445+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f812402d910 con 0x7f812402d4b0 2024-09-16T01:52:54.447 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:54.449 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:54.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.445+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8124207fc0 0x7f812420a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.445+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f812402db50 con 0x7f8124207fc0 2024-09-16T01:52:54.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.445+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8124207fc0 0x7f812420a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.446+0000 7f8123e00640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8124207fc0 0x7f812420a3b0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.446+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f812402db50 con 0x7f8124207fc0 2024-09-16T01:52:54.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.460+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f812402d910 con 0x7f8124207fc0 2024-09-16T01:52:54.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.462+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f812402d910 con 0x7f8124207fc0 2024-09-16T01:52:54.464 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:54.465 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:54.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.462+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f812420ae00 0x7f812420d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.462+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8124210080 con 0x7f812420ae00 2024-09-16T01:52:54.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.462+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f812420ae00 0x7f812420d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.463+0000 7f8123400640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f812420ae00 0x7f812420d1f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.464+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8124210080 con 0x7f812420ae00 2024-09-16T01:52:54.470 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:54.471 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:54.480 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:54.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.477+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8124000950 con 0x7f812420ae00 2024-09-16T01:52:54.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.479+0000 7f811b400640 1 -- 172.21.15.146:0/243801079 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8124000950 con 0x7f812420ae00 2024-09-16T01:52:54.483 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:54.486 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:54.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f81241f2080 msgr2=0x7f812414fab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f81241f2080 0x7f812414fab0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f81241f43f0 msgr2=0x7f81241f47d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f81241f43f0 0x7f81241f47d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f81241f4d10 msgr2=0x7f81240c4c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f81241f4d10 0x7f81240c4c70 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8124063110 msgr2=0x7f8124153170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f8124063110 0x7f8124153170 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f812402c640 msgr2=0x7f812402ca60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f812402c640 0x7f812402ca60 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f812402d4b0 msgr2=0x7f8124207570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f812402d4b0 0x7f8124207570 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8124207fc0 msgr2=0x7f812420a3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f8124207fc0 0x7f812420a3b0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f812420ae00 msgr2=0x7f812420d1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f812420ae00 0x7f812420d1f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f8123e00640 1 -- 172.21.15.146:0/243801079 reap_dead start 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f8123e00640 1 -- 172.21.15.146:0/243801079 reap_dead start 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f8123e00640 1 -- 172.21.15.146:0/243801079 reap_dead start 2024-09-16T01:52:54.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f8123e00640 1 -- 172.21.15.146:0/243801079 reap_dead start 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f8123e00640 1 -- 172.21.15.146:0/243801079 reap_dead start 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f810406c2b0 msgr2=0x7f810406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f810406c2b0 0x7f810406e770 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8124072d40 tx=0x7f8110000bc0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 msgr2=0x7f81240c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 0x7f81240c6180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f811400e870 tx=0x7f811400ed30 comp rx=0 tx=0).stop 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 shutdown_connections 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f810406c2b0 0x7f810406e770 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 --2- 172.21.15.146:0/243801079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f812406f610 0x7f81240c6180 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 >> 172.21.15.146:0/243801079 conn(0x7f812406bc10 msgr2=0x7f81241e63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:54.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 shutdown_connections 2024-09-16T01:52:54.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.480+0000 7f812a800640 1 -- 172.21.15.146:0/243801079 wait complete. 2024-09-16T01:52:54.493 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:54.491+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:54.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:52:54.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.572+0000 7f63b9200640 1 Processor -- start 2024-09-16T01:52:54.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.573+0000 7f63b9200640 1 -- start start 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b9200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63b40c6b70 0x7f63b415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b406d140 con 0x7f63b40cc040 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b9200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b406d420 con 0x7f63b40c75a0 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b9200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f63b415b880 con 0x7f63b40c6b70 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b3400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f63b40c6b70 0x7f63b415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b2a00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f63b40c75a0 0x7f63b40cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36248/0 (socket says 172.21.15.146:36248) 2024-09-16T01:52:54.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b2a00640 1 -- 172.21.15.146:0/2717725852 learned_addr learned my addr 172.21.15.146:0/2717725852 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b3400640 1 -- 172.21.15.146:0/2717725852 >> v1:172.21.15.161:6789/0 conn(0x7f63b40c75a0 legacy=0x7f63b40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b3400640 1 -- 172.21.15.146:0/2717725852 >> v1:172.21.15.146:6789/0 conn(0x7f63b40cc040 legacy=0x7f63b4165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b3400640 1 -- 172.21.15.146:0/2717725852 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b406d700 con 0x7f63b40c6b70 2024-09-16T01:52:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.574+0000 7f63b2000640 1 -- 172.21.15.146:0/2717725852 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1029913064 0 0) 0x7f63b406d420 con 0x7f63b40c75a0 2024-09-16T01:52:54.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.575+0000 7f63b3400640 1 --2- 172.21.15.146:0/2717725852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c6b70 0x7f63b415b340 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f63a40097a0 tx=0x7f63a402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98fb8810ef587061 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b2000640 1 -- 172.21.15.146:0/2717725852 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4037070 con 0x7f63b40c6b70 2024-09-16T01:52:54.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b2000640 1 -- 172.21.15.146:0/2717725852 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a4032440 con 0x7f63b40c6b70 2024-09-16T01:52:54.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b2000640 1 -- 172.21.15.146:0/2717725852 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4032810 con 0x7f63b40c6b70 2024-09-16T01:52:54.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 -- 172.21.15.146:0/2717725852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c6b70 msgr2=0x7f63b415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 --2- 172.21.15.146:0/2717725852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c6b70 0x7f63b415b340 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f63a40097a0 tx=0x7f63a402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 -- 172.21.15.146:0/2717725852 shutdown_connections 2024-09-16T01:52:54.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 --2- 172.21.15.146:0/2717725852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c6b70 0x7f63b415b340 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 -- 172.21.15.146:0/2717725852 >> 172.21.15.146:0/2717725852 conn(0x7f63b40ccc20 msgr2=0x7f63b40cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:54.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 -- 172.21.15.146:0/2717725852 shutdown_connections 2024-09-16T01:52:54.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.576+0000 7f63b9200640 1 -- 172.21.15.146:0/2717725852 wait complete. 2024-09-16T01:52:54.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.578+0000 7f63b9200640 1 Processor -- start 2024-09-16T01:52:54.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.578+0000 7f63b9200640 1 -- start start 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b9200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 0x7f63b40c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b9200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c75a0 0x7f63b41f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63b40cc040 0x7f63b41512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f63b40cb050 con 0x7f63b40cc040 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b9200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f63b40cab20 con 0x7f63b40c75a0 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b9200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f63b40cbc80 con 0x7f63b40c6b70 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b2a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c75a0 0x7f63b41f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 0x7f63b40c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.579+0000 7f63b3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 0x7f63b40c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56716/0 (socket says 172.21.15.146:56716) 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 -- 172.21.15.146:0/3707786524 learned_addr learned my addr 172.21.15.146:0/3707786524 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b2a00640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c75a0 0x7f63b41f0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43372/0 (socket says 172.21.15.146:43372) 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3e00640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63b40cc040 0x7f63b41512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 -- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c75a0 msgr2=0x7f63b41f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c75a0 0x7f63b41f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 -- 172.21.15.146:0/3707786524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63b40cc040 msgr2=0x7f63b41512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63b40cc040 0x7f63b41512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 -- 172.21.15.146:0/3707786524 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b4158250 con 0x7f63b40c6b70 2024-09-16T01:52:54.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b3400640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 0x7f63b40c9110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f63a40098c0 tx=0x7f63a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4037040 con 0x7f63b40c6b70 2024-09-16T01:52:54.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a4042070 con 0x7f63b40c6b70 2024-09-16T01:52:54.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b40ab920 con 0x7f63b40c6b70 2024-09-16T01:52:54.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.580+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4030a70 con 0x7f63b40c6b70 2024-09-16T01:52:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.581+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b41f4a80 con 0x7f63b40c6b70 2024-09-16T01:52:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.582+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b406cd20 con 0x7f63b40c6b70 2024-09-16T01:52:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.582+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f63a4030c10 con 0x7f63b40c6b70 2024-09-16T01:52:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.583+0000 7f63ab400640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f639406c2b0 0x7f639406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.583+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(120..120 src has 1..120) ==== 5894+0+0 (secure 0 0 0) 0x7f63a40abf20 con 0x7f63b40c6b70 2024-09-16T01:52:54.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.584+0000 7f63b2a00640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f639406c2b0 0x7f639406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.584+0000 7f63b2a00640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f639406c2b0 0x7f639406e770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f63a0007f70 tx=0x7f63a0007f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.589 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:54.589 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:54.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.588+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f63a4079e50 con 0x7f63b40c6b70 2024-09-16T01:52:54.592 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:54.591+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:54.598 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:54.694 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:54.693+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:54.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.703+0000 7f0603e00640 1 Processor -- start 2024-09-16T01:52:54.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.703+0000 7f0603e00640 1 -- start start 2024-09-16T01:52:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0603e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f06040c6b70 0x7f060415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0603e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0603e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f060406d140 con 0x7f06040cbab0 2024-09-16T01:52:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0603e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f060415b880 con 0x7f06040c6b70 2024-09-16T01:52:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0603e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f06040cab20 con 0x7f06040c75a0 2024-09-16T01:52:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0602a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f06040c6b70 0x7f060415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.704+0000 7f0603400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f06040cbab0 0x7f06040cc040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60318/0 (socket says 172.21.15.146:60318) 2024-09-16T01:52:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0603400640 1 -- 172.21.15.146:0/3289843839 learned_addr learned my addr 172.21.15.146:0/3289843839 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0602000640 1 --2- 172.21.15.146:0/3289843839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0602000640 1 -- 172.21.15.146:0/3289843839 >> v2:172.21.15.146:3300/0 conn(0x7f06040c6b70 msgr2=0x7f060415b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0602000640 1 --2- 172.21.15.146:0/3289843839 >> v2:172.21.15.146:3300/0 conn(0x7f06040c6b70 0x7f060415b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0602000640 1 -- 172.21.15.146:0/3289843839 >> v1:172.21.15.146:6789/0 conn(0x7f06040cbab0 legacy=0x7f06040cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0601600640 1 -- 172.21.15.146:0/3289843839 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2267768126 0 0) 0x7f060406d140 con 0x7f06040cbab0 2024-09-16T01:52:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.705+0000 7f0602000640 1 -- 172.21.15.146:0/3289843839 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060406d420 con 0x7f06040c75a0 2024-09-16T01:52:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.706+0000 7f0602000640 1 --2- 172.21.15.146:0/3289843839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040cb570 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f05f00097a0 tx=0x7f05f002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31ff5fe0a1e24efa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.706+0000 7f0601600640 1 -- 172.21.15.146:0/3289843839 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0037070 con 0x7f06040c75a0 2024-09-16T01:52:54.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.706+0000 7f0601600640 1 -- 172.21.15.146:0/3289843839 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f002adb0 con 0x7f06040c75a0 2024-09-16T01:52:54.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.706+0000 7f0601600640 1 -- 172.21.15.146:0/3289843839 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0032660 con 0x7f06040c75a0 2024-09-16T01:52:54.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 -- 172.21.15.146:0/3289843839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 msgr2=0x7f06040cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 --2- 172.21.15.146:0/3289843839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040cb570 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f05f00097a0 tx=0x7f05f002a060 comp rx=0 tx=0).stop 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 -- 172.21.15.146:0/3289843839 shutdown_connections 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 --2- 172.21.15.146:0/3289843839 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040cb570 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 --2- 172.21.15.146:0/3289843839 >> v2:172.21.15.146:3300/0 conn(0x7f06040c6b70 0x7f060415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 -- 172.21.15.146:0/3289843839 >> 172.21.15.146:0/3289843839 conn(0x7f06040ccc20 msgr2=0x7f06040cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 -- 172.21.15.146:0/3289843839 shutdown_connections 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.707+0000 7f0603e00640 1 -- 172.21.15.146:0/3289843839 wait complete. 2024-09-16T01:52:54.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.709+0000 7f0603e00640 1 Processor -- start 2024-09-16T01:52:54.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.709+0000 7f0603e00640 1 -- start start 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c6b70 0x7f06040abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06040cbab0 0x7f06041f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f06040cb050 con 0x7f06040cbab0 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f06040cad70 con 0x7f06040c6b70 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f060406cd20 con 0x7f06040c75a0 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06040cbab0 0x7f06041f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040c51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56736/0 (socket says 172.21.15.146:56736) 2024-09-16T01:52:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 -- 172.21.15.146:0/1443375421 learned_addr learned my addr 172.21.15.146:0/1443375421 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:54.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602a00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c6b70 0x7f06040abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 -- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c6b70 msgr2=0x7f06040abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c6b70 0x7f06040abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 -- 172.21.15.146:0/1443375421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06040cbab0 msgr2=0x7f06041f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06040cbab0 0x7f06041f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0602000640 1 -- 172.21.15.146:0/1443375421 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060415bf10 con 0x7f06040c75a0 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.710+0000 7f0603400640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06040cbab0 0x7f06041f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.711+0000 7f0602000640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040c51a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000c00 tx=0x7f05f00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.711+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0041070 con 0x7f06040c75a0 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.711+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f003c460 con 0x7f06040c75a0 2024-09-16T01:52:54.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.711+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0037040 con 0x7f06040c75a0 2024-09-16T01:52:54.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.711+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0604157690 con 0x7f06040c75a0 2024-09-16T01:52:54.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.711+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0604157c50 con 0x7f06040c75a0 2024-09-16T01:52:54.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.713+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f060415ad30 con 0x7f06040c75a0 2024-09-16T01:52:54.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.713+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f05f0044070 con 0x7f06040c75a0 2024-09-16T01:52:54.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.714+0000 7f05fb400640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f05e406c2b0 0x7f05e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.714+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(120..120 src has 1..120) ==== 5894+0+0 (secure 0 0 0) 0x7f05f00abe60 con 0x7f06040c75a0 2024-09-16T01:52:54.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.715+0000 7f0602a00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f05e406c2b0 0x7f05e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.715+0000 7f0602a00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f05e406c2b0 0x7f05e406e770 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009aa0 tx=0x7f05f4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.719+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f05f0079d90 con 0x7f06040c75a0 2024-09-16T01:52:54.744 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:54.744 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:54.753 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:54.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.768+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63b40cda30 con 0x7f639406c2b0 2024-09-16T01:52:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.775+0000 7f63ab400640 1 -- 172.21.15.146:0/3707786524 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254836 (secure 0 0 0) 0x7f63b40cda30 con 0x7f639406c2b0 2024-09-16T01:52:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:54.781 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:54.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f639406c2b0 msgr2=0x7f639406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f639406c2b0 0x7f639406e770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f63a0007f70 tx=0x7f63a0007f00 comp rx=0 tx=0).stop 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 msgr2=0x7f63b40c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 0x7f63b40c9110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f63a40098c0 tx=0x7f63a40046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 shutdown_connections 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f639406c2b0 0x7f639406e770 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63b40cc040 0x7f63b41512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f63b40c75a0 0x7f63b41f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 --2- 172.21.15.146:0/3707786524 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f63b40c6b70 0x7f63b40c9110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 >> 172.21.15.146:0/3707786524 conn(0x7f63b40ccc20 msgr2=0x7f63b40c8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 shutdown_connections 2024-09-16T01:52:54.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.779+0000 7f63b9200640 1 -- 172.21.15.146:0/3707786524 wait complete. 2024-09-16T01:52:54.794 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:54.793+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:54.797 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-16T01:52:54.564909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":7207,"num_write_kb":25168808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3154,"num_bytes_recovered":9722644997,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7648,"ondisk_log_size":7648,"up":194,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":205,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1544292,"kb_used_data":1344660,"kb_used_omap":9,"kb_used_meta":199478,"kb_avail":564686748,"statfs":{"total":579820584960,"available":578239229952,"internally_reserved":0,"allocated":1376931840,"data_stored":21888582785,"data_compressed":1094557287,"data_compressed_allocated":1367805952,"data_compressed_original":21884112896,"omap_allocated":9562,"internal_metadata":204266150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-6,"num_objects_missing":-6,"num_objects_degraded":-2,"num_objects_misplaced":-12,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":1950352582,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.074358"},"pg_stats":[{"pgid":"3.3f","version":"112'104","reported_seq":388,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503950+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:51.503950+0000","last_peered":"2024-09-16T01:52:51.503950+0000","last_clean":"2024-09-16T01:52:51.503950+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:51.503950+0000","last_undegraded":"2024-09-16T01:52:51.503950+0000","last_fullsized":"2024-09-16T01:52:51.503950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"113'94","reported_seq":296,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495254+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:50.495254+0000","last_peered":"2024-09-16T01:52:50.495254+0000","last_clean":"2024-09-16T01:52:50.495254+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:50.495254+0000","last_undegraded":"2024-09-16T01:52:50.495254+0000","last_fullsized":"2024-09-16T01:52:50.495254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":555,"reported_epoch":116,"state":"peering","last_fresh":"2024-09-16T01:52:49.890306+0000","last_change":"2024-09-16T01:52:50.495187+0000","last_active":"2024-09-16T01:52:50.495187+0000","last_peered":"2024-09-16T01:52:49.890306+0000","last_clean":"2024-09-16T01:52:45.091998+0000","last_became_active":"2024-09-16T01:52:47.550146+0000","last_became_peered":"2024-09-16T01:52:47.550146+0000","last_unstale":"2024-09-16T01:52:49.890306+0000","last_undegraded":"2024-09-16T01:52:49.890306+0000","last_fullsized":"2024-09-16T01:52:47.535688+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"113'125","reported_seq":335,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500613+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:51.500613+0000","last_peered":"2024-09-16T01:52:51.500613+0000","last_clean":"2024-09-16T01:52:51.500613+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:51.500613+0000","last_undegraded":"2024-09-16T01:52:51.500613+0000","last_fullsized":"2024-09-16T01:52:51.500613+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"114'112","reported_seq":288,"reported_epoch":117,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T01:52:50.495126+0000","last_change":"2024-09-16T01:52:47.557414+0000","last_active":"2024-09-16T01:52:50.495126+0000","last_peered":"2024-09-16T01:52:50.495126+0000","last_clean":"2024-09-16T01:52:44.507567+0000","last_became_active":"2024-09-16T01:52:47.551851+0000","last_became_peered":"2024-09-16T01:52:47.551851+0000","last_unstale":"2024-09-16T01:52:50.495126+0000","last_undegraded":"2024-09-16T01:52:47.536446+0000","last_fullsized":"2024-09-16T01:52:47.535390+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":36},{"shards":"5","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"112'102","reported_seq":359,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500752+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:51.500752+0000","last_peered":"2024-09-16T01:52:51.500752+0000","last_clean":"2024-09-16T01:52:51.500752+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:51.500752+0000","last_undegraded":"2024-09-16T01:52:51.500752+0000","last_fullsized":"2024-09-16T01:52:51.500752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"112'125","reported_seq":504,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496948+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:51.496948+0000","last_peered":"2024-09-16T01:52:51.496948+0000","last_clean":"2024-09-16T01:52:51.496948+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:51.496948+0000","last_undegraded":"2024-09-16T01:52:51.496948+0000","last_fullsized":"2024-09-16T01:52:51.496948+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"113'131","reported_seq":458,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494852+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:50.494852+0000","last_peered":"2024-09-16T01:52:50.494852+0000","last_clean":"2024-09-16T01:52:50.494852+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:50.494852+0000","last_undegraded":"2024-09-16T01:52:50.494852+0000","last_fullsized":"2024-09-16T01:52:50.494852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"113'114","reported_seq":319,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.498045+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:51.498045+0000","last_peered":"2024-09-16T01:52:51.498045+0000","last_clean":"2024-09-16T01:52:51.498045+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:51.498045+0000","last_undegraded":"2024-09-16T01:52:51.498045+0000","last_fullsized":"2024-09-16T01:52:51.498045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"112'117","reported_seq":426,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502248+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:51.502248+0000","last_peered":"2024-09-16T01:52:51.502248+0000","last_clean":"2024-09-16T01:52:51.502248+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:51.502248+0000","last_undegraded":"2024-09-16T01:52:51.502248+0000","last_fullsized":"2024-09-16T01:52:51.502248+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"112'135","reported_seq":469,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503733+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:51.503733+0000","last_peered":"2024-09-16T01:52:51.503733+0000","last_clean":"2024-09-16T01:52:51.503733+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:51.503733+0000","last_undegraded":"2024-09-16T01:52:51.503733+0000","last_fullsized":"2024-09-16T01:52:51.503733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"114'107","reported_seq":232,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497368+0000","last_change":"2024-09-16T01:52:50.499570+0000","last_active":"2024-09-16T01:52:51.497368+0000","last_peered":"2024-09-16T01:52:51.497368+0000","last_clean":"2024-09-16T01:52:51.497368+0000","last_became_active":"2024-09-16T01:52:50.498997+0000","last_became_peered":"2024-09-16T01:52:50.498997+0000","last_unstale":"2024-09-16T01:52:51.497368+0000","last_undegraded":"2024-09-16T01:52:51.497368+0000","last_fullsized":"2024-09-16T01:52:51.497368+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"113'112","reported_seq":438,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493569+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:50.493569+0000","last_peered":"2024-09-16T01:52:50.493569+0000","last_clean":"2024-09-16T01:52:50.493569+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:50.493569+0000","last_undegraded":"2024-09-16T01:52:50.493569+0000","last_fullsized":"2024-09-16T01:52:50.493569+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"113'107","reported_seq":360,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492306+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:50.492306+0000","last_peered":"2024-09-16T01:52:50.492306+0000","last_clean":"2024-09-16T01:52:50.492306+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:50.492306+0000","last_undegraded":"2024-09-16T01:52:50.492306+0000","last_fullsized":"2024-09-16T01:52:50.492306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"113'110","reported_seq":415,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494790+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:50.494790+0000","last_peered":"2024-09-16T01:52:50.494790+0000","last_clean":"2024-09-16T01:52:50.494790+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:50.494790+0000","last_undegraded":"2024-09-16T01:52:50.494790+0000","last_fullsized":"2024-09-16T01:52:50.494790+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"111'107","reported_seq":311,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501551+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:51.501551+0000","last_peered":"2024-09-16T01:52:51.501551+0000","last_clean":"2024-09-16T01:52:51.501551+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:51.501551+0000","last_undegraded":"2024-09-16T01:52:51.501551+0000","last_fullsized":"2024-09-16T01:52:51.501551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"113'104","reported_seq":405,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.504164+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:51.504164+0000","last_peered":"2024-09-16T01:52:51.504164+0000","last_clean":"2024-09-16T01:52:51.504164+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:51.504164+0000","last_undegraded":"2024-09-16T01:52:51.504164+0000","last_fullsized":"2024-09-16T01:52:51.504164+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"114'109","reported_seq":506,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502460+0000","last_change":"2024-09-16T01:52:49.497350+0000","last_active":"2024-09-16T01:52:51.502460+0000","last_peered":"2024-09-16T01:52:51.502460+0000","last_clean":"2024-09-16T01:52:51.502460+0000","last_became_active":"2024-09-16T01:52:49.496908+0000","last_became_peered":"2024-09-16T01:52:49.496908+0000","last_unstale":"2024-09-16T01:52:51.502460+0000","last_undegraded":"2024-09-16T01:52:51.502460+0000","last_fullsized":"2024-09-16T01:52:51.502460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:51.945635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"114'124","reported_seq":560,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492538+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:52:50.492538+0000","last_peered":"2024-09-16T01:52:50.492538+0000","last_clean":"2024-09-16T01:52:50.492538+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:52:50.492538+0000","last_undegraded":"2024-09-16T01:52:50.492538+0000","last_fullsized":"2024-09-16T01:52:50.492538+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"112'114","reported_seq":462,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497504+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:51.497504+0000","last_peered":"2024-09-16T01:52:51.497504+0000","last_clean":"2024-09-16T01:52:51.497504+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:51.497504+0000","last_undegraded":"2024-09-16T01:52:51.497504+0000","last_fullsized":"2024-09-16T01:52:51.497504+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"113'95","reported_seq":337,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501366+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:51.501366+0000","last_peered":"2024-09-16T01:52:51.501366+0000","last_clean":"2024-09-16T01:52:51.501366+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:51.501366+0000","last_undegraded":"2024-09-16T01:52:51.501366+0000","last_fullsized":"2024-09-16T01:52:51.501366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"113'127","reported_seq":578,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497174+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:52:51.497174+0000","last_peered":"2024-09-16T01:52:51.497174+0000","last_clean":"2024-09-16T01:52:51.497174+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:52:51.497174+0000","last_undegraded":"2024-09-16T01:52:51.497174+0000","last_fullsized":"2024-09-16T01:52:51.497174+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":295,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494914+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:50.494914+0000","last_peered":"2024-09-16T01:52:50.494914+0000","last_clean":"2024-09-16T01:52:50.494914+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:50.494914+0000","last_undegraded":"2024-09-16T01:52:50.494914+0000","last_fullsized":"2024-09-16T01:52:50.494914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"111'132","reported_seq":336,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503002+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:51.503002+0000","last_peered":"2024-09-16T01:52:51.503002+0000","last_clean":"2024-09-16T01:52:51.503002+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:51.503002+0000","last_undegraded":"2024-09-16T01:52:51.503002+0000","last_fullsized":"2024-09-16T01:52:51.503002+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"113'104","reported_seq":306,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495482+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:50.495482+0000","last_peered":"2024-09-16T01:52:50.495482+0000","last_clean":"2024-09-16T01:52:50.495482+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:50.495482+0000","last_undegraded":"2024-09-16T01:52:50.495482+0000","last_fullsized":"2024-09-16T01:52:50.495482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"112'142","reported_seq":347,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502561+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:51.502561+0000","last_peered":"2024-09-16T01:52:51.502561+0000","last_clean":"2024-09-16T01:52:51.502561+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:51.502561+0000","last_undegraded":"2024-09-16T01:52:51.502561+0000","last_fullsized":"2024-09-16T01:52:51.502561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"114'105","reported_seq":538,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493440+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:52:50.493440+0000","last_peered":"2024-09-16T01:52:50.493440+0000","last_clean":"2024-09-16T01:52:50.493440+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:52:50.493440+0000","last_undegraded":"2024-09-16T01:52:50.493440+0000","last_fullsized":"2024-09-16T01:52:50.493440+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"113'100","reported_seq":396,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492476+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:50.492476+0000","last_peered":"2024-09-16T01:52:50.492476+0000","last_clean":"2024-09-16T01:52:50.492476+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:50.492476+0000","last_undegraded":"2024-09-16T01:52:50.492476+0000","last_fullsized":"2024-09-16T01:52:50.492476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"113'88","reported_seq":573,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495412+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:52:50.495412+0000","last_peered":"2024-09-16T01:52:50.495412+0000","last_clean":"2024-09-16T01:52:50.495412+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:52:50.495412+0000","last_undegraded":"2024-09-16T01:52:50.495412+0000","last_fullsized":"2024-09-16T01:52:50.495412+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"112'98","reported_seq":302,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501689+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:51.501689+0000","last_peered":"2024-09-16T01:52:51.501689+0000","last_clean":"2024-09-16T01:52:51.501689+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:51.501689+0000","last_undegraded":"2024-09-16T01:52:51.501689+0000","last_fullsized":"2024-09-16T01:52:51.501689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"113'102","reported_seq":499,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497066+0000","last_change":"2024-09-16T01:52:47.475996+0000","last_active":"2024-09-16T01:52:51.497066+0000","last_peered":"2024-09-16T01:52:51.497066+0000","last_clean":"2024-09-16T01:52:51.497066+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:52:51.497066+0000","last_undegraded":"2024-09-16T01:52:51.497066+0000","last_fullsized":"2024-09-16T01:52:51.497066+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:05.418025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"113'112","reported_seq":428,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492019+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:50.492019+0000","last_peered":"2024-09-16T01:52:50.492019+0000","last_clean":"2024-09-16T01:52:50.492019+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:50.492019+0000","last_undegraded":"2024-09-16T01:52:50.492019+0000","last_fullsized":"2024-09-16T01:52:50.492019+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"112'98","reported_seq":303,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497739+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:51.497739+0000","last_peered":"2024-09-16T01:52:51.497739+0000","last_clean":"2024-09-16T01:52:51.497739+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:51.497739+0000","last_undegraded":"2024-09-16T01:52:51.497739+0000","last_fullsized":"2024-09-16T01:52:51.497739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":209,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495509+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:50.495509+0000","last_peered":"2024-09-16T01:52:50.495509+0000","last_clean":"2024-09-16T01:52:50.495509+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:50.495509+0000","last_undegraded":"2024-09-16T01:52:50.495509+0000","last_fullsized":"2024-09-16T01:52:50.495509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"114'105","reported_seq":636,"reported_epoch":117,"state":"peering","last_fresh":"2024-09-16T01:52:50.626211+0000","last_change":"2024-09-16T01:52:51.501876+0000","last_active":"2024-09-16T01:52:51.501876+0000","last_peered":"2024-09-16T01:52:50.626211+0000","last_clean":"2024-09-16T01:52:44.543722+0000","last_became_active":"2024-09-16T01:52:47.551382+0000","last_became_peered":"2024-09-16T01:52:47.551382+0000","last_unstale":"2024-09-16T01:52:50.626211+0000","last_undegraded":"2024-09-16T01:52:50.626211+0000","last_fullsized":"2024-09-16T01:52:47.534787+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":226,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501889+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:51.501889+0000","last_peered":"2024-09-16T01:52:51.501889+0000","last_clean":"2024-09-16T01:52:51.501889+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:51.501889+0000","last_undegraded":"2024-09-16T01:52:51.501889+0000","last_fullsized":"2024-09-16T01:52:51.501889+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":230,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492167+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:52:50.492167+0000","last_peered":"2024-09-16T01:52:50.492167+0000","last_clean":"2024-09-16T01:52:50.492167+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:52:50.492167+0000","last_undegraded":"2024-09-16T01:52:50.492167+0000","last_fullsized":"2024-09-16T01:52:50.492167+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"112'122","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494972+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:50.494972+0000","last_peered":"2024-09-16T01:52:50.494972+0000","last_clean":"2024-09-16T01:52:50.494972+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:50.494972+0000","last_undegraded":"2024-09-16T01:52:50.494972+0000","last_fullsized":"2024-09-16T01:52:50.494972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":209,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503062+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:51.503062+0000","last_peered":"2024-09-16T01:52:51.503062+0000","last_clean":"2024-09-16T01:52:51.503062+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:51.503062+0000","last_undegraded":"2024-09-16T01:52:51.503062+0000","last_fullsized":"2024-09-16T01:52:51.503062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"112'114","reported_seq":284,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496946+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:51.496946+0000","last_peered":"2024-09-16T01:52:51.496946+0000","last_clean":"2024-09-16T01:52:51.496946+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:51.496946+0000","last_undegraded":"2024-09-16T01:52:51.496946+0000","last_fullsized":"2024-09-16T01:52:51.496946+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":238,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.498045+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:51.498045+0000","last_peered":"2024-09-16T01:52:51.498045+0000","last_clean":"2024-09-16T01:52:51.498045+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:51.498045+0000","last_undegraded":"2024-09-16T01:52:51.498045+0000","last_fullsized":"2024-09-16T01:52:51.498045+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"113'167","reported_seq":383,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495029+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:50.495029+0000","last_peered":"2024-09-16T01:52:50.495029+0000","last_clean":"2024-09-16T01:52:50.495029+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:50.495029+0000","last_undegraded":"2024-09-16T01:52:50.495029+0000","last_fullsized":"2024-09-16T01:52:50.495029+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.498047+0000","last_change":"2024-09-16T01:52:46.621488+0000","last_active":"2024-09-16T01:52:51.498047+0000","last_peered":"2024-09-16T01:52:51.498047+0000","last_clean":"2024-09-16T01:52:51.498047+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:52:51.498047+0000","last_undegraded":"2024-09-16T01:52:51.498047+0000","last_fullsized":"2024-09-16T01:52:51.498047+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"113'101","reported_seq":303,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495598+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:50.495598+0000","last_peered":"2024-09-16T01:52:50.495598+0000","last_clean":"2024-09-16T01:52:50.495598+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:50.495598+0000","last_undegraded":"2024-09-16T01:52:50.495598+0000","last_fullsized":"2024-09-16T01:52:50.495598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502831+0000","last_change":"2024-09-16T01:52:46.632501+0000","last_active":"2024-09-16T01:52:51.502831+0000","last_peered":"2024-09-16T01:52:51.502831+0000","last_clean":"2024-09-16T01:52:51.502831+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:52:51.502831+0000","last_undegraded":"2024-09-16T01:52:51.502831+0000","last_fullsized":"2024-09-16T01:52:51.502831+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"112'107","reported_seq":311,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502768+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:51.502768+0000","last_peered":"2024-09-16T01:52:51.502768+0000","last_clean":"2024-09-16T01:52:51.502768+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:51.502768+0000","last_undegraded":"2024-09-16T01:52:51.502768+0000","last_fullsized":"2024-09-16T01:52:51.502768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":396,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500717+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:51.500717+0000","last_peered":"2024-09-16T01:52:51.500717+0000","last_clean":"2024-09-16T01:52:51.500717+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:51.500717+0000","last_undegraded":"2024-09-16T01:52:51.500717+0000","last_fullsized":"2024-09-16T01:52:51.500717+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":417,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493346+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:50.493346+0000","last_peered":"2024-09-16T01:52:50.493346+0000","last_clean":"2024-09-16T01:52:50.493346+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:50.493346+0000","last_undegraded":"2024-09-16T01:52:50.493346+0000","last_fullsized":"2024-09-16T01:52:50.493346+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":285,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502300+0000","last_change":"2024-09-16T01:52:46.615925+0000","last_active":"2024-09-16T01:52:51.502300+0000","last_peered":"2024-09-16T01:52:51.502300+0000","last_clean":"2024-09-16T01:52:51.502300+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:52:51.502300+0000","last_undegraded":"2024-09-16T01:52:51.502300+0000","last_fullsized":"2024-09-16T01:52:51.502300+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":340,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"112'117","reported_seq":320,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492401+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:50.492401+0000","last_peered":"2024-09-16T01:52:50.492401+0000","last_clean":"2024-09-16T01:52:50.492401+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:50.492401+0000","last_undegraded":"2024-09-16T01:52:50.492401+0000","last_fullsized":"2024-09-16T01:52:50.492401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":289,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501983+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:52:51.501983+0000","last_peered":"2024-09-16T01:52:51.501983+0000","last_clean":"2024-09-16T01:52:51.501983+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:52:51.501983+0000","last_undegraded":"2024-09-16T01:52:51.501983+0000","last_fullsized":"2024-09-16T01:52:51.501983+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"112'121","reported_seq":456,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502402+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:51.502402+0000","last_peered":"2024-09-16T01:52:51.502402+0000","last_clean":"2024-09-16T01:52:51.502402+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:51.502402+0000","last_undegraded":"2024-09-16T01:52:51.502402+0000","last_fullsized":"2024-09-16T01:52:51.502402+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501563+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:51.501563+0000","last_peered":"2024-09-16T01:52:51.501563+0000","last_clean":"2024-09-16T01:52:51.501563+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:51.501563+0000","last_undegraded":"2024-09-16T01:52:51.501563+0000","last_fullsized":"2024-09-16T01:52:51.501563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"113'133","reported_seq":338,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496970+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:51.496970+0000","last_peered":"2024-09-16T01:52:51.496970+0000","last_clean":"2024-09-16T01:52:51.496970+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:51.496970+0000","last_undegraded":"2024-09-16T01:52:51.496970+0000","last_fullsized":"2024-09-16T01:52:51.496970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":142,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496880+0000","last_change":"2024-09-16T01:52:46.599623+0000","last_active":"2024-09-16T01:52:51.496880+0000","last_peered":"2024-09-16T01:52:51.496880+0000","last_clean":"2024-09-16T01:52:51.496880+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:52:51.496880+0000","last_undegraded":"2024-09-16T01:52:51.496880+0000","last_fullsized":"2024-09-16T01:52:51.496880+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:22.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":784,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"113'145","reported_seq":349,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501711+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:51.501711+0000","last_peered":"2024-09-16T01:52:51.501711+0000","last_clean":"2024-09-16T01:52:51.501711+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:51.501711+0000","last_undegraded":"2024-09-16T01:52:51.501711+0000","last_fullsized":"2024-09-16T01:52:51.501711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":77,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495388+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:50.495388+0000","last_peered":"2024-09-16T01:52:50.495388+0000","last_clean":"2024-09-16T01:52:50.495388+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:50.495388+0000","last_undegraded":"2024-09-16T01:52:50.495388+0000","last_fullsized":"2024-09-16T01:52:50.495388+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"112'106","reported_seq":412,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502668+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:51.502668+0000","last_peered":"2024-09-16T01:52:51.502668+0000","last_clean":"2024-09-16T01:52:51.502668+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:51.502668+0000","last_undegraded":"2024-09-16T01:52:51.502668+0000","last_fullsized":"2024-09-16T01:52:51.502668+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495361+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:50.495361+0000","last_peered":"2024-09-16T01:52:50.495361+0000","last_clean":"2024-09-16T01:52:50.495361+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:50.495361+0000","last_undegraded":"2024-09-16T01:52:50.495361+0000","last_fullsized":"2024-09-16T01:52:50.495361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"112'103","reported_seq":404,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497913+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:51.497913+0000","last_peered":"2024-09-16T01:52:51.497913+0000","last_clean":"2024-09-16T01:52:51.497913+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:51.497913+0000","last_undegraded":"2024-09-16T01:52:51.497913+0000","last_fullsized":"2024-09-16T01:52:51.497913+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":207,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495615+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:50.495615+0000","last_peered":"2024-09-16T01:52:50.495615+0000","last_clean":"2024-09-16T01:52:50.495615+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:50.495615+0000","last_undegraded":"2024-09-16T01:52:50.495615+0000","last_fullsized":"2024-09-16T01:52:50.495615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"114'102","reported_seq":634,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501795+0000","last_change":"2024-09-16T01:52:50.499005+0000","last_active":"2024-09-16T01:52:51.501795+0000","last_peered":"2024-09-16T01:52:51.501795+0000","last_clean":"2024-09-16T01:52:51.501795+0000","last_became_active":"2024-09-16T01:52:50.498634+0000","last_became_peered":"2024-09-16T01:52:50.498634+0000","last_unstale":"2024-09-16T01:52:51.501795+0000","last_undegraded":"2024-09-16T01:52:51.501795+0000","last_fullsized":"2024-09-16T01:52:51.501795+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":223,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492672+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:50.492672+0000","last_peered":"2024-09-16T01:52:50.492672+0000","last_clean":"2024-09-16T01:52:50.492672+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:50.492672+0000","last_undegraded":"2024-09-16T01:52:50.492672+0000","last_fullsized":"2024-09-16T01:52:50.492672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"112'112","reported_seq":316,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503064+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:51.503064+0000","last_peered":"2024-09-16T01:52:51.503064+0000","last_clean":"2024-09-16T01:52:51.503064+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:51.503064+0000","last_undegraded":"2024-09-16T01:52:51.503064+0000","last_fullsized":"2024-09-16T01:52:51.503064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":364,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497043+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:51.497043+0000","last_peered":"2024-09-16T01:52:51.497043+0000","last_clean":"2024-09-16T01:52:51.497043+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:51.497043+0000","last_undegraded":"2024-09-16T01:52:51.497043+0000","last_fullsized":"2024-09-16T01:52:51.497043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":212,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502584+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:51.502584+0000","last_peered":"2024-09-16T01:52:51.502584+0000","last_clean":"2024-09-16T01:52:51.502584+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:51.502584+0000","last_undegraded":"2024-09-16T01:52:51.502584+0000","last_fullsized":"2024-09-16T01:52:51.502584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"113'112","reported_seq":457,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500657+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:51.500657+0000","last_peered":"2024-09-16T01:52:51.500657+0000","last_clean":"2024-09-16T01:52:51.500657+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:51.500657+0000","last_undegraded":"2024-09-16T01:52:51.500657+0000","last_fullsized":"2024-09-16T01:52:51.500657+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":210,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503111+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:51.503111+0000","last_peered":"2024-09-16T01:52:51.503111+0000","last_clean":"2024-09-16T01:52:51.503111+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:51.503111+0000","last_undegraded":"2024-09-16T01:52:51.503111+0000","last_fullsized":"2024-09-16T01:52:51.503111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"113'107","reported_seq":310,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492106+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:50.492106+0000","last_peered":"2024-09-16T01:52:50.492106+0000","last_clean":"2024-09-16T01:52:50.492106+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:50.492106+0000","last_undegraded":"2024-09-16T01:52:50.492106+0000","last_fullsized":"2024-09-16T01:52:50.492106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":265,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494812+0000","last_change":"2024-09-16T01:52:46.569921+0000","last_active":"2024-09-16T01:52:50.494812+0000","last_peered":"2024-09-16T01:52:50.494812+0000","last_clean":"2024-09-16T01:52:50.494812+0000","last_became_active":"2024-09-16T01:52:46.544102+0000","last_became_peered":"2024-09-16T01:52:46.544102+0000","last_unstale":"2024-09-16T01:52:50.494812+0000","last_undegraded":"2024-09-16T01:52:50.494812+0000","last_fullsized":"2024-09-16T01:52:50.494812+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:14.446115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"113'96","reported_seq":515,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495047+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:52:50.495047+0000","last_peered":"2024-09-16T01:52:50.495047+0000","last_clean":"2024-09-16T01:52:50.495047+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:52:50.495047+0000","last_undegraded":"2024-09-16T01:52:50.495047+0000","last_fullsized":"2024-09-16T01:52:50.495047+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":211,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502105+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:51.502105+0000","last_peered":"2024-09-16T01:52:51.502105+0000","last_clean":"2024-09-16T01:52:51.502105+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:51.502105+0000","last_undegraded":"2024-09-16T01:52:51.502105+0000","last_fullsized":"2024-09-16T01:52:51.502105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"113'111","reported_seq":415,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500536+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:51.500536+0000","last_peered":"2024-09-16T01:52:51.500536+0000","last_clean":"2024-09-16T01:52:51.500536+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:51.500536+0000","last_undegraded":"2024-09-16T01:52:51.500536+0000","last_fullsized":"2024-09-16T01:52:51.500536+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":205,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495040+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:50.495040+0000","last_peered":"2024-09-16T01:52:50.495040+0000","last_clean":"2024-09-16T01:52:50.495040+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:50.495040+0000","last_undegraded":"2024-09-16T01:52:50.495040+0000","last_fullsized":"2024-09-16T01:52:50.495040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"113'110","reported_seq":315,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501011+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:51.501011+0000","last_peered":"2024-09-16T01:52:51.501011+0000","last_clean":"2024-09-16T01:52:51.501011+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:51.501011+0000","last_undegraded":"2024-09-16T01:52:51.501011+0000","last_fullsized":"2024-09-16T01:52:51.501011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500909+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:52:51.500909+0000","last_peered":"2024-09-16T01:52:51.500909+0000","last_clean":"2024-09-16T01:52:51.500909+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:52:51.500909+0000","last_undegraded":"2024-09-16T01:52:51.500909+0000","last_fullsized":"2024-09-16T01:52:51.500909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"113'103","reported_seq":306,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492575+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:50.492575+0000","last_peered":"2024-09-16T01:52:50.492575+0000","last_clean":"2024-09-16T01:52:50.492575+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:50.492575+0000","last_undegraded":"2024-09-16T01:52:50.492575+0000","last_fullsized":"2024-09-16T01:52:50.492575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":203,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495195+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:50.495195+0000","last_peered":"2024-09-16T01:52:50.495195+0000","last_clean":"2024-09-16T01:52:50.495195+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:50.495195+0000","last_undegraded":"2024-09-16T01:52:50.495195+0000","last_fullsized":"2024-09-16T01:52:50.495195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"113'114","reported_seq":362,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493268+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:50.493268+0000","last_peered":"2024-09-16T01:52:50.493268+0000","last_clean":"2024-09-16T01:52:50.493268+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:50.493268+0000","last_undegraded":"2024-09-16T01:52:50.493268+0000","last_fullsized":"2024-09-16T01:52:50.493268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":228,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.504195+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:51.504195+0000","last_peered":"2024-09-16T01:52:51.504195+0000","last_clean":"2024-09-16T01:52:51.504195+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:51.504195+0000","last_undegraded":"2024-09-16T01:52:51.504195+0000","last_fullsized":"2024-09-16T01:52:51.504195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"114'116","reported_seq":405,"reported_epoch":117,"state":"peering","last_fresh":"2024-09-16T01:52:51.250680+0000","last_change":"2024-09-16T01:52:51.498002+0000","last_active":"2024-09-16T01:52:51.498002+0000","last_peered":"2024-09-16T01:52:51.250680+0000","last_clean":"2024-09-16T01:52:44.870429+0000","last_became_active":"2024-09-16T01:52:47.547843+0000","last_became_peered":"2024-09-16T01:52:47.547843+0000","last_unstale":"2024-09-16T01:52:51.250680+0000","last_undegraded":"2024-09-16T01:52:51.250680+0000","last_fullsized":"2024-09-16T01:52:47.537632+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500451+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:52:51.500451+0000","last_peered":"2024-09-16T01:52:51.500451+0000","last_clean":"2024-09-16T01:52:51.500451+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:52:51.500451+0000","last_undegraded":"2024-09-16T01:52:51.500451+0000","last_fullsized":"2024-09-16T01:52:51.500451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"112'114","reported_seq":373,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501828+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:51.501828+0000","last_peered":"2024-09-16T01:52:51.501828+0000","last_clean":"2024-09-16T01:52:51.501828+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:51.501828+0000","last_undegraded":"2024-09-16T01:52:51.501828+0000","last_fullsized":"2024-09-16T01:52:51.501828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492054+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:50.492054+0000","last_peered":"2024-09-16T01:52:50.492054+0000","last_clean":"2024-09-16T01:52:50.492054+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:50.492054+0000","last_undegraded":"2024-09-16T01:52:50.492054+0000","last_fullsized":"2024-09-16T01:52:50.492054+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"113'116","reported_seq":376,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500330+0000","last_change":"2024-09-16T01:52:46.782699+0000","last_active":"2024-09-16T01:52:51.500330+0000","last_peered":"2024-09-16T01:52:51.500330+0000","last_clean":"2024-09-16T01:52:51.500330+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:52:51.500330+0000","last_undegraded":"2024-09-16T01:52:51.500330+0000","last_fullsized":"2024-09-16T01:52:51.500330+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:30.580835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":208,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492505+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:50.492505+0000","last_peered":"2024-09-16T01:52:50.492505+0000","last_clean":"2024-09-16T01:52:50.492505+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:50.492505+0000","last_undegraded":"2024-09-16T01:52:50.492505+0000","last_fullsized":"2024-09-16T01:52:50.492505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"114'105","reported_seq":512,"reported_epoch":116,"state":"peering","last_fresh":"2024-09-16T01:52:50.241765+0000","last_change":"2024-09-16T01:52:50.492781+0000","last_active":"2024-09-16T01:52:50.492781+0000","last_peered":"2024-09-16T01:52:50.241765+0000","last_clean":"2024-09-16T01:52:44.861424+0000","last_became_active":"2024-09-16T01:52:47.546951+0000","last_became_peered":"2024-09-16T01:52:47.546951+0000","last_unstale":"2024-09-16T01:52:50.241765+0000","last_undegraded":"2024-09-16T01:52:50.241765+0000","last_fullsized":"2024-09-16T01:52:47.532896+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":211,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502491+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:51.502491+0000","last_peered":"2024-09-16T01:52:51.502491+0000","last_clean":"2024-09-16T01:52:51.502491+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:51.502491+0000","last_undegraded":"2024-09-16T01:52:51.502491+0000","last_fullsized":"2024-09-16T01:52:51.502491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"111'94","reported_seq":298,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501786+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:51.501786+0000","last_peered":"2024-09-16T01:52:51.501786+0000","last_clean":"2024-09-16T01:52:51.501786+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:51.501786+0000","last_undegraded":"2024-09-16T01:52:51.501786+0000","last_fullsized":"2024-09-16T01:52:51.501786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"112'131","reported_seq":454,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502967+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:51.502967+0000","last_peered":"2024-09-16T01:52:51.502967+0000","last_clean":"2024-09-16T01:52:51.502967+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:51.502967+0000","last_undegraded":"2024-09-16T01:52:51.502967+0000","last_fullsized":"2024-09-16T01:52:51.502967+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497567+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:51.497567+0000","last_peered":"2024-09-16T01:52:51.497567+0000","last_clean":"2024-09-16T01:52:51.497567+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:51.497567+0000","last_undegraded":"2024-09-16T01:52:51.497567+0000","last_fullsized":"2024-09-16T01:52:51.497567+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"111'106","reported_seq":406,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497668+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:51.497668+0000","last_peered":"2024-09-16T01:52:51.497668+0000","last_clean":"2024-09-16T01:52:51.497668+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:51.497668+0000","last_undegraded":"2024-09-16T01:52:51.497668+0000","last_fullsized":"2024-09-16T01:52:51.497668+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":48,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497378+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:51.497378+0000","last_peered":"2024-09-16T01:52:51.497378+0000","last_clean":"2024-09-16T01:52:51.497378+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:51.497378+0000","last_undegraded":"2024-09-16T01:52:51.497378+0000","last_fullsized":"2024-09-16T01:52:51.497378+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"113'97","reported_seq":299,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495490+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:50.495490+0000","last_peered":"2024-09-16T01:52:50.495490+0000","last_clean":"2024-09-16T01:52:50.495490+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:50.495490+0000","last_undegraded":"2024-09-16T01:52:50.495490+0000","last_fullsized":"2024-09-16T01:52:50.495490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":236,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495382+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:52:50.495382+0000","last_peered":"2024-09-16T01:52:50.495382+0000","last_clean":"2024-09-16T01:52:50.495382+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:52:50.495382+0000","last_undegraded":"2024-09-16T01:52:50.495382+0000","last_fullsized":"2024-09-16T01:52:50.495382+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"112'115","reported_seq":317,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495288+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:50.495288+0000","last_peered":"2024-09-16T01:52:50.495288+0000","last_clean":"2024-09-16T01:52:50.495288+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:50.495288+0000","last_undegraded":"2024-09-16T01:52:50.495288+0000","last_fullsized":"2024-09-16T01:52:50.495288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.504067+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:51.504067+0000","last_peered":"2024-09-16T01:52:51.504067+0000","last_clean":"2024-09-16T01:52:51.504067+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:51.504067+0000","last_undegraded":"2024-09-16T01:52:51.504067+0000","last_fullsized":"2024-09-16T01:52:51.504067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2936,"num_bytes_recovered":9722396672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1408827392,"data_stored":22541238272,"data_compressed":1124772320,"data_compressed_allocated":1408827392,"data_compressed_original":22541238272,"omap_allocated":0,"internal_metadata":0},"log_size":7171,"ondisk_log_size":7171,"up":128,"acting":127,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":248325,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118208,"data_stored":995820,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509493,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333660,"kb_used_data":302468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94038180,"statfs":{"total":96636764160,"available":96295096320,"internally_reserved":0,"allocated":309727232,"data_stored":4933159299,"data_compressed":246680448,"data_compressed_allocated":308281344,"data_compressed_original":4932501504,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509493,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300472,"kb_used_data":269280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94071368,"statfs":{"total":96636764160,"available":96329080832,"internally_reserved":0,"allocated":275742720,"data_stored":4384079069,"data_compressed":219220992,"data_compressed_allocated":273940480,"data_compressed_original":4383047680,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509492,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240132,"kb_used_data":213100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94131708,"statfs":{"total":96636764160,"available":96390868992,"internally_reserved":0,"allocated":218214400,"data_stored":3466116710,"data_compressed":173345635,"data_compressed_allocated":216621056,"data_compressed_original":3465302016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509493,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542196,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148800,"kb_used_data":121768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94223040,"statfs":{"total":96636764160,"available":96484392960,"internally_reserved":0,"allocated":124690432,"data_stored":1973020073,"data_compressed":98654688,"data_compressed_allocated":123273216,"data_compressed_original":1972371456,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542196,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255748,"kb_used_data":212076,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94116092,"statfs":{"total":96636764160,"available":96374878208,"internally_reserved":0,"allocated":217165824,"data_stored":3452562337,"data_compressed":172647168,"data_compressed_allocated":215744512,"data_compressed_original":3451912192,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542196,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265480,"kb_used_data":225968,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94106360,"statfs":{"total":96636764160,"available":96364912640,"internally_reserved":0,"allocated":231391232,"data_stored":3679645297,"data_compressed":184008356,"data_compressed_allocated":229945344,"data_compressed_original":3678978048,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574901,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":377634,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":377785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196345856,"data_stored":3141533696,"data_compressed":157134208,"data_compressed_allocated":196345856,"data_compressed_original":3141533696,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":273940480,"data_stored":4383047680,"data_compressed":219220992,"data_compressed_allocated":273940480,"data_compressed_original":4383047680,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308281344,"data_stored":4932501504,"data_compressed":246680448,"data_compressed_allocated":308281344,"data_compressed_original":4932501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":3678404608,"data_compressed":183975936,"data_compressed_allocated":229900288,"data_compressed_original":3678404608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":205520896,"data_stored":3288334336,"data_compressed":164476928,"data_compressed_allocated":205520896,"data_compressed_original":3288334336,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":135331840,"data_stored":2165309440,"data_compressed":108305120,"data_compressed_allocated":135331840,"data_compressed_original":2165309440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:54.801 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:54.895 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:54.893+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:54.897 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:54.897 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:54.906 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:54.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.909+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f060406da90 con 0x7f06040c75a0 2024-09-16T01:52:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.990+0000 7f0ad7200640 1 Processor -- start 2024-09-16T01:52:54.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.990+0000 7f0ad7200640 1 -- start start 2024-09-16T01:52:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0ad7200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0ad0071b30 0x7f0ad00c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0ad7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad01578a0 con 0x7f0ad00c98d0 2024-09-16T01:52:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0ad7200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ad0072620 con 0x7f0ad0150bf0 2024-09-16T01:52:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0ad7200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0ad00c7a80 con 0x7f0ad0071b30 2024-09-16T01:52:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0acf400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0ad00c98d0 0x7f0ad0071f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60326/0 (socket says 172.21.15.146:60326) 2024-09-16T01:52:54.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0acf400640 1 -- 172.21.15.146:0/397458575 learned_addr learned my addr 172.21.15.146:0/397458575 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.991+0000 7f0acfe00640 1 --2- 172.21.15.146:0/397458575 >> v2:172.21.15.161:3300/0 conn(0x7f0ad0071b30 0x7f0ad00c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.992+0000 7f0acfe00640 1 -- 172.21.15.146:0/397458575 >> v1:172.21.15.161:6789/0 conn(0x7f0ad0150bf0 legacy=0x7f0ad00c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.992+0000 7f0acea00640 1 -- 172.21.15.146:0/397458575 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3497474998 0 0) 0x7f0ad01578a0 con 0x7f0ad00c98d0 2024-09-16T01:52:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.992+0000 7f0acfe00640 1 -- 172.21.15.146:0/397458575 >> v1:172.21.15.146:6789/0 conn(0x7f0ad00c98d0 legacy=0x7f0ad0071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.992+0000 7f0acfe00640 1 -- 172.21.15.146:0/397458575 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad0072f30 con 0x7f0ad0071b30 2024-09-16T01:52:54.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0acfe00640 1 --2- 172.21.15.146:0/397458575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad00c6dc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000aea0 tx=0x7f0ac002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6018ea0486f59d9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0acea00640 1 -- 172.21.15.146:0/397458575 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0037070 con 0x7f0ad0071b30 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0acea00640 1 -- 172.21.15.146:0/397458575 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac0008640 con 0x7f0ad0071b30 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0acea00640 1 -- 172.21.15.146:0/397458575 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0008a10 con 0x7f0ad0071b30 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0ad7200640 1 -- 172.21.15.146:0/397458575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 msgr2=0x7f0ad00c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0ad7200640 1 --2- 172.21.15.146:0/397458575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad00c6dc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000aea0 tx=0x7f0ac002b7f0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0ad7200640 1 -- 172.21.15.146:0/397458575 shutdown_connections 2024-09-16T01:52:54.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.993+0000 7f0ad7200640 1 --2- 172.21.15.146:0/397458575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad00c6dc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:54.996 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:54.994+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.994+0000 7f0ad7200640 1 -- 172.21.15.146:0/397458575 >> 172.21.15.146:0/397458575 conn(0x7f0ad006f3c0 msgr2=0x7f0ad006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.994+0000 7f0ad7200640 1 -- 172.21.15.146:0/397458575 shutdown_connections 2024-09-16T01:52:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.994+0000 7f0ad7200640 1 -- 172.21.15.146:0/397458575 wait complete. 2024-09-16T01:52:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.995+0000 7f0ad7200640 1 Processor -- start 2024-09-16T01:52:54.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.995+0000 7f0ad7200640 1 -- start start 2024-09-16T01:52:54.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad7200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad01f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad7200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00c98d0 0x7f0ad014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0150bf0 0x7f0ad00c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:54.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad00c8620 con 0x7f0ad0150bf0 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad7200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad00731f0 con 0x7f0ad0071b30 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad7200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad00c87a0 con 0x7f0ad00c98d0 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad01f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0acfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad01f0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43402/0 (socket says 172.21.15.146:43402) 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0acfe00640 1 -- 172.21.15.146:0/762867406 learned_addr learned my addr 172.21.15.146:0/762867406 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0acf400640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00c98d0 0x7f0ad014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.996+0000 7f0ad5200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0150bf0 0x7f0ad00c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0acfe00640 1 -- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00c98d0 msgr2=0x7f0ad014f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0acfe00640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00c98d0 0x7f0ad014f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0acfe00640 1 -- 172.21.15.146:0/762867406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0150bf0 msgr2=0x7f0ad00c25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0acfe00640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0150bf0 0x7f0ad00c25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0acfe00640 1 -- 172.21.15.146:0/762867406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad0156840 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0acfe00640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad01f0a30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0035040 tx=0x7f0ac0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0037050 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac0043070 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac000a560 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad0156b20 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.997+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad01f0510 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.999+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad01f0020 con 0x7f0ad0071b30 2024-09-16T01:52:55.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:54.999+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0ac0032c80 con 0x7f0ad0071b30 2024-09-16T01:52:55.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.000+0000 7f0ac7e00640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ab006c2e0 0x7f0ab006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:55.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.000+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 5894+0+0 (secure 0 0 0) 0x7f0ac00ac570 con 0x7f0ad0071b30 2024-09-16T01:52:55.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.001+0000 7f0acf400640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ab006c2e0 0x7f0ab006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:55.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.001+0000 7f0acf400640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ab006c2e0 0x7f0ab006e7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ad014fce0 tx=0x7f0abc007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:55.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.005+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0ac007a710 con 0x7f0ad0071b30 2024-09-16T01:52:55.011 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.011 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:55.097 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:55.096+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:55.123 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.123 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.131 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:52:55.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.177+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ad0155e60 con 0x7f0ab006c2e0 2024-09-16T01:52:55.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.184+0000 7f0ac7e00640 1 -- 172.21.15.146:0/762867406 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254836 (secure 0 0 0) 0x7f0abc004c70 con 0x7f0ab006c2e0 2024-09-16T01:52:55.189 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:55.190 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:55.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ab006c2e0 msgr2=0x7f0ab006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ab006c2e0 0x7f0ab006e7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ad014fce0 tx=0x7f0abc007e40 comp rx=0 tx=0).stop 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 msgr2=0x7f0ad01f0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad01f0a30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0035040 tx=0x7f0ac0004a40 comp rx=0 tx=0).stop 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 shutdown_connections 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0ab006c2e0 0x7f0ab006e7a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0150bf0 0x7f0ad00c25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00c98d0 0x7f0ad014f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 --2- 172.21.15.146:0/762867406 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0071b30 0x7f0ad01f0a30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 >> 172.21.15.146:0/762867406 conn(0x7f0ad006f3c0 msgr2=0x7f0ad0071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 shutdown_connections 2024-09-16T01:52:55.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.188+0000 7f0ad7200640 1 -- 172.21.15.146:0/762867406 wait complete. 2024-09-16T01:52:55.199 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:55.198+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:55.206 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-16T01:52:54.564909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":7207,"num_write_kb":25168808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3154,"num_bytes_recovered":9722644997,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7648,"ondisk_log_size":7648,"up":194,"acting":193,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":205,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1544292,"kb_used_data":1344660,"kb_used_omap":9,"kb_used_meta":199478,"kb_avail":564686748,"statfs":{"total":579820584960,"available":578239229952,"internally_reserved":0,"allocated":1376931840,"data_stored":21888582785,"data_compressed":1094557287,"data_compressed_allocated":1367805952,"data_compressed_original":21884112896,"omap_allocated":9562,"internal_metadata":204266150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-6,"num_objects_missing":-6,"num_objects_degraded":-2,"num_objects_misplaced":-12,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":1950352582,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.074358"},"pg_stats":[{"pgid":"3.3f","version":"112'104","reported_seq":388,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503950+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:51.503950+0000","last_peered":"2024-09-16T01:52:51.503950+0000","last_clean":"2024-09-16T01:52:51.503950+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:51.503950+0000","last_undegraded":"2024-09-16T01:52:51.503950+0000","last_fullsized":"2024-09-16T01:52:51.503950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"113'94","reported_seq":296,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495254+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:50.495254+0000","last_peered":"2024-09-16T01:52:50.495254+0000","last_clean":"2024-09-16T01:52:50.495254+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:50.495254+0000","last_undegraded":"2024-09-16T01:52:50.495254+0000","last_fullsized":"2024-09-16T01:52:50.495254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":555,"reported_epoch":116,"state":"peering","last_fresh":"2024-09-16T01:52:49.890306+0000","last_change":"2024-09-16T01:52:50.495187+0000","last_active":"2024-09-16T01:52:50.495187+0000","last_peered":"2024-09-16T01:52:49.890306+0000","last_clean":"2024-09-16T01:52:45.091998+0000","last_became_active":"2024-09-16T01:52:47.550146+0000","last_became_peered":"2024-09-16T01:52:47.550146+0000","last_unstale":"2024-09-16T01:52:49.890306+0000","last_undegraded":"2024-09-16T01:52:49.890306+0000","last_fullsized":"2024-09-16T01:52:47.535688+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"113'125","reported_seq":335,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500613+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:51.500613+0000","last_peered":"2024-09-16T01:52:51.500613+0000","last_clean":"2024-09-16T01:52:51.500613+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:51.500613+0000","last_undegraded":"2024-09-16T01:52:51.500613+0000","last_fullsized":"2024-09-16T01:52:51.500613+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"114'112","reported_seq":288,"reported_epoch":117,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T01:52:50.495126+0000","last_change":"2024-09-16T01:52:47.557414+0000","last_active":"2024-09-16T01:52:50.495126+0000","last_peered":"2024-09-16T01:52:50.495126+0000","last_clean":"2024-09-16T01:52:44.507567+0000","last_became_active":"2024-09-16T01:52:47.551851+0000","last_became_peered":"2024-09-16T01:52:47.551851+0000","last_unstale":"2024-09-16T01:52:50.495126+0000","last_undegraded":"2024-09-16T01:52:47.536446+0000","last_fullsized":"2024-09-16T01:52:47.535390+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":36},{"shards":"5","objects":3}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"112'102","reported_seq":359,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500752+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:51.500752+0000","last_peered":"2024-09-16T01:52:51.500752+0000","last_clean":"2024-09-16T01:52:51.500752+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:51.500752+0000","last_undegraded":"2024-09-16T01:52:51.500752+0000","last_fullsized":"2024-09-16T01:52:51.500752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"112'125","reported_seq":504,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496948+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:51.496948+0000","last_peered":"2024-09-16T01:52:51.496948+0000","last_clean":"2024-09-16T01:52:51.496948+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:51.496948+0000","last_undegraded":"2024-09-16T01:52:51.496948+0000","last_fullsized":"2024-09-16T01:52:51.496948+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"113'131","reported_seq":458,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494852+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:50.494852+0000","last_peered":"2024-09-16T01:52:50.494852+0000","last_clean":"2024-09-16T01:52:50.494852+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:50.494852+0000","last_undegraded":"2024-09-16T01:52:50.494852+0000","last_fullsized":"2024-09-16T01:52:50.494852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"113'114","reported_seq":319,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.498045+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:51.498045+0000","last_peered":"2024-09-16T01:52:51.498045+0000","last_clean":"2024-09-16T01:52:51.498045+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:51.498045+0000","last_undegraded":"2024-09-16T01:52:51.498045+0000","last_fullsized":"2024-09-16T01:52:51.498045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"112'117","reported_seq":426,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502248+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:51.502248+0000","last_peered":"2024-09-16T01:52:51.502248+0000","last_clean":"2024-09-16T01:52:51.502248+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:51.502248+0000","last_undegraded":"2024-09-16T01:52:51.502248+0000","last_fullsized":"2024-09-16T01:52:51.502248+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"112'135","reported_seq":469,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503733+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:51.503733+0000","last_peered":"2024-09-16T01:52:51.503733+0000","last_clean":"2024-09-16T01:52:51.503733+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:51.503733+0000","last_undegraded":"2024-09-16T01:52:51.503733+0000","last_fullsized":"2024-09-16T01:52:51.503733+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"114'107","reported_seq":232,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497368+0000","last_change":"2024-09-16T01:52:50.499570+0000","last_active":"2024-09-16T01:52:51.497368+0000","last_peered":"2024-09-16T01:52:51.497368+0000","last_clean":"2024-09-16T01:52:51.497368+0000","last_became_active":"2024-09-16T01:52:50.498997+0000","last_became_peered":"2024-09-16T01:52:50.498997+0000","last_unstale":"2024-09-16T01:52:51.497368+0000","last_undegraded":"2024-09-16T01:52:51.497368+0000","last_fullsized":"2024-09-16T01:52:51.497368+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"113'112","reported_seq":438,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493569+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:50.493569+0000","last_peered":"2024-09-16T01:52:50.493569+0000","last_clean":"2024-09-16T01:52:50.493569+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:50.493569+0000","last_undegraded":"2024-09-16T01:52:50.493569+0000","last_fullsized":"2024-09-16T01:52:50.493569+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"113'107","reported_seq":360,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492306+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:50.492306+0000","last_peered":"2024-09-16T01:52:50.492306+0000","last_clean":"2024-09-16T01:52:50.492306+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:50.492306+0000","last_undegraded":"2024-09-16T01:52:50.492306+0000","last_fullsized":"2024-09-16T01:52:50.492306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"113'110","reported_seq":415,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494790+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:50.494790+0000","last_peered":"2024-09-16T01:52:50.494790+0000","last_clean":"2024-09-16T01:52:50.494790+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:50.494790+0000","last_undegraded":"2024-09-16T01:52:50.494790+0000","last_fullsized":"2024-09-16T01:52:50.494790+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"111'107","reported_seq":311,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501551+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:51.501551+0000","last_peered":"2024-09-16T01:52:51.501551+0000","last_clean":"2024-09-16T01:52:51.501551+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:51.501551+0000","last_undegraded":"2024-09-16T01:52:51.501551+0000","last_fullsized":"2024-09-16T01:52:51.501551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"113'104","reported_seq":405,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.504164+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:51.504164+0000","last_peered":"2024-09-16T01:52:51.504164+0000","last_clean":"2024-09-16T01:52:51.504164+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:51.504164+0000","last_undegraded":"2024-09-16T01:52:51.504164+0000","last_fullsized":"2024-09-16T01:52:51.504164+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"114'109","reported_seq":506,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502460+0000","last_change":"2024-09-16T01:52:49.497350+0000","last_active":"2024-09-16T01:52:51.502460+0000","last_peered":"2024-09-16T01:52:51.502460+0000","last_clean":"2024-09-16T01:52:51.502460+0000","last_became_active":"2024-09-16T01:52:49.496908+0000","last_became_peered":"2024-09-16T01:52:49.496908+0000","last_unstale":"2024-09-16T01:52:51.502460+0000","last_undegraded":"2024-09-16T01:52:51.502460+0000","last_fullsized":"2024-09-16T01:52:51.502460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:51.945635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"114'124","reported_seq":560,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492538+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:52:50.492538+0000","last_peered":"2024-09-16T01:52:50.492538+0000","last_clean":"2024-09-16T01:52:50.492538+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:52:50.492538+0000","last_undegraded":"2024-09-16T01:52:50.492538+0000","last_fullsized":"2024-09-16T01:52:50.492538+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"112'114","reported_seq":462,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497504+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:51.497504+0000","last_peered":"2024-09-16T01:52:51.497504+0000","last_clean":"2024-09-16T01:52:51.497504+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:51.497504+0000","last_undegraded":"2024-09-16T01:52:51.497504+0000","last_fullsized":"2024-09-16T01:52:51.497504+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"113'95","reported_seq":337,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501366+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:51.501366+0000","last_peered":"2024-09-16T01:52:51.501366+0000","last_clean":"2024-09-16T01:52:51.501366+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:51.501366+0000","last_undegraded":"2024-09-16T01:52:51.501366+0000","last_fullsized":"2024-09-16T01:52:51.501366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"113'127","reported_seq":578,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497174+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:52:51.497174+0000","last_peered":"2024-09-16T01:52:51.497174+0000","last_clean":"2024-09-16T01:52:51.497174+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:52:51.497174+0000","last_undegraded":"2024-09-16T01:52:51.497174+0000","last_fullsized":"2024-09-16T01:52:51.497174+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":295,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494914+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:50.494914+0000","last_peered":"2024-09-16T01:52:50.494914+0000","last_clean":"2024-09-16T01:52:50.494914+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:50.494914+0000","last_undegraded":"2024-09-16T01:52:50.494914+0000","last_fullsized":"2024-09-16T01:52:50.494914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"111'132","reported_seq":336,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503002+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:51.503002+0000","last_peered":"2024-09-16T01:52:51.503002+0000","last_clean":"2024-09-16T01:52:51.503002+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:51.503002+0000","last_undegraded":"2024-09-16T01:52:51.503002+0000","last_fullsized":"2024-09-16T01:52:51.503002+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"113'104","reported_seq":306,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495482+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:50.495482+0000","last_peered":"2024-09-16T01:52:50.495482+0000","last_clean":"2024-09-16T01:52:50.495482+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:50.495482+0000","last_undegraded":"2024-09-16T01:52:50.495482+0000","last_fullsized":"2024-09-16T01:52:50.495482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"112'142","reported_seq":347,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502561+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:51.502561+0000","last_peered":"2024-09-16T01:52:51.502561+0000","last_clean":"2024-09-16T01:52:51.502561+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:51.502561+0000","last_undegraded":"2024-09-16T01:52:51.502561+0000","last_fullsized":"2024-09-16T01:52:51.502561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"114'105","reported_seq":538,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493440+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:52:50.493440+0000","last_peered":"2024-09-16T01:52:50.493440+0000","last_clean":"2024-09-16T01:52:50.493440+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:52:50.493440+0000","last_undegraded":"2024-09-16T01:52:50.493440+0000","last_fullsized":"2024-09-16T01:52:50.493440+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"113'100","reported_seq":396,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492476+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:50.492476+0000","last_peered":"2024-09-16T01:52:50.492476+0000","last_clean":"2024-09-16T01:52:50.492476+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:50.492476+0000","last_undegraded":"2024-09-16T01:52:50.492476+0000","last_fullsized":"2024-09-16T01:52:50.492476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"113'88","reported_seq":573,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495412+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:52:50.495412+0000","last_peered":"2024-09-16T01:52:50.495412+0000","last_clean":"2024-09-16T01:52:50.495412+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:52:50.495412+0000","last_undegraded":"2024-09-16T01:52:50.495412+0000","last_fullsized":"2024-09-16T01:52:50.495412+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"112'98","reported_seq":302,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501689+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:51.501689+0000","last_peered":"2024-09-16T01:52:51.501689+0000","last_clean":"2024-09-16T01:52:51.501689+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:51.501689+0000","last_undegraded":"2024-09-16T01:52:51.501689+0000","last_fullsized":"2024-09-16T01:52:51.501689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"113'102","reported_seq":499,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497066+0000","last_change":"2024-09-16T01:52:47.475996+0000","last_active":"2024-09-16T01:52:51.497066+0000","last_peered":"2024-09-16T01:52:51.497066+0000","last_clean":"2024-09-16T01:52:51.497066+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:52:51.497066+0000","last_undegraded":"2024-09-16T01:52:51.497066+0000","last_fullsized":"2024-09-16T01:52:51.497066+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:05.418025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"113'112","reported_seq":428,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492019+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:50.492019+0000","last_peered":"2024-09-16T01:52:50.492019+0000","last_clean":"2024-09-16T01:52:50.492019+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:50.492019+0000","last_undegraded":"2024-09-16T01:52:50.492019+0000","last_fullsized":"2024-09-16T01:52:50.492019+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"112'98","reported_seq":303,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497739+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:51.497739+0000","last_peered":"2024-09-16T01:52:51.497739+0000","last_clean":"2024-09-16T01:52:51.497739+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:51.497739+0000","last_undegraded":"2024-09-16T01:52:51.497739+0000","last_fullsized":"2024-09-16T01:52:51.497739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":209,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495509+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:50.495509+0000","last_peered":"2024-09-16T01:52:50.495509+0000","last_clean":"2024-09-16T01:52:50.495509+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:50.495509+0000","last_undegraded":"2024-09-16T01:52:50.495509+0000","last_fullsized":"2024-09-16T01:52:50.495509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"114'105","reported_seq":636,"reported_epoch":117,"state":"peering","last_fresh":"2024-09-16T01:52:50.626211+0000","last_change":"2024-09-16T01:52:51.501876+0000","last_active":"2024-09-16T01:52:51.501876+0000","last_peered":"2024-09-16T01:52:50.626211+0000","last_clean":"2024-09-16T01:52:44.543722+0000","last_became_active":"2024-09-16T01:52:47.551382+0000","last_became_peered":"2024-09-16T01:52:47.551382+0000","last_unstale":"2024-09-16T01:52:50.626211+0000","last_undegraded":"2024-09-16T01:52:50.626211+0000","last_fullsized":"2024-09-16T01:52:47.534787+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":226,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501889+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:51.501889+0000","last_peered":"2024-09-16T01:52:51.501889+0000","last_clean":"2024-09-16T01:52:51.501889+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:51.501889+0000","last_undegraded":"2024-09-16T01:52:51.501889+0000","last_fullsized":"2024-09-16T01:52:51.501889+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":230,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492167+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:52:50.492167+0000","last_peered":"2024-09-16T01:52:50.492167+0000","last_clean":"2024-09-16T01:52:50.492167+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:52:50.492167+0000","last_undegraded":"2024-09-16T01:52:50.492167+0000","last_fullsized":"2024-09-16T01:52:50.492167+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"112'122","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494972+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:50.494972+0000","last_peered":"2024-09-16T01:52:50.494972+0000","last_clean":"2024-09-16T01:52:50.494972+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:50.494972+0000","last_undegraded":"2024-09-16T01:52:50.494972+0000","last_fullsized":"2024-09-16T01:52:50.494972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":209,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503062+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:51.503062+0000","last_peered":"2024-09-16T01:52:51.503062+0000","last_clean":"2024-09-16T01:52:51.503062+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:51.503062+0000","last_undegraded":"2024-09-16T01:52:51.503062+0000","last_fullsized":"2024-09-16T01:52:51.503062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"112'114","reported_seq":284,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496946+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:51.496946+0000","last_peered":"2024-09-16T01:52:51.496946+0000","last_clean":"2024-09-16T01:52:51.496946+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:51.496946+0000","last_undegraded":"2024-09-16T01:52:51.496946+0000","last_fullsized":"2024-09-16T01:52:51.496946+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":238,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.498045+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:51.498045+0000","last_peered":"2024-09-16T01:52:51.498045+0000","last_clean":"2024-09-16T01:52:51.498045+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:51.498045+0000","last_undegraded":"2024-09-16T01:52:51.498045+0000","last_fullsized":"2024-09-16T01:52:51.498045+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"113'167","reported_seq":383,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495029+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:50.495029+0000","last_peered":"2024-09-16T01:52:50.495029+0000","last_clean":"2024-09-16T01:52:50.495029+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:50.495029+0000","last_undegraded":"2024-09-16T01:52:50.495029+0000","last_fullsized":"2024-09-16T01:52:50.495029+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.498047+0000","last_change":"2024-09-16T01:52:46.621488+0000","last_active":"2024-09-16T01:52:51.498047+0000","last_peered":"2024-09-16T01:52:51.498047+0000","last_clean":"2024-09-16T01:52:51.498047+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:52:51.498047+0000","last_undegraded":"2024-09-16T01:52:51.498047+0000","last_fullsized":"2024-09-16T01:52:51.498047+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"113'101","reported_seq":303,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495598+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:50.495598+0000","last_peered":"2024-09-16T01:52:50.495598+0000","last_clean":"2024-09-16T01:52:50.495598+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:50.495598+0000","last_undegraded":"2024-09-16T01:52:50.495598+0000","last_fullsized":"2024-09-16T01:52:50.495598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502831+0000","last_change":"2024-09-16T01:52:46.632501+0000","last_active":"2024-09-16T01:52:51.502831+0000","last_peered":"2024-09-16T01:52:51.502831+0000","last_clean":"2024-09-16T01:52:51.502831+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:52:51.502831+0000","last_undegraded":"2024-09-16T01:52:51.502831+0000","last_fullsized":"2024-09-16T01:52:51.502831+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"112'107","reported_seq":311,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502768+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:51.502768+0000","last_peered":"2024-09-16T01:52:51.502768+0000","last_clean":"2024-09-16T01:52:51.502768+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:51.502768+0000","last_undegraded":"2024-09-16T01:52:51.502768+0000","last_fullsized":"2024-09-16T01:52:51.502768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":396,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500717+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:51.500717+0000","last_peered":"2024-09-16T01:52:51.500717+0000","last_clean":"2024-09-16T01:52:51.500717+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:51.500717+0000","last_undegraded":"2024-09-16T01:52:51.500717+0000","last_fullsized":"2024-09-16T01:52:51.500717+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":417,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493346+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:50.493346+0000","last_peered":"2024-09-16T01:52:50.493346+0000","last_clean":"2024-09-16T01:52:50.493346+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:50.493346+0000","last_undegraded":"2024-09-16T01:52:50.493346+0000","last_fullsized":"2024-09-16T01:52:50.493346+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":285,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502300+0000","last_change":"2024-09-16T01:52:46.615925+0000","last_active":"2024-09-16T01:52:51.502300+0000","last_peered":"2024-09-16T01:52:51.502300+0000","last_clean":"2024-09-16T01:52:51.502300+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:52:51.502300+0000","last_undegraded":"2024-09-16T01:52:51.502300+0000","last_fullsized":"2024-09-16T01:52:51.502300+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":340,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"112'117","reported_seq":320,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492401+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:50.492401+0000","last_peered":"2024-09-16T01:52:50.492401+0000","last_clean":"2024-09-16T01:52:50.492401+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:50.492401+0000","last_undegraded":"2024-09-16T01:52:50.492401+0000","last_fullsized":"2024-09-16T01:52:50.492401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":289,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501983+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:52:51.501983+0000","last_peered":"2024-09-16T01:52:51.501983+0000","last_clean":"2024-09-16T01:52:51.501983+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:52:51.501983+0000","last_undegraded":"2024-09-16T01:52:51.501983+0000","last_fullsized":"2024-09-16T01:52:51.501983+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"112'121","reported_seq":456,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502402+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:51.502402+0000","last_peered":"2024-09-16T01:52:51.502402+0000","last_clean":"2024-09-16T01:52:51.502402+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:51.502402+0000","last_undegraded":"2024-09-16T01:52:51.502402+0000","last_fullsized":"2024-09-16T01:52:51.502402+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501563+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:51.501563+0000","last_peered":"2024-09-16T01:52:51.501563+0000","last_clean":"2024-09-16T01:52:51.501563+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:51.501563+0000","last_undegraded":"2024-09-16T01:52:51.501563+0000","last_fullsized":"2024-09-16T01:52:51.501563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"113'133","reported_seq":338,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496970+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:51.496970+0000","last_peered":"2024-09-16T01:52:51.496970+0000","last_clean":"2024-09-16T01:52:51.496970+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:51.496970+0000","last_undegraded":"2024-09-16T01:52:51.496970+0000","last_fullsized":"2024-09-16T01:52:51.496970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":142,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.496880+0000","last_change":"2024-09-16T01:52:46.599623+0000","last_active":"2024-09-16T01:52:51.496880+0000","last_peered":"2024-09-16T01:52:51.496880+0000","last_clean":"2024-09-16T01:52:51.496880+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:52:51.496880+0000","last_undegraded":"2024-09-16T01:52:51.496880+0000","last_fullsized":"2024-09-16T01:52:51.496880+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:22.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":784,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"113'145","reported_seq":349,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501711+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:51.501711+0000","last_peered":"2024-09-16T01:52:51.501711+0000","last_clean":"2024-09-16T01:52:51.501711+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:51.501711+0000","last_undegraded":"2024-09-16T01:52:51.501711+0000","last_fullsized":"2024-09-16T01:52:51.501711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":77,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495388+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:50.495388+0000","last_peered":"2024-09-16T01:52:50.495388+0000","last_clean":"2024-09-16T01:52:50.495388+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:50.495388+0000","last_undegraded":"2024-09-16T01:52:50.495388+0000","last_fullsized":"2024-09-16T01:52:50.495388+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"112'106","reported_seq":412,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502668+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:51.502668+0000","last_peered":"2024-09-16T01:52:51.502668+0000","last_clean":"2024-09-16T01:52:51.502668+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:51.502668+0000","last_undegraded":"2024-09-16T01:52:51.502668+0000","last_fullsized":"2024-09-16T01:52:51.502668+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495361+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:50.495361+0000","last_peered":"2024-09-16T01:52:50.495361+0000","last_clean":"2024-09-16T01:52:50.495361+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:50.495361+0000","last_undegraded":"2024-09-16T01:52:50.495361+0000","last_fullsized":"2024-09-16T01:52:50.495361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"112'103","reported_seq":404,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497913+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:51.497913+0000","last_peered":"2024-09-16T01:52:51.497913+0000","last_clean":"2024-09-16T01:52:51.497913+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:51.497913+0000","last_undegraded":"2024-09-16T01:52:51.497913+0000","last_fullsized":"2024-09-16T01:52:51.497913+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":207,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495615+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:50.495615+0000","last_peered":"2024-09-16T01:52:50.495615+0000","last_clean":"2024-09-16T01:52:50.495615+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:50.495615+0000","last_undegraded":"2024-09-16T01:52:50.495615+0000","last_fullsized":"2024-09-16T01:52:50.495615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"114'102","reported_seq":634,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501795+0000","last_change":"2024-09-16T01:52:50.499005+0000","last_active":"2024-09-16T01:52:51.501795+0000","last_peered":"2024-09-16T01:52:51.501795+0000","last_clean":"2024-09-16T01:52:51.501795+0000","last_became_active":"2024-09-16T01:52:50.498634+0000","last_became_peered":"2024-09-16T01:52:50.498634+0000","last_unstale":"2024-09-16T01:52:51.501795+0000","last_undegraded":"2024-09-16T01:52:51.501795+0000","last_fullsized":"2024-09-16T01:52:51.501795+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":223,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492672+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:50.492672+0000","last_peered":"2024-09-16T01:52:50.492672+0000","last_clean":"2024-09-16T01:52:50.492672+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:50.492672+0000","last_undegraded":"2024-09-16T01:52:50.492672+0000","last_fullsized":"2024-09-16T01:52:50.492672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"112'112","reported_seq":316,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503064+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:51.503064+0000","last_peered":"2024-09-16T01:52:51.503064+0000","last_clean":"2024-09-16T01:52:51.503064+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:51.503064+0000","last_undegraded":"2024-09-16T01:52:51.503064+0000","last_fullsized":"2024-09-16T01:52:51.503064+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":364,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497043+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:51.497043+0000","last_peered":"2024-09-16T01:52:51.497043+0000","last_clean":"2024-09-16T01:52:51.497043+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:51.497043+0000","last_undegraded":"2024-09-16T01:52:51.497043+0000","last_fullsized":"2024-09-16T01:52:51.497043+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":212,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502584+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:51.502584+0000","last_peered":"2024-09-16T01:52:51.502584+0000","last_clean":"2024-09-16T01:52:51.502584+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:51.502584+0000","last_undegraded":"2024-09-16T01:52:51.502584+0000","last_fullsized":"2024-09-16T01:52:51.502584+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"113'112","reported_seq":457,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500657+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:51.500657+0000","last_peered":"2024-09-16T01:52:51.500657+0000","last_clean":"2024-09-16T01:52:51.500657+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:51.500657+0000","last_undegraded":"2024-09-16T01:52:51.500657+0000","last_fullsized":"2024-09-16T01:52:51.500657+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":210,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.503111+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:51.503111+0000","last_peered":"2024-09-16T01:52:51.503111+0000","last_clean":"2024-09-16T01:52:51.503111+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:51.503111+0000","last_undegraded":"2024-09-16T01:52:51.503111+0000","last_fullsized":"2024-09-16T01:52:51.503111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"113'107","reported_seq":310,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492106+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:50.492106+0000","last_peered":"2024-09-16T01:52:50.492106+0000","last_clean":"2024-09-16T01:52:50.492106+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:50.492106+0000","last_undegraded":"2024-09-16T01:52:50.492106+0000","last_fullsized":"2024-09-16T01:52:50.492106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":265,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.494812+0000","last_change":"2024-09-16T01:52:46.569921+0000","last_active":"2024-09-16T01:52:50.494812+0000","last_peered":"2024-09-16T01:52:50.494812+0000","last_clean":"2024-09-16T01:52:50.494812+0000","last_became_active":"2024-09-16T01:52:46.544102+0000","last_became_peered":"2024-09-16T01:52:46.544102+0000","last_unstale":"2024-09-16T01:52:50.494812+0000","last_undegraded":"2024-09-16T01:52:50.494812+0000","last_fullsized":"2024-09-16T01:52:50.494812+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:14.446115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"113'96","reported_seq":515,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495047+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:52:50.495047+0000","last_peered":"2024-09-16T01:52:50.495047+0000","last_clean":"2024-09-16T01:52:50.495047+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:52:50.495047+0000","last_undegraded":"2024-09-16T01:52:50.495047+0000","last_fullsized":"2024-09-16T01:52:50.495047+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":211,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502105+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:51.502105+0000","last_peered":"2024-09-16T01:52:51.502105+0000","last_clean":"2024-09-16T01:52:51.502105+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:51.502105+0000","last_undegraded":"2024-09-16T01:52:51.502105+0000","last_fullsized":"2024-09-16T01:52:51.502105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"113'111","reported_seq":415,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500536+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:51.500536+0000","last_peered":"2024-09-16T01:52:51.500536+0000","last_clean":"2024-09-16T01:52:51.500536+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:51.500536+0000","last_undegraded":"2024-09-16T01:52:51.500536+0000","last_fullsized":"2024-09-16T01:52:51.500536+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":205,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495040+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:50.495040+0000","last_peered":"2024-09-16T01:52:50.495040+0000","last_clean":"2024-09-16T01:52:50.495040+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:50.495040+0000","last_undegraded":"2024-09-16T01:52:50.495040+0000","last_fullsized":"2024-09-16T01:52:50.495040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"113'110","reported_seq":315,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501011+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:51.501011+0000","last_peered":"2024-09-16T01:52:51.501011+0000","last_clean":"2024-09-16T01:52:51.501011+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:51.501011+0000","last_undegraded":"2024-09-16T01:52:51.501011+0000","last_fullsized":"2024-09-16T01:52:51.501011+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500909+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:52:51.500909+0000","last_peered":"2024-09-16T01:52:51.500909+0000","last_clean":"2024-09-16T01:52:51.500909+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:52:51.500909+0000","last_undegraded":"2024-09-16T01:52:51.500909+0000","last_fullsized":"2024-09-16T01:52:51.500909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"113'103","reported_seq":306,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492575+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:50.492575+0000","last_peered":"2024-09-16T01:52:50.492575+0000","last_clean":"2024-09-16T01:52:50.492575+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:50.492575+0000","last_undegraded":"2024-09-16T01:52:50.492575+0000","last_fullsized":"2024-09-16T01:52:50.492575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":203,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495195+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:50.495195+0000","last_peered":"2024-09-16T01:52:50.495195+0000","last_clean":"2024-09-16T01:52:50.495195+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:50.495195+0000","last_undegraded":"2024-09-16T01:52:50.495195+0000","last_fullsized":"2024-09-16T01:52:50.495195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"113'114","reported_seq":362,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.493268+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:50.493268+0000","last_peered":"2024-09-16T01:52:50.493268+0000","last_clean":"2024-09-16T01:52:50.493268+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:50.493268+0000","last_undegraded":"2024-09-16T01:52:50.493268+0000","last_fullsized":"2024-09-16T01:52:50.493268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":228,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.504195+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:51.504195+0000","last_peered":"2024-09-16T01:52:51.504195+0000","last_clean":"2024-09-16T01:52:51.504195+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:51.504195+0000","last_undegraded":"2024-09-16T01:52:51.504195+0000","last_fullsized":"2024-09-16T01:52:51.504195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"114'116","reported_seq":405,"reported_epoch":117,"state":"peering","last_fresh":"2024-09-16T01:52:51.250680+0000","last_change":"2024-09-16T01:52:51.498002+0000","last_active":"2024-09-16T01:52:51.498002+0000","last_peered":"2024-09-16T01:52:51.250680+0000","last_clean":"2024-09-16T01:52:44.870429+0000","last_became_active":"2024-09-16T01:52:47.547843+0000","last_became_peered":"2024-09-16T01:52:47.547843+0000","last_unstale":"2024-09-16T01:52:51.250680+0000","last_undegraded":"2024-09-16T01:52:51.250680+0000","last_fullsized":"2024-09-16T01:52:47.537632+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":92,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500451+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:52:51.500451+0000","last_peered":"2024-09-16T01:52:51.500451+0000","last_clean":"2024-09-16T01:52:51.500451+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:52:51.500451+0000","last_undegraded":"2024-09-16T01:52:51.500451+0000","last_fullsized":"2024-09-16T01:52:51.500451+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"112'114","reported_seq":373,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501828+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:51.501828+0000","last_peered":"2024-09-16T01:52:51.501828+0000","last_clean":"2024-09-16T01:52:51.501828+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:51.501828+0000","last_undegraded":"2024-09-16T01:52:51.501828+0000","last_fullsized":"2024-09-16T01:52:51.501828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492054+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:50.492054+0000","last_peered":"2024-09-16T01:52:50.492054+0000","last_clean":"2024-09-16T01:52:50.492054+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:50.492054+0000","last_undegraded":"2024-09-16T01:52:50.492054+0000","last_fullsized":"2024-09-16T01:52:50.492054+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"113'116","reported_seq":376,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.500330+0000","last_change":"2024-09-16T01:52:46.782699+0000","last_active":"2024-09-16T01:52:51.500330+0000","last_peered":"2024-09-16T01:52:51.500330+0000","last_clean":"2024-09-16T01:52:51.500330+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:52:51.500330+0000","last_undegraded":"2024-09-16T01:52:51.500330+0000","last_fullsized":"2024-09-16T01:52:51.500330+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:30.580835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":208,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.492505+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:50.492505+0000","last_peered":"2024-09-16T01:52:50.492505+0000","last_clean":"2024-09-16T01:52:50.492505+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:50.492505+0000","last_undegraded":"2024-09-16T01:52:50.492505+0000","last_fullsized":"2024-09-16T01:52:50.492505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"114'105","reported_seq":512,"reported_epoch":116,"state":"peering","last_fresh":"2024-09-16T01:52:50.241765+0000","last_change":"2024-09-16T01:52:50.492781+0000","last_active":"2024-09-16T01:52:50.492781+0000","last_peered":"2024-09-16T01:52:50.241765+0000","last_clean":"2024-09-16T01:52:44.861424+0000","last_became_active":"2024-09-16T01:52:47.546951+0000","last_became_peered":"2024-09-16T01:52:47.546951+0000","last_unstale":"2024-09-16T01:52:50.241765+0000","last_undegraded":"2024-09-16T01:52:50.241765+0000","last_fullsized":"2024-09-16T01:52:47.532896+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":211,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502491+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:51.502491+0000","last_peered":"2024-09-16T01:52:51.502491+0000","last_clean":"2024-09-16T01:52:51.502491+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:51.502491+0000","last_undegraded":"2024-09-16T01:52:51.502491+0000","last_fullsized":"2024-09-16T01:52:51.502491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"111'94","reported_seq":298,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.501786+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:51.501786+0000","last_peered":"2024-09-16T01:52:51.501786+0000","last_clean":"2024-09-16T01:52:51.501786+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:51.501786+0000","last_undegraded":"2024-09-16T01:52:51.501786+0000","last_fullsized":"2024-09-16T01:52:51.501786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"112'131","reported_seq":454,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.502967+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:51.502967+0000","last_peered":"2024-09-16T01:52:51.502967+0000","last_clean":"2024-09-16T01:52:51.502967+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:51.502967+0000","last_undegraded":"2024-09-16T01:52:51.502967+0000","last_fullsized":"2024-09-16T01:52:51.502967+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497567+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:51.497567+0000","last_peered":"2024-09-16T01:52:51.497567+0000","last_clean":"2024-09-16T01:52:51.497567+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:51.497567+0000","last_undegraded":"2024-09-16T01:52:51.497567+0000","last_fullsized":"2024-09-16T01:52:51.497567+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"111'106","reported_seq":406,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497668+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:51.497668+0000","last_peered":"2024-09-16T01:52:51.497668+0000","last_clean":"2024-09-16T01:52:51.497668+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:51.497668+0000","last_undegraded":"2024-09-16T01:52:51.497668+0000","last_fullsized":"2024-09-16T01:52:51.497668+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":48,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.497378+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:51.497378+0000","last_peered":"2024-09-16T01:52:51.497378+0000","last_clean":"2024-09-16T01:52:51.497378+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:51.497378+0000","last_undegraded":"2024-09-16T01:52:51.497378+0000","last_fullsized":"2024-09-16T01:52:51.497378+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"113'97","reported_seq":299,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495490+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:50.495490+0000","last_peered":"2024-09-16T01:52:50.495490+0000","last_clean":"2024-09-16T01:52:50.495490+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:50.495490+0000","last_undegraded":"2024-09-16T01:52:50.495490+0000","last_fullsized":"2024-09-16T01:52:50.495490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":236,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495382+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:52:50.495382+0000","last_peered":"2024-09-16T01:52:50.495382+0000","last_clean":"2024-09-16T01:52:50.495382+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:52:50.495382+0000","last_undegraded":"2024-09-16T01:52:50.495382+0000","last_fullsized":"2024-09-16T01:52:50.495382+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"112'115","reported_seq":317,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-16T01:52:50.495288+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:50.495288+0000","last_peered":"2024-09-16T01:52:50.495288+0000","last_clean":"2024-09-16T01:52:50.495288+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:50.495288+0000","last_undegraded":"2024-09-16T01:52:50.495288+0000","last_fullsized":"2024-09-16T01:52:50.495288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-16T01:52:51.504067+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:51.504067+0000","last_peered":"2024-09-16T01:52:51.504067+0000","last_clean":"2024-09-16T01:52:51.504067+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:51.504067+0000","last_undegraded":"2024-09-16T01:52:51.504067+0000","last_fullsized":"2024-09-16T01:52:51.504067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2936,"num_bytes_recovered":9722396672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1408827392,"data_stored":22541238272,"data_compressed":1124772320,"data_compressed_allocated":1408827392,"data_compressed_original":22541238272,"omap_allocated":0,"internal_metadata":0},"log_size":7171,"ondisk_log_size":7171,"up":128,"acting":127,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":248325,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118208,"data_stored":995820,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509493,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":333660,"kb_used_data":302468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94038180,"statfs":{"total":96636764160,"available":96295096320,"internally_reserved":0,"allocated":309727232,"data_stored":4933159299,"data_compressed":246680448,"data_compressed_allocated":308281344,"data_compressed_original":4932501504,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509493,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300472,"kb_used_data":269280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94071368,"statfs":{"total":96636764160,"available":96329080832,"internally_reserved":0,"allocated":275742720,"data_stored":4384079069,"data_compressed":219220992,"data_compressed_allocated":273940480,"data_compressed_original":4383047680,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509492,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":240132,"kb_used_data":213100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94131708,"statfs":{"total":96636764160,"available":96390868992,"internally_reserved":0,"allocated":218214400,"data_stored":3466116710,"data_compressed":173345635,"data_compressed_allocated":216621056,"data_compressed_original":3465302016,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509493,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542196,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148800,"kb_used_data":121768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94223040,"statfs":{"total":96636764160,"available":96484392960,"internally_reserved":0,"allocated":124690432,"data_stored":1973020073,"data_compressed":98654688,"data_compressed_allocated":123273216,"data_compressed_original":1972371456,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542196,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255748,"kb_used_data":212076,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94116092,"statfs":{"total":96636764160,"available":96374878208,"internally_reserved":0,"allocated":217165824,"data_stored":3452562337,"data_compressed":172647168,"data_compressed_allocated":215744512,"data_compressed_original":3451912192,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542196,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265480,"kb_used_data":225968,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94106360,"statfs":{"total":96636764160,"available":96364912640,"internally_reserved":0,"allocated":231391232,"data_stored":3679645297,"data_compressed":184008356,"data_compressed_allocated":229945344,"data_compressed_original":3678978048,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574901,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":377634,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":377785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196345856,"data_stored":3141533696,"data_compressed":157134208,"data_compressed_allocated":196345856,"data_compressed_original":3141533696,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":273940480,"data_stored":4383047680,"data_compressed":219220992,"data_compressed_allocated":273940480,"data_compressed_original":4383047680,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":308281344,"data_stored":4932501504,"data_compressed":246680448,"data_compressed_allocated":308281344,"data_compressed_original":4932501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":3678404608,"data_compressed":183975936,"data_compressed_allocated":229900288,"data_compressed_original":3678404608,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":205520896,"data_stored":3288334336,"data_compressed":164476928,"data_compressed_allocated":205520896,"data_compressed_original":3288334336,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":135331840,"data_stored":2165309440,"data_compressed":108305120,"data_compressed_allocated":135331840,"data_compressed_original":2165309440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:55.236 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.236 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.245 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:52:55.300 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:55.299+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:55.389 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.389 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.397 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:52:55.401 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:55.399+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:55.501 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:55.500+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:55.502 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.502 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.510 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:52:55.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.517+0000 7f05fb400640 1 -- 172.21.15.146:0/1443375421 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v121) ==== 87+0+0 (secure 0 0 0) 0x7f05f007e090 con 0x7f06040c75a0 2024-09-16T01:52:55.519 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:52:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f05e406c2b0 msgr2=0x7f05e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f05e406c2b0 0x7f05e406e770 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009aa0 tx=0x7f05f4009190 comp rx=0 tx=0).stop 2024-09-16T01:52:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 msgr2=0x7f06040c51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040c51a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000c00 tx=0x7f05f00046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 shutdown_connections 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f05e406c2b0 0x7f05e406e770 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06040cbab0 0x7f06041f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06040c75a0 0x7f06040c51a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 --2- 172.21.15.146:0/1443375421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06040c6b70 0x7f06040abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 >> 172.21.15.146:0/1443375421 conn(0x7f06040ccc20 msgr2=0x7f06040ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 shutdown_connections 2024-09-16T01:52:55.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:55.519+0000 7f0603e00640 1 -- 172.21.15.146:0/1443375421 wait complete. 2024-09-16T01:52:55.602 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:55.601+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:55.615 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.616 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:52:55.704 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:55.703+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:55.783 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.783 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:52:55.804 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:55.804+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:55.905 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:55.903+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:55.960 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:55.960 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:55.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:52:56.015 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:56.013+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:56.108 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:56.106+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:56.182 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:56.182 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:56.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:52:56.209 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:56.208+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:56.310 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:56.309+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:56.404 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:56.404 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:56.411 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:56.410+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:56.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:52:56.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:52:56.512 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:56.511+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:56.588 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:56.588 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:56.602 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:52:56.613 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:56.612+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:56.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.663+0000 7fbd96c00640 1 Processor -- start 2024-09-16T01:52:56.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.663+0000 7fbd96c00640 1 -- start start 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd96c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd90152c30 0x7fbd900c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd96c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbd900c6e90 0x7fbd900c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd96c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd900c4240 con 0x7fbd900c6ab0 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd96c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fbd900c4da0 con 0x7fbd90152c30 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd96c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbd900c4500 con 0x7fbd900c6e90 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd90152c30 0x7fbd900c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd90152c30 0x7fbd900c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57610/0 (socket says 172.21.15.146:57610) 2024-09-16T01:52:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 -- 172.21.15.146:0/1062645462 learned_addr learned my addr 172.21.15.146:0/1062645462 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd94c00640 1 --2- 172.21.15.146:0/1062645462 >> v2:172.21.15.161:3300/0 conn(0x7fbd900c6e90 0x7fbd900c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 -- 172.21.15.146:0/1062645462 >> v2:172.21.15.161:3300/0 conn(0x7fbd900c6e90 msgr2=0x7fbd900c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/1062645462 >> v2:172.21.15.161:3300/0 conn(0x7fbd900c6e90 0x7fbd900c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 -- 172.21.15.146:0/1062645462 >> v1:172.21.15.161:6789/0 conn(0x7fbd900c6ab0 legacy=0x7fbd900c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd8fe00640 1 -- 172.21.15.146:0/1062645462 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd900c47e0 con 0x7fbd90152c30 2024-09-16T01:52:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.665+0000 7fbd94c00640 1 --2- 172.21.15.146:0/1062645462 >> v2:172.21.15.161:3300/0 conn(0x7fbd900c6e90 0x7fbd900c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:52:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.667+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/1062645462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd90152c30 0x7fbd900c36b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbd9015cdc0 tx=0x7fbd8002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f40e02ffb7a220e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.667+0000 7fbd8ea00640 1 -- 172.21.15.146:0/1062645462 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80037070 con 0x7fbd90152c30 2024-09-16T01:52:56.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.667+0000 7fbd8ea00640 1 -- 172.21.15.146:0/1062645462 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd80032440 con 0x7fbd90152c30 2024-09-16T01:52:56.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.667+0000 7fbd8ea00640 1 -- 172.21.15.146:0/1062645462 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80032810 con 0x7fbd90152c30 2024-09-16T01:52:56.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 -- 172.21.15.146:0/1062645462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd90152c30 msgr2=0x7fbd900c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 --2- 172.21.15.146:0/1062645462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd90152c30 0x7fbd900c36b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbd9015cdc0 tx=0x7fbd8002a270 comp rx=0 tx=0).stop 2024-09-16T01:52:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 -- 172.21.15.146:0/1062645462 shutdown_connections 2024-09-16T01:52:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 --2- 172.21.15.146:0/1062645462 >> v2:172.21.15.161:3300/0 conn(0x7fbd900c6e90 0x7fbd900c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 --2- 172.21.15.146:0/1062645462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd90152c30 0x7fbd900c36b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 -- 172.21.15.146:0/1062645462 >> 172.21.15.146:0/1062645462 conn(0x7fbd9006bc10 msgr2=0x7fbd9006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 -- 172.21.15.146:0/1062645462 shutdown_connections 2024-09-16T01:52:56.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.668+0000 7fbd96c00640 1 -- 172.21.15.146:0/1062645462 wait complete. 2024-09-16T01:52:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.671+0000 7fbd96c00640 1 Processor -- start 2024-09-16T01:52:56.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.671+0000 7fbd96c00640 1 -- start start 2024-09-16T01:52:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.672+0000 7fbd96c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd900c6ab0 0x7fbd9014c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.672+0000 7fbd96c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900c6e90 0x7fbd900cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.672+0000 7fbd96c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd90152c30 0x7fbd9014a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.672+0000 7fbd96c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbd900c9350 con 0x7fbd900c6e90 2024-09-16T01:52:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.672+0000 7fbd96c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbd900c6630 con 0x7fbd900c6ab0 2024-09-16T01:52:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.672+0000 7fbd96c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbd900c94d0 con 0x7fbd90152c30 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900c6e90 0x7fbd900cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900c6e90 0x7fbd900cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57622/0 (socket says 172.21.15.146:57622) 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd900c6ab0 0x7fbd9014c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8f400640 1 -- 172.21.15.146:0/4161811749 learned_addr learned my addr 172.21.15.146:0/4161811749 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd90152c30 0x7fbd9014a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd90152c30 msgr2=0x7fbd9014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbd90152c30 0x7fbd9014a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900c6e90 msgr2=0x7fbd900cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900c6e90 0x7fbd900cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901499f0 con 0x7fbd900c6ab0 2024-09-16T01:52:56.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900c6e90 0x7fbd900cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.673+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd900c6ab0 0x7fbd9014c250 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbd80031000 tx=0x7fbd800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.674+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80041020 con 0x7fbd900c6ab0 2024-09-16T01:52:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.674+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd80030700 con 0x7fbd900c6ab0 2024-09-16T01:52:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.674+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80037040 con 0x7fbd900c6ab0 2024-09-16T01:52:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.674+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd90149c60 con 0x7fbd900c6ab0 2024-09-16T01:52:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.674+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd900cb1d0 con 0x7fbd900c6ab0 2024-09-16T01:52:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.675+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbd900c36b0 con 0x7fbd900c6ab0 2024-09-16T01:52:56.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.676+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbd80044020 con 0x7fbd900c6ab0 2024-09-16T01:52:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.677+0000 7fbd87e00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd7006c2b0 0x7fbd7006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.677+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 5894+0+0 (secure 0 0 0) 0x7fbd800abb80 con 0x7fbd900c6ab0 2024-09-16T01:52:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.677+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v121) ==== 54+0+16 (secure 0 0 0) 0x7fbd8004b030 con 0x7fbd900c6ab0 2024-09-16T01:52:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.677+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd7006c2b0 0x7fbd7006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.677+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbd900cccd0 0x7fbd9014a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.677+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd900c7530 con 0x7fbd900cccd0 2024-09-16T01:52:56.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.678+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbd900cccd0 0x7fbd9014a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.678+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd7006c2b0 0x7fbd7006e770 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbd90140bc0 tx=0x7fbd7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.679+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbd900cccd0 0x7fbd9014a540 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.680+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd900c7530 con 0x7fbd900cccd0 2024-09-16T01:52:56.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.695+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd900c9050 con 0x7fbd900cccd0 2024-09-16T01:52:56.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.698+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd900c9050 con 0x7fbd900cccd0 2024-09-16T01:52:56.700 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:56.701 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:56.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.698+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbd9014aa80 0x7fbd901f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.698+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbd90149140 con 0x7fbd9014aa80 2024-09-16T01:52:56.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.698+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbd9014aa80 0x7fbd901f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.699+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbd9014aa80 0x7fbd901f2140 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.700+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd90149140 con 0x7fbd9014aa80 2024-09-16T01:52:56.714 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:56.713+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:56.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.715+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd901f2680 con 0x7fbd9014aa80 2024-09-16T01:52:56.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.717+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd901f2680 con 0x7fbd9014aa80 2024-09-16T01:52:56.719 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:56.720 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:56.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.717+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbd900ca5a0 0x7fbd900ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.717+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbd900caec0 con 0x7fbd900ca5a0 2024-09-16T01:52:56.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.717+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbd900ca5a0 0x7fbd900ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.718+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbd900ca5a0 0x7fbd900ca980 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.719+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd900caec0 con 0x7fbd900ca5a0 2024-09-16T01:52:56.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.733+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd90062900 con 0x7fbd900ca5a0 2024-09-16T01:52:56.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.736+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd90062900 con 0x7fbd900ca5a0 2024-09-16T01:52:56.738 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:56.739 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:56.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.736+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbd9006ea70 0x7fbd9006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.736+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbd900633e0 con 0x7fbd9006ea70 2024-09-16T01:52:56.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.737+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbd9006ea70 0x7fbd9006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.737+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbd9006ea70 0x7fbd9006eeb0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.739+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd900633e0 con 0x7fbd9006ea70 2024-09-16T01:52:56.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:52:56.749 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:52:56.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.754+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd900633e0 con 0x7fbd9006ea70 2024-09-16T01:52:56.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.757+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd900633e0 con 0x7fbd9006ea70 2024-09-16T01:52:56.759 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:56.760 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:56.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.757+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbd9006f720 0x7fbd9002cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.758+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbd900633e0 con 0x7fbd9006f720 2024-09-16T01:52:56.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.758+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbd9006f720 0x7fbd9002cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.759+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbd9006f720 0x7fbd9002cb50 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.760+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd900633e0 con 0x7fbd9006f720 2024-09-16T01:52:56.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.775+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd900633e0 con 0x7fbd9006f720 2024-09-16T01:52:56.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.777+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd900633e0 con 0x7fbd9006f720 2024-09-16T01:52:56.779 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.777+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbd9002d5a0 0x7fbd90204f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.777+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbd9002d9e0 con 0x7fbd9002d5a0 2024-09-16T01:52:56.780 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:56.780 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 249.00915079200001, 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:52:56.781 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:56.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:52:56.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:52:56.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:52:56.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:52:56.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:56.789 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 116.391698452, 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.790 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:52:56.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:52:56.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:52:56.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:52:56.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:52:56.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.599681502999999, 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:52:56.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:52:56.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:52:56.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:52:56.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:56.805 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 96.564148880000005, 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:52:56.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:52:56.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:52:56.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:52:56.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:52:56.812 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.198059943000004, 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.815 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:52:56.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.633848404999995, 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:52:56.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:52:56.824 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.572061102999996, 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:52:56.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:52:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.829 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.730278656999999, 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.830 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:52:56.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.834 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:52:56.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:52:56.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.986839904, 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.840 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:56.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:52:56.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.942618957999997, 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.847 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.778+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbd9002d5a0 0x7fbd90204f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:56.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:52:56.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:52:56.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.855 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.934202472000003, 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:52:56.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:52:56.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:52:56.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.863 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4107 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e99)", 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.501564985000002, 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31541509600000001, 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4107 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275941+0000", 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6959999999999999e-06 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303032+0000", 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027090797 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303053+0000", 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1019999999999999e-05 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303063+0000", 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6479999999999998e-06 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303176+0000", 2024-09-16T01:52:56.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011292 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303185+0000", 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.491825+0000", 2024-09-16T01:52:56.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.18864050500000001 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.498314+0000", 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064884210000000003 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551379+0000", 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053065042999999999 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551446+0000", 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7477999999999993e-05 2024-09-16T01:52:56.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551537+0000", 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1055000000000001e-05 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.576787+0000", 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025250156999999999 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591154+0000", 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01436693 2024-09-16T01:52:56.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591228+0000", 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4103000000000005e-05 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591355+0000", 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126367 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4250 3.10 3:0a05e26b:::100000001f8.00000297:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 33.467345447, 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.149078401, 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4250 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310162+0000", 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6599999999999999e-06 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407037+0000", 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096875366000000004 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407073+0000", 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5969000000000001e-05 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407088+0000", 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5034999999999999e-05 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407218+0000", 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129462 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407332+0000", 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114183 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:56.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407516+0000", 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018366500000000001 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.446008+0000", 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038492551999999999 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456958+0000", 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01095023 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.458896+0000", 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019376650000000001 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.459238+0000", 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034161400000000001 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4504 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.247718250999998, 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22042661699999999, 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.883 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4504 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:52:56.884 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529789+0000", 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5409999999999999e-06 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577075+0000", 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047285897 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.885 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577104+0000", 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9173999999999999e-05 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577115+0000", 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0895e-05 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577223+0000", 2024-09-16T01:52:56.886 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108194 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577330+0000", 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107089 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577485+0000", 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154532 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.887 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.599121+0000", 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021636636000000001 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750098+0000", 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15097622499999999 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.888 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750164+0000", 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6172000000000003e-05 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750213+0000", 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9261999999999997e-05 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4512 3.19 3:987f889e:::100000001f8.0000039d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.145524594000001, 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12513411299999999, 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4512 2024-09-16T01:52:56.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.891 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631984+0000", 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8999999999999999e-06 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679187+0000", 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047202698000000001 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.892 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679218+0000", 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1775e-05 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679233+0000", 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4182000000000001e-05 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682406+0000", 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031734889999999998 2024-09-16T01:52:56.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682504+0000", 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7374999999999995e-05 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.683089+0000", 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058535 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.894 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.705462+0000", 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022372670000000001 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757013+0000", 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051551620999999999 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.895 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757066+0000", 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3044999999999998e-05 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757114+0000", 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8007999999999998e-05 2024-09-16T01:52:56.896 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4553 3.6 3:62a9ab61:::100000001f8.000003c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.610865162, 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.118770112, 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.897 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4553 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.898 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166642+0000", 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0470000000000001e-06 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.899 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206911+0000", 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040269395999999999 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206944+0000", 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3185000000000002e-05 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206957+0000", 2024-09-16T01:52:56.900 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2894999999999999e-05 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207157+0000", 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199908 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207262+0000", 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105509 2024-09-16T01:52:56.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207423+0000", 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016019999999999999 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.902 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.221084+0000", 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013661240999999999 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285289+0000", 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064205110999999995 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.903 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285358+0000", 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9015000000000002e-05 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285410+0000", 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1604999999999998e-05 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.904 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4560 3.39 3:9e09c0a8:::100000001f8.000003cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.531779615000001, 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11688955500000001, 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4560 2024-09-16T01:52:56.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245728+0000", 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.548e-06 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.907 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301561+0000", 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055832859999999998 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301606+0000", 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5423000000000002e-05 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301620+0000", 2024-09-16T01:52:56.908 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.453e-05 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301728+0000", 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107836 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301847+0000", 2024-09-16T01:52:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119127 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.302620+0000", 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000772461 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.328630+0000", 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026010253000000001 2024-09-16T01:52:56.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362241+0000", 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033611125999999998 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362382+0000", 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014118400000000001 2024-09-16T01:52:56.911 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362615+0000", 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023220700000000001 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5800 3.8 3:139dc79b:::100000001f8.000008a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.40512262, 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.57801824300000004, 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.913 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5800 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372384+0000", 2024-09-16T01:52:56.915 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2840000000000001e-06 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387317+0000", 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014932126 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.916 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387330+0000", 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3658e-05 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387338+0000", 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.917 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389339+0000", 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020019460000000001 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389354+0000", 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.472e-05 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.918 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899029+0000", 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50967473200000002 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899208+0000", 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017872499999999999 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899396+0000", 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018809 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927572+0000", 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028175951000000001 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.920 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950173+0000", 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022601711 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950246+0000", 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2533000000000005e-05 2024-09-16T01:52:56.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950400+0000", 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154418 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5815 3.21 3:84e25a9c:::100000001f8.000008b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:56.922 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.252945436999999, 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.133812769, 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5815 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.923 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.924 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524561+0000", 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.968e-06 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641841+0000", 2024-09-16T01:52:56.925 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11728021 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641861+0000", 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9939e-05 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641868+0000", 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5339999999999996e-06 2024-09-16T01:52:56.926 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641942+0000", 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4416e-05 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642034+0000", 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1916999999999994e-05 2024-09-16T01:52:56.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642172+0000", 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013773100000000001 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656243+0000", 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014070719000000001 2024-09-16T01:52:56.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658259+0000", 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020165389999999999 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658323+0000", 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3876999999999998e-05 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.929 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658372+0000", 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8918999999999997e-05 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5821 3.8 3:10a53e03:::100000001f8.000008ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 14.100286678, 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27338652600000002, 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:56.930 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5821 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:56.931 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677219+0000", 2024-09-16T01:52:56.932 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.344e-06 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702527+0000", 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025308081999999999 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702546+0000", 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8698999999999998e-05 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.933 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702555+0000", 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2609999999999999e-06 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702663+0000", 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010725 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.934 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702675+0000", 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2743e-05 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910154+0000", 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.207478311 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910331+0000", 2024-09-16T01:52:56.935 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000177057 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910525+0000", 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000194074 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927630+0000", 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017105187000000001 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.936 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950420+0000", 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022789713 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950473+0000", 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3115999999999998e-05 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:56.937 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950605+0000", 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013168899999999999 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:56.938 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:56.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.778+0000 7fbd8f400640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbd9002d5a0 0x7fbd90204f50 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.779+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd9002d9e0 con 0x7fbd9002d5a0 2024-09-16T01:52:56.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:52:56.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.793+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd9002d9e0 con 0x7fbd9002d5a0 2024-09-16T01:52:56.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.796+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd9002d9e0 con 0x7fbd9002d5a0 2024-09-16T01:52:56.941 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.796+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbd902059a0 0x7fbd90207d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.796+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbd9020abe0 con 0x7fbd902059a0 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.796+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbd902059a0 0x7fbd90207d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.797+0000 7fbd94c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbd902059a0 0x7fbd90207d90 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.798+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd9020abe0 con 0x7fbd902059a0 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.812+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd90208350 con 0x7fbd902059a0 2024-09-16T01:52:56.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.814+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd90208350 con 0x7fbd902059a0 2024-09-16T01:52:56.944 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:56.945 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:56.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.814+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbd90208a70 0x7fbd90208eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:56.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.815+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbd9020df50 con 0x7fbd90208a70 2024-09-16T01:52:56.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.815+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbd90208a70 0x7fbd90208eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:56.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.816+0000 7fbd8fe00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbd90208a70 0x7fbd90208eb0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:56.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.816+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd9020df50 con 0x7fbd90208a70 2024-09-16T01:52:56.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.830+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fbd90209460 con 0x7fbd90208a70 2024-09-16T01:52:56.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.832+0000 7fbd87e00640 1 -- 172.21.15.146:0/4161811749 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbd90209460 con 0x7fbd90208a70 2024-09-16T01:52:56.946 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbd900cccd0 msgr2=0x7fbd9014a540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fbd900cccd0 0x7fbd9014a540 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbd9014aa80 msgr2=0x7fbd901f2140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fbd9014aa80 0x7fbd901f2140 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbd900ca5a0 msgr2=0x7fbd900ca980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fbd900ca5a0 0x7fbd900ca980 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbd9006ea70 msgr2=0x7fbd9006eeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fbd9006ea70 0x7fbd9006eeb0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbd9006f720 msgr2=0x7fbd9002cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbd9006f720 0x7fbd9002cb50 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbd9002d5a0 msgr2=0x7fbd90204f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fbd9002d5a0 0x7fbd90204f50 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbd902059a0 msgr2=0x7fbd90207d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbd902059a0 0x7fbd90207d90 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbd90208a70 msgr2=0x7fbd90208eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fbd90208a70 0x7fbd90208eb0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd94c00640 1 -- 172.21.15.146:0/4161811749 reap_dead start 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd94c00640 1 -- 172.21.15.146:0/4161811749 reap_dead start 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd94c00640 1 -- 172.21.15.146:0/4161811749 reap_dead start 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd94c00640 1 -- 172.21.15.146:0/4161811749 reap_dead start 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd94c00640 1 -- 172.21.15.146:0/4161811749 reap_dead start 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd7006c2b0 msgr2=0x7fbd7006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd7006c2b0 0x7fbd7006e770 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbd90140bc0 tx=0x7fbd7c009040 comp rx=0 tx=0).stop 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd900c6ab0 msgr2=0x7fbd9014c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd900c6ab0 0x7fbd9014c250 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbd80031000 tx=0x7fbd800046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 shutdown_connections 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbd7006c2b0 0x7fbd7006e770 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 --2- 172.21.15.146:0/4161811749 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbd900c6ab0 0x7fbd9014c250 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbd80031000 tx=0x7fbd800046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:56.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 >> 172.21.15.146:0/4161811749 conn(0x7fbd9006bc10 msgr2=0x7fbd9014d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:56.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 shutdown_connections 2024-09-16T01:52:56.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:56.833+0000 7fbd96c00640 1 -- 172.21.15.146:0/4161811749 wait complete. 2024-09-16T01:52:56.981 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:56.979+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:57.043 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:57.041+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:57.144 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:57.142+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:57.149 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:57.149 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:57.150 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:57.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:52:57.250 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:57.248+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:57.346 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:57.345+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:57.374 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:57.374 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:57.374 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:57.374 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:57.374 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:57.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:52:57.447 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:57.446+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:57.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:52:57.547 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:57.546+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:57.602 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:57.602 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.307154373000003, 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.603 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.604 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.605 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.606 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:52:57.607 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.608 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.277993518000002, 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.609 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:52:57.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:52:57.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.614 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.245641950000007, 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:52:57.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:52:57.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:52:57.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.229694683999995, 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.621 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:52:57.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:52:57.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:52:57.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:52:57.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:57.627 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.682619518999999, 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:52:57.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.497690163000001, 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.634 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:52:57.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:52:57.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.495152355000002, 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.641 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:52:57.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:52:57.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.646 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.885741123000003, 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:57.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.648 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:52:57.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:52:57.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:52:57.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.847484688000002, 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.653 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:52:57.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.658 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 41.772412953, 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:52:57.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:52:57.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4142 3.16 3:6ba609e9:::100000001f8.0000022b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.811316398999999, 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108516207, 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4142 2024-09-16T01:52:57.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787022+0000", 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.61e-06 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842519+0000", 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055496399000000002 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842539+0000", 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0820000000000001e-05 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842550+0000", 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0414999999999999e-05 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842641+0000", 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1657999999999996e-05 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842742+0000", 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010075 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842888+0000", 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014623699999999999 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.892001+0000", 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112211000000003 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895434+0000", 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034334869999999998 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895499+0000", 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5313000000000003e-05 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895537+0000", 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7307e-05 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.670 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4246 3.34 3:2f44359e:::100000001f8.00000293:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.268703836999997, 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.142262732, 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4246 2024-09-16T01:52:57.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329635+0000", 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9180000000000001e-06 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410104+0000", 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080469499 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410128+0000", 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3243000000000001e-05 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410141+0000", 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3067999999999999e-05 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419779+0000", 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096380589999999992 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419889+0000", 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-09-16T01:52:57.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.420057+0000", 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016837699999999999 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456784+0000", 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036727237000000003 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471769+0000", 2024-09-16T01:52:57.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014985112 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471839+0000", 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9275000000000003e-05 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471896+0000", 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7259000000000002e-05 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4247 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.188148730000002, 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120085209, 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4247 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.677 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410190+0000", 2024-09-16T01:52:57.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451530+0000", 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041340783999999998 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451572+0000", 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1690000000000002e-05 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451620+0000", 2024-09-16T01:52:57.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8180999999999998e-05 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452310+0000", 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068996900000000002 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452413+0000", 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010248599999999999 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452588+0000", 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017574800000000001 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.518054+0000", 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065465819999999994 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530168+0000", 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012113413999999999 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530225+0000", 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7284999999999998e-05 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530273+0000", 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8381999999999999e-05 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4503 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:57.682 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.086593375, 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13319203700000001, 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4503 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511746+0000", 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0250000000000001e-06 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576368+0000", 2024-09-16T01:52:57.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064622151000000003 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576386+0000", 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7878999999999999e-05 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576398+0000", 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578343+0000", 2024-09-16T01:52:57.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001945397 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578445+0000", 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102001 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578601+0000", 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015608399999999999 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.643878+0000", 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065277247999999996 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644839+0000", 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096038899999999997 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644895+0000", 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6484000000000003e-05 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644936+0000", 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0324999999999998e-05 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4504 3.f e101/94 3:f3dac423:::100000001f8.00000395:head v 101'76, mlcod=100'74)", 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 31.003667258, 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14827673899999999, 2024-09-16T01:52:57.688 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594672+0000", 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3309999999999998e-06 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629004+0000", 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034332342000000002 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629020+0000", 2024-09-16T01:52:57.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6121e-05 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629029+0000", 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7479999999999999e-06 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635523+0000", 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064935909999999999 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635549+0000", 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5808000000000001e-05 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742856+0000", 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107307556 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742946+0000", 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0241999999999994e-05 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.692 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4549 3.2 3:428d9500:::100000001f8.000003c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 30.428703810999998, 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11639424900000001, 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4549 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169636+0000", 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8660000000000002e-06 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212280+0000", 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042644350999999997 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212312+0000", 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2088999999999997e-05 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212327+0000", 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521e-05 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212426+0000", 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9245000000000003e-05 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212539+0000", 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112711 2024-09-16T01:52:57.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212725+0000", 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018583400000000001 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285881+0000", 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073156321999999996 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285914+0000", 2024-09-16T01:52:57.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2929999999999998e-05 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285979+0000", 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5048000000000005e-05 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.286027+0000", 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8331999999999997e-05 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5790 3.16 3:687a9a46:::100000001f8.0000089b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.387668675, 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.65913515700000003, 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5790 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210670+0000", 2024-09-16T01:52:57.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244952+0000", 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034282188999999998 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244970+0000", 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8179999999999999e-05 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244982+0000", 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1736000000000001e-05 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245078+0000", 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5439000000000006e-05 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245090+0000", 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2177e-05 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791600+0000", 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.546509829 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791770+0000", 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017051699999999999 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791950+0000", 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018029299999999999 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829483+0000", 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037532101999999998 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867462+0000", 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037979295000000003 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869661+0000", 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021987059999999999 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869702+0000", 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1148e-05 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869803+0000", 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101696 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.705 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5793 3.16 3:68c3c174:::100000001f8.0000089e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.326958055, 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.59860013000000001, 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5793 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271381+0000", 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0669999999999999e-06 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311762+0000", 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040380644 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311785+0000", 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3090000000000001e-05 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311798+0000", 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3057999999999999e-05 2024-09-16T01:52:57.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311911+0000", 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113429 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311923+0000", 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1915e-05 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829575+0000", 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51765227000000003 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829754+0000", 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017901700000000001 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829935+0000", 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018041800000000001 2024-09-16T01:52:57.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867483+0000", 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037548040999999997 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869178+0000", 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016955379999999999 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869843+0000", 2024-09-16T01:52:57.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000664345 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869882+0000", 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8976000000000001e-05 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869979+0000", 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7322000000000001e-05 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5945 3.2b 3:d7b2ba59:::100000001f8.00000936:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.200817589, 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11205129899999999, 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5945 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397521+0000", 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:52:57.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429779+0000", 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032257503999999999 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429805+0000", 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.716 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.716 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429818+0000", 2024-09-16T01:52:57.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2687999999999999e-05 2024-09-16T01:52:57.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429896+0000", 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7898999999999994e-05 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430008+0000", 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011241100000000001 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430198+0000", 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000189197 2024-09-16T01:52:57.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.448700+0000", 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018502761999999999 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509440+0000", 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060739425999999999 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509514+0000", 2024-09-16T01:52:57.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4673999999999997e-05 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509571+0000", 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6249999999999998e-05 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5948 3.2d 3:b677c6b2:::100000001f8.00000939:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.168122448, 2024-09-16T01:52:57.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122297751, 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5948 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:57.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430216+0000", 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453110+0000", 2024-09-16T01:52:57.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022894134 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453131+0000", 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1120000000000001e-05 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453141+0000", 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5340000000000005e-06 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495855+0000", 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042714162 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495997+0000", 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014160999999999999 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537228+0000", 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041231297 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537360+0000", 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264400000000001 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6,7", 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537537+0000", 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017664800000000001 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548049+0000", 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511915 2024-09-16T01:52:57.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548065+0000", 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5812e-05 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552281+0000", 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042165600000000003 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552355+0000", 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3712000000000002e-05 2024-09-16T01:52:57.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552512+0000", 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015707300000000001 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:57.726 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:57.727 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:57.726+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:57.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:52:57.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.702+0000 7fb4b8600640 1 Processor -- start 2024-09-16T01:52:57.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.703+0000 7fb4b8600640 1 -- start start 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b8600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb4b00c6980 0x7fb4b00cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b8600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb4b00c7770 0x7fb4b015aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b8600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b015b420 con 0x7fb4b00cb410 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b8600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b015b9e0 con 0x7fb4b00c7770 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b8600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b015b700 con 0x7fb4b00c6980 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b6600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb4b00cb410 0x7fb4b00cbfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60342/0 (socket says 172.21.15.146:60342) 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b6600640 1 -- 172.21.15.146:0/681302542 learned_addr learned my addr 172.21.15.146:0/681302542 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b5c00640 1 --2- 172.21.15.146:0/681302542 >> v2:172.21.15.161:3300/0 conn(0x7fb4b00c6980 0x7fb4b00cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b5200640 1 --2- 172.21.15.146:0/681302542 >> v2:172.21.15.146:3300/0 conn(0x7fb4b00c7770 0x7fb4b015aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4afe00640 1 -- 172.21.15.146:0/681302542 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446844105 0 0) 0x7fb4b015b420 con 0x7fb4b00cb410 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.704+0000 7fb4b5c00640 1 -- 172.21.15.146:0/681302542 >> v2:172.21.15.146:3300/0 conn(0x7fb4b00c7770 msgr2=0x7fb4b015aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.705+0000 7fb4b5c00640 1 --2- 172.21.15.146:0/681302542 >> v2:172.21.15.146:3300/0 conn(0x7fb4b00c7770 0x7fb4b015aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:57.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.705+0000 7fb4b5c00640 1 -- 172.21.15.146:0/681302542 >> v1:172.21.15.146:6789/0 conn(0x7fb4b00cb410 legacy=0x7fb4b00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.705+0000 7fb4b5c00640 1 -- 172.21.15.146:0/681302542 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b00ccc50 con 0x7fb4b00c6980 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.706+0000 7fb4b5c00640 1 --2- 172.21.15.146:0/681302542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b00cd730 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00cde90 tx=0x7fb4a402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f896e40f12556a76 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.706+0000 7fb4afe00640 1 -- 172.21.15.146:0/681302542 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4037070 con 0x7fb4b00c6980 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.706+0000 7fb4afe00640 1 -- 172.21.15.146:0/681302542 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a4032440 con 0x7fb4b00c6980 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.706+0000 7fb4afe00640 1 -- 172.21.15.146:0/681302542 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a402ae60 con 0x7fb4b00c6980 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 -- 172.21.15.146:0/681302542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 msgr2=0x7fb4b00cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 --2- 172.21.15.146:0/681302542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b00cd730 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00cde90 tx=0x7fb4a402a110 comp rx=0 tx=0).stop 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 -- 172.21.15.146:0/681302542 shutdown_connections 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 --2- 172.21.15.146:0/681302542 >> v2:172.21.15.146:3300/0 conn(0x7fb4b00c7770 0x7fb4b015aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 --2- 172.21.15.146:0/681302542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b00cd730 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 -- 172.21.15.146:0/681302542 >> 172.21.15.146:0/681302542 conn(0x7fb4b006cb70 msgr2=0x7fb4b006e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 -- 172.21.15.146:0/681302542 shutdown_connections 2024-09-16T01:52:57.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.707+0000 7fb4b8600640 1 -- 172.21.15.146:0/681302542 wait complete. 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.709+0000 7fb4b8600640 1 Processor -- start 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.709+0000 7fb4b8600640 1 -- start start 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.710+0000 7fb4b8600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b01e5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.710+0000 7fb4b8600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b00c7770 0x7fb4b01f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.710+0000 7fb4b8600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 0x7fb4b01e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.710+0000 7fb4b8600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b00d0fe0 con 0x7fb4b00cb410 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.710+0000 7fb4b8600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b00cd210 con 0x7fb4b00c6980 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.710+0000 7fb4b8600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b00cc6f0 con 0x7fb4b00c7770 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 0x7fb4b01e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 0x7fb4b01e0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57636/0 (socket says 172.21.15.146:57636) 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 -- 172.21.15.146:0/3062580824 learned_addr learned my addr 172.21.15.146:0/3062580824 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b5c00640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b01e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:57.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b5200640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b00c7770 0x7fb4b01f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 -- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b00c7770 msgr2=0x7fb4b01f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b00c7770 0x7fb4b01f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 -- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 msgr2=0x7fb4b01e5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b01e5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 -- 172.21.15.146:0/3062580824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b01e15c0 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4b6600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 0x7fb4b01e0b10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800caf0 tx=0x7fb4a80021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.711+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a800aba0 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.712+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a800ad40 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.712+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b01570b0 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.712+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8012640 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.712+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b0157670 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.713+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b015b420 con 0x7fb4b00cb410 2024-09-16T01:52:57.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.714+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb4a80127e0 con 0x7fb4b00cb410 2024-09-16T01:52:57.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.714+0000 7fb4ad600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb49006c2e0 0x7fb49006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:57.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.715+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 5894+0+0 (secure 0 0 0) 0x7fb4a808b870 con 0x7fb4b00cb410 2024-09-16T01:52:57.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.715+0000 7fb4b5c00640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb49006c2e0 0x7fb49006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:57.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.716+0000 7fb4b5c00640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb49006c2e0 0x7fb49006e7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4034000 tx=0x7fb4a402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:57.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.720+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb4b015b420 con 0x7fb4b00cb410 2024-09-16T01:52:57.829 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:57.828+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:57.907 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:57.907 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:57.907 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:52:57.907 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:57.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:57.917+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb4b015bbf0 con 0x7fb4b00cb410 2024-09-16T01:52:57.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:52:57.929 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:52:57.928+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:52:58.030 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:58.029+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:58.090 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:58.090 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:52:58.090 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:52:58.090 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:52:58.090 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:58.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:52:58.149 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:58.147+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:58.211 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 118.421479623, 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.756 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:52:58.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.758 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.759 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.760 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:52:58.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:52:58.762 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 118.27901995800001, 2024-09-16T01:52:58.763 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:52:58.765 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.766 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.767 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:52:58.768 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:52:58.769 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.770 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 117.93155373499999, 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.771 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:52:58.772 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.773 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.774 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:52:58.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.776 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.777 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 98.047262371000002, 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.778 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.779 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.780 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:52:58.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:52:58.782 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.783 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.009522868000005, 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.784 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:52:58.785 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.786 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.787 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.788 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:52:58.789 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.150535868000006, 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.790 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.791 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:52:58.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:52:58.793 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.794 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.795 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.283496893999995, 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:52:58.798 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:52:58.799 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.800 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:52:58.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.802 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.087544672999996, 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.803 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:52:58.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:52:58.805 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.806 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:52:58.807 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.808 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 69.046542729999999, 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.809 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.810 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:52:58.811 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.812 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.813 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.814 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 68.822315169999996, 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:52:58.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.816 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.817 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:52:58.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:52:58.819 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.820 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 62.865533616999997, 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.821 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:52:58.822 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.823 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.824 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.825 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.826 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 56.954553267999998, 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.827 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.828 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:52:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:52:58.830 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:52:58.831 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.832 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.833 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.518022285999997, 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.834 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.835 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.836 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:52:58.837 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.838 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.839 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.840 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.494868367999999, 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.841 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.842 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.843 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.844 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.845 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.483932525, 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.846 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.847 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:52:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:52:58.849 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.453738100999999, 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.850 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:52:58.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:52:58.853 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:52:58.854 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 42.412332648000003, 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:52:58.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.857 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.858 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.859 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.860 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.861 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5551 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.431525558000001, 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13476244500000001, 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5551 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.862 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838694+0000", 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.306e-06 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.863 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866535+0000", 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027841175999999999 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866552+0000", 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7387000000000001e-05 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866563+0000", 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0447e-05 2024-09-16T01:52:58.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866665+0000", 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101894 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866788+0000", 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123194 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866953+0000", 2024-09-16T01:52:58.865 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165365 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.918808+0000", 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051855367999999999 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973333+0000", 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054524250000000003 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.866 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973402+0000", 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9648999999999997e-05 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973454+0000", 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.867 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5812 3.a 3:51094797:::100000001f8.000008b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 15.737826442999999, 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14779089000000001, 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5812 2024-09-16T01:52:58.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.869 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532392+0000", 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8679999999999999e-06 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653045+0000", 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120652599 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653062+0000", 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6929e-05 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.870 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653073+0000", 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0925e-05 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656653+0000", 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035800049999999998 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656738+0000", 2024-09-16T01:52:58.871 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5599000000000005e-05 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656890+0000", 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151874 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.676041+0000", 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019150358999999999 2024-09-16T01:52:58.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680071+0000", 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004030624 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680131+0000", 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9704000000000003e-05 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.873 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680181+0000", 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0404000000000001e-05 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5944 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 13.882997298999999, 2024-09-16T01:52:58.874 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.115163742, 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5944 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:52:58.875 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387222+0000", 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418222+0000", 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031000603000000002 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:52:58.877 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418239+0000", 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6833e-05 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418249+0000", 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8179999999999999e-06 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:52:58.878 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418846+0000", 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059710099999999995 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418945+0000", 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8826000000000006e-05 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.879 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.419093+0000", 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148553 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.442472+0000", 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023378423999999998 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:52:58.880 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502282+0000", 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059810025000000003 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502341+0000", 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8922000000000003e-05 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502383+0000", 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2590999999999999e-05 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.881 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:52:58.882 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:52:58.882 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:52:58.882 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:52:58.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.524+0000 7fb4ad600640 1 -- 172.21.15.146:0/3062580824 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v122) ==== 97+0+0 (secure 0 0 0) 0x7fb4b015bbf0 con 0x7fb4b00cb410 2024-09-16T01:52:58.884 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:52:58.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb49006c2e0 msgr2=0x7fb49006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:58.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb49006c2e0 0x7fb49006e7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4034000 tx=0x7fb4a402e000 comp rx=0 tx=0).stop 2024-09-16T01:52:58.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 msgr2=0x7fb4b01e0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 0x7fb4b01e0b10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800caf0 tx=0x7fb4a80021c0 comp rx=0 tx=0).stop 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 shutdown_connections 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb49006c2e0 0x7fb49006e7a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b00cb410 0x7fb4b01e0b10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b00c7770 0x7fb4b01f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.526+0000 7fb4b8600640 1 --2- 172.21.15.146:0/3062580824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4b00c6980 0x7fb4b01e5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.527+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 >> 172.21.15.146:0/3062580824 conn(0x7fb4b006cb70 msgr2=0x7fb4b00c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.527+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 shutdown_connections 2024-09-16T01:52:58.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:58.527+0000 7fb4b8600640 1 -- 172.21.15.146:0/3062580824 wait complete. 2024-09-16T01:52:58.886 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:58.753+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:58.890 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:52:58.888+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:52:58.936 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:52:58.936 INFO:tasks.workunit.client.0.smithi146.stdout: Children see throughput for 1 re-readers = 3387241.00 kB/sec 2024-09-16T01:52:58.936 INFO:tasks.workunit.client.0.smithi146.stdout: Parent sees throughput for 1 re-readers = 3386312.30 kB/sec 2024-09-16T01:52:58.936 INFO:tasks.workunit.client.0.smithi146.stdout: Min throughput per process = 3387241.00 kB/sec 2024-09-16T01:52:58.936 INFO:tasks.workunit.client.0.smithi146.stdout: Max throughput per process = 3387241.00 kB/sec 2024-09-16T01:52:58.936 INFO:tasks.workunit.client.0.smithi146.stdout: Avg throughput per process = 3387241.00 kB/sec 2024-09-16T01:52:58.937 INFO:tasks.workunit.client.0.smithi146.stdout: Min xfer = 10485760.00 kB 2024-09-16T01:52:58.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:52:58.988 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:52:58.986+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:52:59.002 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:59.002 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:59.002 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:52:59.002 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:59.011 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:52:59.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.046+0000 7fdd1f200640 1 Processor -- start 2024-09-16T01:52:59.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.047+0000 7fdd1f200640 1 -- start start 2024-09-16T01:52:59.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1f200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd200c6b70 0x7fdd2015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdd200cc040 0x7fdd200cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1f200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd2006d140 con 0x7fdd200c75a0 2024-09-16T01:52:59.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdd2015b880 con 0x7fdd200cc040 2024-09-16T01:52:59.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1f200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdd200cab20 con 0x7fdd200c6b70 2024-09-16T01:52:59.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1de00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd200c6b70 0x7fdd2015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1d400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdd200c75a0 0x7fdd200cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36274/0 (socket says 172.21.15.146:36274) 2024-09-16T01:52:59.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdd200cc040 0x7fdd200cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.048+0000 7fdd1d400640 1 -- 172.21.15.146:0/281912547 learned_addr learned my addr 172.21.15.146:0/281912547 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:59.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.049+0000 7fdd1ca00640 1 -- 172.21.15.146:0/281912547 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347799883 0 0) 0x7fdd2006d140 con 0x7fdd200c75a0 2024-09-16T01:52:59.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.049+0000 7fdd1de00640 1 -- 172.21.15.146:0/281912547 >> v2:172.21.15.146:3300/0 conn(0x7fdd200cc040 msgr2=0x7fdd200cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:52:59.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.049+0000 7fdd1de00640 1 --2- 172.21.15.146:0/281912547 >> v2:172.21.15.146:3300/0 conn(0x7fdd200cc040 0x7fdd200cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.049+0000 7fdd1de00640 1 -- 172.21.15.146:0/281912547 >> v1:172.21.15.161:6789/0 conn(0x7fdd200c75a0 legacy=0x7fdd200cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.049+0000 7fdd1de00640 1 -- 172.21.15.146:0/281912547 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2006d420 con 0x7fdd200c6b70 2024-09-16T01:52:59.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.050+0000 7fdd1de00640 1 --2- 172.21.15.146:0/281912547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd2015b340 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdd20075ec0 tx=0x7fdd1002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b6af289871042a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.050+0000 7fdd1ca00640 1 -- 172.21.15.146:0/281912547 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10037070 con 0x7fdd200c6b70 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1ca00640 1 -- 172.21.15.146:0/281912547 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd10032440 con 0x7fdd200c6b70 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1ca00640 1 -- 172.21.15.146:0/281912547 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1002ae60 con 0x7fdd200c6b70 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 -- 172.21.15.146:0/281912547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 msgr2=0x7fdd2015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 --2- 172.21.15.146:0/281912547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd2015b340 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdd20075ec0 tx=0x7fdd1002a110 comp rx=0 tx=0).stop 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 -- 172.21.15.146:0/281912547 shutdown_connections 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 --2- 172.21.15.146:0/281912547 >> v2:172.21.15.146:3300/0 conn(0x7fdd200cc040 0x7fdd200cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 --2- 172.21.15.146:0/281912547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd2015b340 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 -- 172.21.15.146:0/281912547 >> 172.21.15.146:0/281912547 conn(0x7fdd200ccc20 msgr2=0x7fdd200cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 -- 172.21.15.146:0/281912547 shutdown_connections 2024-09-16T01:52:59.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.051+0000 7fdd1f200640 1 -- 172.21.15.146:0/281912547 wait complete. 2024-09-16T01:52:59.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.053+0000 7fdd1f200640 1 Processor -- start 2024-09-16T01:52:59.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.054+0000 7fdd1f200640 1 -- start start 2024-09-16T01:52:59.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd201f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd200c75a0 0x7fdd200cf390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdd200cc040 0x7fdd2015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200cb050 con 0x7fdd200c75a0 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd200cad70 con 0x7fdd200c6b70 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdd200cbc80 con 0x7fdd200cc040 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd201f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd201f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43468/0 (socket says 172.21.15.146:43468) 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1de00640 1 -- 172.21.15.146:0/2472080369 learned_addr learned my addr 172.21.15.146:0/2472080369 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1e800640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdd200cc040 0x7fdd2015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1d400640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd200c75a0 0x7fdd200cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1de00640 1 -- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdd200cc040 msgr2=0x7fdd2015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1de00640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdd200cc040 0x7fdd2015cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.055+0000 7fdd1de00640 1 -- 172.21.15.146:0/2472080369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd200c75a0 msgr2=0x7fdd200cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd1de00640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd200c75a0 0x7fdd200cf390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd1de00640 1 -- 172.21.15.146:0/2472080369 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2015db40 con 0x7fdd200c6b70 2024-09-16T01:52:59.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd1de00640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd201f3e10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000c00 tx=0x7fdd100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10037040 con 0x7fdd200c6b70 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd10042070 con 0x7fdd200c6b70 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10030920 con 0x7fdd200c6b70 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd20158000 con 0x7fdd200c6b70 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.056+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd201f4350 con 0x7fdd200c6b70 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.057+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2006cd20 con 0x7fdd200c6b70 2024-09-16T01:52:59.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.058+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdd10030ac0 con 0x7fdd200c6b70 2024-09-16T01:52:59.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.059+0000 7fdd16000640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdd0006c2b0 0x7fdd0006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.059+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5894+0+0 (secure 0 0 0) 0x7fdd100abd90 con 0x7fdd200c6b70 2024-09-16T01:52:59.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.059+0000 7fdd1d400640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdd0006c2b0 0x7fdd0006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.060+0000 7fdd1d400640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdd0006c2b0 0x7fdd0006e770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdd201f19d0 tx=0x7fdd0c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.063+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdd10079de0 con 0x7fdd200c6b70 2024-09-16T01:52:59.096 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:59.095+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:59.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.106+0000 7f14f0800640 1 Processor -- start 2024-09-16T01:52:59.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.106+0000 7f14f0800640 1 -- start start 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14f0800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e80c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14f0800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e806c050 con 0x7f14e80c1e70 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14f0800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14e80731b0 con 0x7f14e8073f30 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14f0800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f14e8073490 con 0x7f14e815b6f0 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14ee800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f14e8073f30 0x7f14e8165330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36286/0 (socket says 172.21.15.146:36286) 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14ee800640 1 -- 172.21.15.146:0/494539191 learned_addr learned my addr 172.21.15.146:0/494539191 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:52:59.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14ed400640 1 --2- 172.21.15.146:0/494539191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e80c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14eca00640 1 -- 172.21.15.146:0/494539191 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 250521168 0 0) 0x7f14e80731b0 con 0x7f14e8073f30 2024-09-16T01:52:59.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14eca00640 1 -- 172.21.15.146:0/494539191 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d0003520 con 0x7f14e8073f30 2024-09-16T01:52:59.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14ed400640 1 -- 172.21.15.146:0/494539191 >> v1:172.21.15.161:6789/0 conn(0x7f14e8073f30 legacy=0x7f14e8165330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14eca00640 1 -- 172.21.15.146:0/494539191 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3621733494 0 0) 0x7f14e806c050 con 0x7f14e80c1e70 2024-09-16T01:52:59.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14ed400640 1 -- 172.21.15.146:0/494539191 >> v1:172.21.15.146:6789/0 conn(0x7f14e80c1e70 legacy=0x7f14e815afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.108+0000 7f14ed400640 1 -- 172.21.15.146:0/494539191 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e8073770 con 0x7f14e815b6f0 2024-09-16T01:52:59.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.110+0000 7f14ed400640 1 --2- 172.21.15.146:0/494539191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e80c2250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f14d80097d0 tx=0x7f14d802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=592eb8f63028fcd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.110+0000 7f14eca00640 1 -- 172.21.15.146:0/494539191 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8037070 con 0x7f14e815b6f0 2024-09-16T01:52:59.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.110+0000 7f14eca00640 1 -- 172.21.15.146:0/494539191 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d8032440 con 0x7f14e815b6f0 2024-09-16T01:52:59.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.110+0000 7f14eca00640 1 -- 172.21.15.146:0/494539191 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d80327f0 con 0x7f14e815b6f0 2024-09-16T01:52:59.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 -- 172.21.15.146:0/494539191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 msgr2=0x7f14e80c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 --2- 172.21.15.146:0/494539191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e80c2250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f14d80097d0 tx=0x7f14d802a290 comp rx=0 tx=0).stop 2024-09-16T01:52:59.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 -- 172.21.15.146:0/494539191 shutdown_connections 2024-09-16T01:52:59.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 --2- 172.21.15.146:0/494539191 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e80c2250 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 -- 172.21.15.146:0/494539191 >> 172.21.15.146:0/494539191 conn(0x7f14e80ccb10 msgr2=0x7f14e80ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:59.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 -- 172.21.15.146:0/494539191 shutdown_connections 2024-09-16T01:52:59.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.111+0000 7f14f0800640 1 -- 172.21.15.146:0/494539191 wait complete. 2024-09-16T01:52:59.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.113+0000 7f14f0800640 1 Processor -- start 2024-09-16T01:52:59.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.113+0000 7f14f0800640 1 -- start start 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14f0800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e8073f30 0x7f14e80cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14f0800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14e80c1e70 0x7f14e814abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14f0800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e81e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14f0800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f14e80b3130 con 0x7f14e8073f30 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14f0800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f14e815aa70 con 0x7f14e80c1e70 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14f0800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f14e8155040 con 0x7f14e815b6f0 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14ee800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e81e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14ed400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14e80c1e70 0x7f14e814abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14ee800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e81e41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:56772/0 (socket says 172.21.15.146:56772) 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 learned_addr learned my addr 172.21.15.146:0/424284475 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14ed400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14e80c1e70 0x7f14e814abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:43474/0 (socket says 172.21.15.146:43474) 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.114+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e8073f30 0x7f14e80cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14e80c1e70 msgr2=0x7f14e814abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14e80c1e70 0x7f14e814abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e8073f30 msgr2=0x7f14e80cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14e8073f30 0x7f14e80cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e8152a70 con 0x7f14e815b6f0 2024-09-16T01:52:59.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14e80c1e70 0x7f14e814abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:52:59.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e81e41c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f14e400caa0 tx=0x7f14e400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4017070 con 0x7f14e815b6f0 2024-09-16T01:52:59.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e400aba0 con 0x7f14e815b6f0 2024-09-16T01:52:59.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e8152d50 con 0x7f14e815b6f0 2024-09-16T01:52:59.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4012400 con 0x7f14e815b6f0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.115+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e8153310 con 0x7f14e815b6f0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.116+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f14e80731b0 con 0x7f14e815b6f0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.117+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f14e401a070 con 0x7f14e815b6f0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.118+0000 7f14e2000640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f14cc06c2b0 0x7f14cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.118+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(122..122 src has 1..122) ==== 5894+0+0 (secure 0 0 0) 0x7f14e408b190 con 0x7f14e815b6f0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.118+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v122) ==== 54+0+16 (secure 0 0 0) 0x7f14e408d730 con 0x7f14e815b6f0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.118+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f14cc06c2b0 0x7f14cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.119+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f14e81e4700 0x7f14e814ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.119+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14e80c2710 con 0x7f14e81e4700 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.119+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f14e81e4700 0x7f14e814ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.119+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f14cc06c2b0 0x7f14cc06e770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f14dc000f60 tx=0x7f14dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.120+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f14e81e4700 0x7f14e814ba10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.121+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e80c2710 con 0x7f14e81e4700 2024-09-16T01:52:59.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.136+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e80c2960 con 0x7f14e81e4700 2024-09-16T01:52:59.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.139+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e80c2960 con 0x7f14e81e4700 2024-09-16T01:52:59.141 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:52:59.142 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:52:59.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.139+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f14e81453a0 0x7f14e81f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.139+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f14e80cc720 con 0x7f14e81453a0 2024-09-16T01:52:59.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.140+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f14e81453a0 0x7f14e81f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.140+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f14e81453a0 0x7f14e81f3b90 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.142+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e80cc720 con 0x7f14e81453a0 2024-09-16T01:52:59.157 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:59.158 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:52:59.158 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:52:59.158 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:52:59.158 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:59.166 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:52:59.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.173+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e815b1f0 con 0x7f14e81453a0 2024-09-16T01:52:59.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.176+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e815b1f0 con 0x7f14e81453a0 2024-09-16T01:52:59.178 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:52:59.179 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:52:59.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.176+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f14e81f45e0 0x7f14e8062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.176+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f14e81f49c0 con 0x7f14e81f45e0 2024-09-16T01:52:59.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.176+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f14e81f45e0 0x7f14e8062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.177+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f14e81f45e0 0x7f14e8062630 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.178+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e81f49c0 con 0x7f14e81f45e0 2024-09-16T01:52:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.192+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e81e4ae0 con 0x7f14e81f45e0 2024-09-16T01:52:59.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.194+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e81e4ae0 con 0x7f14e81f45e0 2024-09-16T01:52:59.197 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:52:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:52:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.195+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f14e8063080 0x7f14e806c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.195+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f14e80634c0 con 0x7f14e8063080 2024-09-16T01:52:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.195+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f14e8063080 0x7f14e806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.195+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f14e8063080 0x7f14e806c330 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.196+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e80634c0 con 0x7f14e8063080 2024-09-16T01:52:59.207 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:52:59.206+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:52:59.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.210+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e806cb40 con 0x7f14e8063080 2024-09-16T01:52:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.212+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e806cb40 con 0x7f14e8063080 2024-09-16T01:52:59.214 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:52:59.214 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:52:59.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.212+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f14e802c640 0x7f14e802caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.212+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f14e806d260 con 0x7f14e802c640 2024-09-16T01:52:59.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.212+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f14e802c640 0x7f14e802caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.213+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f14e802c640 0x7f14e802caa0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.214+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e806d260 con 0x7f14e802c640 2024-09-16T01:52:59.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.227+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e806d260 con 0x7f14e802c640 2024-09-16T01:52:59.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.229+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e806d260 con 0x7f14e802c640 2024-09-16T01:52:59.231 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:52:59.232 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:52:59.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.229+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f14e802d4f0 0x7f14e8206f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.229+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f14e802d970 con 0x7f14e802d4f0 2024-09-16T01:52:59.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.230+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f14e802d4f0 0x7f14e8206f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.230+0000 7f14ed400640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f14e802d4f0 0x7f14e8206f20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.231+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e802d970 con 0x7f14e802d4f0 2024-09-16T01:52:59.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.244+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e802d970 con 0x7f14e802d4f0 2024-09-16T01:52:59.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.246+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e802d970 con 0x7f14e802d4f0 2024-09-16T01:52:59.248 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:52:59.249 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:52:59.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.246+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f14e8207970 0x7f14e8209d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.246+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f14e820cb30 con 0x7f14e8207970 2024-09-16T01:52:59.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.247+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f14e8207970 0x7f14e8209d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.247+0000 7f14ee800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f14e8207970 0x7f14e8209d60 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.248+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e820cb30 con 0x7f14e8207970 2024-09-16T01:52:59.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.262+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e820a2a0 con 0x7f14e8207970 2024-09-16T01:52:59.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.264+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e820a2a0 con 0x7f14e8207970 2024-09-16T01:52:59.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:52:59.266 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:52:59.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.264+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f14e820a9c0 0x7f14e820ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:52:59.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.264+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f14e820fea0 con 0x7f14e820a9c0 2024-09-16T01:52:59.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.264+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f14e820a9c0 0x7f14e820ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:52:59.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.265+0000 7f14ede00640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f14e820a9c0 0x7f14e820ae00 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:52:59.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.266+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14e820fea0 con 0x7f14e820a9c0 2024-09-16T01:52:59.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.275+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd20157db0 con 0x7fdd0006c2b0 2024-09-16T01:52:59.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.279+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f14e820b450 con 0x7f14e820a9c0 2024-09-16T01:52:59.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.281+0000 7f14e2000640 1 -- 172.21.15.146:0/424284475 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14e820b450 con 0x7f14e820a9c0 2024-09-16T01:52:59.284 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:52:59.284 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f14e81e4700 msgr2=0x7f14e814ba10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f14e81e4700 0x7f14e814ba10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f14e81453a0 msgr2=0x7f14e81f3b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f14e81453a0 0x7f14e81f3b90 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f14e81f45e0 msgr2=0x7f14e8062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f14e81f45e0 0x7f14e8062630 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f14e8063080 msgr2=0x7f14e806c330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f14e8063080 0x7f14e806c330 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f14e802c640 msgr2=0x7f14e802caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f14e802c640 0x7f14e802caa0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 reap_dead start 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 reap_dead start 2024-09-16T01:52:59.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f14e802d4f0 msgr2=0x7f14e8206f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f14e802d4f0 0x7f14e8206f20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f14e8207970 msgr2=0x7f14e8209d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f14e8207970 0x7f14e8209d60 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f14e820a9c0 msgr2=0x7f14e820ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f14e820a9c0 0x7f14e820ae00 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7fdd16000640 1 -- 172.21.15.146:0/2472080369 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254882 (secure 0 0 0) 0x7fdd20157db0 con 0x7fdd0006c2b0 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f14cc06c2b0 msgr2=0x7f14cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f14cc06c2b0 0x7f14cc06e770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f14dc000f60 tx=0x7f14dc009040 comp rx=0 tx=0).stop 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14ee800640 1 -- 172.21.15.146:0/424284475 reap_dead start 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 msgr2=0x7f14e81e41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e81e41c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f14e400caa0 tx=0x7f14e400cf60 comp rx=0 tx=0).stop 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 shutdown_connections 2024-09-16T01:52:59.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 --2- 172.21.15.146:0/424284475 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f14e815b6f0 0x7f14e81e41c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 >> 172.21.15.146:0/424284475 conn(0x7f14e80ccb10 msgr2=0x7f14e81590a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:59.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 shutdown_connections 2024-09-16T01:52:59.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.282+0000 7f14f0800640 1 -- 172.21.15.146:0/424284475 wait complete. 2024-09-16T01:52:59.290 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:52:59.292 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:52:59.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdd0006c2b0 msgr2=0x7fdd0006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdd0006c2b0 0x7fdd0006e770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdd201f19d0 tx=0x7fdd0c007de0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 msgr2=0x7fdd201f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:52:59.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd201f3e10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000c00 tx=0x7fdd100046c0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 shutdown_connections 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdd0006c2b0 0x7fdd0006e770 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdd200cc040 0x7fdd2015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd200c75a0 0x7fdd200cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 --2- 172.21.15.146:0/2472080369 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd200c6b70 0x7fdd201f3e10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 >> 172.21.15.146:0/2472080369 conn(0x7fdd200ccc20 msgr2=0x7fdd200c7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 shutdown_connections 2024-09-16T01:52:59.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:52:59.286+0000 7fdd1f200640 1 -- 172.21.15.146:0/2472080369 wait complete. 2024-09-16T01:52:59.294 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:59.288+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:52:59.306 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T01:52:58.565871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738391577,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":118,"num_read_kb":102,"num_write":7207,"num_write_kb":25168808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":9886222853,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7648,"ondisk_log_size":7648,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":195,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1507164,"kb_used_data":1320008,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":564723876,"statfs":{"total":579820584960,"available":578277249024,"internally_reserved":0,"allocated":1351688192,"data_stored":21480815382,"data_compressed":1074236551,"data_compressed_allocated":1342312448,"data_compressed_original":21476216832,"omap_allocated":9562,"internal_metadata":191486630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":163577856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001266"},"pg_stats":[{"pgid":"3.3f","version":"112'104","reported_seq":392,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718260+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:52:53.718260+0000","last_peered":"2024-09-16T01:52:53.718260+0000","last_clean":"2024-09-16T01:52:53.718260+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:52:53.718260+0000","last_undegraded":"2024-09-16T01:52:53.718260+0000","last_fullsized":"2024-09-16T01:52:53.718260+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"113'94","reported_seq":302,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514319+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:52:53.514319+0000","last_peered":"2024-09-16T01:52:53.514319+0000","last_clean":"2024-09-16T01:52:53.514319+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:52:53.514319+0000","last_undegraded":"2024-09-16T01:52:53.514319+0000","last_fullsized":"2024-09-16T01:52:53.514319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"111'109","reported_seq":567,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513986+0000","last_change":"2024-09-16T01:52:51.502920+0000","last_active":"2024-09-16T01:52:53.513986+0000","last_peered":"2024-09-16T01:52:53.513986+0000","last_clean":"2024-09-16T01:52:53.513986+0000","last_became_active":"2024-09-16T01:52:51.502433+0000","last_became_peered":"2024-09-16T01:52:51.502433+0000","last_unstale":"2024-09-16T01:52:53.513986+0000","last_undegraded":"2024-09-16T01:52:53.513986+0000","last_fullsized":"2024-09-16T01:52:53.513986+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"113'125","reported_seq":339,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331930+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:52:54.331930+0000","last_peered":"2024-09-16T01:52:54.331930+0000","last_clean":"2024-09-16T01:52:54.331930+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:52:54.331930+0000","last_undegraded":"2024-09-16T01:52:54.331930+0000","last_fullsized":"2024-09-16T01:52:54.331930+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"114'112","reported_seq":397,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.516521+0000","last_change":"2024-09-16T01:52:53.516521+0000","last_active":"2024-09-16T01:52:53.516521+0000","last_peered":"2024-09-16T01:52:53.516521+0000","last_clean":"2024-09-16T01:52:53.516521+0000","last_became_active":"2024-09-16T01:52:53.516052+0000","last_became_peered":"2024-09-16T01:52:53.516052+0000","last_unstale":"2024-09-16T01:52:53.516521+0000","last_undegraded":"2024-09-16T01:52:53.516521+0000","last_fullsized":"2024-09-16T01:52:53.516521+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"112'102","reported_seq":363,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331633+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:52:54.331633+0000","last_peered":"2024-09-16T01:52:54.331633+0000","last_clean":"2024-09-16T01:52:54.331633+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:52:54.331633+0000","last_undegraded":"2024-09-16T01:52:54.331633+0000","last_fullsized":"2024-09-16T01:52:54.331633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"112'125","reported_seq":508,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511694+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:52:53.511694+0000","last_peered":"2024-09-16T01:52:53.511694+0000","last_clean":"2024-09-16T01:52:53.511694+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:52:53.511694+0000","last_undegraded":"2024-09-16T01:52:53.511694+0000","last_fullsized":"2024-09-16T01:52:53.511694+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"113'131","reported_seq":464,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513081+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:52:53.513081+0000","last_peered":"2024-09-16T01:52:53.513081+0000","last_clean":"2024-09-16T01:52:53.513081+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:52:53.513081+0000","last_undegraded":"2024-09-16T01:52:53.513081+0000","last_fullsized":"2024-09-16T01:52:53.513081+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"113'114","reported_seq":323,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512350+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:52:53.512350+0000","last_peered":"2024-09-16T01:52:53.512350+0000","last_clean":"2024-09-16T01:52:53.512350+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:52:53.512350+0000","last_undegraded":"2024-09-16T01:52:53.512350+0000","last_fullsized":"2024-09-16T01:52:53.512350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"112'117","reported_seq":430,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332086+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:52:54.332086+0000","last_peered":"2024-09-16T01:52:54.332086+0000","last_clean":"2024-09-16T01:52:54.332086+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:52:54.332086+0000","last_undegraded":"2024-09-16T01:52:54.332086+0000","last_fullsized":"2024-09-16T01:52:54.332086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"112'135","reported_seq":473,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717935+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:52:53.717935+0000","last_peered":"2024-09-16T01:52:53.717935+0000","last_clean":"2024-09-16T01:52:53.717935+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:52:53.717935+0000","last_undegraded":"2024-09-16T01:52:53.717935+0000","last_fullsized":"2024-09-16T01:52:53.717935+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"114'107","reported_seq":236,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511834+0000","last_change":"2024-09-16T01:52:50.499570+0000","last_active":"2024-09-16T01:52:53.511834+0000","last_peered":"2024-09-16T01:52:53.511834+0000","last_clean":"2024-09-16T01:52:53.511834+0000","last_became_active":"2024-09-16T01:52:50.498997+0000","last_became_peered":"2024-09-16T01:52:50.498997+0000","last_unstale":"2024-09-16T01:52:53.511834+0000","last_undegraded":"2024-09-16T01:52:53.511834+0000","last_fullsized":"2024-09-16T01:52:53.511834+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"113'112","reported_seq":446,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518184+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:52:55.518184+0000","last_peered":"2024-09-16T01:52:55.518184+0000","last_clean":"2024-09-16T01:52:55.518184+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:52:55.518184+0000","last_undegraded":"2024-09-16T01:52:55.518184+0000","last_fullsized":"2024-09-16T01:52:55.518184+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"113'107","reported_seq":368,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518070+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:52:55.518070+0000","last_peered":"2024-09-16T01:52:55.518070+0000","last_clean":"2024-09-16T01:52:55.518070+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:52:55.518070+0000","last_undegraded":"2024-09-16T01:52:55.518070+0000","last_fullsized":"2024-09-16T01:52:55.518070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"113'110","reported_seq":421,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513606+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:52:53.513606+0000","last_peered":"2024-09-16T01:52:53.513606+0000","last_clean":"2024-09-16T01:52:53.513606+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:52:53.513606+0000","last_undegraded":"2024-09-16T01:52:53.513606+0000","last_fullsized":"2024-09-16T01:52:53.513606+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"111'107","reported_seq":315,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717400+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:52:53.717400+0000","last_peered":"2024-09-16T01:52:53.717400+0000","last_clean":"2024-09-16T01:52:53.717400+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:52:53.717400+0000","last_undegraded":"2024-09-16T01:52:53.717400+0000","last_fullsized":"2024-09-16T01:52:53.717400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"113'104","reported_seq":409,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718369+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:52:53.718369+0000","last_peered":"2024-09-16T01:52:53.718369+0000","last_clean":"2024-09-16T01:52:53.718369+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:52:53.718369+0000","last_undegraded":"2024-09-16T01:52:53.718369+0000","last_fullsized":"2024-09-16T01:52:53.718369+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"114'109","reported_seq":510,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332177+0000","last_change":"2024-09-16T01:52:49.497350+0000","last_active":"2024-09-16T01:52:54.332177+0000","last_peered":"2024-09-16T01:52:54.332177+0000","last_clean":"2024-09-16T01:52:54.332177+0000","last_became_active":"2024-09-16T01:52:49.496908+0000","last_became_peered":"2024-09-16T01:52:49.496908+0000","last_unstale":"2024-09-16T01:52:54.332177+0000","last_undegraded":"2024-09-16T01:52:54.332177+0000","last_fullsized":"2024-09-16T01:52:54.332177+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:51.945635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"114'124","reported_seq":568,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518059+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:52:55.518059+0000","last_peered":"2024-09-16T01:52:55.518059+0000","last_clean":"2024-09-16T01:52:55.518059+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:52:55.518059+0000","last_undegraded":"2024-09-16T01:52:55.518059+0000","last_fullsized":"2024-09-16T01:52:55.518059+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"112'114","reported_seq":466,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511940+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:52:53.511940+0000","last_peered":"2024-09-16T01:52:53.511940+0000","last_clean":"2024-09-16T01:52:53.511940+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:52:53.511940+0000","last_undegraded":"2024-09-16T01:52:53.511940+0000","last_fullsized":"2024-09-16T01:52:53.511940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"113'95","reported_seq":341,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331805+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:52:54.331805+0000","last_peered":"2024-09-16T01:52:54.331805+0000","last_clean":"2024-09-16T01:52:54.331805+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:52:54.331805+0000","last_undegraded":"2024-09-16T01:52:54.331805+0000","last_fullsized":"2024-09-16T01:52:54.331805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"113'127","reported_seq":582,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511698+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:52:53.511698+0000","last_peered":"2024-09-16T01:52:53.511698+0000","last_clean":"2024-09-16T01:52:53.511698+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:52:53.511698+0000","last_undegraded":"2024-09-16T01:52:53.511698+0000","last_fullsized":"2024-09-16T01:52:53.511698+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"111'93","reported_seq":301,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513855+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:52:53.513855+0000","last_peered":"2024-09-16T01:52:53.513855+0000","last_clean":"2024-09-16T01:52:53.513855+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:52:53.513855+0000","last_undegraded":"2024-09-16T01:52:53.513855+0000","last_fullsized":"2024-09-16T01:52:53.513855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"111'132","reported_seq":340,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717469+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:52:53.717469+0000","last_peered":"2024-09-16T01:52:53.717469+0000","last_clean":"2024-09-16T01:52:53.717469+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:52:53.717469+0000","last_undegraded":"2024-09-16T01:52:53.717469+0000","last_fullsized":"2024-09-16T01:52:53.717469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"113'104","reported_seq":312,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513765+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:52:53.513765+0000","last_peered":"2024-09-16T01:52:53.513765+0000","last_clean":"2024-09-16T01:52:53.513765+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:52:53.513765+0000","last_undegraded":"2024-09-16T01:52:53.513765+0000","last_fullsized":"2024-09-16T01:52:53.513765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"112'142","reported_seq":351,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332199+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:52:54.332199+0000","last_peered":"2024-09-16T01:52:54.332199+0000","last_clean":"2024-09-16T01:52:54.332199+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:52:54.332199+0000","last_undegraded":"2024-09-16T01:52:54.332199+0000","last_fullsized":"2024-09-16T01:52:54.332199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"114'105","reported_seq":546,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518407+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:52:55.518407+0000","last_peered":"2024-09-16T01:52:55.518407+0000","last_clean":"2024-09-16T01:52:55.518407+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:52:55.518407+0000","last_undegraded":"2024-09-16T01:52:55.518407+0000","last_fullsized":"2024-09-16T01:52:55.518407+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"113'100","reported_seq":404,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518407+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:52:55.518407+0000","last_peered":"2024-09-16T01:52:55.518407+0000","last_clean":"2024-09-16T01:52:55.518407+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:52:55.518407+0000","last_undegraded":"2024-09-16T01:52:55.518407+0000","last_fullsized":"2024-09-16T01:52:55.518407+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"113'88","reported_seq":579,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513621+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:52:53.513621+0000","last_peered":"2024-09-16T01:52:53.513621+0000","last_clean":"2024-09-16T01:52:53.513621+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:52:53.513621+0000","last_undegraded":"2024-09-16T01:52:53.513621+0000","last_fullsized":"2024-09-16T01:52:53.513621+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"112'98","reported_seq":306,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717589+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:52:53.717589+0000","last_peered":"2024-09-16T01:52:53.717589+0000","last_clean":"2024-09-16T01:52:53.717589+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:52:53.717589+0000","last_undegraded":"2024-09-16T01:52:53.717589+0000","last_fullsized":"2024-09-16T01:52:53.717589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"113'102","reported_seq":503,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512059+0000","last_change":"2024-09-16T01:52:47.475996+0000","last_active":"2024-09-16T01:52:53.512059+0000","last_peered":"2024-09-16T01:52:53.512059+0000","last_clean":"2024-09-16T01:52:53.512059+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:52:53.512059+0000","last_undegraded":"2024-09-16T01:52:53.512059+0000","last_fullsized":"2024-09-16T01:52:53.512059+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:05.418025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"113'112","reported_seq":436,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.517858+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:52:55.517858+0000","last_peered":"2024-09-16T01:52:55.517858+0000","last_clean":"2024-09-16T01:52:55.517858+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:52:55.517858+0000","last_undegraded":"2024-09-16T01:52:55.517858+0000","last_fullsized":"2024-09-16T01:52:55.517858+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"112'98","reported_seq":307,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511902+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:52:53.511902+0000","last_peered":"2024-09-16T01:52:53.511902+0000","last_clean":"2024-09-16T01:52:53.511902+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:52:53.511902+0000","last_undegraded":"2024-09-16T01:52:53.511902+0000","last_fullsized":"2024-09-16T01:52:53.511902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":215,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513765+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:52:53.513765+0000","last_peered":"2024-09-16T01:52:53.513765+0000","last_clean":"2024-09-16T01:52:53.513765+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:52:53.513765+0000","last_undegraded":"2024-09-16T01:52:53.513765+0000","last_fullsized":"2024-09-16T01:52:53.513765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"114'105","reported_seq":646,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717764+0000","last_change":"2024-09-16T01:52:52.506119+0000","last_active":"2024-09-16T01:52:53.717764+0000","last_peered":"2024-09-16T01:52:53.717764+0000","last_clean":"2024-09-16T01:52:53.717764+0000","last_became_active":"2024-09-16T01:52:52.505498+0000","last_became_peered":"2024-09-16T01:52:52.505498+0000","last_unstale":"2024-09-16T01:52:53.717764+0000","last_undegraded":"2024-09-16T01:52:53.717764+0000","last_fullsized":"2024-09-16T01:52:53.717764+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:00.694119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":230,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717717+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:52:53.717717+0000","last_peered":"2024-09-16T01:52:53.717717+0000","last_clean":"2024-09-16T01:52:53.717717+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:52:53.717717+0000","last_undegraded":"2024-09-16T01:52:53.717717+0000","last_fullsized":"2024-09-16T01:52:53.717717+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":238,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518013+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:52:55.518013+0000","last_peered":"2024-09-16T01:52:55.518013+0000","last_clean":"2024-09-16T01:52:55.518013+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:52:55.518013+0000","last_undegraded":"2024-09-16T01:52:55.518013+0000","last_fullsized":"2024-09-16T01:52:55.518013+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"112'122","reported_seq":330,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513199+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:52:53.513199+0000","last_peered":"2024-09-16T01:52:53.513199+0000","last_clean":"2024-09-16T01:52:53.513199+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:52:53.513199+0000","last_undegraded":"2024-09-16T01:52:53.513199+0000","last_fullsized":"2024-09-16T01:52:53.513199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":213,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717518+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:52:53.717518+0000","last_peered":"2024-09-16T01:52:53.717518+0000","last_clean":"2024-09-16T01:52:53.717518+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:52:53.717518+0000","last_undegraded":"2024-09-16T01:52:53.717518+0000","last_fullsized":"2024-09-16T01:52:53.717518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"112'114","reported_seq":288,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511940+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:52:53.511940+0000","last_peered":"2024-09-16T01:52:53.511940+0000","last_clean":"2024-09-16T01:52:53.511940+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:52:53.511940+0000","last_undegraded":"2024-09-16T01:52:53.511940+0000","last_fullsized":"2024-09-16T01:52:53.511940+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":242,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512382+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:52:53.512382+0000","last_peered":"2024-09-16T01:52:53.512382+0000","last_clean":"2024-09-16T01:52:53.512382+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:52:53.512382+0000","last_undegraded":"2024-09-16T01:52:53.512382+0000","last_fullsized":"2024-09-16T01:52:53.512382+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"113'167","reported_seq":389,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514123+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:52:53.514123+0000","last_peered":"2024-09-16T01:52:53.514123+0000","last_clean":"2024-09-16T01:52:53.514123+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:52:53.514123+0000","last_undegraded":"2024-09-16T01:52:53.514123+0000","last_fullsized":"2024-09-16T01:52:53.514123+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512275+0000","last_change":"2024-09-16T01:52:46.621488+0000","last_active":"2024-09-16T01:52:53.512275+0000","last_peered":"2024-09-16T01:52:53.512275+0000","last_clean":"2024-09-16T01:52:53.512275+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:52:53.512275+0000","last_undegraded":"2024-09-16T01:52:53.512275+0000","last_fullsized":"2024-09-16T01:52:53.512275+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"113'101","reported_seq":309,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513994+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:52:53.513994+0000","last_peered":"2024-09-16T01:52:53.513994+0000","last_clean":"2024-09-16T01:52:53.513994+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:52:53.513994+0000","last_undegraded":"2024-09-16T01:52:53.513994+0000","last_fullsized":"2024-09-16T01:52:53.513994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":122,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717760+0000","last_change":"2024-09-16T01:52:46.632501+0000","last_active":"2024-09-16T01:52:53.717760+0000","last_peered":"2024-09-16T01:52:53.717760+0000","last_clean":"2024-09-16T01:52:53.717760+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:52:53.717760+0000","last_undegraded":"2024-09-16T01:52:53.717760+0000","last_fullsized":"2024-09-16T01:52:53.717760+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"112'107","reported_seq":315,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718163+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:52:53.718163+0000","last_peered":"2024-09-16T01:52:53.718163+0000","last_clean":"2024-09-16T01:52:53.718163+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:52:53.718163+0000","last_undegraded":"2024-09-16T01:52:53.718163+0000","last_fullsized":"2024-09-16T01:52:53.718163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"104'168","reported_seq":400,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332005+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:52:54.332005+0000","last_peered":"2024-09-16T01:52:54.332005+0000","last_clean":"2024-09-16T01:52:54.332005+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:52:54.332005+0000","last_undegraded":"2024-09-16T01:52:54.332005+0000","last_fullsized":"2024-09-16T01:52:54.332005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376086,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"111'111","reported_seq":425,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518261+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:52:55.518261+0000","last_peered":"2024-09-16T01:52:55.518261+0000","last_clean":"2024-09-16T01:52:55.518261+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:52:55.518261+0000","last_undegraded":"2024-09-16T01:52:55.518261+0000","last_fullsized":"2024-09-16T01:52:55.518261+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":289,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717859+0000","last_change":"2024-09-16T01:52:46.615925+0000","last_active":"2024-09-16T01:52:53.717859+0000","last_peered":"2024-09-16T01:52:53.717859+0000","last_clean":"2024-09-16T01:52:53.717859+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:52:53.717859+0000","last_undegraded":"2024-09-16T01:52:53.717859+0000","last_fullsized":"2024-09-16T01:52:53.717859+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":340,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"112'117","reported_seq":328,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518175+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:52:55.518175+0000","last_peered":"2024-09-16T01:52:55.518175+0000","last_clean":"2024-09-16T01:52:55.518175+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:52:55.518175+0000","last_undegraded":"2024-09-16T01:52:55.518175+0000","last_fullsized":"2024-09-16T01:52:55.518175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":293,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717981+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:52:53.717981+0000","last_peered":"2024-09-16T01:52:53.717981+0000","last_clean":"2024-09-16T01:52:53.717981+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:52:53.717981+0000","last_undegraded":"2024-09-16T01:52:53.717981+0000","last_fullsized":"2024-09-16T01:52:53.717981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"112'121","reported_seq":460,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717927+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:52:53.717927+0000","last_peered":"2024-09-16T01:52:53.717927+0000","last_clean":"2024-09-16T01:52:53.717927+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:52:53.717927+0000","last_undegraded":"2024-09-16T01:52:53.717927+0000","last_fullsized":"2024-09-16T01:52:53.717927+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717530+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:52:53.717530+0000","last_peered":"2024-09-16T01:52:53.717530+0000","last_clean":"2024-09-16T01:52:53.717530+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:52:53.717530+0000","last_undegraded":"2024-09-16T01:52:53.717530+0000","last_fullsized":"2024-09-16T01:52:53.717530+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"113'133","reported_seq":342,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511731+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:52:53.511731+0000","last_peered":"2024-09-16T01:52:53.511731+0000","last_clean":"2024-09-16T01:52:53.511731+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:52:53.511731+0000","last_undegraded":"2024-09-16T01:52:53.511731+0000","last_fullsized":"2024-09-16T01:52:53.511731+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":146,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511605+0000","last_change":"2024-09-16T01:52:46.599623+0000","last_active":"2024-09-16T01:52:53.511605+0000","last_peered":"2024-09-16T01:52:53.511605+0000","last_clean":"2024-09-16T01:52:53.511605+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:52:53.511605+0000","last_undegraded":"2024-09-16T01:52:53.511605+0000","last_fullsized":"2024-09-16T01:52:53.511605+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:22.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":784,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"113'145","reported_seq":353,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717589+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:52:53.717589+0000","last_peered":"2024-09-16T01:52:53.717589+0000","last_clean":"2024-09-16T01:52:53.717589+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:52:53.717589+0000","last_undegraded":"2024-09-16T01:52:53.717589+0000","last_fullsized":"2024-09-16T01:52:53.717589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":83,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513643+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:52:53.513643+0000","last_peered":"2024-09-16T01:52:53.513643+0000","last_clean":"2024-09-16T01:52:53.513643+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:52:53.513643+0000","last_undegraded":"2024-09-16T01:52:53.513643+0000","last_fullsized":"2024-09-16T01:52:53.513643+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"112'106","reported_seq":416,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332235+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:52:54.332235+0000","last_peered":"2024-09-16T01:52:54.332235+0000","last_clean":"2024-09-16T01:52:54.332235+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:52:54.332235+0000","last_undegraded":"2024-09-16T01:52:54.332235+0000","last_fullsized":"2024-09-16T01:52:54.332235+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514439+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:52:53.514439+0000","last_peered":"2024-09-16T01:52:53.514439+0000","last_clean":"2024-09-16T01:52:53.514439+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:52:53.514439+0000","last_undegraded":"2024-09-16T01:52:53.514439+0000","last_fullsized":"2024-09-16T01:52:53.514439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"112'103","reported_seq":408,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512264+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:52:53.512264+0000","last_peered":"2024-09-16T01:52:53.512264+0000","last_clean":"2024-09-16T01:52:53.512264+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:52:53.512264+0000","last_undegraded":"2024-09-16T01:52:53.512264+0000","last_fullsized":"2024-09-16T01:52:53.512264+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":213,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514096+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:52:53.514096+0000","last_peered":"2024-09-16T01:52:53.514096+0000","last_clean":"2024-09-16T01:52:53.514096+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:52:53.514096+0000","last_undegraded":"2024-09-16T01:52:53.514096+0000","last_fullsized":"2024-09-16T01:52:53.514096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"114'102","reported_seq":638,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717963+0000","last_change":"2024-09-16T01:52:50.499005+0000","last_active":"2024-09-16T01:52:53.717963+0000","last_peered":"2024-09-16T01:52:53.717963+0000","last_clean":"2024-09-16T01:52:53.717963+0000","last_became_active":"2024-09-16T01:52:50.498634+0000","last_became_peered":"2024-09-16T01:52:50.498634+0000","last_unstale":"2024-09-16T01:52:53.717963+0000","last_undegraded":"2024-09-16T01:52:53.717963+0000","last_fullsized":"2024-09-16T01:52:53.717963+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":231,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518512+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:52:55.518512+0000","last_peered":"2024-09-16T01:52:55.518512+0000","last_clean":"2024-09-16T01:52:55.518512+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:52:55.518512+0000","last_undegraded":"2024-09-16T01:52:55.518512+0000","last_fullsized":"2024-09-16T01:52:55.518512+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"112'112","reported_seq":320,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718146+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:52:53.718146+0000","last_peered":"2024-09-16T01:52:53.718146+0000","last_clean":"2024-09-16T01:52:53.718146+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:52:53.718146+0000","last_undegraded":"2024-09-16T01:52:53.718146+0000","last_fullsized":"2024-09-16T01:52:53.718146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":368,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511717+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:52:53.511717+0000","last_peered":"2024-09-16T01:52:53.511717+0000","last_clean":"2024-09-16T01:52:53.511717+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:52:53.511717+0000","last_undegraded":"2024-09-16T01:52:53.511717+0000","last_fullsized":"2024-09-16T01:52:53.511717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":216,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718037+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:52:53.718037+0000","last_peered":"2024-09-16T01:52:53.718037+0000","last_clean":"2024-09-16T01:52:53.718037+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:52:53.718037+0000","last_undegraded":"2024-09-16T01:52:53.718037+0000","last_fullsized":"2024-09-16T01:52:53.718037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"113'112","reported_seq":461,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331441+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:52:54.331441+0000","last_peered":"2024-09-16T01:52:54.331441+0000","last_clean":"2024-09-16T01:52:54.331441+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:52:54.331441+0000","last_undegraded":"2024-09-16T01:52:54.331441+0000","last_fullsized":"2024-09-16T01:52:54.331441+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":214,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717706+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:52:53.717706+0000","last_peered":"2024-09-16T01:52:53.717706+0000","last_clean":"2024-09-16T01:52:53.717706+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:52:53.717706+0000","last_undegraded":"2024-09-16T01:52:53.717706+0000","last_fullsized":"2024-09-16T01:52:53.717706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"113'107","reported_seq":318,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.517901+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:52:55.517901+0000","last_peered":"2024-09-16T01:52:55.517901+0000","last_clean":"2024-09-16T01:52:55.517901+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:52:55.517901+0000","last_undegraded":"2024-09-16T01:52:55.517901+0000","last_fullsized":"2024-09-16T01:52:55.517901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"74'9","reported_seq":271,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513761+0000","last_change":"2024-09-16T01:52:46.569921+0000","last_active":"2024-09-16T01:52:53.513761+0000","last_peered":"2024-09-16T01:52:53.513761+0000","last_clean":"2024-09-16T01:52:53.513761+0000","last_became_active":"2024-09-16T01:52:46.544102+0000","last_became_peered":"2024-09-16T01:52:46.544102+0000","last_unstale":"2024-09-16T01:52:53.513761+0000","last_undegraded":"2024-09-16T01:52:53.513761+0000","last_fullsized":"2024-09-16T01:52:53.513761+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:14.446115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"113'96","reported_seq":521,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513173+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:52:53.513173+0000","last_peered":"2024-09-16T01:52:53.513173+0000","last_clean":"2024-09-16T01:52:53.513173+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:52:53.513173+0000","last_undegraded":"2024-09-16T01:52:53.513173+0000","last_fullsized":"2024-09-16T01:52:53.513173+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":215,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718096+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:52:53.718096+0000","last_peered":"2024-09-16T01:52:53.718096+0000","last_clean":"2024-09-16T01:52:53.718096+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:52:53.718096+0000","last_undegraded":"2024-09-16T01:52:53.718096+0000","last_fullsized":"2024-09-16T01:52:53.718096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"113'111","reported_seq":419,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331709+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:52:54.331709+0000","last_peered":"2024-09-16T01:52:54.331709+0000","last_clean":"2024-09-16T01:52:54.331709+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:52:54.331709+0000","last_undegraded":"2024-09-16T01:52:54.331709+0000","last_fullsized":"2024-09-16T01:52:54.331709+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":211,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513507+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:52:53.513507+0000","last_peered":"2024-09-16T01:52:53.513507+0000","last_clean":"2024-09-16T01:52:53.513507+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:52:53.513507+0000","last_undegraded":"2024-09-16T01:52:53.513507+0000","last_fullsized":"2024-09-16T01:52:53.513507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"113'110","reported_seq":319,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332065+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:52:54.332065+0000","last_peered":"2024-09-16T01:52:54.332065+0000","last_clean":"2024-09-16T01:52:54.332065+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:52:54.332065+0000","last_undegraded":"2024-09-16T01:52:54.332065+0000","last_fullsized":"2024-09-16T01:52:54.332065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":42,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.332060+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:52:54.332060+0000","last_peered":"2024-09-16T01:52:54.332060+0000","last_clean":"2024-09-16T01:52:54.332060+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:52:54.332060+0000","last_undegraded":"2024-09-16T01:52:54.332060+0000","last_fullsized":"2024-09-16T01:52:54.332060+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"113'103","reported_seq":314,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518458+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:52:55.518458+0000","last_peered":"2024-09-16T01:52:55.518458+0000","last_clean":"2024-09-16T01:52:55.518458+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:52:55.518458+0000","last_undegraded":"2024-09-16T01:52:55.518458+0000","last_fullsized":"2024-09-16T01:52:55.518458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'18","reported_seq":209,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514250+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:52:53.514250+0000","last_peered":"2024-09-16T01:52:53.514250+0000","last_clean":"2024-09-16T01:52:53.514250+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:52:53.514250+0000","last_undegraded":"2024-09-16T01:52:53.514250+0000","last_fullsized":"2024-09-16T01:52:53.514250+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"113'114","reported_seq":370,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518236+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:52:55.518236+0000","last_peered":"2024-09-16T01:52:55.518236+0000","last_clean":"2024-09-16T01:52:55.518236+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:52:55.518236+0000","last_undegraded":"2024-09-16T01:52:55.518236+0000","last_fullsized":"2024-09-16T01:52:55.518236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":232,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718434+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:52:53.718434+0000","last_peered":"2024-09-16T01:52:53.718434+0000","last_clean":"2024-09-16T01:52:53.718434+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:52:53.718434+0000","last_undegraded":"2024-09-16T01:52:53.718434+0000","last_fullsized":"2024-09-16T01:52:53.718434+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"114'116","reported_seq":415,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512168+0000","last_change":"2024-09-16T01:52:52.504293+0000","last_active":"2024-09-16T01:52:53.512168+0000","last_peered":"2024-09-16T01:52:53.512168+0000","last_clean":"2024-09-16T01:52:53.512168+0000","last_became_active":"2024-09-16T01:52:52.503886+0000","last_became_peered":"2024-09-16T01:52:52.503886+0000","last_unstale":"2024-09-16T01:52:53.512168+0000","last_undegraded":"2024-09-16T01:52:53.512168+0000","last_fullsized":"2024-09-16T01:52:53.512168+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:29.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331393+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:52:54.331393+0000","last_peered":"2024-09-16T01:52:54.331393+0000","last_clean":"2024-09-16T01:52:54.331393+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:52:54.331393+0000","last_undegraded":"2024-09-16T01:52:54.331393+0000","last_fullsized":"2024-09-16T01:52:54.331393+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"112'114","reported_seq":377,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717732+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:52:53.717732+0000","last_peered":"2024-09-16T01:52:53.717732+0000","last_clean":"2024-09-16T01:52:53.717732+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:52:53.717732+0000","last_undegraded":"2024-09-16T01:52:53.717732+0000","last_fullsized":"2024-09-16T01:52:53.717732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":93,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.517888+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:52:55.517888+0000","last_peered":"2024-09-16T01:52:55.517888+0000","last_clean":"2024-09-16T01:52:55.517888+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:52:55.517888+0000","last_undegraded":"2024-09-16T01:52:55.517888+0000","last_fullsized":"2024-09-16T01:52:55.517888+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"113'116","reported_seq":380,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:54.331257+0000","last_change":"2024-09-16T01:52:46.782699+0000","last_active":"2024-09-16T01:52:54.331257+0000","last_peered":"2024-09-16T01:52:54.331257+0000","last_clean":"2024-09-16T01:52:54.331257+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:52:54.331257+0000","last_undegraded":"2024-09-16T01:52:54.331257+0000","last_fullsized":"2024-09-16T01:52:54.331257+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:30.580835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":216,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518265+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:52:55.518265+0000","last_peered":"2024-09-16T01:52:55.518265+0000","last_clean":"2024-09-16T01:52:55.518265+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:52:55.518265+0000","last_undegraded":"2024-09-16T01:52:55.518265+0000","last_fullsized":"2024-09-16T01:52:55.518265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"114'105","reported_seq":526,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-16T01:52:55.518065+0000","last_change":"2024-09-16T01:52:51.501914+0000","last_active":"2024-09-16T01:52:55.518065+0000","last_peered":"2024-09-16T01:52:55.518065+0000","last_clean":"2024-09-16T01:52:55.518065+0000","last_became_active":"2024-09-16T01:52:51.501494+0000","last_became_peered":"2024-09-16T01:52:51.501494+0000","last_unstale":"2024-09-16T01:52:55.518065+0000","last_undegraded":"2024-09-16T01:52:55.518065+0000","last_fullsized":"2024-09-16T01:52:55.518065+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:29.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":215,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717978+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:52:53.717978+0000","last_peered":"2024-09-16T01:52:53.717978+0000","last_clean":"2024-09-16T01:52:53.717978+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:52:53.717978+0000","last_undegraded":"2024-09-16T01:52:53.717978+0000","last_fullsized":"2024-09-16T01:52:53.717978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"111'94","reported_seq":302,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.717857+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:52:53.717857+0000","last_peered":"2024-09-16T01:52:53.717857+0000","last_clean":"2024-09-16T01:52:53.717857+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:52:53.717857+0000","last_undegraded":"2024-09-16T01:52:53.717857+0000","last_fullsized":"2024-09-16T01:52:53.717857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"112'131","reported_seq":458,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718046+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:52:53.718046+0000","last_peered":"2024-09-16T01:52:53.718046+0000","last_clean":"2024-09-16T01:52:53.718046+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:52:53.718046+0000","last_undegraded":"2024-09-16T01:52:53.718046+0000","last_fullsized":"2024-09-16T01:52:53.718046+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":84,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511951+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:52:53.511951+0000","last_peered":"2024-09-16T01:52:53.511951+0000","last_clean":"2024-09-16T01:52:53.511951+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:52:53.511951+0000","last_undegraded":"2024-09-16T01:52:53.511951+0000","last_fullsized":"2024-09-16T01:52:53.511951+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"111'106","reported_seq":410,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.512035+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:52:53.512035+0000","last_peered":"2024-09-16T01:52:53.512035+0000","last_clean":"2024-09-16T01:52:53.512035+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:52:53.512035+0000","last_undegraded":"2024-09-16T01:52:53.512035+0000","last_fullsized":"2024-09-16T01:52:53.512035+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":52,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.511998+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:52:53.511998+0000","last_peered":"2024-09-16T01:52:53.511998+0000","last_clean":"2024-09-16T01:52:53.511998+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:52:53.511998+0000","last_undegraded":"2024-09-16T01:52:53.511998+0000","last_fullsized":"2024-09-16T01:52:53.511998+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"113'97","reported_seq":305,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.513892+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:52:53.513892+0000","last_peered":"2024-09-16T01:52:53.513892+0000","last_clean":"2024-09-16T01:52:53.513892+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:52:53.513892+0000","last_undegraded":"2024-09-16T01:52:53.513892+0000","last_fullsized":"2024-09-16T01:52:53.513892+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":242,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514535+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:52:53.514535+0000","last_peered":"2024-09-16T01:52:53.514535+0000","last_clean":"2024-09-16T01:52:53.514535+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:52:53.514535+0000","last_undegraded":"2024-09-16T01:52:53.514535+0000","last_fullsized":"2024-09-16T01:52:53.514535+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"112'115","reported_seq":323,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.514376+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:52:53.514376+0000","last_peered":"2024-09-16T01:52:53.514376+0000","last_clean":"2024-09-16T01:52:53.514376+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:52:53.514376+0000","last_undegraded":"2024-09-16T01:52:53.514376+0000","last_fullsized":"2024-09-16T01:52:53.514376+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":121,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-16T01:52:53.718318+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:52:53.718318+0000","last_peered":"2024-09-16T01:52:53.718318+0000","last_clean":"2024-09-16T01:52:53.718318+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:52:53.718318+0000","last_undegraded":"2024-09-16T01:52:53.718318+0000","last_fullsized":"2024-09-16T01:52:53.718318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1401683968,"data_stored":22426943488,"data_compressed":1119113728,"data_compressed_allocated":1401683968,"data_compressed_original":22426943488,"omap_allocated":0,"internal_metadata":0},"log_size":7171,"ondisk_log_size":7171,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":382969,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":392,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":248325,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110016,"data_stored":995148,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":360,"ondisk_log_size":360,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509494,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323712,"kb_used_data":292520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048128,"statfs":{"total":96636764160,"available":96305283072,"internally_reserved":0,"allocated":299540480,"data_stored":4769599877,"data_compressed":238533504,"data_compressed_allocated":298057728,"data_compressed_original":4768923648,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509494,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288988,"kb_used_data":257796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94082852,"statfs":{"total":96636764160,"available":96340840448,"internally_reserved":0,"allocated":263983104,"data_stored":4195353823,"data_compressed":209792000,"data_compressed_allocated":262144000,"data_compressed_original":4194304000,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509493,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220456,"kb_used_data":193424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94151384,"statfs":{"total":96636764160,"available":96411017216,"internally_reserved":0,"allocated":198066176,"data_stored":3143173736,"data_compressed":157203299,"data_compressed_allocated":196435968,"data_compressed_original":3142340608,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542197,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188276,"kb_used_data":157080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94183564,"statfs":{"total":96636764160,"available":96443969536,"internally_reserved":0,"allocated":160849920,"data_stored":2550810126,"data_compressed":127553536,"data_compressed_allocated":159383552,"data_compressed_original":2550136832,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542197,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229176,"kb_used_data":202140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94142664,"statfs":{"total":96636764160,"available":96402087936,"internally_reserved":0,"allocated":206991360,"data_stored":3289009158,"data_compressed":164476928,"data_compressed_allocated":205520896,"data_compressed_original":3288334336,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542197,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256556,"kb_used_data":217048,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94115284,"statfs":{"total":96636764160,"available":96374050816,"internally_reserved":0,"allocated":222257152,"data_stored":3532868662,"data_compressed":176677284,"data_compressed_allocated":220770304,"data_compressed_original":3532177408,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574901,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":377634,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":377785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196345856,"data_stored":3141533696,"data_compressed":157134208,"data_compressed_allocated":196345856,"data_compressed_original":3141533696,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":262144000,"data_stored":4194304000,"data_compressed":209792000,"data_compressed_allocated":262144000,"data_compressed_original":4194304000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298057728,"data_stored":4768923648,"data_compressed":238533504,"data_compressed_allocated":298057728,"data_compressed_original":4768923648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":220725248,"data_stored":3531603968,"data_compressed":176644864,"data_compressed_allocated":220725248,"data_compressed_original":3531603968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":205520896,"data_stored":3288334336,"data_compressed":164476928,"data_compressed_allocated":205520896,"data_compressed_original":3288334336,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":159383552,"data_stored":2550136832,"data_compressed":127553536,"data_compressed_allocated":159383552,"data_compressed_original":2550136832,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:52:59.310 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T01:52:59.310 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:52:59.310 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T01:52:59.310 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T01:52:59.310 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T01:52:59.311 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-09-16T01:52:59.311 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:59.312 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:59.320 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:52:59.391 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:59.389+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:59.416 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:52:59.417 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "" 2024-09-16T01:52:59.417 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:52:59.467 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:59.467 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:59.476 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:52:59.492 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:52:59.490+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:52:59.582 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:59.582 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:59.591 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:52:59.592 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:59.591+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:59.693 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:59.691+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:59.695 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:59.695 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:59.704 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:52:59.794 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:52:59.793+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:52:59.848 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:52:59.849 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:52:59.857 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:52:59.895 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:52:59.894+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:52:59.996 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:52:59.994+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:00.002 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.002 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.012 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:53:00.097 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:00.096+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:00.117 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.117 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:53:00.197 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:00.196+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:00.229 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.229 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.238 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:53:00.298 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:00.297+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:00.341 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.341 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.351 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:53:00.436 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:00.435+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:00.495 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.496 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.501 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:00.500+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:00.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:53:00.602 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:00.602+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:00.667 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.668 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:53:00.704 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:00.704+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:00.805 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:00.804+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:00.838 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:00.838 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:00.849 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:53:00.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:53:00.940 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:00.938+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:01.008 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:01.007+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:01.011 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:01.011 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:01.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:53:01.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.051+0000 7f5b15400640 1 Processor -- start 2024-09-16T01:53:01.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.051+0000 7f5b15400640 1 -- start start 2024-09-16T01:53:01.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b15400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5b10154d30 0x7f5b100c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b15400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b100c7a60 0x7f5b1006dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b15400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b10155110 con 0x7f5b100c7f30 2024-09-16T01:53:01.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b15400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5b100c6bb0 con 0x7f5b10154d30 2024-09-16T01:53:01.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b15400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5b10073140 con 0x7f5b100c7a60 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b100c7a60 0x7f5b1006dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0de00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5b100c7f30 0x7f5b100c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:36288/0 (socket says 172.21.15.146:36288) 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0de00640 1 -- 172.21.15.146:0/69842579 learned_addr learned my addr 172.21.15.146:0/69842579 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0e800640 1 --2- 172.21.15.146:0/69842579 >> v2:172.21.15.146:3300/0 conn(0x7f5b10154d30 0x7f5b100c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0f200640 1 -- 172.21.15.146:0/69842579 >> v2:172.21.15.146:3300/0 conn(0x7f5b10154d30 msgr2=0x7f5b100c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0f200640 1 --2- 172.21.15.146:0/69842579 >> v2:172.21.15.146:3300/0 conn(0x7f5b10154d30 0x7f5b100c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0d400640 1 -- 172.21.15.146:0/69842579 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651439558 0 0) 0x7f5b10155110 con 0x7f5b100c7f30 2024-09-16T01:53:01.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0f200640 1 -- 172.21.15.146:0/69842579 >> v1:172.21.15.161:6789/0 conn(0x7f5b100c7f30 legacy=0x7f5b100c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.053+0000 7f5b0f200640 1 -- 172.21.15.146:0/69842579 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b100c6e00 con 0x7f5b100c7a60 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.054+0000 7f5b0f200640 1 --2- 172.21.15.146:0/69842579 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b100c7a60 0x7f5b1006dae0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009d90 tx=0x7f5b0802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db76fbaad330b209 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b0d400640 1 -- 172.21.15.146:0/69842579 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08007740 con 0x7f5b100c7a60 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b0d400640 1 -- 172.21.15.146:0/69842579 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b080078e0 con 0x7f5b100c7a60 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b0d400640 1 -- 172.21.15.146:0/69842579 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08007cb0 con 0x7f5b100c7a60 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 -- 172.21.15.146:0/69842579 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b100c7a60 msgr2=0x7f5b1006dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 --2- 172.21.15.146:0/69842579 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b100c7a60 0x7f5b1006dae0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009d90 tx=0x7f5b0802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 -- 172.21.15.146:0/69842579 shutdown_connections 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 --2- 172.21.15.146:0/69842579 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b100c7a60 0x7f5b1006dae0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 --2- 172.21.15.146:0/69842579 >> v2:172.21.15.146:3300/0 conn(0x7f5b10154d30 0x7f5b100c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 -- 172.21.15.146:0/69842579 >> 172.21.15.146:0/69842579 conn(0x7f5b1006f3c0 msgr2=0x7f5b1006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 -- 172.21.15.146:0/69842579 shutdown_connections 2024-09-16T01:53:01.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.055+0000 7f5b15400640 1 -- 172.21.15.146:0/69842579 wait complete. 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.057+0000 7f5b15400640 1 Processor -- start 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.057+0000 7f5b15400640 1 -- start start 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b15400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b100c7f30 0x7f5b101e5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b15400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 0x7f5b101e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b15400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b101e4930 0x7f5b101f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b15400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10072e00 con 0x7f5b10154d30 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b15400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10072920 con 0x7f5b100c7f30 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b15400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5b100718c0 con 0x7f5b101e4930 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b0de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 0x7f5b101e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b0de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 0x7f5b101e43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57658/0 (socket says 172.21.15.146:57658) 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b0de00640 1 -- 172.21.15.146:0/877955446 learned_addr learned my addr 172.21.15.146:0/877955446 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b0e800640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b100c7f30 0x7f5b101e5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.058+0000 7f5b0f200640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b101e4930 0x7f5b101f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0de00640 1 -- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b101e4930 msgr2=0x7f5b101f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0de00640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b101e4930 0x7f5b101f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0de00640 1 -- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b100c7f30 msgr2=0x7f5b101e5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0de00640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b100c7f30 0x7f5b101e5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0de00640 1 -- 172.21.15.146:0/877955446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b101e5cf0 con 0x7f5b10154d30 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0e800640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b100c7f30 0x7f5b101e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:01.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b0de00640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 0x7f5b101e43f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5afc00c4d0 tx=0x7f5afc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5afc018070 con 0x7f5b10154d30 2024-09-16T01:53:01.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5afc007500 con 0x7f5b10154d30 2024-09-16T01:53:01.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5afc004a20 con 0x7f5b10154d30 2024-09-16T01:53:01.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.059+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b101e5fd0 con 0x7f5b10154d30 2024-09-16T01:53:01.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.060+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b101e6590 con 0x7f5b10154d30 2024-09-16T01:53:01.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.061+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b100c9570 con 0x7f5b10154d30 2024-09-16T01:53:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.061+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5afc004bc0 con 0x7f5b10154d30 2024-09-16T01:53:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.062+0000 7f5b06a00640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5af006c2e0 0x7f5af006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.062+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5894+0+0 (secure 0 0 0) 0x7f5afc08a780 con 0x7f5b10154d30 2024-09-16T01:53:01.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.063+0000 7f5b0e800640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5af006c2e0 0x7f5af006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.063+0000 7f5b0e800640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5af006c2e0 0x7f5af006e7a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009e90 tx=0x7f5b00009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.067+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5afc058740 con 0x7f5b10154d30 2024-09-16T01:53:01.108 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:01.107+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:01.209 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:01.207+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:01.216 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:01.216 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:01.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:53:01.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.259+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5b100c9b60 con 0x7f5b10154d30 2024-09-16T01:53:01.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:53:01.311 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:01.309+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:01.411 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:01.409+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:01.423 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:01.423 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:01.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:53:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.459+0000 7f7291400640 1 Processor -- start 2024-09-16T01:53:01.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.460+0000 7f7291400640 1 -- start start 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7291400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f728c0c7980 0x7f728c0c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7291400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c071a10 0x7f728c0c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7291400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f728c1585b0 con 0x7f728c151820 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7291400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f728c1580a0 con 0x7f728c0c7980 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7291400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f728c0c6680 con 0x7f728c071a10 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f728a800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f728c151820 0x7f728c0c8070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:60362/0 (socket says 172.21.15.146:60362) 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f728a800640 1 -- 172.21.15.146:0/4237255856 learned_addr learned my addr 172.21.15.146:0/4237255856 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:01.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7289e00640 1 --2- 172.21.15.146:0/4237255856 >> v2:172.21.15.161:3300/0 conn(0x7f728c0c7980 0x7f728c0c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f728b200640 1 --2- 172.21.15.146:0/4237255856 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c071a10 0x7f728c0c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7289400640 1 -- 172.21.15.146:0/4237255856 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1413369023 0 0) 0x7f728c1585b0 con 0x7f728c151820 2024-09-16T01:53:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.461+0000 7f7289400640 1 -- 172.21.15.146:0/4237255856 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7270003520 con 0x7f728c151820 2024-09-16T01:53:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.462+0000 7f728b200640 1 -- 172.21.15.146:0/4237255856 >> v2:172.21.15.161:3300/0 conn(0x7f728c0c7980 msgr2=0x7f728c0c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.462+0000 7f728b200640 1 --2- 172.21.15.146:0/4237255856 >> v2:172.21.15.161:3300/0 conn(0x7f728c0c7980 0x7f728c0c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.462+0000 7f728b200640 1 -- 172.21.15.146:0/4237255856 >> v1:172.21.15.146:6789/0 conn(0x7f728c151820 legacy=0x7f728c0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.462+0000 7f728b200640 1 -- 172.21.15.146:0/4237255856 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c1582d0 con 0x7f728c071a10 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.463+0000 7f728b200640 1 --2- 172.21.15.146:0/4237255856 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c071a10 0x7f728c0c5180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7284009d90 tx=0x7f728402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e48b545b5eeb46 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.463+0000 7f7289400640 1 -- 172.21.15.146:0/4237255856 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284007740 con 0x7f728c071a10 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.463+0000 7f7289400640 1 -- 172.21.15.146:0/4237255856 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72840078e0 con 0x7f728c071a10 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.463+0000 7f7289400640 1 -- 172.21.15.146:0/4237255856 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284007cb0 con 0x7f728c071a10 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 -- 172.21.15.146:0/4237255856 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c071a10 msgr2=0x7f728c0c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 --2- 172.21.15.146:0/4237255856 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c071a10 0x7f728c0c5180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7284009d90 tx=0x7f728402a6f0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 -- 172.21.15.146:0/4237255856 shutdown_connections 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 --2- 172.21.15.146:0/4237255856 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c071a10 0x7f728c0c5180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 --2- 172.21.15.146:0/4237255856 >> v2:172.21.15.161:3300/0 conn(0x7f728c0c7980 0x7f728c0c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 -- 172.21.15.146:0/4237255856 >> 172.21.15.146:0/4237255856 conn(0x7f728c06bc10 msgr2=0x7f728c06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 -- 172.21.15.146:0/4237255856 shutdown_connections 2024-09-16T01:53:01.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.464+0000 7f7291400640 1 -- 172.21.15.146:0/4237255856 wait complete. 2024-09-16T01:53:01.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.465+0000 7f7291400640 1 Processor -- start 2024-09-16T01:53:01.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.465+0000 7f7291400640 1 -- start start 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.466+0000 7f7291400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f728c071a10 0x7f728c153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.466+0000 7f7291400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f728c0c7980 0x7f728c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.466+0000 7f7291400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c151820 0x7f728c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.466+0000 7f7291400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f728c0c6ea0 con 0x7f728c071a10 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.466+0000 7f7291400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f728c0c74f0 con 0x7f728c0c7980 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.466+0000 7f7291400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f728c0c7180 con 0x7f728c151820 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f728c071a10 0x7f728c153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c151820 0x7f728c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f728c071a10 0x7f728c153ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57664/0 (socket says 172.21.15.146:57664) 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728a800640 1 -- 172.21.15.146:0/61966397 learned_addr learned my addr 172.21.15.146:0/61966397 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:01.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f728c0c7980 0x7f728c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f728c0c7980 msgr2=0x7f728c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f728c0c7980 0x7f728c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f728c071a10 msgr2=0x7f728c153ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f728c071a10 0x7f728c153ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c0ccda0 con 0x7f728c151820 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f728c0c7980 0x7f728c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.467+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c151820 0x7f728c1f3860 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7284031000 tx=0x7f7284004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.468+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728403f070 con 0x7f728c151820 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.468+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7284031c00 con 0x7f728c151820 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.468+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284009450 con 0x7f728c151820 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.468+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c163db0 con 0x7f728c151820 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.468+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c164370 con 0x7f728c151820 2024-09-16T01:53:01.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.469+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f728c151c00 con 0x7f728c151820 2024-09-16T01:53:01.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.470+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7284064680 con 0x7f728c151820 2024-09-16T01:53:01.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.471+0000 7f7282a00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f726c06c2e0 0x7f726c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.471+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(122..122 src has 1..122) ==== 5894+0+0 (secure 0 0 0) 0x7f72840ac950 con 0x7f728c151820 2024-09-16T01:53:01.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.471+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v122) ==== 54+0+16 (secure 0 0 0) 0x7f728404b030 con 0x7f728c151820 2024-09-16T01:53:01.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.471+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f726c06c2e0 0x7f726c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.471+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f728c0cc860 0x7f728c148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.471+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f728c156310 con 0x7f728c0cc860 2024-09-16T01:53:01.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.472+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f728c0cc860 0x7f728c148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.472+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f726c06c2e0 0x7f726c06e7a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f727c005880 tx=0x7f727c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.472+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f728c0cc860 0x7f728c148590 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.473+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c156310 con 0x7f728c0cc860 2024-09-16T01:53:01.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.488+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c0008d0 con 0x7f728c0cc860 2024-09-16T01:53:01.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.490+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c0008d0 con 0x7f728c0cc860 2024-09-16T01:53:01.492 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:53:01.492 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:53:01.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.490+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f728c162f10 0x7f728c13ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.490+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f728c164850 con 0x7f728c162f10 2024-09-16T01:53:01.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.490+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f728c162f10 0x7f728c13ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.491+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f728c162f10 0x7f728c13ce90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.492+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c164850 con 0x7f728c162f10 2024-09-16T01:53:01.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.507+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c1509b0 con 0x7f728c162f10 2024-09-16T01:53:01.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.509+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c1509b0 con 0x7f728c162f10 2024-09-16T01:53:01.511 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:53:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:53:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.509+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f728c062630 0x7f728c062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.509+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f728c13d3d0 con 0x7f728c062630 2024-09-16T01:53:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.509+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f728c062630 0x7f728c062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.510+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f728c062630 0x7f728c062a50 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.513 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:01.511+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:01.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.510+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c13d3d0 con 0x7f728c062630 2024-09-16T01:53:01.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.524+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c062f90 con 0x7f728c062630 2024-09-16T01:53:01.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.526+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c062f90 con 0x7f728c062630 2024-09-16T01:53:01.528 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:53:01.529 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:53:01.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.527+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f728c0ca4c0 0x7f728c0ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.527+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f728c063560 con 0x7f728c0ca4c0 2024-09-16T01:53:01.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.527+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f728c0ca4c0 0x7f728c0ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.527+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f728c0ca4c0 0x7f728c0ca8a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.528+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c063560 con 0x7f728c0ca4c0 2024-09-16T01:53:01.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.542+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c02c640 con 0x7f728c0ca4c0 2024-09-16T01:53:01.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.543+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c02c640 con 0x7f728c0ca4c0 2024-09-16T01:53:01.545 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:53:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:53:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.544+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f728c02cd60 0x7f728c02d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.544+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f728c02d680 con 0x7f728c02cd60 2024-09-16T01:53:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.544+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f728c02cd60 0x7f728c02d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.545+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f728c02cd60 0x7f728c02d140 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.546+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c02d680 con 0x7f728c02cd60 2024-09-16T01:53:01.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.559+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c02d680 con 0x7f728c02cd60 2024-09-16T01:53:01.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.562+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c02d680 con 0x7f728c02cd60 2024-09-16T01:53:01.564 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:53:01.565 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:53:01.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.562+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f728c204be0 0x7f728c206fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.562+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f728c209da0 con 0x7f728c204be0 2024-09-16T01:53:01.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.562+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f728c204be0 0x7f728c206fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.563+0000 7f7289e00640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f728c204be0 0x7f728c206fd0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.564+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c209da0 con 0x7f728c204be0 2024-09-16T01:53:01.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.577+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c207510 con 0x7f728c204be0 2024-09-16T01:53:01.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.579+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c207510 con 0x7f728c204be0 2024-09-16T01:53:01.581 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:53:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:53:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.579+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f728c207b90 0x7f728c209f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.579+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f728c20cd50 con 0x7f728c207b90 2024-09-16T01:53:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.580+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f728c207b90 0x7f728c209f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.580+0000 7f728b200640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f728c207b90 0x7f728c209f80 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.581+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c20cd50 con 0x7f728c207b90 2024-09-16T01:53:01.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.595+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c20a4c0 con 0x7f728c207b90 2024-09-16T01:53:01.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.597+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c20a4c0 con 0x7f728c207b90 2024-09-16T01:53:01.599 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:53:01.600 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:53:01.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.598+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f728c20ab40 0x7f728c20cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:01.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.598+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f728c20fd00 con 0x7f728c20ab40 2024-09-16T01:53:01.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.598+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f728c20ab40 0x7f728c20cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:01.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.598+0000 7f728a800640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f728c20ab40 0x7f728c20cf30 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:01.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.600+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f728c20fd00 con 0x7f728c20ab40 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 253.832919226, 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34925853899999998, 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.604 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4201", 2024-09-16T01:53:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.161:0/2749638142", 2024-09-16T01:53:01.605 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768354+0000", 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768356+0000", 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5889999999999999e-06 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768380+0000", 2024-09-16T01:53:01.610 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4239e-05 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768393+0000", 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3066e-05 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768402+0000", 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1330000000000005e-06 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.611 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768465+0000", 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2923999999999996e-05 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:47.768474+0000", 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0799999999999995e-06 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115353+0000", 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.34687946200000003 2024-09-16T01:53:01.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115547+0000", 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000193475 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.115834+0000", 2024-09-16T01:53:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000287452 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117364+0000", 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0015301500000000001 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117512+0000", 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000147532 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117554+0000", 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1588000000000002e-05 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:48:48.117612+0000", 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8848999999999998e-05 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:513 3.21 3:86aaefcc:::10000000002.000000fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 121.215466886, 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50821042299999997, 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 513 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385806+0000", 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.385808+0000", 2024-09-16T01:53:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8369999999999999e-06 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413182+0000", 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027373728 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413200+0000", 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8629000000000001e-05 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413209+0000", 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3909999999999998e-06 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413279+0000", 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9904e-05 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.413288+0000", 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.217e-06 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.828630+0000", 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.41534226000000002 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.831001+0000", 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0023711959999999999 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865186+0000", 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034184565 2024-09-16T01:53:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865282+0000", 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5817999999999996e-05 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.865430+0000", 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014839399999999999 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.883319+0000", 2024-09-16T01:53:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017888733 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893819+0000", 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010500449 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.893897+0000", 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7644000000000004e-05 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.894017+0000", 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119658 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:772 3.21 3:848ebb0b:::100000001f7.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.423449937, 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14771456899999999, 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 772 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177823+0000", 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.177825+0000", 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9609999999999999e-06 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213269+0000", 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.035444147000000002 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213303+0000", 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3698000000000002e-05 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213316+0000", 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:53:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214511+0000", 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0011954559999999999 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214639+0000", 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127224 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.214742+0000", 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010360400000000001 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.237849+0000", 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023107114000000002 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325447+0000", 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.087597224000000001 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325503+0000", 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6795000000000001e-05 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325538+0000", 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.4477999999999998e-05 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:773 3.1c 3:398da958:::100000001f7.00000002:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e66)", 2024-09-16T01:53:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 101.38791731400001, 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11201401800000001, 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 773 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213356+0000", 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.213358+0000", 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8840000000000001e-06 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246782+0000", 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033424545 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246812+0000", 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0127000000000001e-05 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.246824+0000", 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1922999999999999e-05 2024-09-16T01:53:01.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248637+0000", 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0018131499999999999 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248740+0000", 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102484 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.248890+0000", 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014989100000000001 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.283086+0000", 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034196077999999998 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325056+0000", 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041969845999999998 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325156+0000", 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100208 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.325370+0000", 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00021388200000000001 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.635 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1477 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e80)", 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.021828377000006, 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.124707233, 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1477 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.636 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579445+0000", 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.579447+0000", 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8029999999999999e-06 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628509+0000", 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049062891999999997 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628542+0000", 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2032000000000001e-05 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628556+0000", 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4328000000000001e-05 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.628858+0000", 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000302195 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.629531+0000", 2024-09-16T01:53:01.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00067345999999999997 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.630276+0000", 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00074494000000000003 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.679022+0000", 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048745905999999999 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.640 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704038+0000", 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025015827000000001 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704104+0000", 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5430000000000002e-05 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:43.704152+0000", 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8420000000000001e-05 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.641 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1945 3.10 3:09a17e20:::100000001f8.00000396:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.457616838999996, 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.208683534, 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.642 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1945 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.643 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143656+0000", 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.143658+0000", 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1069999999999999e-06 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254641+0000", 2024-09-16T01:53:01.644 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.110982097 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254656+0000", 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5157000000000001e-05 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254664+0000", 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.0220000000000004e-06 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.645 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254762+0000", 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8228999999999998e-05 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.254910+0000", 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148011 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.255061+0000", 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015151699999999999 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.646 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.337875+0000", 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082813734999999999 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352225+0000", 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014350224999999999 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352292+0000", 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6338000000000001e-05 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.647 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.352340+0000", 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8096000000000002e-05 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:1944 3.f e84/20 3:f3dac423:::100000001f8.00000395:head v 84'39, mlcod=81'37)", 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.395829536999997, 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14484328099999999, 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.648 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:53:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.205444+0000", 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9100000000000004e-07 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228030+0000", 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022585411999999999 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.650 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228039+0000", 2024-09-16T01:53:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6339999999999996e-06 2024-09-16T01:53:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.651 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228047+0000", 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1790000000000008e-06 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228171+0000", 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012380900000000001 2024-09-16T01:53:01.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.228195+0000", 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3804000000000001e-05 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350239+0000", 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122043526 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.653 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.350287+0000", 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8325999999999998e-05 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2006 3.10 3:0a0745c3:::100000001f8.000003d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 71.554047091000001, 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11491246400000001, 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.654 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2006 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.655 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047226+0000", 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.047228+0000", 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2350000000000002e-06 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131295+0000", 2024-09-16T01:53:01.656 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.084066503000000001 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131324+0000", 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9388999999999999e-05 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131338+0000", 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3562e-05 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.657 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131509+0000", 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000171676 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131679+0000", 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016956999999999999 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.131834+0000", 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015484799999999999 2024-09-16T01:53:01.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.159881+0000", 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028046712000000001 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162064+0000", 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002183375 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.659 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162107+0000", 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2809000000000003e-05 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.162139+0000", 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1785000000000001e-05 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2848 3.10 3:089d96a6:::100000001f8.0000071d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e89)", 2024-09-16T01:53:01.660 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.810608338000002, 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12258813, 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2848 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.661 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790665+0000", 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.790666+0000", 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6500000000000001e-06 2024-09-16T01:53:01.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814465+0000", 2024-09-16T01:53:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023798574999999999 2024-09-16T01:53:01.663 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814490+0000", 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5092999999999998e-05 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814501+0000", 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1258e-05 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.664 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814629+0000", 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012716999999999999 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814801+0000", 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172807 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.814974+0000", 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000172409 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.665 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.822167+0000", 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0071928169999999998 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912637+0000", 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090470043999999999 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.912870+0000", 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023319700000000001 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.666 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:59.913253+0000", 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00038310999999999998 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3674 3.37 3:ee844fa5:::100000001f8.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.766387391999999, 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14415167700000001, 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.667 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3674 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.668 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834886+0000", 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.834887+0000", 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6530000000000001e-06 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843220+0000", 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0083322640000000007 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.669 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843246+0000", 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6597e-05 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843259+0000", 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2533e-05 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843342+0000", 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3039000000000002e-05 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.670 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843454+0000", 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011199799999999999 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843639+0000", 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018544000000000001 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.888003+0000", 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.044364210000000001 2024-09-16T01:53:01.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978925+0000", 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.090921743999999999 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.978989+0000", 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4071000000000004e-05 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.979037+0000", 2024-09-16T01:53:01.672 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8127999999999997e-05 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3677 3.2c 3:36e2673e:::100000001f8.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.757970905999997, 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15400875999999999, 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.673 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3677 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843302+0000", 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.674 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.843304+0000", 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5239999999999999e-06 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919840+0000", 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.076536495999999996 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919864+0000", 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4009999999999999e-05 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.675 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.919878+0000", 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4124999999999999e-05 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.676 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.923948+0000", 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0040693079999999998 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924056+0000", 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108725 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.924222+0000", 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166002 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.677 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.988627+0000", 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064404536999999998 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997188+0000", 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0085607719999999995 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997259+0000", 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1394000000000001e-05 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.678 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.997311+0000", 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1867000000000003e-05 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4107 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e99)", 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.325333419000003, 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.31541509600000001, 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4107 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.680 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275940+0000", 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.275941+0000", 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6959999999999999e-06 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303032+0000", 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027090797 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.681 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303053+0000", 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1019999999999999e-05 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303063+0000", 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6479999999999998e-06 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303176+0000", 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011292 2024-09-16T01:53:01.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.303185+0000", 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9560000000000003e-06 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.491825+0000", 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.18864050500000001 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.498314+0000", 2024-09-16T01:53:01.683 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064884210000000003 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551379+0000", 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053065042999999999 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551446+0000", 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7477999999999993e-05 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.684 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.551537+0000", 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1055000000000001e-05 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.576787+0000", 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025250156999999999 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591154+0000", 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01436693 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.685 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591228+0000", 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4103000000000005e-05 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.591355+0000", 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000126367 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.686 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4250 3.10 3:0a05e26b:::100000001f8.00000297:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.291113881000001, 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.149078401, 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4250 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.687 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310159+0000", 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.310162+0000", 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6599999999999999e-06 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.688 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407037+0000", 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096875366000000004 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407073+0000", 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.5969000000000001e-05 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407088+0000", 2024-09-16T01:53:01.689 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5034999999999999e-05 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407218+0000", 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000129462 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407332+0000", 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114183 2024-09-16T01:53:01.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.407516+0000", 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018366500000000001 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.446008+0000", 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038492551999999999 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456958+0000", 2024-09-16T01:53:01.691 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.01095023 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.458896+0000", 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0019376650000000001 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.459238+0000", 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00034161400000000001 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4504 3.f 3:f3dac423:::100000001f8.00000395:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.071486684999996, 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.22042661699999999, 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4504 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.693 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529786+0000", 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.694 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.529789+0000", 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5409999999999999e-06 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577075+0000", 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047285897 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577104+0000", 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9173999999999999e-05 2024-09-16T01:53:01.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577115+0000", 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0895e-05 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577223+0000", 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108194 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.696 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577330+0000", 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107089 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.577485+0000", 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154532 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.599121+0000", 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021636636000000001 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.697 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750098+0000", 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15097622499999999 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750164+0000", 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6172000000000003e-05 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.750213+0000", 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9261999999999997e-05 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.698 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4512 3.19 3:987f889e:::100000001f8.0000039d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.969293028000003, 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12513411299999999, 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4512 2024-09-16T01:53:01.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631980+0000", 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.700 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.631984+0000", 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8999999999999999e-06 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679187+0000", 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.047202698000000001 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679218+0000", 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1775e-05 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.701 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.679233+0000", 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4182000000000001e-05 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682406+0000", 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0031734889999999998 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.682504+0000", 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7374999999999995e-05 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.683089+0000", 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00058535 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.705462+0000", 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022372670000000001 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757013+0000", 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051551620999999999 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.703 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757066+0000", 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3044999999999998e-05 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.757114+0000", 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8007999999999998e-05 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.704 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4553 3.6 3:62a9ab61:::100000001f8.000003c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.434633595999998, 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.118770112, 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4553 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.705 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166640+0000", 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.166642+0000", 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0470000000000001e-06 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.706 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206911+0000", 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040269395999999999 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206944+0000", 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.3185000000000002e-05 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.206957+0000", 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2894999999999999e-05 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207157+0000", 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000199908 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207262+0000", 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105509 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.207423+0000", 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016019999999999999 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.708 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.221084+0000", 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013661240999999999 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285289+0000", 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064205110999999995 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285358+0000", 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9015000000000002e-05 2024-09-16T01:53:01.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285410+0000", 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1604999999999998e-05 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4560 3.39 3:9e09c0a8:::100000001f8.000003cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 34.355548048999999, 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11688955500000001, 2024-09-16T01:53:01.710 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4560 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.711 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245725+0000", 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.245728+0000", 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.548e-06 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301561+0000", 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055832859999999998 2024-09-16T01:53:01.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301606+0000", 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5423000000000002e-05 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301620+0000", 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.453e-05 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301728+0000", 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107836 2024-09-16T01:53:01.713 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.301847+0000", 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000119127 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.302620+0000", 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000772461 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.328630+0000", 2024-09-16T01:53:01.714 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026010253000000001 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362241+0000", 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033611125999999998 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362382+0000", 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014118400000000001 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.715 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.362615+0000", 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023220700000000001 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5800 3.8 3:139dc79b:::100000001f8.000008a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.228891054000002, 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.57801824300000004, 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.716 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5800 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372382+0000", 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.717 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.372384+0000", 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2840000000000001e-06 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387317+0000", 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014932126 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387330+0000", 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3658e-05 2024-09-16T01:53:01.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.387338+0000", 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3490000000000003e-06 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389339+0000", 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020019460000000001 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.389354+0000", 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.472e-05 2024-09-16T01:53:01.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899029+0000", 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.50967473200000002 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899208+0000", 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017872499999999999 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.899396+0000", 2024-09-16T01:53:01.720 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018809 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927572+0000", 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028175951000000001 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950173+0000", 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022601711 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.721 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950246+0000", 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.2533000000000005e-05 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950400+0000", 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000154418 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5815 3.21 3:84e25a9c:::100000001f8.000008b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:53:01.722 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 19.076713870999999, 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.133812769, 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5815 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524559+0000", 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.524561+0000", 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.968e-06 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.724 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641841+0000", 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11728021 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641861+0000", 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9939e-05 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641868+0000", 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5339999999999996e-06 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.641942+0000", 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4416e-05 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642034+0000", 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1916999999999994e-05 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.642172+0000", 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013773100000000001 2024-09-16T01:53:01.726 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656243+0000", 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014070719000000001 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658259+0000", 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0020165389999999999 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658323+0000", 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3876999999999998e-05 2024-09-16T01:53:01.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.658372+0000", 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8918999999999997e-05 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5821 3.8 3:10a53e03:::100000001f8.000008ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.924055112000001, 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.27338652600000002, 2024-09-16T01:53:01.728 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5821 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.729 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677218+0000", 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.677219+0000", 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.344e-06 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702527+0000", 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025308081999999999 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.730 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702546+0000", 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8698999999999998e-05 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702555+0000", 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2609999999999999e-06 2024-09-16T01:53:01.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702663+0000", 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010725 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.702675+0000", 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2743e-05 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:01.732 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910154+0000", 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.207478311 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910331+0000", 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000177057 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.910525+0000", 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000194074 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.733 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.927630+0000", 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017105187000000001 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950420+0000", 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022789713 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950473+0000", 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3115999999999998e-05 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:01.734 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.950605+0000", 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013168899999999999 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:01.735 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:01.737 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:53:01.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.613+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f728c20d4f0 con 0x7f728c20ab40 2024-09-16T01:53:01.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7282a00640 1 -- 172.21.15.146:0/61966397 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f728c20d4f0 con 0x7f728c20ab40 2024-09-16T01:53:01.738 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f728c0cc860 msgr2=0x7f728c148590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f728c0cc860 0x7f728c148590 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f728c162f10 msgr2=0x7f728c13ce90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f728c162f10 0x7f728c13ce90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f728c062630 msgr2=0x7f728c062a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f728c062630 0x7f728c062a50 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f728c0ca4c0 msgr2=0x7f728c0ca8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f728c0ca4c0 0x7f728c0ca8a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f728c02cd60 msgr2=0x7f728c02d140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f728c02cd60 0x7f728c02d140 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f728c204be0 msgr2=0x7f728c206fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f728c204be0 0x7f728c206fd0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 reap_dead start 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f728c207b90 msgr2=0x7f728c209f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f728c207b90 0x7f728c209f80 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 reap_dead start 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f728c20ab40 msgr2=0x7f728c20cf30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f728c20ab40 0x7f728c20cf30 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 reap_dead start 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f728b200640 1 -- 172.21.15.146:0/61966397 reap_dead start 2024-09-16T01:53:01.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f726c06c2e0 msgr2=0x7f726c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f726c06c2e0 0x7f726c06e7a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f727c005880 tx=0x7f727c009040 comp rx=0 tx=0).stop 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c151820 msgr2=0x7f728c1f3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c151820 0x7f728c1f3860 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7284031000 tx=0x7f7284004a40 comp rx=0 tx=0).stop 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 shutdown_connections 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f726c06c2e0 0x7f726c06e7a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 --2- 172.21.15.146:0/61966397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f728c151820 0x7f728c1f3860 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 >> 172.21.15.146:0/61966397 conn(0x7f728c06bc10 msgr2=0x7f728c157240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 shutdown_connections 2024-09-16T01:53:01.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.616+0000 7f7291400640 1 -- 172.21.15.146:0/61966397 wait complete. 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.622+0000 7f5b06a00640 1 -- 172.21.15.146:0/877955446 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v123) ==== 91+0+0 (secure 0 0 0) 0x7f5afc05ca40 con 0x7f5b10154d30 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5af006c2e0 msgr2=0x7f5af006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5af006c2e0 0x7f5af006e7a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009e90 tx=0x7f5b00009190 comp rx=0 tx=0).stop 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 msgr2=0x7f5b101e43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 0x7f5b101e43f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5afc00c4d0 tx=0x7f5afc00c990 comp rx=0 tx=0).stop 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 shutdown_connections 2024-09-16T01:53:01.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5af006c2e0 0x7f5af006e7a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.624+0000 7f5b15400640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5b101e4930 0x7f5b101f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.625+0000 7f5b15400640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b10154d30 0x7f5b101e43f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.625+0000 7f5b15400640 1 --2- 172.21.15.146:0/877955446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b100c7f30 0x7f5b101e5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.625+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 >> 172.21.15.146:0/877955446 conn(0x7f5b1006f3c0 msgr2=0x7f5b1014f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.625+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 shutdown_connections 2024-09-16T01:53:01.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:01.625+0000 7f5b15400640 1 -- 172.21.15.146:0/877955446 wait complete. 2024-09-16T01:53:01.744 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:01.735+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:01.837 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:01.835+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:01.938 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:01.936+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:01.938 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:53:01.938 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:53:01.938 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:01.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:53:02.039 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:02.037+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:02.139 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:02.137+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:02.149 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:02.149 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:53:02.149 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:53:02.149 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:53:02.149 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:02.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:53:02.249 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:02.247+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:02.342 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:02.340+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:02.362 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:02.362 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:53:02.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:53:02.362 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:53:02.362 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.068469176999997, 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.363 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.039308321999997, 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:53:02.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.374 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.006956754000001, 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.375 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:53:02.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:53:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:53:02.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:53:02.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.991009488000003, 2024-09-16T01:53:02.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:53:02.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:53:02.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.443934322999993, 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:53:02.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:53:02.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:53:02.395 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.259004966999996, 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.398 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:53:02.399 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:02.400 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.401 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.402 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.256467158999996, 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.403 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:53:02.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:53:02.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:53:02.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:53:02.408 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.647055926999997, 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.409 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.410 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.411 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.412 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:53:02.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.415 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:02.416 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.608799492000003, 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.417 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.418 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:53:02.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:53:02.420 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:53:02.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.422 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 46.533727757000001, 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.424 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:53:02.425 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.426 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:53:02.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.428 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4142 3.16 3:6ba609e9:::100000001f8.0000022b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.572631203, 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108516207, 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4142 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.430 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787022+0000", 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.61e-06 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842519+0000", 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055496399000000002 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.431 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842539+0000", 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0820000000000001e-05 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842550+0000", 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0414999999999999e-05 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842641+0000", 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1657999999999996e-05 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.432 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842742+0000", 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010075 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842888+0000", 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014623699999999999 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.892001+0000", 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112211000000003 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895434+0000", 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034334869999999998 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895499+0000", 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5313000000000003e-05 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895537+0000", 2024-09-16T01:53:02.434 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7307e-05 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4246 3.34 3:2f44359e:::100000001f8.00000293:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.030018640999998, 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.142262732, 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.435 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4246 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:53:02.436 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329635+0000", 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9180000000000001e-06 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410104+0000", 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080469499 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.437 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410128+0000", 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3243000000000001e-05 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410141+0000", 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3067999999999999e-05 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419779+0000", 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096380589999999992 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.438 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419889+0000", 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.420057+0000", 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016837699999999999 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456784+0000", 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036727237000000003 2024-09-16T01:53:02.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471769+0000", 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014985112 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471839+0000", 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9275000000000003e-05 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.440 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471896+0000", 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7259000000000002e-05 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4247 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 38.949463534000003, 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120085209, 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.441 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4247 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:53:02.442 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410190+0000", 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451530+0000", 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041340783999999998 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.443 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451572+0000", 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1690000000000002e-05 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451620+0000", 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8180999999999998e-05 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452310+0000", 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068996900000000002 2024-09-16T01:53:02.444 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452413+0000", 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010248599999999999 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452588+0000", 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017574800000000001 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.518054+0000", 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065465819999999994 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530168+0000", 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012113413999999999 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.446 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530225+0000", 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7284999999999998e-05 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530273+0000", 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8381999999999999e-05 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4503 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.847908179000001, 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13319203700000001, 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4503 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.448 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511746+0000", 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0250000000000001e-06 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.449 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576368+0000", 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064622151000000003 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576386+0000", 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7878999999999999e-05 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576398+0000", 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-09-16T01:53:02.450 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578343+0000", 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001945397 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578445+0000", 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102001 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.451 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578601+0000", 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015608399999999999 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.643878+0000", 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065277247999999996 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644839+0000", 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096038899999999997 2024-09-16T01:53:02.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644895+0000", 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6484000000000003e-05 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644936+0000", 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0324999999999998e-05 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4504 3.f e101/94 3:f3dac423:::100000001f8.00000395:head v 101'76, mlcod=100'74)", 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.764982062000001, 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14827673899999999, 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:53:02.454 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594672+0000", 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3309999999999998e-06 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.455 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629004+0000", 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034332342000000002 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629020+0000", 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6121e-05 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629029+0000", 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7479999999999999e-06 2024-09-16T01:53:02.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635523+0000", 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064935909999999999 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635549+0000", 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5808000000000001e-05 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.457 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742856+0000", 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107307556 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742946+0000", 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0241999999999994e-05 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.458 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4549 3.2 3:428d9500:::100000001f8.000003c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 35.190018615, 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11639424900000001, 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4549 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.459 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169636+0000", 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8660000000000002e-06 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.460 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212280+0000", 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042644350999999997 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212312+0000", 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2088999999999997e-05 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.461 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212327+0000", 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521e-05 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212426+0000", 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9245000000000003e-05 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212539+0000", 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112711 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.462 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212725+0000", 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018583400000000001 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285881+0000", 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073156321999999996 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285914+0000", 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2929999999999998e-05 2024-09-16T01:53:02.463 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285979+0000", 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5048000000000005e-05 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.286027+0000", 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8331999999999997e-05 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.464 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5790 3.16 3:687a9a46:::100000001f8.0000089b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.148983479000002, 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.65913515700000003, 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5790 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.465 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210670+0000", 2024-09-16T01:53:02.466 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244952+0000", 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034282188999999998 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244970+0000", 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8179999999999999e-05 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.467 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244982+0000", 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1736000000000001e-05 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245078+0000", 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5439000000000006e-05 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245090+0000", 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2177e-05 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.468 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791600+0000", 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.546509829 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791770+0000", 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017051699999999999 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791950+0000", 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018029299999999999 2024-09-16T01:53:02.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829483+0000", 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037532101999999998 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867462+0000", 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037979295000000003 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869661+0000", 2024-09-16T01:53:02.470 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021987059999999999 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869702+0000", 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1148e-05 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869803+0000", 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101696 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5793 3.16 3:68c3c174:::100000001f8.0000089e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.088272859, 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.59860013000000001, 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.472 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5793 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271381+0000", 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0669999999999999e-06 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311762+0000", 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040380644 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.474 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311785+0000", 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3090000000000001e-05 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311798+0000", 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3057999999999999e-05 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.475 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311911+0000", 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113429 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311923+0000", 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1915e-05 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829575+0000", 2024-09-16T01:53:02.476 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51765227000000003 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829754+0000", 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017901700000000001 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829935+0000", 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018041800000000001 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867483+0000", 2024-09-16T01:53:02.477 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037548040999999997 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869178+0000", 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016955379999999999 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869843+0000", 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000664345 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.478 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869882+0000", 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8976000000000001e-05 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869979+0000", 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7322000000000001e-05 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.479 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5945 3.2b 3:d7b2ba59:::100000001f8.00000936:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.962132393000001, 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11205129899999999, 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5945 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397521+0000", 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.481 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429779+0000", 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032257503999999999 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429805+0000", 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429818+0000", 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2687999999999999e-05 2024-09-16T01:53:02.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429896+0000", 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7898999999999994e-05 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430008+0000", 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011241100000000001 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430198+0000", 2024-09-16T01:53:02.483 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000189197 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.448700+0000", 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018502761999999999 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509440+0000", 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060739425999999999 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.484 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509514+0000", 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4673999999999997e-05 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509571+0000", 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6249999999999998e-05 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.485 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5948 3.2d 3:b677c6b2:::100000001f8.00000939:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 17.929437252, 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122297751, 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5948 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.486 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:02.487 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430216+0000", 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453110+0000", 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022894134 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453131+0000", 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1120000000000001e-05 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.488 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453141+0000", 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5340000000000005e-06 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495855+0000", 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042714162 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495997+0000", 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014160999999999999 2024-09-16T01:53:02.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537228+0000", 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041231297 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537360+0000", 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264400000000001 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6,7", 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537537+0000", 2024-09-16T01:53:02.490 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017664800000000001 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548049+0000", 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511915 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548065+0000", 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5812e-05 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:02.491 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552281+0000", 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042165600000000003 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552355+0000", 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3712000000000002e-05 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552512+0000", 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015707300000000001 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.492 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.493 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:02.493 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:02.493 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:02.494 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:53:02.495 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:02.493+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:02.595 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:02.594+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:02.694 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:53:02.694 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 0 2024-09-16T01:53:02.694 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:02.695 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:02.694+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:02.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:53:02.796 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:02.796+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:02.897 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:02.896+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:02.905 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:02.905 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:53:02.905 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:53:02.906 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:53:02.906 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:02.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:53:03.004 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:03.002+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:03.099 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:03.097+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:03.118 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:03.118 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:53:03.118 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:53:03.118 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 123.26700241899999, 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:53:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.120 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.121 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.122 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:53:03.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.124 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.125 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 123.124542754, 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:53:03.127 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.128 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.129 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:53:03.130 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:53:03.131 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.132 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:53:03.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 122.77707653100001, 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:53:03.134 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.135 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:53:03.136 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.137 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.138 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.139 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:53:03.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.141 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 102.892785167, 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.142 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:53:03.144 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.145 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.146 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.147 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 83.855045664000002, 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.148 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.149 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.150 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.151 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:53:03.152 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.153 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 82.996058664000003, 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.154 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:53:03.155 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.156 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:53:03.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 74.129019690000007, 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:53:03.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:53:03.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.933067468999994, 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:53:03.167 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:53:03.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:53:03.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.892065525999996, 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:53:03.173 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:53:03.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:53:03.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:53:03.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:53:03.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 73.667837965999993, 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:53:03.180 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:53:03.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:53:03.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:53:03.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 67.711056412999994, 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.186 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.188 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:53:03.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:53:03.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:53:03.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:53:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:53:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:53:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:53:03.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:53:03.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:53:03.195 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 61.800076064000002, 2024-09-16T01:53:03.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:53:03.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:03.202 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.363545082000002, 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:53:03.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.340391164000003, 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:53:03.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.213 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.329455320999998, 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:53:03.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.299260897000003, 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.218 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:53:03.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:53:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:53:03.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:03.224 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 47.257855444, 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:53:03.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5551 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:53:03.230 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.277048354000001, 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13476244500000001, 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5551 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838694+0000", 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.306e-06 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866535+0000", 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027841175999999999 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866552+0000", 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7387000000000001e-05 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866563+0000", 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0447e-05 2024-09-16T01:53:03.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866665+0000", 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101894 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866788+0000", 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123194 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866953+0000", 2024-09-16T01:53:03.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165365 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.918808+0000", 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051855367999999999 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973333+0000", 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054524250000000003 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973402+0000", 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9648999999999997e-05 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973454+0000", 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5812 3.a 3:51094797:::100000001f8.000008b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:53:03.236 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 20.583349239, 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14779089000000001, 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5812 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:53:03.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532392+0000", 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8679999999999999e-06 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653045+0000", 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120652599 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653062+0000", 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6929e-05 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653073+0000", 2024-09-16T01:53:03.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0925e-05 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656653+0000", 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035800049999999998 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656738+0000", 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5599000000000005e-05 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656890+0000", 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151874 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.676041+0000", 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019150358999999999 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680071+0000", 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004030624 2024-09-16T01:53:03.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680131+0000", 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9704000000000003e-05 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680181+0000", 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0404000000000001e-05 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5944 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 18.728520095, 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.115163742, 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.246 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5944 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387222+0000", 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418222+0000", 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031000603000000002 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418239+0000", 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6833e-05 2024-09-16T01:53:03.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418249+0000", 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8179999999999999e-06 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418846+0000", 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059710099999999995 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:03.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418945+0000", 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8826000000000006e-05 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.419093+0000", 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148553 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.442472+0000", 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023378423999999998 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502282+0000", 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059810025000000003 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502341+0000", 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8922000000000003e-05 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502383+0000", 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2590999999999999e-05 2024-09-16T01:53:03.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.252 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:03.252 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:03.252 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:03.253 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:53:03.254 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:03.253+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:03.354 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:03.352+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:03.370 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:03.370 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:03.371 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:53:03.371 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:03.380 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:53:03.455 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:03.454+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:03.484 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:03.485 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:03.485 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:53:03.485 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:53:03.485 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:03.494 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:53:03.555 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:03.554+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:03.609 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:03.609 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 132.742987642, 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.610 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.611 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:53:03.612 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:53:03.613 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.614 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:53:03.615 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:53:03.616 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 131.83140933000001, 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.617 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.618 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.619 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:53:03.620 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:53:03.621 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.622 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 131.78714909999999, 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.623 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.624 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.625 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:53:03.626 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.627 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.628 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 131.71509375799999, 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:53:03.629 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:53:03.631 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:53:03.632 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:53:03.633 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.634 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.635 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 123.45808733699999, 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.636 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.637 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.638 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:53:03.639 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.640 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:03.641 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.642 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.643 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.643 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:53:03.643 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:53:03.643 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.643 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:53:03.644 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.645 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 123.423291038, 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:53:03.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.647 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.648 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:53:03.649 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:53:03.650 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:53:03.653 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:53:03.654 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.655 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 123.339842085, 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:53:03.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.657 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:53:03.658 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.659 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.661 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.662 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.663 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.663 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:53:03.663 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:53:03.663 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.663 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.663 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 109.14053929799999, 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.664 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.665 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:53:03.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:53:03.667 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.668 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.669 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 108.99566236, 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:53:03.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.671 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.672 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:53:03.673 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.674 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:53:03.675 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.676 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:53:03.678 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 108.955017936, 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.679 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.680 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.681 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.682 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:53:03.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.684 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 108.79707510199999, 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.685 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:53:03.686 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.687 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.688 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.689 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:53:03.690 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 108.67194784100001, 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.691 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:53:03.692 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.693 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.694 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:53:03.695 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.696 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 107.63514791, 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.697 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.698 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.699 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.700 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:53:03.701 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:53:03.702 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 102.47212226800001, 2024-09-16T01:53:03.703 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:53:03.704 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.705 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.706 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:53:03.707 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.708 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:53:03.709 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 84.338384478999998, 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.710 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.711 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:53:03.712 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:53:03.713 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.714 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.715 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 83.643994364999998, 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.716 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.717 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.718 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:53:03.719 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.720 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.721 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 79.317019798999993, 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.722 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.723 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:53:03.724 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.725 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.726 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:53:03.727 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 79.176109659000005, 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.728 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.729 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:53:03.730 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:53:03.731 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.732 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.733 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 79.123306714999998, 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.734 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.735 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:53:03.736 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.737 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.738 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:53:03.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 77.804048175000005, 2024-09-16T01:53:03.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:53:03.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:53:03.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:03.746 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:03.747 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:53:03.747 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:03.677+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:03.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:53:03.748 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:53:03.817 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:03.816+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:03.852 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:03.853 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:03.853 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:53:03.853 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:03.861 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:53:03.879 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:03.877+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:03.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.911+0000 7fe0e9000640 1 Processor -- start 2024-09-16T01:53:03.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.911+0000 7fe0e9000640 1 -- start start 2024-09-16T01:53:03.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.912+0000 7fe0e9000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0e40c36b0 0x7fe0e40c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.912+0000 7fe0e9000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e40c9180 0x7fe0e40c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.912+0000 7fe0e9000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e40c4240 con 0x7fe0e4152c30 2024-09-16T01:53:03.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.912+0000 7fe0e9000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e40c71c0 con 0x7fe0e40c36b0 2024-09-16T01:53:03.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.912+0000 7fe0e9000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e40c3a90 con 0x7fe0e40c9180 2024-09-16T01:53:03.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e1e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe0e4152c30 0x7fe0e40c6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55370/0 (socket says 172.21.15.146:55370) 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e1e00640 1 -- 172.21.15.146:0/3956814761 learned_addr learned my addr 172.21.15.146:0/3956814761 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e2800640 1 --2- 172.21.15.146:0/3956814761 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e40c9180 0x7fe0e40c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e1400640 1 --2- 172.21.15.146:0/3956814761 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40c36b0 0x7fe0e40c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e0a00640 1 -- 172.21.15.146:0/3956814761 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887843084 0 0) 0x7fe0e40c4240 con 0x7fe0e4152c30 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e0a00640 1 -- 172.21.15.146:0/3956814761 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c8003520 con 0x7fe0e4152c30 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e2800640 1 -- 172.21.15.146:0/3956814761 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40c36b0 msgr2=0x7fe0e40c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e2800640 1 --2- 172.21.15.146:0/3956814761 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40c36b0 0x7fe0e40c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e2800640 1 -- 172.21.15.146:0/3956814761 >> v1:172.21.15.161:6789/0 conn(0x7fe0e4152c30 legacy=0x7fe0e40c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.914+0000 7fe0e2800640 1 -- 172.21.15.146:0/3956814761 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e40c4500 con 0x7fe0e40c9180 2024-09-16T01:53:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e2800640 1 --2- 172.21.15.146:0/3956814761 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e40c9180 0x7fe0e40c7ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009d90 tx=0x7fe0dc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6b1b94fa136fe40 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e0a00640 1 -- 172.21.15.146:0/3956814761 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc007740 con 0x7fe0e40c9180 2024-09-16T01:53:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e0a00640 1 -- 172.21.15.146:0/3956814761 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0078e0 con 0x7fe0e40c9180 2024-09-16T01:53:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e0a00640 1 -- 172.21.15.146:0/3956814761 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc007c90 con 0x7fe0e40c9180 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e9000640 1 -- 172.21.15.146:0/3956814761 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e40c9180 msgr2=0x7fe0e40c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e9000640 1 --2- 172.21.15.146:0/3956814761 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e40c9180 0x7fe0e40c7ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009d90 tx=0x7fe0dc02a660 comp rx=0 tx=0).stop 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e9000640 1 -- 172.21.15.146:0/3956814761 shutdown_connections 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e9000640 1 --2- 172.21.15.146:0/3956814761 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e40c9180 0x7fe0e40c7ff0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e9000640 1 --2- 172.21.15.146:0/3956814761 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40c36b0 0x7fe0e40c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.915+0000 7fe0e9000640 1 -- 172.21.15.146:0/3956814761 >> 172.21.15.146:0/3956814761 conn(0x7fe0e406bc10 msgr2=0x7fe0e406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.916+0000 7fe0e9000640 1 -- 172.21.15.146:0/3956814761 shutdown_connections 2024-09-16T01:53:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.916+0000 7fe0e9000640 1 -- 172.21.15.146:0/3956814761 wait complete. 2024-09-16T01:53:03.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.917+0000 7fe0e9000640 1 Processor -- start 2024-09-16T01:53:03.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.917+0000 7fe0e9000640 1 -- start start 2024-09-16T01:53:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e40c36b0 0x7fe0e40c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 0x7fe0e41f0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e4152c30 0x7fe0e4148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e40c4ac0 con 0x7fe0e40c36b0 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e40c4da0 con 0x7fe0e40c9180 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e40c6630 con 0x7fe0e4152c30 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 0x7fe0e41f0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e40c36b0 0x7fe0e40c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 0x7fe0e41f0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55744/0 (socket says 172.21.15.146:55744) 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 -- 172.21.15.146:0/2854106940 learned_addr learned my addr 172.21.15.146:0/2854106940 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e4152c30 0x7fe0e4148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e4152c30 msgr2=0x7fe0e4148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe0e4152c30 0x7fe0e4148ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e40c36b0 msgr2=0x7fe0e40c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e40c36b0 0x7fe0e40c14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e41596d0 con 0x7fe0e40c9180 2024-09-16T01:53:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 0x7fe0e41f0750 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000e870 tx=0x7fe0d000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0017070 con 0x7fe0e40c9180 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d000c940 con 0x7fe0e40c9180 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d000cd30 con 0x7fe0e40c9180 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e4071b20 con 0x7fe0e40c9180 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.919+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e41f3fa0 con 0x7fe0e40c9180 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.921+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe0e404ffb0 con 0x7fe0e40c9180 2024-09-16T01:53:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.921+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe0d001a070 con 0x7fe0e40c9180 2024-09-16T01:53:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.923+0000 7fe0da000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0c406c2b0 0x7fe0c406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.923+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5894+0+0 (secure 0 0 0) 0x7fe0d008b420 con 0x7fe0e40c9180 2024-09-16T01:53:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.923+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0c406c2b0 0x7fe0c406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.923+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v123) ==== 54+0+16 (secure 0 0 0) 0x7fe0e404ffb0 con 0x7fe0e40c9180 2024-09-16T01:53:03.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.923+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe0e4148ff0 0x7fe0e41f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.924+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0e41f46b0 con 0x7fe0e4148ff0 2024-09-16T01:53:03.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.924+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0c406c2b0 0x7fe0c406e770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005880 tx=0x7fe0d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.924+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe0e4148ff0 0x7fe0e41f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.924+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe0e4148ff0 0x7fe0e41f4170 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.925+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0e41f46b0 con 0x7fe0e4148ff0 2024-09-16T01:53:03.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.926+0000 7f83dda00640 1 Processor -- start 2024-09-16T01:53:03.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.926+0000 7f83dda00640 1 -- start start 2024-09-16T01:53:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83dda00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d80cc0c0 0x7f83d815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83dda00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83d80cb2f0 0x7f83d80cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83dda00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d815b1d0 con 0x7f83d815e9d0 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83dda00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f83d80cbd00 con 0x7f83d80cb2f0 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83dda00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f83d815ba70 con 0x7f83d80cc0c0 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f83d80cb2f0 0x7f83d80cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d80cc0c0 0x7f83d815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d7400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f83d815e9d0 0x7f83d80cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55374/0 (socket says 172.21.15.146:55374) 2024-09-16T01:53:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d7400640 1 -- 172.21.15.146:0/2351531482 learned_addr learned my addr 172.21.15.146:0/2351531482 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d5600640 1 -- 172.21.15.146:0/2351531482 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143138684 0 0) 0x7f83d815b1d0 con 0x7f83d815e9d0 2024-09-16T01:53:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6a00640 1 -- 172.21.15.146:0/2351531482 >> v2:172.21.15.161:3300/0 conn(0x7f83d80cb2f0 msgr2=0x7f83d80cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6a00640 1 --2- 172.21.15.146:0/2351531482 >> v2:172.21.15.161:3300/0 conn(0x7f83d80cb2f0 0x7f83d80cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6a00640 1 -- 172.21.15.146:0/2351531482 >> v1:172.21.15.161:6789/0 conn(0x7f83d815e9d0 legacy=0x7f83d80cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6a00640 1 -- 172.21.15.146:0/2351531482 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d815b4b0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.928+0000 7f83d6000640 1 --2- 172.21.15.146:0/2351531482 >> v2:172.21.15.161:3300/0 conn(0x7f83d80cb2f0 0x7f83d80cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:03.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.929+0000 7f83d6a00640 1 --2- 172.21.15.146:0/2351531482 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d80cc0c0 0x7f83d815aab0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f83c80097a0 tx=0x7f83c802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42bec86e6bb784db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.929+0000 7f83d5600640 1 -- 172.21.15.146:0/2351531482 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8037070 con 0x7f83d80cc0c0 2024-09-16T01:53:03.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.929+0000 7f83d5600640 1 -- 172.21.15.146:0/2351531482 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c8032440 con 0x7f83d80cc0c0 2024-09-16T01:53:03.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.929+0000 7f83d5600640 1 -- 172.21.15.146:0/2351531482 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8032810 con 0x7f83d80cc0c0 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 -- 172.21.15.146:0/2351531482 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d80cc0c0 msgr2=0x7f83d815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 --2- 172.21.15.146:0/2351531482 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d80cc0c0 0x7f83d815aab0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f83c80097a0 tx=0x7f83c802a180 comp rx=0 tx=0).stop 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 -- 172.21.15.146:0/2351531482 shutdown_connections 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 --2- 172.21.15.146:0/2351531482 >> v2:172.21.15.161:3300/0 conn(0x7f83d80cb2f0 0x7f83d80cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 --2- 172.21.15.146:0/2351531482 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d80cc0c0 0x7f83d815aab0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 -- 172.21.15.146:0/2351531482 >> 172.21.15.146:0/2351531482 conn(0x7f83d806d560 msgr2=0x7f83d806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 -- 172.21.15.146:0/2351531482 shutdown_connections 2024-09-16T01:53:03.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.930+0000 7f83dda00640 1 -- 172.21.15.146:0/2351531482 wait complete. 2024-09-16T01:53:03.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.933+0000 7f83dda00640 1 Processor -- start 2024-09-16T01:53:03.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.933+0000 7f83dda00640 1 -- start start 2024-09-16T01:53:03.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83dda00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83d80cb2f0 0x7f83d814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83dda00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83d80cc0c0 0x7f83d815c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83dda00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 0x7f83d815cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83dda00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f83d815e1d0 con 0x7f83d80cb2f0 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83dda00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83d815e540 con 0x7f83d80cc0c0 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83dda00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f83d80c5560 con 0x7f83d815e9d0 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 0x7f83d815cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83d80cb2f0 0x7f83d814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d7400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 0x7f83d815cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43158/0 (socket says 172.21.15.146:43158) 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d7400640 1 -- 172.21.15.146:0/1282398503 learned_addr learned my addr 172.21.15.146:0/1282398503 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6000640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83d80cc0c0 0x7f83d815c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6000640 1 -- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 msgr2=0x7f83d815cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6000640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 0x7f83d815cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6000640 1 -- 172.21.15.146:0/1282398503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83d80cb2f0 msgr2=0x7f83d814b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:03.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6000640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83d80cb2f0 0x7f83d814b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:03.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d6000640 1 -- 172.21.15.146:0/1282398503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d81f0ec0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.934+0000 7f83d7400640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 0x7f83d815cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:03.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83d6000640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83d80cc0c0 0x7f83d815c6c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f83c400e900 tx=0x7f83c400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4017070 con 0x7f83d80cc0c0 2024-09-16T01:53:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c400ca00 con 0x7f83d80cc0c0 2024-09-16T01:53:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c400cdf0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d81f11a0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d815d8f0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.935+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d80cbea0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.936+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f83c401a070 con 0x7f83d80cc0c0 2024-09-16T01:53:03.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.937+0000 7f83cea00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83b806c2b0 0x7f83b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.937+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5894+0+0 (secure 0 0 0) 0x7f83c408b320 con 0x7f83d80cc0c0 2024-09-16T01:53:03.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.937+0000 7f83d6a00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83b806c2b0 0x7f83b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.938+0000 7f83d6a00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83b806c2b0 0x7f83b806e770 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f83c8034000 tx=0x7f83c802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.941+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f83c40592e0 con 0x7f83d80cc0c0 2024-09-16T01:53:03.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.955+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e40c4240 con 0x7fe0e4148ff0 2024-09-16T01:53:03.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.957+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e40c4240 con 0x7fe0e4148ff0 2024-09-16T01:53:03.958 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:53:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:53:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.957+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe0e40cb850 0x7fe0e40be330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.957+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe0e40c2ad0 con 0x7fe0e40cb850 2024-09-16T01:53:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.957+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe0e40cb850 0x7fe0e40be330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.957+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe0e40cb850 0x7fe0e40be330 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.959+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0e40c2ad0 con 0x7fe0e40cb850 2024-09-16T01:53:03.968 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:03.968 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:03.968 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:53:03.968 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:53:03.968 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:03.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.974+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e41505f0 con 0x7fe0e40cb850 2024-09-16T01:53:03.978 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:53:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.976+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e41505f0 con 0x7fe0e40cb850 2024-09-16T01:53:03.981 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:53:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:53:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.978+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe0e40bed80 0x7fe0e4062630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.978+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe0e41f4bf0 con 0x7fe0e40bed80 2024-09-16T01:53:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.978+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe0e40bed80 0x7fe0e4062630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.978+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe0e40bed80 0x7fe0e4062630 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.983 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:03.978+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.978+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0d0059350 con 0x7fe0e40bed80 2024-09-16T01:53:03.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.993+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e40bf1a0 con 0x7fe0e40bed80 2024-09-16T01:53:03.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.995+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e40bf1a0 con 0x7fe0e40bed80 2024-09-16T01:53:03.997 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:53:03.998 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:53:03.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.995+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe0e40631f0 0x7fe0e40c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:03.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.995+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe0e40635f0 con 0x7fe0e40631f0 2024-09-16T01:53:03.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.995+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe0e40631f0 0x7fe0e40c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:03.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.996+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe0e40631f0 0x7fe0e40c00b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:03.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:03.998+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0dc045ac0 con 0x7fe0e40631f0 2024-09-16T01:53:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.011+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e40c0840 con 0x7fe0e40631f0 2024-09-16T01:53:04.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.013+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e40c0840 con 0x7fe0e40631f0 2024-09-16T01:53:04.014 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:53:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:53:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.013+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe0e402c640 0x7fe0e402caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.013+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe0e40c0ec0 con 0x7fe0e402c640 2024-09-16T01:53:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.013+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe0e402c640 0x7fe0e402caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.013+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe0e402c640 0x7fe0e402caa0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.015+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0e40c0ec0 con 0x7fe0e402c640 2024-09-16T01:53:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.028+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e40c0ec0 con 0x7fe0e402c640 2024-09-16T01:53:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.029+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e40c0ec0 con 0x7fe0e402c640 2024-09-16T01:53:04.032 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:53:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:53:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.030+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe0e402d4f0 0x7fe0e4207050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.030+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe0e402d930 con 0x7fe0e402d4f0 2024-09-16T01:53:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.030+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe0e402d4f0 0x7fe0e4207050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.030+0000 7fe0e1400640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe0e402d4f0 0x7fe0e4207050 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.031+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0e402d930 con 0x7fe0e402d4f0 2024-09-16T01:53:04.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.045+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e402db70 con 0x7fe0e402d4f0 2024-09-16T01:53:04.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.048+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e402db70 con 0x7fe0e402d4f0 2024-09-16T01:53:04.049 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:53:04.049 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:53:04.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.048+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe0e4207aa0 0x7fe0e4209e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.048+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe0e420cc60 con 0x7fe0e4207aa0 2024-09-16T01:53:04.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.048+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe0e4207aa0 0x7fe0e4209e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.048+0000 7fe0e2800640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe0e4207aa0 0x7fe0e4209e90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.049+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0e420cc60 con 0x7fe0e4207aa0 2024-09-16T01:53:04.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.076+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e420a450 con 0x7fe0e4207aa0 2024-09-16T01:53:04.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.078+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e420a450 con 0x7fe0e4207aa0 2024-09-16T01:53:04.080 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:53:04.080 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:53:04.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.078+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe0e420ab70 0x7fe0e420afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.078+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe0e4210030 con 0x7fe0e420ab70 2024-09-16T01:53:04.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.078+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe0e420ab70 0x7fe0e420afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.080+0000 7fe0e1e00640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe0e420ab70 0x7fe0e420afb0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.080+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0e4210030 con 0x7fe0e420ab70 2024-09-16T01:53:04.082 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:04.081+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:04.091 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.091 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.094+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0e420eeb0 con 0x7fe0e420ab70 2024-09-16T01:53:04.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.094+0000 7fe0da000640 1 -- 172.21.15.146:0/2854106940 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0e420eeb0 con 0x7fe0e420ab70 2024-09-16T01:53:04.096 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe0e4148ff0 msgr2=0x7fe0e41f4170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fe0e4148ff0 0x7fe0e41f4170 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe0e40cb850 msgr2=0x7fe0e40be330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fe0e40cb850 0x7fe0e40be330 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe0e40bed80 msgr2=0x7fe0e4062630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fe0e40bed80 0x7fe0e4062630 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe0e40631f0 msgr2=0x7fe0e40c00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fe0e40631f0 0x7fe0e40c00b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe0e402c640 msgr2=0x7fe0e402caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe0e402c640 0x7fe0e402caa0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe0e402d4f0 msgr2=0x7fe0e4207050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fe0e402d4f0 0x7fe0e4207050 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe0e4207aa0 msgr2=0x7fe0e4209e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fe0e4207aa0 0x7fe0e4209e90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe0e420ab70 msgr2=0x7fe0e420afb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fe0e420ab70 0x7fe0e420afb0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e2800640 1 -- 172.21.15.146:0/2854106940 reap_dead start 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e2800640 1 -- 172.21.15.146:0/2854106940 reap_dead start 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e2800640 1 -- 172.21.15.146:0/2854106940 reap_dead start 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e2800640 1 -- 172.21.15.146:0/2854106940 reap_dead start 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e2800640 1 -- 172.21.15.146:0/2854106940 reap_dead start 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0c406c2b0 msgr2=0x7fe0c406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0c406c2b0 0x7fe0c406e770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4005880 tx=0x7fe0d4009040 comp rx=0 tx=0).stop 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 msgr2=0x7fe0e41f0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 0x7fe0e41f0750 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000e870 tx=0x7fe0d000ed30 comp rx=0 tx=0).stop 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 shutdown_connections 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe0c406c2b0 0x7fe0c406e770 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 --2- 172.21.15.146:0/2854106940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e40c9180 0x7fe0e41f0750 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 >> 172.21.15.146:0/2854106940 conn(0x7fe0e406bc10 msgr2=0x7fe0e406ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 shutdown_connections 2024-09-16T01:53:04.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.096+0000 7fe0e9000640 1 -- 172.21.15.146:0/2854106940 wait complete. 2024-09-16T01:53:04.101 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:53:04.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.139+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f83d80cfee0 con 0x7f83d80cc0c0 2024-09-16T01:53:04.186 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:04.185+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:04.245 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.245 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.254 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:53:04.284 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:04.283+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:04.385 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:04.384+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:04.398 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.399 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.407 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:53:04.426 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T01:53:04.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T01:53:04.486 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:04.485+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:04.510 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.510 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.519 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:53:04.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.586+0000 7f3cd6e00640 1 Processor -- start 2024-09-16T01:53:04.587 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:04.586+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:04.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.586+0000 7f3cd6e00640 1 -- start start 2024-09-16T01:53:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3cd6e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cd0155000 0x7f3cd015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3cd6e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd00b31e0 con 0x7f3cd00d09e0 2024-09-16T01:53:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3cd6e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd00b4110 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3cd6e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f3cd00b43f0 con 0x7f3cd0155000 2024-09-16T01:53:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3cd4e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cd0155000 0x7f3cd015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3ccfe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3cd00d09e0 0x7f3cd015c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33808/0 (socket says 172.21.15.146:33808) 2024-09-16T01:53:04.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3ccfe00640 1 -- 172.21.15.146:0/3033492116 learned_addr learned my addr 172.21.15.146:0/3033492116 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:04.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.587+0000 7f3ccf400640 1 --1- 172.21.15.146:0/3033492116 >> v1:172.21.15.161:6789/0 conn(0x7f3cd00d0dc0 0x7f3cd01548f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55380/0 (socket says 172.21.15.146:55380) 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.588+0000 7f3ccea00640 1 -- 172.21.15.146:0/3033492116 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891139115 0 0) 0x7f3cd00b4110 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.588+0000 7f3ccea00640 1 -- 172.21.15.146:0/3033492116 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb4003520 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.588+0000 7f3cd4e00640 1 -- 172.21.15.146:0/3033492116 >> v1:172.21.15.161:6789/0 conn(0x7f3cd00d0dc0 legacy=0x7f3cd01548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.588+0000 7f3cd4e00640 1 -- 172.21.15.146:0/3033492116 >> v1:172.21.15.146:6789/0 conn(0x7f3cd00d09e0 legacy=0x7f3cd015c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.588+0000 7f3cd4e00640 1 -- 172.21.15.146:0/3033492116 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd00b46d0 con 0x7f3cd0155000 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.588+0000 7f3ccea00640 1 -- 172.21.15.146:0/3033492116 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732342046 0 0) 0x7f3cd00b31e0 con 0x7f3cd00d09e0 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.589+0000 7f3cd4e00640 1 --2- 172.21.15.146:0/3033492116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd0155000 0x7f3cd015adc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009d90 tx=0x7f3cc802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22d3886a501ee55 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.589+0000 7f3ccea00640 1 -- 172.21.15.146:0/3033492116 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8037070 con 0x7f3cd0155000 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.589+0000 7f3ccea00640 1 -- 172.21.15.146:0/3033492116 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc8007740 con 0x7f3cd0155000 2024-09-16T01:53:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.589+0000 7f3ccea00640 1 -- 172.21.15.146:0/3033492116 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8007b10 con 0x7f3cd0155000 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.589+0000 7f3cd6e00640 1 -- 172.21.15.146:0/3033492116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd0155000 msgr2=0x7f3cd015adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.589+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/3033492116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd0155000 0x7f3cd015adc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009d90 tx=0x7f3cc802a780 comp rx=0 tx=0).stop 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.590+0000 7f3cd6e00640 1 -- 172.21.15.146:0/3033492116 shutdown_connections 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.590+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/3033492116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd0155000 0x7f3cd015adc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.590+0000 7f3cd6e00640 1 -- 172.21.15.146:0/3033492116 >> 172.21.15.146:0/3033492116 conn(0x7f3cd006c920 msgr2=0x7f3cd006cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.590+0000 7f3cd6e00640 1 -- 172.21.15.146:0/3033492116 shutdown_connections 2024-09-16T01:53:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.590+0000 7f3cd6e00640 1 -- 172.21.15.146:0/3033492116 wait complete. 2024-09-16T01:53:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.591+0000 7f3cd6e00640 1 Processor -- start 2024-09-16T01:53:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.591+0000 7f3cd6e00640 1 -- start start 2024-09-16T01:53:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd6e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3cd00d09e0 0x7f3cd00c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd6e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 0x7f3cd01e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd6e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cd0155000 0x7f3cd00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd6e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd00b4f70 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd6e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd00b4c90 con 0x7f3cd0155000 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd6e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd015c7a0 con 0x7f3cd00d09e0 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3ccf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 0x7f3cd01e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3ccfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3cd00d09e0 0x7f3cd00c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3cd4e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cd0155000 0x7f3cd00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.592+0000 7f3ccfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3cd00d09e0 0x7f3cd00c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43166/0 (socket says 172.21.15.146:43166) 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccfe00640 1 -- 172.21.15.146:0/756329306 learned_addr learned my addr 172.21.15.146:0/756329306 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:04.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 0x7f3cd01e3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49390/0 (socket says 172.21.15.146:49390) 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 -- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3cd00d09e0 msgr2=0x7f3cd00c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3cd00d09e0 0x7f3cd00c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 -- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cd0155000 msgr2=0x7f3cd00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cd0155000 0x7f3cd00c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 -- 172.21.15.146:0/756329306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd00c5b70 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3cd4e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cd0155000 0x7f3cd00c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3ccf400640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 0x7f3cd01e3e20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c400 tx=0x7f3cbc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0074e0 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cbc007680 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc004a00 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.593+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd015da90 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.594+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd00c9fb0 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.595+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd015c090 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.596+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3cbc01a020 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.597+0000 7f3cc7e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3cb006c2b0 0x7f3cb006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:04.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.597+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5894+0+0 (secure 0 0 0) 0x7f3cbc08ab80 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.597+0000 7f3ccfe00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3cb006c2b0 0x7f3cb006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:04.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.598+0000 7f3ccfe00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3cb006c2b0 0x7f3cb006e770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0000f60 tx=0x7f3cc0007e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:04.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.602+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3cbc007e20 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.625 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.625 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.634+0000 7f83cea00640 1 -- 172.21.15.146:0/1282398503 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v124) ==== 101+0+0 (secure 0 0 0) 0x7f83c405d5e0 con 0x7f83d80cc0c0 2024-09-16T01:53:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:53:04.636 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:53:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83b806c2b0 msgr2=0x7f83b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83b806c2b0 0x7f83b806e770 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f83c8034000 tx=0x7f83c802e000 comp rx=0 tx=0).stop 2024-09-16T01:53:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83d80cc0c0 msgr2=0x7f83d815c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83d80cc0c0 0x7f83d815c6c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f83c400e900 tx=0x7f83c400edc0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 shutdown_connections 2024-09-16T01:53:04.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83b806c2b0 0x7f83b806e770 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83d815e9d0 0x7f83d815cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83d80cc0c0 0x7f83d815c6c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 --2- 172.21.15.146:0/1282398503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83d80cb2f0 0x7f83d814b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:04.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 >> 172.21.15.146:0/1282398503 conn(0x7f83d806d560 msgr2=0x7f83d80cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:04.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 shutdown_connections 2024-09-16T01:53:04.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.638+0000 7f83dda00640 1 -- 172.21.15.146:0/1282398503 wait complete. 2024-09-16T01:53:04.688 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:04.687+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:04.739 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.739 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.748 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:53:04.789 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:04.788+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:04.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:04.809+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f3cd015d850 con 0x7f3cd00d0dc0 2024-09-16T01:53:04.890 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:04.889+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:04.893 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:04.893 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:04.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:53:04.991 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:04.990+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:05.064 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:05.065 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:05.077 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:53:05.092 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:05.091+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:05.193 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:05.192+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:05.243 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:05.243 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:05.257 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:53:05.294 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:05.293+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:05.395 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:05.394+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:05.459 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:05.459 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:05.472 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:53:05.513 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:05.512+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:05.597 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:05.597+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:05.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.637+0000 7f3cc7e00640 1 -- 172.21.15.146:0/756329306 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v125) ==== 113+0+0 (secure 0 0 0) 0x7f3cbc058b40 con 0x7f3cd00d0dc0 2024-09-16T01:53:05.639 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T01:53:05.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3cb006c2b0 msgr2=0x7f3cb006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:05.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3cb006c2b0 0x7f3cb006e770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0000f60 tx=0x7f3cc0007e10 comp rx=0 tx=0).stop 2024-09-16T01:53:05.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 msgr2=0x7f3cd01e3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 0x7f3cd01e3e20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c400 tx=0x7f3cbc00c8c0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 shutdown_connections 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f3cb006c2b0 0x7f3cb006e770 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cd0155000 0x7f3cd00c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cd00d0dc0 0x7f3cd01e3e20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 --2- 172.21.15.146:0/756329306 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3cd00d09e0 0x7f3cd00c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 >> 172.21.15.146:0/756329306 conn(0x7f3cd006c920 msgr2=0x7f3cd0153150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 shutdown_connections 2024-09-16T01:53:05.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.640+0000 7f3cd6e00640 1 -- 172.21.15.146:0/756329306 wait complete. 2024-09-16T01:53:05.660 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T01:53:05.675 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:05.676 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:05.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:53:05.698 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:05.697+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:05.799 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:05.798+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:05.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.818+0000 7f7eb1800640 1 Processor -- start 2024-09-16T01:53:05.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.818+0000 7f7eb1800640 1 -- start start 2024-09-16T01:53:05.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.819+0000 7f7eb1800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7eac0d09e0 0x7f7eac15c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.819+0000 7f7eb1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7eac0d0dc0 0x7f7eac15c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.819+0000 7f7eb1800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac1548f0 0x7f7eac154d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.819+0000 7f7eb1800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f7eac0b31e0 con 0x7f7eac0d0dc0 2024-09-16T01:53:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.819+0000 7f7eb1800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7eac155260 con 0x7f7eac0d09e0 2024-09-16T01:53:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.819+0000 7f7eb1800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7eac0b4110 con 0x7f7eac1548f0 2024-09-16T01:53:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7eac0d09e0 0x7f7eac15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7eac0d0dc0 0x7f7eac15c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7eac0d09e0 0x7f7eac15c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55772/0 (socket says 172.21.15.146:55772) 2024-09-16T01:53:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eab400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac1548f0 0x7f7eac154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 -- 172.21.15.146:0/711897645 learned_addr learned my addr 172.21.15.146:0/711897645 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 -- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac1548f0 msgr2=0x7f7eac154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:05.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 --2- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac1548f0 0x7f7eac154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 -- 172.21.15.146:0/711897645 >> v2:172.21.15.146:3300/0 conn(0x7f7eac0d0dc0 msgr2=0x7f7eac15c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 --2- 172.21.15.146:0/711897645 >> v2:172.21.15.146:3300/0 conn(0x7f7eac0d0dc0 0x7f7eac15c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.820+0000 7f7eaaa00640 1 -- 172.21.15.146:0/711897645 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eac0b43f0 con 0x7f7eac0d09e0 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.821+0000 7f7eaaa00640 1 --2- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac0d09e0 0x7f7eac15c090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7eac08dc90 tx=0x7f7e9c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=541309630b4fbc74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7ea9600640 1 -- 172.21.15.146:0/711897645 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c037070 con 0x7f7eac0d09e0 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7ea9600640 1 -- 172.21.15.146:0/711897645 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e9c032440 con 0x7f7eac0d09e0 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7ea9600640 1 -- 172.21.15.146:0/711897645 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c032810 con 0x7f7eac0d09e0 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 -- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac0d09e0 msgr2=0x7f7eac15c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 --2- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac0d09e0 0x7f7eac15c090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f7eac08dc90 tx=0x7f7e9c02a270 comp rx=0 tx=0).stop 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 -- 172.21.15.146:0/711897645 shutdown_connections 2024-09-16T01:53:05.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 --2- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac1548f0 0x7f7eac154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 --2- 172.21.15.146:0/711897645 >> v2:172.21.15.146:3300/0 conn(0x7f7eac0d0dc0 0x7f7eac15c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 --2- 172.21.15.146:0/711897645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac0d09e0 0x7f7eac15c090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.822+0000 7f7eb1800640 1 -- 172.21.15.146:0/711897645 >> 172.21.15.146:0/711897645 conn(0x7f7eac06c920 msgr2=0x7f7eac06cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.823+0000 7f7eb1800640 1 -- 172.21.15.146:0/711897645 shutdown_connections 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.823+0000 7f7eb1800640 1 -- 172.21.15.146:0/711897645 wait complete. 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.824+0000 7f7eb1800640 1 Processor -- start 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.824+0000 7f7eb1800640 1 -- start start 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eb1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 0x7f7eac0c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eb1800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac0d0dc0 0x7f7eac1f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eb1800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac1548f0 0x7f7eac0c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eb1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7eac0b4c90 con 0x7f7eac0d09e0 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eb1800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7eac0b49b0 con 0x7f7eac1548f0 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eb1800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7eac0b4f70 con 0x7f7eac0d0dc0 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.825+0000 7f7eaaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 0x7f7eac0c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 0x7f7eac0c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49402/0 (socket says 172.21.15.146:49402) 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 -- 172.21.15.146:0/3574497419 learned_addr learned my addr 172.21.15.146:0/3574497419 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaa000640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac0d0dc0 0x7f7eac1f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eab400640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac1548f0 0x7f7eac0c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 -- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac0d0dc0 msgr2=0x7f7eac1f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac0d0dc0 0x7f7eac1f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 -- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac1548f0 msgr2=0x7f7eac0c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:05.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac1548f0 0x7f7eac0c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 -- 172.21.15.146:0/3574497419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eac0ca130 con 0x7f7eac0d09e0 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.826+0000 7f7eaaa00640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 0x7f7eac0c7a20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c02a2b0 tx=0x7f7e9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.827+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c041020 con 0x7f7eac0d09e0 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.827+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e9c03d620 con 0x7f7eac0d09e0 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.827+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c037040 con 0x7f7eac0d09e0 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.827+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eac1f4570 con 0x7f7eac0d09e0 2024-09-16T01:53:05.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.827+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eac0c4300 con 0x7f7eac0d09e0 2024-09-16T01:53:05.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.828+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eac0b3b00 con 0x7f7eac0d09e0 2024-09-16T01:53:05.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.829+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7e9c044020 con 0x7f7eac0d09e0 2024-09-16T01:53:05.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.830+0000 7f7ea2a00640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e8c06c2b0 0x7f7e8c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:05.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.830+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5894+0+0 (secure 0 0 0) 0x7f7e9c0abad0 con 0x7f7eac0d09e0 2024-09-16T01:53:05.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.830+0000 7f7eaa000640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e8c06c2b0 0x7f7e8c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:05.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.831+0000 7f7eaa000640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e8c06c2b0 0x7f7e8c06e770 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0cfec0 tx=0x7f7e98009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:05.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:05.834+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7eac0b3b00 con 0x7f7eac0d09e0 2024-09-16T01:53:05.894 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:05.895 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:05.900 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:05.899+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:05.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:53:06.001 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:06.000+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:06.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.029+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7f7eac154d20 con 0x7f7eac0d09e0 2024-09-16T01:53:06.101 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:06.101+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:06.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T01:53:06.139 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:06.139 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:06.159 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:53:06.205 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:06.201+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:06.308 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:06.303+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.376+0000 7f0842800640 1 Processor -- start 2024-09-16T01:53:06.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.376+0000 7f0842800640 1 -- start start 2024-09-16T01:53:06.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.378+0000 7f0842800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f083c0d0490 0x7f083c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.378+0000 7f0842800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f083c0d0870 0x7f083c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.378+0000 7f0842800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 0x7f083c154b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.378+0000 7f0842800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f083c0c6160 con 0x7f083c0d0490 2024-09-16T01:53:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.378+0000 7f0842800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f083c15e220 con 0x7f083c0d0870 2024-09-16T01:53:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.378+0000 7f0842800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f083c15e500 con 0x7f083c154690 2024-09-16T01:53:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.379+0000 7f0841e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 0x7f083c154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.379+0000 7f0841e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 0x7f083c154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43194/0 (socket says 172.21.15.146:43194) 2024-09-16T01:53:06.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.379+0000 7f0841e00640 1 -- 172.21.15.146:0/2652678006 learned_addr learned my addr 172.21.15.146:0/2652678006 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:06.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.380+0000 7f0840a00640 1 --2- 172.21.15.146:0/2652678006 >> v2:172.21.15.161:3300/0 conn(0x7f083c0d0870 0x7f083c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.380+0000 7f0841e00640 1 -- 172.21.15.146:0/2652678006 >> v2:172.21.15.161:3300/0 conn(0x7f083c0d0870 msgr2=0x7f083c0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.380+0000 7f0841e00640 1 --2- 172.21.15.146:0/2652678006 >> v2:172.21.15.161:3300/0 conn(0x7f083c0d0870 0x7f083c0c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.380+0000 7f0841e00640 1 -- 172.21.15.146:0/2652678006 >> v2:172.21.15.146:3300/0 conn(0x7f083c0d0490 msgr2=0x7f083c0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.380+0000 7f0841e00640 1 --2- 172.21.15.146:0/2652678006 >> v2:172.21.15.146:3300/0 conn(0x7f083c0d0490 0x7f083c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.380+0000 7f0841e00640 1 -- 172.21.15.146:0/2652678006 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c15e7e0 con 0x7f083c154690 2024-09-16T01:53:06.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.382+0000 7f0841e00640 1 --2- 172.21.15.146:0/2652678006 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 0x7f083c154b00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f083c0c8730 tx=0x7f083402a280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b34a6557655fd6e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.382+0000 7f083be00640 1 -- 172.21.15.146:0/2652678006 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834037070 con 0x7f083c154690 2024-09-16T01:53:06.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.382+0000 7f083be00640 1 -- 172.21.15.146:0/2652678006 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0834032440 con 0x7f083c154690 2024-09-16T01:53:06.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.382+0000 7f083be00640 1 -- 172.21.15.146:0/2652678006 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834032850 con 0x7f083c154690 2024-09-16T01:53:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 -- 172.21.15.146:0/2652678006 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 msgr2=0x7f083c154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 --2- 172.21.15.146:0/2652678006 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 0x7f083c154b00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f083c0c8730 tx=0x7f083402a280 comp rx=0 tx=0).stop 2024-09-16T01:53:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 -- 172.21.15.146:0/2652678006 shutdown_connections 2024-09-16T01:53:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 --2- 172.21.15.146:0/2652678006 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c154690 0x7f083c154b00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 --2- 172.21.15.146:0/2652678006 >> v2:172.21.15.161:3300/0 conn(0x7f083c0d0870 0x7f083c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 --2- 172.21.15.146:0/2652678006 >> v2:172.21.15.146:3300/0 conn(0x7f083c0d0490 0x7f083c0c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.384+0000 7f0842800640 1 -- 172.21.15.146:0/2652678006 >> 172.21.15.146:0/2652678006 conn(0x7f083c06d6f0 msgr2=0x7f083c06e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:06.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.385+0000 7f0842800640 1 -- 172.21.15.146:0/2652678006 shutdown_connections 2024-09-16T01:53:06.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.386+0000 7f0842800640 1 -- 172.21.15.146:0/2652678006 wait complete. 2024-09-16T01:53:06.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.389+0000 7f0842800640 1 Processor -- start 2024-09-16T01:53:06.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.389+0000 7f0842800640 1 -- start start 2024-09-16T01:53:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.391+0000 7f0842800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c0d0490 0x7f083c06ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.391+0000 7f0842800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f083c0d0870 0x7f083c150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.391+0000 7f0842800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f083c154690 0x7f083c150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.391+0000 7f0842800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f083c155040 con 0x7f083c154690 2024-09-16T01:53:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.391+0000 7f0842800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f083c15eda0 con 0x7f083c0d0870 2024-09-16T01:53:06.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.391+0000 7f0842800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f083c15dec0 con 0x7f083c0d0490 2024-09-16T01:53:06.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.392+0000 7f0840a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f083c0d0870 0x7f083c150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.392+0000 7f0840a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f083c0d0870 0x7f083c150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55792/0 (socket says 172.21.15.146:55792) 2024-09-16T01:53:06.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.392+0000 7f0840a00640 1 -- 172.21.15.146:0/2730033669 learned_addr learned my addr 172.21.15.146:0/2730033669 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:06.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.392+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c0d0490 0x7f083c06ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.392+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f083c154690 0x7f083c150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.394+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c0d0490 msgr2=0x7f083c06ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.394+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f083c0d0490 0x7f083c06ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.394+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f083c0d0870 msgr2=0x7f083c150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.394+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f083c0d0870 0x7f083c150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.394+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c1515e0 con 0x7f083c154690 2024-09-16T01:53:06.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.395+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f083c154690 0x7f083c150ff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0834031000 tx=0x7f08340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.395+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834041020 con 0x7f083c154690 2024-09-16T01:53:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.396+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f083403d6e0 con 0x7f083c154690 2024-09-16T01:53:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.396+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834037040 con 0x7f083c154690 2024-09-16T01:53:06.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.396+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083c072c60 con 0x7f083c154690 2024-09-16T01:53:06.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.397+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083c06e1b0 con 0x7f083c154690 2024-09-16T01:53:06.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.403+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0834044020 con 0x7f083c154690 2024-09-16T01:53:06.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.404+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f083c0c95d0 con 0x7f083c154690 2024-09-16T01:53:06.405 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:06.404+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:06.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.405+0000 7f0839600640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f081c06c380 0x7f081c06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.405+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5894+0+0 (secure 0 0 0) 0x7f08340aba80 con 0x7f083c154690 2024-09-16T01:53:06.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.405+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f081c06c380 0x7f081c06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.406+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v125) ==== 54+0+16 (secure 0 0 0) 0x7f083c0c95d0 con 0x7f083c154690 2024-09-16T01:53:06.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.407+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f083c14efd0 0x7f083c14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.407+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f083c0cc3f0 con 0x7f083c14efd0 2024-09-16T01:53:06.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.407+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f081c06c380 0x7f081c06e840 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f083000a400 tx=0x7f083000d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.409+0000 7f0840a00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f083c14efd0 0x7f083c14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.410+0000 7f0840a00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f083c14efd0 0x7f083c14f3b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.412+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c0cc3f0 con 0x7f083c14efd0 2024-09-16T01:53:06.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.441+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c0cb430 con 0x7f083c14efd0 2024-09-16T01:53:06.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.446+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c0cb430 con 0x7f083c14efd0 2024-09-16T01:53:06.447 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:53:06.448 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:53:06.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.447+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f083c157f70 0x7f083c1565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.447+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f083c158350 con 0x7f083c157f70 2024-09-16T01:53:06.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.447+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f083c157f70 0x7f083c1565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.449+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f083c157f70 0x7f083c1565a0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.452+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c158350 con 0x7f083c157f70 2024-09-16T01:53:06.489 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:06.489 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:06.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.489+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c156d30 con 0x7f083c157f70 2024-09-16T01:53:06.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.492+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c156d30 con 0x7f083c157f70 2024-09-16T01:53:06.494 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:53:06.494 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:53:06.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.493+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f083c157300 0x7f083c062b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.493+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f083c063080 con 0x7f083c157300 2024-09-16T01:53:06.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.497+0000 7f0840a00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f083c157300 0x7f083c062b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.499+0000 7f0840a00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f083c157300 0x7f083c062b40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.501+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c063080 con 0x7f083c157300 2024-09-16T01:53:06.506 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:06.505+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:06.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:53:06.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.544+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c063590 con 0x7f083c157300 2024-09-16T01:53:06.555 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:53:06.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.548+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c063590 con 0x7f083c157300 2024-09-16T01:53:06.556 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:53:06.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.548+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f083c0b41d0 0x7f083c206840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.548+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f083c0b45f0 con 0x7f083c0b41d0 2024-09-16T01:53:06.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.550+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f083c0b41d0 0x7f083c206840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.552+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f083c0b41d0 0x7f083c206840 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.556+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c0b45f0 con 0x7f083c0b41d0 2024-09-16T01:53:06.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.587+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c206d80 con 0x7f083c0b41d0 2024-09-16T01:53:06.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.593+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c206d80 con 0x7f083c0b41d0 2024-09-16T01:53:06.594 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:53:06.595 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:53:06.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.593+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f083c2075b0 0x7f083c2099a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.593+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f083c20c8b0 con 0x7f083c2075b0 2024-09-16T01:53:06.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.593+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f083c2075b0 0x7f083c2099a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.594+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f083c2075b0 0x7f083c2099a0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.595+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c20c8b0 con 0x7f083c2075b0 2024-09-16T01:53:06.608 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:06.607+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:06.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.621+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c209ee0 con 0x7f083c2075b0 2024-09-16T01:53:06.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.623+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c209ee0 con 0x7f083c2075b0 2024-09-16T01:53:06.625 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:53:06.625 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:53:06.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.627+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f083c20a9c0 0x7f083c20ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.627+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f083c20c4e0 con 0x7f083c20a9c0 2024-09-16T01:53:06.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.628+0000 7f0840a00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f083c20a9c0 0x7f083c20ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.630+0000 7f0840a00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f083c20a9c0 0x7f083c20ae20 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.631+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c20c4e0 con 0x7f083c20a9c0 2024-09-16T01:53:06.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.641+0000 7f7ea2a00640 1 -- 172.21.15.146:0/3574497419 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v126) ==== 113+0+0 (secure 0 0 0) 0x7f7e9c04b030 con 0x7f7eac0d09e0 2024-09-16T01:53:06.643 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T01:53:06.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e8c06c2b0 msgr2=0x7f7e8c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e8c06c2b0 0x7f7e8c06e770 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0cfec0 tx=0x7f7e98009190 comp rx=0 tx=0).stop 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 msgr2=0x7f7eac0c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 0x7f7eac0c7a20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c02a2b0 tx=0x7f7e9c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 shutdown_connections 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e8c06c2b0 0x7f7e8c06e770 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0cfec0 tx=0x7f7e98009190 comp rx=0 tx=0).stop 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7eac1548f0 0x7f7eac0c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7eac0d0dc0 0x7f7eac1f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 --2- 172.21.15.146:0/3574497419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eac0d09e0 0x7f7eac0c7a20 secure :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c02a2b0 tx=0x7f7e9c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.644+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 >> 172.21.15.146:0/3574497419 conn(0x7f7eac06c920 msgr2=0x7f7eac067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:06.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.647+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 shutdown_connections 2024-09-16T01:53:06.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.647+0000 7f7eb1800640 1 -- 172.21.15.146:0/3574497419 wait complete. 2024-09-16T01:53:06.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.657+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c20b360 con 0x7f083c20a9c0 2024-09-16T01:53:06.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.659+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c20b360 con 0x7f083c20a9c0 2024-09-16T01:53:06.661 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:53:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:53:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.660+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f083c20ba80 0x7f083c20bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.660+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f083c2133a0 con 0x7f083c20ba80 2024-09-16T01:53:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.660+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f083c20ba80 0x7f083c20bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.661+0000 7f0841e00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f083c20ba80 0x7f083c20bec0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.663+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c2133a0 con 0x7f083c20ba80 2024-09-16T01:53:06.667 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T01:53:06.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T01:53:06.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.697+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c210b10 con 0x7f083c20ba80 2024-09-16T01:53:06.702 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:53:06.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.700+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c210b10 con 0x7f083c20ba80 2024-09-16T01:53:06.703 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:53:06.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.700+0000 7f082ea00640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f083c211190 0x7f083c213580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.700+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f083c2164e0 con 0x7f083c211190 2024-09-16T01:53:06.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.701+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f083c211190 0x7f083c213580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.702+0000 7f0841400640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f083c211190 0x7f083c213580 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.703+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f083c2164e0 con 0x7f083c211190 2024-09-16T01:53:06.707 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:06.706+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:06.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.733+0000 7f082ea00640 1 -- 172.21.15.146:0/2730033669 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f083c216510 con 0x7f083c211190 2024-09-16T01:53:06.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.735+0000 7f0839600640 1 -- 172.21.15.146:0/2730033669 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f083c216510 con 0x7f083c211190 2024-09-16T01:53:06.737 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:53:06.737 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:53:06.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f083c14efd0 msgr2=0x7f083c14f3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f083c14efd0 0x7f083c14f3b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f083c157f70 msgr2=0x7f083c1565a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f083c157f70 0x7f083c1565a0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f083c157300 msgr2=0x7f083c062b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f083c157300 0x7f083c062b40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f083c0b41d0 msgr2=0x7f083c206840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f083c0b41d0 0x7f083c206840 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f083c2075b0 msgr2=0x7f083c2099a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7f083c2075b0 0x7f083c2099a0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f083c20a9c0 msgr2=0x7f083c20ae20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f083c20a9c0 0x7f083c20ae20 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f083c20ba80 msgr2=0x7f083c20bec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7f083c20ba80 0x7f083c20bec0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f083c211190 msgr2=0x7f083c213580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.736+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f083c211190 0x7f083c213580 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f081c06c380 msgr2=0x7f081c06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f081c06c380 0x7f081c06e840 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f083000a400 tx=0x7f083000d040 comp rx=0 tx=0).stop 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f083c154690 msgr2=0x7f083c150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0842800640 1 --2- 172.21.15.146:0/2730033669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f083c154690 0x7f083c150ff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0834031000 tx=0x7f08340046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 reap_dead start 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 reap_dead start 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 reap_dead start 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 reap_dead start 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 reap_dead start 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.738+0000 7f0841e00640 1 -- 172.21.15.146:0/2730033669 reap_dead start 2024-09-16T01:53:06.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.739+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 shutdown_connections 2024-09-16T01:53:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.739+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 >> 172.21.15.146:0/2730033669 conn(0x7f083c06d6f0 msgr2=0x7f083c0cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.740+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 shutdown_connections 2024-09-16T01:53:06.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.741+0000 7f0842800640 1 -- 172.21.15.146:0/2730033669 wait complete. 2024-09-16T01:53:06.816 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:06.815+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:06.832 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:06.832 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:53:06.832 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:53:06.832 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:53:06.832 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:06.854 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:53:06.944 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:06.943+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.970+0000 7f192e800640 1 Processor -- start 2024-09-16T01:53:06.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.970+0000 7f192e800640 1 -- start start 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.972+0000 7f192e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1920005a00 0x7f19200461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.972+0000 7f192e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1920034c40 con 0x7f1920046730 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.972+0000 7f192e800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1920034f20 con 0x7f19200d5590 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.972+0000 7f192e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f19200d5970 con 0x7f1920005a00 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.973+0000 7f192d400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f19200d5590 0x7f1920044fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55386/0 (socket says 172.21.15.146:55386) 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.973+0000 7f192d400640 1 -- 172.21.15.146:0/301843563 learned_addr learned my addr 172.21.15.146:0/301843563 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.974+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2597464116 0 0) 0x7f1920034f20 con 0x7f19200d5590 2024-09-16T01:53:06.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.974+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1910003520 con 0x7f19200d5590 2024-09-16T01:53:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.974+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2966219948 0 0) 0x7f1910003520 con 0x7f19200d5590 2024-09-16T01:53:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.974+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 >> v2:172.21.15.146:3300/0 conn(0x7f1920005a00 msgr2=0x7f19200461f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:53:06.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.975+0000 7f192ca00640 1 --2- 172.21.15.146:0/301843563 >> v2:172.21.15.146:3300/0 conn(0x7f1920005a00 0x7f19200461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.978+0000 7f1927e00640 1 --2- 172.21.15.146:0/301843563 >> v2:172.21.15.146:3300/0 conn(0x7f1920005a00 0x7f19200461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.978+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 >> v1:172.21.15.146:6789/0 conn(0x7f1920046730 legacy=0x7f19200db650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:53:06.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1920035200 con 0x7f19200d5590 2024-09-16T01:53:06.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f191c001670 con 0x7f19200d5590 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 >> v1:172.21.15.161:6789/0 conn(0x7f19200d5590 legacy=0x7f1920044fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19100039b0 0x7f1910003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1910004800 0x7f1910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100052e0 0x7f191000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f19200d5970 con 0x7f19100052e0 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f191000c1d0 con 0x7f1910004800 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.979+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f191000c3d0 con 0x7f19100039b0 2024-09-16T01:53:06.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.980+0000 7f192d400640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19100039b0 0x7f1910003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.980+0000 7f192de00640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100052e0 0x7f191000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.981+0000 7f192d400640 1 -- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1910004800 msgr2=0x7f1910004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:53:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.981+0000 7f192d400640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1910004800 0x7f1910004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.981+0000 7f192d400640 1 -- 172.21.15.146:0/301843563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100052e0 msgr2=0x7f191000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.981+0000 7f192d400640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100052e0 0x7f191000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.981+0000 7f192d400640 1 -- 172.21.15.146:0/301843563 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1910003520 con 0x7f19100039b0 2024-09-16T01:53:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.982+0000 7f192de00640 1 -- 172.21.15.146:0/301843563 reap_dead start 2024-09-16T01:53:06.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.983+0000 7f192ca00640 1 --2- 172.21.15.146:0/301843563 >> v2:172.21.15.146:3300/0 conn(0x7f1920005a00 0x7f19200461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:06.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.984+0000 7f192d400640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19100039b0 0x7f1910003dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f191c007fd0 tx=0x7f1910005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9a2b2346a3c32e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:06.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.985+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c007ca0 con 0x7f19100039b0 2024-09-16T01:53:06.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.985+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191c005ad0 con 0x7f19100039b0 2024-09-16T01:53:06.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.985+0000 7f1927e00640 1 -- 172.21.15.146:0/301843563 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c007070 con 0x7f19100039b0 2024-09-16T01:53:06.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.988+0000 7f192e800640 1 -- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19100039b0 msgr2=0x7f1910003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:06.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.989+0000 7f192e800640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19100039b0 0x7f1910003dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f191c007fd0 tx=0x7f1910005830 comp rx=0 tx=0).stop 2024-09-16T01:53:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.990+0000 7f192e800640 1 -- 172.21.15.146:0/301843563 shutdown_connections 2024-09-16T01:53:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.990+0000 7f192e800640 1 --2- 172.21.15.146:0/301843563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19100039b0 0x7f1910003dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:06.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.990+0000 7f192e800640 1 -- 172.21.15.146:0/301843563 >> 172.21.15.146:0/301843563 conn(0x7f192004fc50 msgr2=0x7f1920050060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:06.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.991+0000 7f192e800640 1 -- 172.21.15.146:0/301843563 shutdown_connections 2024-09-16T01:53:06.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.991+0000 7f192e800640 1 -- 172.21.15.146:0/301843563 wait complete. 2024-09-16T01:53:06.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.994+0000 7f192e800640 1 Processor -- start 2024-09-16T01:53:06.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.994+0000 7f192e800640 1 -- start start 2024-09-16T01:53:06.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.995+0000 7f192e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 0x7f1920049520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.995+0000 7f192e800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19200c4520 0x7f19200ba420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.995+0000 7f192e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19200db070 0x7f19200d8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.996+0000 7f192e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f19200060b0 con 0x7f19100039b0 2024-09-16T01:53:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.996+0000 7f192e800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1920005e50 con 0x7f19200db070 2024-09-16T01:53:06.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.996+0000 7f192e800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1920050990 con 0x7f19200c4520 2024-09-16T01:53:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.996+0000 7f192d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 0x7f1920049520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.996+0000 7f192d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 0x7f1920049520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49432/0 (socket says 172.21.15.146:49432) 2024-09-16T01:53:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.996+0000 7f192d400640 1 -- 172.21.15.146:0/848794894 learned_addr learned my addr 172.21.15.146:0/848794894 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.997+0000 7f192de00640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19200db070 0x7f19200d8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.997+0000 7f192ca00640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19200c4520 0x7f19200ba420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:06.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.997+0000 7f192d400640 1 -- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19200c4520 msgr2=0x7f19200ba420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.998+0000 7f192d400640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19200c4520 0x7f19200ba420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.998+0000 7f192d400640 1 -- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19200db070 msgr2=0x7f19200d8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.998+0000 7f192d400640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19200db070 0x7f19200d8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.998+0000 7f192d400640 1 -- 172.21.15.146:0/848794894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1920049a60 con 0x7f19100039b0 2024-09-16T01:53:07.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:06.998+0000 7f192ca00640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19200c4520 0x7f19200ba420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.000+0000 7f192d400640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 0x7f1920049520 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f191c009000 tx=0x7f1920034a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:07.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.000+0000 7f192de00640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19200db070 0x7f19200d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:07.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.001+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c03f070 con 0x7f19100039b0 2024-09-16T01:53:07.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.002+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191c0013b0 con 0x7f19100039b0 2024-09-16T01:53:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.002+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c007070 con 0x7f19100039b0 2024-09-16T01:53:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.004+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1920047ae0 con 0x7f19100039b0 2024-09-16T01:53:07.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.004+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19200daad0 con 0x7f19100039b0 2024-09-16T01:53:07.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.009+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19200461f0 con 0x7f19100039b0 2024-09-16T01:53:07.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.010+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f191c042070 con 0x7f19100039b0 2024-09-16T01:53:07.013 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:07.009+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:07.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.012+0000 7f1925600640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f190806c2e0 0x7f190806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.012+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5894+0+0 (secure 0 0 0) 0x7f191c0ab170 con 0x7f19100039b0 2024-09-16T01:53:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.012+0000 7f192ca00640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f190806c2e0 0x7f190806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:07.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.014+0000 7f192ca00640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f190806c2e0 0x7f190806e7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f19200e9cc0 tx=0x7f1914008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:07.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.019+0000 7f31ec800640 1 Processor -- start 2024-09-16T01:53:07.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.019+0000 7f31ec800640 1 -- start start 2024-09-16T01:53:07.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.022+0000 7f31ec800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31e41528e0 0x7f31e40c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.022+0000 7f31ec800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e4152cc0 0x7f31e415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.023+0000 7f31e9e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31e41528e0 0x7f31e40c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:07.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.023+0000 7f31e9e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f31e41528e0 0x7f31e40c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55808/0 (socket says 172.21.15.146:55808) 2024-09-16T01:53:07.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.024+0000 7f31e9e00640 1 -- 172.21.15.146:0/2363713569 learned_addr learned my addr 172.21.15.146:0/2363713569 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:07.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.024+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e40c34b0 con 0x7f31e4070e00 2024-09-16T01:53:07.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.024+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f31e40c3790 con 0x7f31e41528e0 2024-09-16T01:53:07.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.024+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f31e40c3a70 con 0x7f31e4152cc0 2024-09-16T01:53:07.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.024+0000 7f31e9400640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e4152cc0 0x7f31e415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:07.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.025+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f191c049030 con 0x7f19100039b0 2024-09-16T01:53:07.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.025+0000 7f31e9e00640 1 -- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e4152cc0 msgr2=0x7f31e415c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:07.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.025+0000 7f31e8a00640 1 -- 172.21.15.146:0/2363713569 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4137889155 0 0) 0x7f31e40c34b0 con 0x7f31e4070e00 2024-09-16T01:53:07.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.025+0000 7f31e9e00640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e4152cc0 0x7f31e415c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.025+0000 7f31e9e00640 1 -- 172.21.15.146:0/2363713569 >> v1:172.21.15.146:6789/0 conn(0x7f31e4070e00 legacy=0x7f31e40ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.026+0000 7f31e9e00640 1 -- 172.21.15.146:0/2363713569 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e415bd00 con 0x7f31e41528e0 2024-09-16T01:53:07.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.027+0000 7f31e9400640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e4152cc0 0x7f31e415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:07.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.027+0000 7f31e9e00640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e41528e0 0x7f31e40c9fc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009870 tx=0x7f31d802a320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35cbaf2dc785e7c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:07.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.028+0000 7f31e8a00640 1 -- 172.21.15.146:0/2363713569 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8037070 con 0x7f31e41528e0 2024-09-16T01:53:07.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.028+0000 7f31e8a00640 1 -- 172.21.15.146:0/2363713569 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d8032640 con 0x7f31e41528e0 2024-09-16T01:53:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.029+0000 7f31e8a00640 1 -- 172.21.15.146:0/2363713569 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8037070 con 0x7f31e41528e0 2024-09-16T01:53:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.029+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e41528e0 msgr2=0x7f31e40c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.029+0000 7f31ec800640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e41528e0 0x7f31e40c9fc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009870 tx=0x7f31d802a320 comp rx=0 tx=0).stop 2024-09-16T01:53:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.030+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 shutdown_connections 2024-09-16T01:53:07.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.030+0000 7f31ec800640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e4152cc0 0x7f31e415c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.030+0000 7f31ec800640 1 --2- 172.21.15.146:0/2363713569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e41528e0 0x7f31e40c9fc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.030+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 >> 172.21.15.146:0/2363713569 conn(0x7f31e406c240 msgr2=0x7f31e4067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:07.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.030+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 shutdown_connections 2024-09-16T01:53:07.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.030+0000 7f31ec800640 1 -- 172.21.15.146:0/2363713569 wait complete. 2024-09-16T01:53:07.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.033+0000 7f31ec800640 1 Processor -- start 2024-09-16T01:53:07.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.033+0000 7f31ec800640 1 -- start start 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.035+0000 7f31ec800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 0x7f31e40c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.035+0000 7f31ec800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e41528e0 0x7f31e40c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.035+0000 7f31ec800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e4152cc0 0x7f31e40b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.035+0000 7f31ec800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f31e40cea90 con 0x7f31e4070e00 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 0x7f31e40c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31ec800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f31e415c2c0 con 0x7f31e4152cc0 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31ec800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f31e40cec10 con 0x7f31e41528e0 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 0x7f31e40c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49434/0 (socket says 172.21.15.146:49434) 2024-09-16T01:53:07.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9e00640 1 -- 172.21.15.146:0/1222687780 learned_addr learned my addr 172.21.15.146:0/1222687780 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:07.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9400640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e41528e0 0x7f31e40c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:07.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9400640 1 -- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e4152cc0 msgr2=0x7f31e40b2ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9400640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e4152cc0 0x7f31e40b2ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9400640 1 -- 172.21.15.146:0/1222687780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 msgr2=0x7f31e40c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9400640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 0x7f31e40c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9400640 1 -- 172.21.15.146:0/1222687780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e41f8c50 con 0x7f31e41528e0 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.036+0000 7f31e9e00640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 0x7f31e40c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.037+0000 7f31e9400640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e41528e0 0x7f31e40c7840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d5f0 tx=0x7f31d400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.037+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4004030 con 0x7f31e41528e0 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.037+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d40041d0 con 0x7f31e41528e0 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.037+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4014620 con 0x7f31e41528e0 2024-09-16T01:53:07.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.037+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e41f8ef0 con 0x7f31e41528e0 2024-09-16T01:53:07.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.038+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e4155d60 con 0x7f31e41528e0 2024-09-16T01:53:07.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.040+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f31d40147c0 con 0x7f31e41528e0 2024-09-16T01:53:07.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.042+0000 7f31de000640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f31c806c2e0 0x7f31c806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:07.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.042+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e40cf4e0 con 0x7f31e41528e0 2024-09-16T01:53:07.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.042+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(126..126 src has 1..126) ==== 5894+0+0 (secure 0 0 0) 0x7f31d408bb30 con 0x7f31e41528e0 2024-09-16T01:53:07.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.042+0000 7f31e9e00640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f31c806c2e0 0x7f31c806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:07.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.044+0000 7f31e9e00640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f31c806c2e0 0x7f31c806e7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f31d8000c00 tx=0x7f31d802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:07.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.048+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f31d405a1d0 con 0x7f31e41528e0 2024-09-16T01:53:07.115 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:07.114+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:07.155 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1124 3.26 3:660a31fe:::100000001f8.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.861380118, 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.17110440499999999, 2024-09-16T01:53:07.156 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1124 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291183+0000", 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.291185+0000", 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8009999999999999e-06 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320257+0000", 2024-09-16T01:53:07.158 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029072132000000001 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320289+0000", 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1884999999999997e-05 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320305+0000", 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5457000000000001e-05 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.159 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320539+0000", 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00023418500000000001 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320705+0000", 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016638300000000001 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320857+0000", 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151693 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.160 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.364757+0000", 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.043900420000000002 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462213+0000", 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.097455080999999999 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462255+0000", 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2178999999999997e-05 2024-09-16T01:53:07.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462287+0000", 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2189e-05 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1125 3.3c 3:3cea676c:::100000001f8.00000062:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.832219262999999, 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15940902100000001, 2024-09-16T01:53:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1125 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.163 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320343+0000", 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.320345+0000", 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9e-06 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.164 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.351910+0000", 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031564927999999999 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352531+0000", 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00062034999999999998 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.165 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352612+0000", 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1075000000000006e-05 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.354881+0000", 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0022689120000000001 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355079+0000", 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000198462 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.166 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.355271+0000", 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00019189399999999999 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.451943+0000", 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096671963999999999 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479630+0000", 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027687125 2024-09-16T01:53:07.167 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479699+0000", 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9460000000000002e-05 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479752+0000", 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2951000000000002e-05 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.168 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1129 3.26 3:662203f5:::100000001f8.00000066:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.799867695000003, 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.127357531, 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1129 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.169 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352695+0000", 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.352697+0000", 2024-09-16T01:53:07.170 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.703e-06 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365574+0000", 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012876881 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365694+0000", 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000120679 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365756+0000", 2024-09-16T01:53:07.171 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1535e-05 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.365881+0000", 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012552900000000001 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366039+0000", 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015748699999999999 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.172 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.366200+0000", 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016154599999999999 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.462205+0000", 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.096004594999999998 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.479999+0000", 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.017794239999999999 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.173 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480028+0000", 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8839999999999998e-05 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480052+0000", 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4496000000000001e-05 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.174 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1131 3.16 3:699ec301:::100000001f8.00000068:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 87.783920429000005, 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11145630199999999, 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1131 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.175 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368642+0000", 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.368644+0000", 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.406e-06 2024-09-16T01:53:07.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391116+0000", 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022471944000000001 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391135+0000", 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9360000000000001e-05 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.177 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.391148+0000", 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.272e-05 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397341+0000", 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.006193507 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397495+0000", 2024-09-16T01:53:07.178 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000153502 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.397661+0000", 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000166692 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.469312+0000", 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.071650723999999999 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.179 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480059+0000", 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010747286999999999 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480081+0000", 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1909e-05 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.480099+0000", 2024-09-16T01:53:07.180 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7251000000000001e-05 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1927 3.8 3:13b44a54:::100000001f8.00000384:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.236845263999996, 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.29772541899999999, 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1927 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.182 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915717+0000", 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.915720+0000", 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.4719999999999998e-06 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949853+0000", 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034133234999999998 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.183 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949883+0000", 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9428999999999999e-05 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949895+0000", 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2744e-05 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949977+0000", 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1502000000000007e-05 2024-09-16T01:53:07.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.949989+0000", 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2075999999999999e-05 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157328+0000", 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.20733884699999999 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.185 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.157484+0000", 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155833 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.161682+0000", 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0041989319999999998 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.203765+0000", 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042082149999999999 2024-09-16T01:53:07.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213330+0000", 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0095658370000000006 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213376+0000", 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.5457000000000002e-05 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.187 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.213443+0000", 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6904999999999997e-05 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1187 2.14 2:2e2fa760:::200.00000001:head [write 231230~2088 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83)", 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.051915907999998, 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0832443309999999, 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.188 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100647+0000", 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100649+0000", 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3089999999999998e-06 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100662+0000", 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3005e-05 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100688+0000", 2024-09-16T01:53:07.190 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5667e-05 2024-09-16T01:53:07.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.191 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.191 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.191 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100700+0000", 2024-09-16T01:53:07.191 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2543e-05 2024-09-16T01:53:07.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100800+0000", 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9566000000000002e-05 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.100812+0000", 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2605e-05 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.192 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153150+0000", 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052337967999999999 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.153157+0000", 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7970000000000003e-06 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175044+0000", 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0218864809999999 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.193 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175147+0000", 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010305 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175296+0000", 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014963500000000001 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.178295+0000", 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029986309999999999 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.194 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183691+0000", 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.005395815 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183761+0000", 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9709999999999995e-05 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183891+0000", 2024-09-16T01:53:07.195 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013054900000000001 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1943 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.049378099999998, 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13049080800000001, 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.196 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1943 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103185+0000", 2024-09-16T01:53:07.197 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.103187+0000", 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180316+0000", 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077129006 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.198 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180328+0000", 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2279000000000001e-05 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180337+0000", 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.081e-06 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180434+0000", 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6534999999999996e-05 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.199 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180604+0000", 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017031899999999999 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.180781+0000", 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017693599999999999 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.230283+0000", 2024-09-16T01:53:07.200 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049502231000000001 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233526+0000", 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032425240000000001 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233622+0000", 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6452000000000004e-05 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.201 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.233675+0000", 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3285000000000002e-05 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:1188 2.14 2:2e2fa760:::200.00000001:head [write 233318~3585 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e84)", 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 77.439966867999999, 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.48008899599999999, 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.202 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712596+0000", 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.203 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712598+0000", 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2740000000000002e-06 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712611+0000", 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2867999999999999e-05 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.204 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712629+0000", 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8309999999999999e-05 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712641+0000", 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.205e-05 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712761+0000", 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011948899999999999 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.205 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.712775+0000", 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4525000000000001e-05 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.175954+0000", 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.46317823600000002 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176054+0000", 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000100176 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.206 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.176192+0000", 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013851099999999999 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.183650+0000", 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0074577810000000001 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192427+0000", 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087771619999999998 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.207 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192554+0000", 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012726600000000001 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:50.192685+0000", 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013034799999999999 2024-09-16T01:53:07.208 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3668 3.26 3:675724e2:::100000001f8.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.401710432999998, 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16146329400000001, 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.209 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3668 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750852+0000", 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.210 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.750854+0000", 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5230000000000001e-06 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782916+0000", 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032062042999999998 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782953+0000", 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7636999999999999e-05 2024-09-16T01:53:07.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.782968+0000", 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4448e-05 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783124+0000", 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000155642 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783223+0000", 2024-09-16T01:53:07.212 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9734000000000004e-05 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.783371+0000", 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014744399999999999 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.796252+0000", 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012880901 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.213 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912197+0000", 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11594560199999999 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912263+0000", 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6056999999999995e-05 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.912316+0000", 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2262999999999998e-05 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3676 3.8 3:1143f9f1:::100000001f8.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 51.326638697999996, 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.16740333199999999, 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.215 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3676 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825924+0000", 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.216 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.825926+0000", 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.204e-06 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886384+0000", 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060457377 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886403+0000", 2024-09-16T01:53:07.217 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9525e-05 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.886415+0000", 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1539000000000001e-05 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895207+0000", 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.008792668 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.218 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895316+0000", 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010839400000000001 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.895476+0000", 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016022700000000001 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.952757+0000", 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.057281350000000002 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993194+0000", 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040437039000000001 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993266+0000", 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1549000000000002e-05 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.993327+0000", 2024-09-16T01:53:07.220 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.1459999999999998e-05 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4142 3.16 3:6ba609e9:::100000001f8.0000022b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 45.365542144000003, 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.108516207, 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.221 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4142 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.222 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787021+0000", 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.787022+0000", 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.61e-06 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842519+0000", 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055496399000000002 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.223 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842539+0000", 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0820000000000001e-05 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842550+0000", 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0414999999999999e-05 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842641+0000", 2024-09-16T01:53:07.224 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1657999999999996e-05 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842742+0000", 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010075 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.842888+0000", 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014623699999999999 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.892001+0000", 2024-09-16T01:53:07.225 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.049112211000000003 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895434+0000", 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0034334869999999998 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895499+0000", 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5313000000000003e-05 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.226 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:21.895537+0000", 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.7307e-05 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4246 3.34 3:2f44359e:::100000001f8.00000293:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.822929582, 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.142262732, 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.227 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4246 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329633+0000", 2024-09-16T01:53:07.228 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.329635+0000", 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9180000000000001e-06 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410104+0000", 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.080469499 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.229 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410128+0000", 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3243000000000001e-05 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410141+0000", 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3067999999999999e-05 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419779+0000", 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0096380589999999992 2024-09-16T01:53:07.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.419889+0000", 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000109685 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.420057+0000", 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016837699999999999 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.456784+0000", 2024-09-16T01:53:07.231 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036727237000000003 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471769+0000", 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.014985112 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471839+0000", 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9275000000000003e-05 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.232 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.471896+0000", 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7259000000000002e-05 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4247 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e100)", 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 43.742374474999998, 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120085209, 2024-09-16T01:53:07.233 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4247 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.234 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410188+0000", 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.410190+0000", 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4500000000000001e-06 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451530+0000", 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041340783999999998 2024-09-16T01:53:07.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451572+0000", 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1690000000000002e-05 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.451620+0000", 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8180999999999998e-05 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.236 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452310+0000", 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00068996900000000002 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452413+0000", 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00010248599999999999 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.452588+0000", 2024-09-16T01:53:07.237 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017574800000000001 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.518054+0000", 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065465819999999994 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530168+0000", 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.012113413999999999 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.238 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530225+0000", 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.7284999999999998e-05 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:23.530273+0000", 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8381999999999999e-05 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.239 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4503 3.16 3:6b1b08a2:::100000001f8.00000394:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.640819120000003, 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13319203700000001, 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4503 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.240 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511744+0000", 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.511746+0000", 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0250000000000001e-06 2024-09-16T01:53:07.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576368+0000", 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064622151000000003 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576386+0000", 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7878999999999999e-05 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.242 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.576398+0000", 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2054e-05 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578343+0000", 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001945397 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578445+0000", 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000102001 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.243 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3", 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.578601+0000", 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015608399999999999 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.643878+0000", 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.065277247999999996 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644839+0000", 2024-09-16T01:53:07.244 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00096038899999999997 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644895+0000", 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6484000000000003e-05 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.644936+0000", 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0324999999999998e-05 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.245 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:4504 3.f e101/94 3:f3dac423:::100000001f8.00000395:head v 101'76, mlcod=100'74)", 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 40.557893002999997, 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14827673899999999, 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.246 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594670+0000", 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.594672+0000", 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3309999999999998e-06 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.247 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629004+0000", 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034332342000000002 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629020+0000", 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6121e-05 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.629029+0000", 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7479999999999999e-06 2024-09-16T01:53:07.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635523+0000", 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0064935909999999999 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.635549+0000", 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5808000000000001e-05 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742856+0000", 2024-09-16T01:53:07.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.107307556 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:26.742946+0000", 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0241999999999994e-05 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:4549 3.2 3:428d9500:::100000001f8.000003c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e101)", 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:53:07.250 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 39.982929556000002, 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11639424900000001, 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 4549 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:53:07.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169633+0000", 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.169636+0000", 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.8660000000000002e-06 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212280+0000", 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042644350999999997 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212312+0000", 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2088999999999997e-05 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212327+0000", 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4521e-05 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212426+0000", 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.9245000000000003e-05 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212539+0000", 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000112711 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.212725+0000", 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018583400000000001 2024-09-16T01:53:07.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285881+0000", 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073156321999999996 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285914+0000", 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2929999999999998e-05 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.285979+0000", 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.5048000000000005e-05 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:27.286027+0000", 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.8331999999999997e-05 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5790 3.16 3:687a9a46:::100000001f8.0000089b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.941894420000001, 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.65913515700000003, 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5790 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210668+0000", 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.210670+0000", 2024-09-16T01:53:07.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8500000000000001e-06 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244952+0000", 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034282188999999998 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244970+0000", 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8179999999999999e-05 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.244982+0000", 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1736000000000001e-05 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245078+0000", 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5439000000000006e-05 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.245090+0000", 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2177e-05 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791600+0000", 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.546509829 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791770+0000", 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017051699999999999 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.791950+0000", 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018029299999999999 2024-09-16T01:53:07.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829483+0000", 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037532101999999998 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867462+0000", 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037979295000000003 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869661+0000", 2024-09-16T01:53:07.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0021987059999999999 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869702+0000", 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.1148e-05 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869803+0000", 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101696 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5793 3.16 3:68c3c174:::100000001f8.0000089e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 24.881183799999999, 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.59860013000000001, 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5793 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.264 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271379+0000", 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.271381+0000", 2024-09-16T01:53:07.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0669999999999999e-06 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311762+0000", 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040380644 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311785+0000", 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3090000000000001e-05 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.266 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311798+0000", 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3057999999999999e-05 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311911+0000", 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000113429 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.311923+0000", 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1915e-05 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.267 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829575+0000", 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.51765227000000003 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829754+0000", 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017901700000000001 2024-09-16T01:53:07.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 3,5", 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.829935+0000", 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018041800000000001 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.867483+0000", 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037548040999999997 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869178+0000", 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0016955379999999999 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869843+0000", 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000664345 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869882+0000", 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8976000000000001e-05 2024-09-16T01:53:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.869979+0000", 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7322000000000001e-05 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5945 3.2b 3:d7b2ba59:::100000001f8.00000936:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.755043334, 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11205129899999999, 2024-09-16T01:53:07.271 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5945 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397519+0000", 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.397521+0000", 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429779+0000", 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032257503999999999 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429805+0000", 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6441999999999999e-05 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429818+0000", 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2687999999999999e-05 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.429896+0000", 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.7898999999999994e-05 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430008+0000", 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011241100000000001 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6", 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430198+0000", 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000189197 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.448700+0000", 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018502761999999999 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509440+0000", 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.060739425999999999 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509514+0000", 2024-09-16T01:53:07.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4673999999999997e-05 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.509571+0000", 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.6249999999999998e-05 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5948 3.2d 3:b677c6b2:::100000001f8.00000939:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:53:07.277 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 22.722348192999998, 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.122297751, 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5948 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.278 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430214+0000", 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.430216+0000", 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.53e-06 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453110+0000", 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022894134 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453131+0000", 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1120000000000001e-05 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.453141+0000", 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.5340000000000005e-06 2024-09-16T01:53:07.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495855+0000", 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042714162 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for rw locks", 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.495997+0000", 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014160999999999999 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537228+0000", 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.041231297 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537360+0000", 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013264400000000001 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 6,7", 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.537537+0000", 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017664800000000001 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548049+0000", 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010511915 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.548065+0000", 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5812e-05 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552281+0000", 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0042165600000000003 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552355+0000", 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.3712000000000002e-05 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.552512+0000", 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015707300000000001 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.284 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:07.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:53:07.286 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:07.284+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:07.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.380+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f19200337b0 con 0x7f19100039b0 2024-09-16T01:53:07.387 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:07.386+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:07.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.435+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f31e4070190 con 0x7f31e41528e0 2024-09-16T01:53:07.489 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:07.487+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:07.589 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:07.588+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:3458 3.16 e126/111 3:68c3c174:::100000001f8.0000089e:head v 126'222, mlcod=126'219)", 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.598620+0000", 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0094642509999999999, 2024-09-16T01:53:07.611 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0094871710000000008, 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598620+0000", 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598620+0000", 2024-09-16T01:53:07.612 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598621+0000", 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5000000000000001e-07 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598627+0000", 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5500000000000002e-06 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.613 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598637+0000", 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0334e-05 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598644+0000", 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6669999999999999e-06 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598719+0000", 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5731999999999999e-05 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.614 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598738+0000", 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8935000000000001e-05 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3460 3.1b 3:d849a4fe:::100000001f8.000008a0:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e126)", 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.598762+0000", 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0093223869999999997, 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.009415596, 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598762+0000", 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598762+0000", 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.616 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598763+0000", 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2750000000000001e-06 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598768+0000", 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2930000000000002e-06 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598778+0000", 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0101999999999999e-05 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.617 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598785+0000", 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9580000000000001e-06 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598836+0000", 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1598999999999998e-05 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.598943+0000", 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000107039 2024-09-16T01:53:07.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.599034+0000", 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.1099000000000003e-05 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.600319+0000", 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0012847830000000001 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.619 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:3462 3.37 e126/20 3:ef3b6139:::100000001f8.000008a2:head v 126'149, mlcod=126'147)", 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.601002+0000", 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0070826780000000002, 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0072502670000000003, 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601002+0000", 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601002+0000", 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601003+0000", 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5899999999999995e-07 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601009+0000", 2024-09-16T01:53:07.621 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.037e-06 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601020+0000", 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1508000000000001e-05 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601027+0000", 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.7989999999999996e-06 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.622 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601086+0000", 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8971000000000002e-05 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601104+0000", 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8448999999999999e-05 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3467 3.1d 3:bb8d9892:::100000001f8.000008a7:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e126)", 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.601412+0000", 2024-09-16T01:53:07.623 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0066724899999999997, 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0069010640000000002, 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601412+0000", 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.624 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601412+0000", 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601413+0000", 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2999999999999999e-07 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601416+0000", 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.551e-06 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.625 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601786+0000", 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00036938800000000001 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601796+0000", 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0618000000000001e-05 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601867+0000", 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0326000000000004e-05 2024-09-16T01:53:07.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.601960+0000", 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.3295999999999996e-05 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.602050+0000", 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.9432999999999996e-05 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.627 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.608105+0000", 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0060552490000000004 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(mds.0.4:3471 3.28 3:15961fc2:::100000001f8.000008ab:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e126)", 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.603293+0000", 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.004791582, 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050966199999999996, 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "waiting for sub ops", 2024-09-16T01:53:07.628 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603293+0000", 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603293+0000", 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603294+0000", 2024-09-16T01:53:07.629 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1480000000000001e-06 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603299+0000", 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1039999999999998e-06 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603311+0000", 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1956e-05 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.630 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603319+0000", 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5889999999999997e-06 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.604185+0000", 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00086619799999999999 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.604264+0000", 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.9223999999999999e-05 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.631 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.604361+0000", 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6996000000000004e-05 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.607590+0000", 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0032289850000000002 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.632 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:3470 3.38 e126/94 3:1db5b4a6:::100000001f8.000008aa:head v 126'172, mlcod=126'170)", 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.603366+0000", 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0047188439999999998, 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050984760000000002, 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603366+0000", 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.633 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603366+0000", 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603366+0000", 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.3e-07 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603371+0000", 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7190000000000001e-06 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.634 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603379+0000", 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1470000000000001e-06 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603385+0000", 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8599999999999998e-06 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603942+0000", 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00055719799999999996 2024-09-16T01:53:07.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603960+0000", 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7677999999999999e-05 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(mds.0.4:3469 3.26 e126/20 3:675f1196:::100000001f8.000008a9:head v 126'183, mlcod=126'181)", 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:53:07.603428+0000", 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 0.0046562859999999999, 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0050935069999999997, 2024-09-16T01:53:07.636 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "started", 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603428+0000", 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603428+0000", 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.637 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603429+0000", 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9299999999999998e-07 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603433+0000", 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.8909999999999997e-06 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603440+0000", 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.6389999999999992e-06 2024-09-16T01:53:07.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603445+0000", 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2870000000000004e-06 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603575+0000", 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012955599999999999 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:53:07.603602+0000", 2024-09-16T01:53:07.639 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7246e-05 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout: "num_ops": 7 2024-09-16T01:53:07.640 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:07.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:53:07.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.661+0000 7f1925600640 1 -- 172.21.15.146:0/848794894 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v127) ==== 87+0+0 (secure 0 0 0) 0x7f191c0796f0 con 0x7f19100039b0 2024-09-16T01:53:07.662 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is set 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f190806c2e0 msgr2=0x7f190806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f190806c2e0 0x7f190806e7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f19200e9cc0 tx=0x7f1914008040 comp rx=0 tx=0).stop 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 msgr2=0x7f1920049520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 0x7f1920049520 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f191c009000 tx=0x7f1920034a10 comp rx=0 tx=0).stop 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 shutdown_connections 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f190806c2e0 0x7f190806e7a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19200db070 0x7f19200d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f19200c4520 0x7f19200ba420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 --2- 172.21.15.146:0/848794894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19100039b0 0x7f1920049520 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 >> 172.21.15.146:0/848794894 conn(0x7f192004fc50 msgr2=0x7f19200c8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:07.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 shutdown_connections 2024-09-16T01:53:07.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.665+0000 7f192e800640 1 -- 172.21.15.146:0/848794894 wait complete. 2024-09-16T01:53:07.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.664+0000 7f31de000640 1 -- 172.21.15.146:0/1222687780 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v127) ==== 113+0+0 (secure 0 0 0) 0x7f31d40045c0 con 0x7f31e41528e0 2024-09-16T01:53:07.668 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T01:53:07.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.671+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f31c806c2e0 msgr2=0x7f31c806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.671+0000 7f31ec800640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f31c806c2e0 0x7f31c806e7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f31d8000c00 tx=0x7f31d802e000 comp rx=0 tx=0).stop 2024-09-16T01:53:07.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.671+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e41528e0 msgr2=0x7f31e40c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:07.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.671+0000 7f31ec800640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e41528e0 0x7f31e40c7840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d5f0 tx=0x7f31d400dab0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.671+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 shutdown_connections 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f31c806c2e0 0x7f31c806e7a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f31e4152cc0 0x7f31e40b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f31e41528e0 0x7f31e40c7840 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d5f0 tx=0x7f31d400dab0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 --2- 172.21.15.146:0/1222687780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31e4070e00 0x7f31e40c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 >> 172.21.15.146:0/1222687780 conn(0x7f31e406c240 msgr2=0x7f31e4150100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:07.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 shutdown_connections 2024-09-16T01:53:07.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:07.672+0000 7f31ec800640 1 -- 172.21.15.146:0/1222687780 wait complete. 2024-09-16T01:53:07.692 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:07.690+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:07.704 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T01:53:07.794 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:07.793+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:07.893 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:07.892+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:07.936 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:07.936 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [], 2024-09-16T01:53:07.936 INFO:teuthology.orchestra.run.smithi146.stdout: "complaint_time": 180, 2024-09-16T01:53:07.936 INFO:teuthology.orchestra.run.smithi146.stdout: "num_blocked_ops": 0 2024-09-16T01:53:07.936 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:07.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:53:07.994 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:07.993+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:08.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.004+0000 7f6fe4000640 1 Processor -- start 2024-09-16T01:53:08.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.005+0000 7f6fe4000640 1 -- start start 2024-09-16T01:53:08.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe4000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 0x7f6fdc156940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:08.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe4000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6fdc06f590 0x7f6fdc0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:08.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe4000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fdc0cc910 con 0x7f6fdc070030 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe4000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6fdc0c9cf0 con 0x7f6fdc06f590 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe4000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6fdc0cd110 con 0x7f6fdc06fc50 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe1600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 0x7f6fdc156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe1600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 0x7f6fdc156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43248/0 (socket says 172.21.15.146:43248) 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.007+0000 7f6fe1600640 1 -- 172.21.15.146:0/2098364830 learned_addr learned my addr 172.21.15.146:0/2098364830 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.008+0000 7f6fe1600640 1 -- 172.21.15.146:0/2098364830 >> v2:172.21.15.161:3300/0 conn(0x7f6fdc06f590 msgr2=0x7f6fdc0cc1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.008+0000 7f6fe1600640 1 --2- 172.21.15.146:0/2098364830 >> v2:172.21.15.161:3300/0 conn(0x7f6fdc06f590 0x7f6fdc0cc1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.008+0000 7f6fe1600640 1 -- 172.21.15.146:0/2098364830 >> v1:172.21.15.161:6789/0 conn(0x7f6fdc070030 legacy=0x7f6fdc0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:08.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.008+0000 7f6fdbe00640 1 -- 172.21.15.146:0/2098364830 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3693887530 0 0) 0x7f6fdc0cc910 con 0x7f6fdc070030 2024-09-16T01:53:08.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.008+0000 7f6fe1600640 1 -- 172.21.15.146:0/2098364830 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc0ccbb0 con 0x7f6fdc06fc50 2024-09-16T01:53:08.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.009+0000 7f6fe1600640 1 --2- 172.21.15.146:0/2098364830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 0x7f6fdc156940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0ccd80 tx=0x7f6fd002e680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310d34716fa7f2ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:08.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.010+0000 7f6fdbe00640 1 -- 172.21.15.146:0/2098364830 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd003a070 con 0x7f6fdc06fc50 2024-09-16T01:53:08.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.010+0000 7f6fdbe00640 1 -- 172.21.15.146:0/2098364830 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd0033710 con 0x7f6fdc06fc50 2024-09-16T01:53:08.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.010+0000 7f6fdbe00640 1 -- 172.21.15.146:0/2098364830 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0033ae0 con 0x7f6fdc06fc50 2024-09-16T01:53:08.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 -- 172.21.15.146:0/2098364830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 msgr2=0x7f6fdc156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:08.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 --2- 172.21.15.146:0/2098364830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 0x7f6fdc156940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0ccd80 tx=0x7f6fd002e680 comp rx=0 tx=0).stop 2024-09-16T01:53:08.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 -- 172.21.15.146:0/2098364830 shutdown_connections 2024-09-16T01:53:08.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 --2- 172.21.15.146:0/2098364830 >> v2:172.21.15.161:3300/0 conn(0x7f6fdc06f590 0x7f6fdc0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 --2- 172.21.15.146:0/2098364830 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc06fc50 0x7f6fdc156940 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 -- 172.21.15.146:0/2098364830 >> 172.21.15.146:0/2098364830 conn(0x7f6fdc06e670 msgr2=0x7f6fdc06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:08.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.011+0000 7f6fe4000640 1 -- 172.21.15.146:0/2098364830 shutdown_connections 2024-09-16T01:53:08.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.012+0000 7f6fe4000640 1 -- 172.21.15.146:0/2098364830 wait complete. 2024-09-16T01:53:08.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.014+0000 7f6fe4000640 1 Processor -- start 2024-09-16T01:53:08.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.014+0000 7f6fe4000640 1 -- start start 2024-09-16T01:53:08.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.015+0000 7f6fe4000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fdc06f590 0x7f6fdc143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:08.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.015+0000 7f6fe4000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fdc06fc50 0x7f6fdc1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:08.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.016+0000 7f6fe4000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc070030 0x7f6fdc143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:08.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.016+0000 7f6fe4000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc0c7470 con 0x7f6fdc06f590 2024-09-16T01:53:08.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.016+0000 7f6fe4000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc0c7190 con 0x7f6fdc06fc50 2024-09-16T01:53:08.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.016+0000 7f6fe4000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6fdc0719a0 con 0x7f6fdc070030 2024-09-16T01:53:08.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.016+0000 7f6fe1600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fdc06f590 0x7f6fdc143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:08.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.016+0000 7f6fe2000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc070030 0x7f6fdc143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:08.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe2000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc070030 0x7f6fdc143850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43260/0 (socket says 172.21.15.146:43260) 2024-09-16T01:53:08.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe2000640 1 -- 172.21.15.146:0/4029541785 learned_addr learned my addr 172.21.15.146:0/4029541785 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:08.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe0c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fdc06fc50 0x7f6fdc1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:08.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe0c00640 1 -- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc070030 msgr2=0x7f6fdc143850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:08.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe0c00640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc070030 0x7f6fdc143850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe0c00640 1 -- 172.21.15.146:0/4029541785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fdc06f590 msgr2=0x7f6fdc143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:08.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe0c00640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fdc06f590 0x7f6fdc143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.017+0000 7f6fe0c00640 1 -- 172.21.15.146:0/4029541785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc14bc10 con 0x7f6fdc06fc50 2024-09-16T01:53:08.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.018+0000 7f6fe0c00640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fdc06fc50 0x7f6fdc1448d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00c470 tx=0x7f6fcc00c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:08.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.019+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc017070 con 0x7f6fdc06fc50 2024-09-16T01:53:08.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.019+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0035f0 con 0x7f6fdc06fc50 2024-09-16T01:53:08.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.019+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0039e0 con 0x7f6fdc06fc50 2024-09-16T01:53:08.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.019+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fdc14be80 con 0x7f6fdc06fc50 2024-09-16T01:53:08.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.019+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fdc14ff20 con 0x7f6fdc06fc50 2024-09-16T01:53:08.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.024+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6fcc01a070 con 0x7f6fdc06fc50 2024-09-16T01:53:08.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.023+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fdc156940 con 0x7f6fdc06fc50 2024-09-16T01:53:08.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.030+0000 7f6fd9600640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6fc006c330 0x7f6fc006e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:08.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.030+0000 7f6fe1600640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6fc006c330 0x7f6fc006e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:08.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.030+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5894+0+0 (secure 0 0 0) 0x7f6fcc08a7b0 con 0x7f6fdc06fc50 2024-09-16T01:53:08.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.032+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6fcc058830 con 0x7f6fdc06fc50 2024-09-16T01:53:08.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.033+0000 7f6fe1600640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6fc006c330 0x7f6fc006e7f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0037000 tx=0x7f6fd0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:08.096 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:08.095+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:08.199 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:08.195+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:08.247 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "size": 20, 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 600, 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "ops": [ 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:462 3.13 3:cbfdaa68:::10000000002.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e58)", 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 128.396153062, 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0916774579999999, 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.248 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 462 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.249 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848737+0000", 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.848740+0000", 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6579999999999998e-06 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.250 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879528+0000", 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.030788236 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879554+0000", 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.5902e-05 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879568+0000", 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3845999999999999e-05 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.251 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879962+0000", 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039379399999999999 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.879979+0000", 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7303e-05 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.835312+0000", 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.95533302399999998 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.252 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837732+0000", 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.002420054 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905687+0000", 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.067954951999999999 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905802+0000", 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000115171 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.905943+0000", 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000140488 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940213+0000", 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.034269885999999999 2024-09-16T01:53:08.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940235+0000", 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2586e-05 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940296+0000", 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0809e-05 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.255 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940415+0000", 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000118749 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:475 3.13 3:c846732e:::10000000002.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 128.25369339700001, 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.94942599699999997, 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.256 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 475 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991197+0000", 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:50:59.991199+0000", 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.441e-06 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.258 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015186+0000", 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023986345999999999 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015209+0000", 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.3635999999999999e-05 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.259 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015220+0000", 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0223e-05 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015294+0000", 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4104999999999996e-05 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.015303+0000", 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.4450000000000002e-06 2024-09-16T01:53:08.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.837892+0000", 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.82258901900000003 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841217+0000", 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0033253789999999998 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.261 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915087+0000", 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073869378999999999 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915148+0000", 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0730999999999998e-05 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.915239+0000", 2024-09-16T01:53:08.262 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.0952000000000006e-05 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940473+0000", 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.025234027999999999 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940485+0000", 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2469999999999999e-05 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.263 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.264 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940526+0000", 2024-09-16T01:53:08.264 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.0639000000000002e-05 2024-09-16T01:53:08.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.264 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.264 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.265 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940623+0000", 2024-09-16T01:53:08.265 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7204e-05 2024-09-16T01:53:08.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.265 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.265 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:509 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 127.90622717399999, 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.60322948300000001, 2024-09-16T01:53:08.266 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 509 2024-09-16T01:53:08.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.268 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.269 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338663+0000", 2024-09-16T01:53:08.269 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.269 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.269 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.338665+0000", 2024-09-16T01:53:08.270 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2419999999999999e-06 2024-09-16T01:53:08.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.270 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.270 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.270 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361724+0000", 2024-09-16T01:53:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023058729 2024-09-16T01:53:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361758+0000", 2024-09-16T01:53:08.271 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.417e-05 2024-09-16T01:53:08.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.272 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.272 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.272 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361772+0000", 2024-09-16T01:53:08.272 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3642000000000001e-05 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361888+0000", 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000116335 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.361902+0000", 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3319000000000001e-05 2024-09-16T01:53:08.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.841276+0000", 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.47937443800000001 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.844376+0000", 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.003100481 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.274 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924129+0000", 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.079752479000000001 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924231+0000", 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101554 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.924391+0000", 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016029600000000001 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.275 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.940892+0000", 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.016500921000000002 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941732+0000", 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00084027400000000001 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.276 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941784+0000", 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1901999999999999e-05 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:00.941893+0000", 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000108701 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:785 3.11 3:8b60708a:::100000001f7.0000000e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:53:08.277 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 108.02193581, 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.114353328, 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 785 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.278 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222954+0000", 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.222958+0000", 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.14e-06 2024-09-16T01:53:08.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252288+0000", 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.029330760000000001 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252308+0000", 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9215000000000002e-05 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.280 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252319+0000", 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.1759999999999999e-05 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252678+0000", 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000359062 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252775+0000", 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7050000000000001e-05 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.281 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.252922+0000", 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014696800000000001 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.263875+0000", 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.010952170000000001 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337207+0000", 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.073332283999999998 2024-09-16T01:53:08.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337263+0000", 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.5995999999999997e-05 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:20.337308+0000", 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.4922999999999997e-05 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1119 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.984196307000005, 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.126297087, 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.284 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1119 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260694+0000", 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.285 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.260696+0000", 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308746+0000", 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.048049847999999999 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308789+0000", 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.3050000000000003e-05 2024-09-16T01:53:08.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308805+0000", 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.5554e-05 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.308978+0000", 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017332799999999999 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309135+0000", 2024-09-16T01:53:08.287 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000156785 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.309293+0000", 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015860099999999999 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.336760+0000", 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027466572000000002 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.288 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386874+0000", 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.050114604 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386941+0000", 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6650999999999996e-05 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:39.386991+0000", 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.9935999999999998e-05 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1209 3.2f 3:f6d139a3:::100000001f8.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e77)", 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 88.125209307000006, 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.117220668, 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1209 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119681+0000", 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.119684+0000", 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.6089999999999999e-06 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.173846+0000", 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054162836999999998 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174719+0000", 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00087305499999999997 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174737+0000", 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7263000000000001e-05 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174829+0000", 2024-09-16T01:53:08.293 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.2656000000000002e-05 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.174990+0000", 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016088800000000001 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.175153+0000", 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016233600000000001 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.294 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.193843+0000", 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018690663 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236802+0000", 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.042958238000000003 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236862+0000", 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0430000000000002e-05 2024-09-16T01:53:08.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:40.236902+0000", 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.9693000000000001e-05 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1936 3.2d 3:b52f6332:::100000001f8.0000038d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.258170332999995, 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.250911512, 2024-09-16T01:53:08.296 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1936 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.297 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986720+0000", 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:48.986723+0000", 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.666e-06 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.298 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039916+0000", 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.053193120000000003 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039943+0000", 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7348e-05 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.299 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.039963+0000", 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9432999999999998e-05 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040067+0000", 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000104307 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for peered", 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.040081+0000", 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4433000000000001e-05 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.300 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178028+0000", 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13794648400000001 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178166+0000", 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013807000000000001 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.178319+0000", 2024-09-16T01:53:08.301 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015350499999999999 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.204917+0000", 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.026597619999999999 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237429+0000", 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032512015999999998 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.302 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237503+0000", 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4534000000000002e-05 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.237631+0000", 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000127976 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.303 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1949 3.2f 3:f7d79abc:::100000001f8.0000039a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e83)", 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.062218111999996, 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15948922400000001, 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1949 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.304 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182672+0000", 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.305 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.182674+0000", 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1509999999999998e-06 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223601+0000", 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040926934999999998 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223624+0000", 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2493e-05 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.306 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223636+0000", 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2347999999999999e-05 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223742+0000", 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000105719 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223904+0000", 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016176 2024-09-16T01:53:08.307 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.224066+0000", 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016200700000000001 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.242364+0000", 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.018298740000000001 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.308 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342038+0000", 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.099673592000000005 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342109+0000", 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.1352000000000004e-05 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.309 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.342161+0000", 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.2126999999999997e-05 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1951 3.a 3:53c75786:::100000001f8.0000039c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 79.021216168999999, 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13168421899999999, 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.310 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1951 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.311 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223674+0000", 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.223676+0000", 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.9309999999999998e-06 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268690+0000", 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045013927000000002 2024-09-16T01:53:08.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268720+0000", 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.0201000000000001e-05 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268733+0000", 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.2349e-05 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.313 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268807+0000", 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.4084000000000004e-05 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.268945+0000", 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013880699999999999 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.269091+0000", 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000145764 2024-09-16T01:53:08.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.309669+0000", 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040578261999999997 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355243+0000", 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045574029000000002 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355310+0000", 2024-09-16T01:53:08.315 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.6945000000000004e-05 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.355358+0000", 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.7920000000000002e-05 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:1965 3.1d 3:b8ffeb6c:::100000001f8.000003aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e84)", 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:53:08.316 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 78.796988608999996, 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12030484499999999, 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 1965 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:53:08.317 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447902+0000", 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.447905+0000", 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.2440000000000002e-06 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.318 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500227+0000", 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.052322472000000002 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500250+0000", 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2874000000000002e-05 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500265+0000", 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4345000000000001e-05 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.319 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500398+0000", 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00013300900000000001 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500552+0000", 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00015409399999999999 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.320 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.500732+0000", 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180648 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.534550+0000", 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033817699 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.321 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.322 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.567980+0000", 2024-09-16T01:53:08.322 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.033430350999999997 2024-09-16T01:53:08.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.322 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.322 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568123+0000", 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014263899999999999 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:49.568207+0000", 2024-09-16T01:53:08.323 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.3469999999999999e-05 2024-09-16T01:53:08.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.324 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.324 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2441 3.11 3:89e800ba:::100000001f8.00000586:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e88)", 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 72.840207055999997, 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12536577300000001, 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.325 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2441 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.326 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:53:08.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.327 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.327 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.327 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.327 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404683+0000", 2024-09-16T01:53:08.327 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.328 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.328 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.328 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.404686+0000", 2024-09-16T01:53:08.328 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.9229999999999998e-06 2024-09-16T01:53:08.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.329 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.329 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441845+0000", 2024-09-16T01:53:08.329 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.037158693999999999 2024-09-16T01:53:08.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.329 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441881+0000", 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.6038999999999998e-05 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.441901+0000", 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.0358000000000001e-05 2024-09-16T01:53:08.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442016+0000", 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000114287 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442199+0000", 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018299899999999999 2024-09-16T01:53:08.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.442384+0000", 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00018498500000000001 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.497778+0000", 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.055394577 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529903+0000", 2024-09-16T01:53:08.332 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.032124728999999998 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.529999+0000", 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.6075000000000004e-05 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.333 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:51:55.530049+0000", 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0107e-05 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:2991 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e90)", 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 66.929226706999998, 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.109327587, 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.334 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 2991 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.335 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315664+0000", 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.315665+0000", 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.832e-06 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361209+0000", 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.045543114000000003 2024-09-16T01:53:08.336 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361240+0000", 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 3.1368999999999997e-05 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361255+0000", 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4664000000000001e-05 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.337 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361361+0000", 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000106115 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361528+0000", 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00016695600000000001 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.361708+0000", 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000180242 2024-09-16T01:53:08.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.384610+0000", 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022901673000000001 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424876+0000", 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.040266587 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.339 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424941+0000", 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.4836000000000001e-05 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:01.424991+0000", 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0198999999999998e-05 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.340 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3669 3.d 3:b351a1ef:::100000001f8.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.492695724999997, 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.15641955299999999, 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.341 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3669 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752195+0000", 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.752197+0000", 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.356e-06 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.788413+0000", 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.036216336000000002 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.343 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815793+0000", 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027379624000000002 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.815914+0000", 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00012110199999999999 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816162+0000", 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00024784999999999998 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.344 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816277+0000", 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00011471100000000001 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816468+0000", 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000191874 2024-09-16T01:53:08.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.899681+0000", 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.083212251000000001 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908472+0000", 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0087916060000000004 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.346 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908533+0000", 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.0698000000000001e-05 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908614+0000", 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.1144999999999996e-05 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3667 3.17 e97/20 3:eab09b46:::100000001f8.00000050:head v 97'68, mlcod=93'66)", 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.469541806999999, 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12887499999999999, 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:53:08.348 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775348+0000", 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.775351+0000", 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1600000000000001e-06 2024-09-16T01:53:08.349 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779273+0000", 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0039221209999999998 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779348+0000", 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.5059000000000001e-05 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.779362+0000", 2024-09-16T01:53:08.350 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.4640999999999999e-05 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780655+0000", 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001292446 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.780677+0000", 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2226e-05 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.351 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.902644+0000", 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12196677 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.904223+0000", 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001579577 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.352 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_repop(client.4619.0:3668 3.26 e97/20 3:675724e2:::100000001f8.00000051:head v 97'95, mlcod=93'93)", 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.458605964, 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12195737700000001, 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.353 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.356 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.356 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.356 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786284+0000", 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.786285+0000", 2024-09-16T01:53:08.357 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 7.0299999999999998e-07 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.787819+0000", 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.001534466 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816616+0000", 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.028796068000000001 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.358 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816628+0000", 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.239e-05 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830137+0000", 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.013508902 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.359 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.830164+0000", 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.7559000000000002e-05 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908145+0000", 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.077980271000000004 2024-09-16T01:53:08.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.908242+0000", 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.7017999999999998e-05 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3672 3.22 3:44862f3f:::100000001f8.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:53:08.361 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.428411539999999, 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.12721389, 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3672 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.362 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816479+0000", 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.816481+0000", 2024-09-16T01:53:08.363 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.2400000000000002e-06 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837694+0000", 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.021212699000000002 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837715+0000", 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1742999999999998e-05 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.364 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.837729+0000", 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.3346999999999999e-05 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840699+0000", 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0029702499999999998 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840801+0000", 2024-09-16T01:53:08.365 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0001021 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 5", 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.840947+0000", 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00014604200000000001 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.905043+0000", 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.064096236000000001 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943551+0000", 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.038507358999999998 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943639+0000", 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.7917999999999994e-05 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.367 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.943693+0000", 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3956000000000003e-05 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:3679 3.14 3:2a097b4c:::100000001f8.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e97)", 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 52.387006087000003, 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.11483824099999999, 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.368 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 3679 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.369 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857884+0000", 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.370 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.857886+0000", 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.1579999999999999e-06 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880333+0000", 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.022446666000000001 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880393+0000", 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9932000000000001e-05 2024-09-16T01:53:08.371 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880409+0000", 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6348e-05 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880805+0000", 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00039615400000000002 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.372 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.880929+0000", 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123292 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.881100+0000", 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00017153900000000001 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.963633+0000", 2024-09-16T01:53:08.373 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.082532787999999996 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972606+0000", 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0089728740000000005 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972669+0000", 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.2691e-05 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.374 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:15.972722+0000", 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.3798999999999998e-05 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5551 3.2f 3:f7814f15:::100000001f8.000007ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e105)", 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 29.406198997000001, 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.13476244500000001, 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5551 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838691+0000", 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.838694+0000", 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.306e-06 2024-09-16T01:53:08.377 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866535+0000", 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.027841175999999999 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866552+0000", 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.7387000000000001e-05 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.378 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866563+0000", 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0447e-05 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866665+0000", 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000101894 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866788+0000", 2024-09-16T01:53:08.379 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000123194 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.866953+0000", 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000165365 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.918808+0000", 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.051855367999999999 2024-09-16T01:53:08.380 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973333+0000", 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.054524250000000003 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973402+0000", 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 6.9648999999999997e-05 2024-09-16T01:53:08.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:38.973454+0000", 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.1409000000000002e-05 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5812 3.a 3:51094797:::100000001f8.000008b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e108)", 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 25.712499881999999, 2024-09-16T01:53:08.382 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.14779089000000001, 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5812 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532390+0000", 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.532392+0000", 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.8679999999999999e-06 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.384 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653045+0000", 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.120652599 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653062+0000", 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6929e-05 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.653073+0000", 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.0925e-05 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.385 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656653+0000", 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.0035800049999999998 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656738+0000", 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 8.5599000000000005e-05 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.656890+0000", 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000151874 2024-09-16T01:53:08.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.676041+0000", 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.019150358999999999 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680071+0000", 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.004030624 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.387 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680131+0000", 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.9704000000000003e-05 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:42.680181+0000", 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.0404000000000001e-05 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "description": "osd_op(client.4619.0:5944 3.30 3:0c844521:::100000001f8.00000935:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e110)", 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "initiated_at": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "age": 23.857670737999999, 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.115163742, 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "continuous": false, 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "type_data": { 2024-09-16T01:53:08.389 INFO:teuthology.orchestra.run.smithi146.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_info": { 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client": "client.4619", 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: "tid": 5944 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: "events": [ 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.390 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "initiated", 2024-09-16T01:53:08.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:53:08.391 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.391 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.391 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "header_read", 2024-09-16T01:53:08.391 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387220+0000", 2024-09-16T01:53:08.392 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0 2024-09-16T01:53:08.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.392 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "throttled", 2024-09-16T01:53:08.392 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.387222+0000", 2024-09-16T01:53:08.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 2.046e-06 2024-09-16T01:53:08.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.393 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "all_read", 2024-09-16T01:53:08.393 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418222+0000", 2024-09-16T01:53:08.393 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.031000603000000002 2024-09-16T01:53:08.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.394 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.394 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "dispatched", 2024-09-16T01:53:08.394 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418239+0000", 2024-09-16T01:53:08.394 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 1.6833e-05 2024-09-16T01:53:08.394 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418249+0000", 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8179999999999999e-06 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "reached_pg", 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418846+0000", 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.00059710099999999995 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "started", 2024-09-16T01:53:08.395 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.418945+0000", 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 9.8826000000000006e-05 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.419093+0000", 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.000148553 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "op_commit", 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.442472+0000", 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.023378423999999998 2024-09-16T01:53:08.396 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502282+0000", 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 0.059810025000000003 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "commit_sent", 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502341+0000", 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 5.8922000000000003e-05 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-09-16T01:53:08.397 INFO:teuthology.orchestra.run.smithi146.stdout: "event": "done", 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: "time": "2024-09-16T01:52:44.502383+0000", 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: "duration": 4.2590999999999999e-05 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-09-16T01:53:08.398 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:08.400 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:53:08.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.310+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f6fdc149930 con 0x7f6fdc06fc50 2024-09-16T01:53:08.401 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-09-16T01:53:08.399+0000 7f3e2bbf6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 37300) UID: 0 2024-09-16T01:53:08.500 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:08.499+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:08.508 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:08.508 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:08.508 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:53:08.508 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:08.517 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:53:08.601 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:08.600+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:08.622 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:08.622 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:08.622 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:53:08.622 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:53:08.622 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:08.631 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:53:08.702 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:08.701+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:08.737 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:08.737 INFO:teuthology.orchestra.run.smithi161.stdout: "size": 20, 2024-09-16T01:53:08.737 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 600, 2024-09-16T01:53:08.737 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [ 2024-09-16T01:53:08.737 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:69 3.8 3:133e839e:::10000000002.00000042:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 137.87905874399999, 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.064976114000000001, 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 69 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.738 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855814+0000", 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.739 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.855815+0000", 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.204e-06 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916789+0000", 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060973912999999998 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916804+0000", 2024-09-16T01:53:08.740 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.5099999999999999e-05 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916816+0000", 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1885999999999999e-05 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916873+0000", 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7438000000000001e-05 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.741 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.916972+0000", 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9261999999999999e-05 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.917061+0000", 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9091000000000005e-05 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.742 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920127+0000", 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0030660029999999999 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920718+0000", 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00059076899999999999 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920752+0000", 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4201e-05 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.743 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:50.920790+0000", 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7246999999999998e-05 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:161 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 136.967480432, 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.071658601000000002, 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.744 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 161 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.745 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767392+0000", 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.767394+0000", 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8610000000000001e-06 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.746 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811603+0000", 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044209713999999997 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811617+0000", 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3772e-05 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811630+0000", 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.3006e-05 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.747 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811681+0000", 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0819999999999998e-05 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811774+0000", 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3012999999999999e-05 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811874+0000", 2024-09-16T01:53:08.748 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9877000000000006e-05 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.814279+0000", 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.002405464 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.838977+0000", 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.024697075999999998 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.749 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839016+0000", 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9721999999999997e-05 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.839051+0000", 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4276000000000002e-05 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:168 3.2b 3:d5eb8d7b:::10000000002.000000a5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 136.92322020200001, 2024-09-16T01:53:08.750 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065178746999999995, 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 168 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.751 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811652+0000", 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.811653+0000", 2024-09-16T01:53:08.752 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0580000000000001e-06 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841171+0000", 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.029518025 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841190+0000", 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9066999999999999e-05 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.753 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841200+0000", 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6439999999999995e-06 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841268+0000", 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.8138000000000004e-05 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841371+0000", 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102678 2024-09-16T01:53:08.754 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.841470+0000", 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9642000000000006e-05 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.845334+0000", 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0038635990000000001 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.755 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876712+0000", 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.031378022999999998 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876775+0000", 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2583999999999996e-05 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.876831+0000", 2024-09-16T01:53:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.6289000000000003e-05 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:179 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e55)", 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 136.85116486000001, 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059554273999999997, 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.757 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 179 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.758 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883707+0000", 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.883710+0000", 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.6290000000000002e-06 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932796+0000", 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.049085877 2024-09-16T01:53:08.759 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932817+0000", 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1013999999999999e-05 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932829+0000", 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1569999999999999e-05 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.760 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.932927+0000", 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8270000000000006e-05 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933085+0000", 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000158639 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.933213+0000", 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012751099999999999 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.761 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.936590+0000", 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033772089999999999 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943181+0000", 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0065910200000000004 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943221+0000", 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.9842000000000002e-05 2024-09-16T01:53:08.762 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:50:51.943262+0000", 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.0692999999999998e-05 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:486 3.6 3:618b61ca:::10000000002.000000e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 128.59415843900001, 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.74452019999999997, 2024-09-16T01:53:08.763 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 486 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.764 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140714+0000", 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.140716+0000", 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1919999999999999e-06 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175467+0000", 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.034750958999999998 2024-09-16T01:53:08.765 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175484+0000", 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7003999999999998e-05 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175491+0000", 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9770000000000003e-06 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.766 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175579+0000", 2024-09-16T01:53:08.767 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.8422000000000001e-05 2024-09-16T01:53:08.767 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.767 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.767 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:53:08.768 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175589+0000", 2024-09-16T01:53:08.768 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5850000000000006e-06 2024-09-16T01:53:08.768 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.768 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.826569+0000", 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.65098032500000003 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.828137+0000", 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00156726 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864620+0000", 2024-09-16T01:53:08.769 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.036482927999999998 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864680+0000", 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0825999999999997e-05 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.864774+0000", 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.3164000000000005e-05 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.770 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.867940+0000", 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031660960000000002 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885114+0000", 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017173996 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885151+0000", 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7419999999999997e-05 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.771 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.885234+0000", 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.3046000000000004e-05 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:488 3.8 3:10dca853:::10000000002.000000e5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 128.55936213999999, 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.72576230399999997, 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.772 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 488 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.773 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175510+0000", 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.175511+0000", 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.02e-06 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220212+0000", 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.044700518000000002 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.774 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220235+0000", 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2901e-05 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220245+0000", 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9019999999999994e-06 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220334+0000", 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9886e-05 2024-09-16T01:53:08.775 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for peered", 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.220344+0000", 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.7990000000000005e-06 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.823626+0000", 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.60328198 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for degraded object", 2024-09-16T01:53:08.776 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.825012+0000", 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.001385828 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855486+0000", 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.030473766999999999 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855577+0000", 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.0921000000000006e-05 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.777 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.855669+0000", 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2046999999999995e-05 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.860608+0000", 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049387670000000002 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901117+0000", 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040509221999999998 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.778 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901178+0000", 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.1068999999999994e-05 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.901273+0000", 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.4677000000000002e-05 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.779 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:490 3.2 3:437e6408:::10000000002.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e59)", 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 128.475913187, 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.061942243000000001, 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 490 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.780 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258959+0000", 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.258961+0000", 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.528e-06 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.781 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299708+0000", 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040746912000000003 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299731+0000", 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.3203000000000001e-05 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299745+0000", 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4299e-05 2024-09-16T01:53:08.782 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299821+0000", 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.6136000000000004e-05 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.299912+0000", 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1267999999999994e-05 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.783 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.300034+0000", 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000121595 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.303382+0000", 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003347455 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320807+0000", 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017425896 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.784 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320858+0000", 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.0136999999999998e-05 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:00.320901+0000", 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3813999999999998e-05 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.785 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:518 3.6 3:613bf181:::100000001f7.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e60)", 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 114.2766104, 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.078742554000000006, 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.786 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 518 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458262+0000", 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.458263+0000", 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.195e-06 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503412+0000", 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.045148718999999997 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503430+0000", 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8561999999999999e-05 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503440+0000", 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.8139999999999995e-06 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.503933+0000", 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000492397 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504074+0000", 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00014114999999999999 2024-09-16T01:53:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.504216+0000", 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000141886 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.509016+0000", 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.004800449 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536826+0000", 2024-09-16T01:53:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027810140000000001 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.536926+0000", 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.9310999999999998e-05 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.537004+0000", 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.8930999999999994e-05 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:539 3.6 3:61d9b40b:::100000001f7.00000018:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 114.131733462, 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.062655269999999999, 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 539 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603139+0000", 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.603141+0000", 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.12e-06 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643726+0000", 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040584506999999999 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643747+0000", 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1801e-05 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643756+0000", 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5490000000000006e-06 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643819+0000", 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.2940000000000004e-05 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643933+0000", 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000113985 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.644025+0000", 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.1990000000000005e-05 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.648243+0000", 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0042180760000000003 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665652+0000", 2024-09-16T01:53:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.017408727999999998 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665727+0000", 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5127000000000001e-05 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.665794+0000", 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.7447000000000007e-05 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:540 3.6 3:62937fa2:::100000001f7.00000019:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 114.09108903800001, 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059040347, 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 540 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643783+0000", 2024-09-16T01:53:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.643785+0000", 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.28e-06 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686276+0000", 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.042491334999999998 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.800 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686294+0000", 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8165999999999999e-05 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686305+0000", 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.1007e-05 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686365+0000", 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9722999999999997e-05 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.801 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686469+0000", 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104075 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.686580+0000", 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110821 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.802 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.689144+0000", 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0025641320000000002 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702734+0000", 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.013589685000000001 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702779+0000", 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.5578000000000003e-05 2024-09-16T01:53:08.803 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.702824+0000", 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4545000000000001e-05 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:558 3.8 3:11e263b4:::100000001f7.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 113.933146204, 2024-09-16T01:53:08.804 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068076880000000006, 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 558 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.805 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801726+0000", 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.801728+0000", 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.984e-06 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834434+0000", 2024-09-16T01:53:08.806 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032705894999999999 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834444+0000", 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0190000000000001e-05 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834451+0000", 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.2370000000000003e-06 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.807 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834501+0000", 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9413999999999998e-05 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834602+0000", 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000100762 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.834694+0000", 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.2491999999999995e-05 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.808 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.837353+0000", 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0026586890000000001 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869732+0000", 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.032379401000000002 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869767+0000", 2024-09-16T01:53:08.809 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4891999999999997e-05 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.869803+0000", 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.5923999999999997e-05 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:569 3.6 3:62f75488:::100000001f7.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:53:08.810 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 113.80801894299999, 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060264180000000001, 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 569 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:53:08.811 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926853+0000", 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.926856+0000", 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4420000000000001e-06 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.812 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968554+0000", 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041698411999999997 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968572+0000", 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.8150000000000001e-05 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968583+0000", 2024-09-16T01:53:08.813 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0159e-05 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968672+0000", 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.9586000000000006e-05 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968779+0000", 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106591 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 1", 2024-09-16T01:53:08.814 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.968889+0000", 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000110519 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.972285+0000", 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.003395534 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987015+0000", 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.014729895 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.815 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987075+0000", 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0158000000000002e-05 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:14.987118+0000", 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.2734000000000001e-05 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.816 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:683 3.8 3:1323b9a3:::100000001f7.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e65)", 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 112.771219012, 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060903035000000001, 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 683 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.817 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963653+0000", 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.963655+0000", 2024-09-16T01:53:08.818 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.1270000000000001e-06 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990205+0000", 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026549288000000001 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990224+0000", 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9675999999999999e-05 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.819 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990235+0000", 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0872e-05 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990294+0000", 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8338000000000003e-05 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990405+0000", 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000111706 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.820 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.990526+0000", 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00012081799999999999 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:15.996339+0000", 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0058132360000000003 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024481+0000", 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.028141347000000001 2024-09-16T01:53:08.821 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024525+0000", 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3930000000000001e-05 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:16.024556+0000", 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.1696999999999998e-05 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.822 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:870 3.2b 3:d419a785:::100000001f7.00000063:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e69)", 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 107.60819337, 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077709022000000003, 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 870 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.823 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126679+0000", 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.824 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.126681+0000", 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.407e-06 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191821+0000", 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.065139694999999997 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191846+0000", 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.4547000000000001e-05 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.825 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191856+0000", 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0458e-05 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.191931+0000", 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.5388000000000004e-05 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192019+0000", 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.7136000000000001e-05 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.826 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.192153+0000", 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00013444499999999999 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.196616+0000", 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044629780000000003 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204276+0000", 2024-09-16T01:53:08.827 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0076599190000000003 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204335+0000", 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.9246999999999997e-05 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:21.204388+0000", 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.2802e-05 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.828 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1122 3.2 3:415a7f8d:::100000001f8.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 89.474455581000001, 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.058795700999999999, 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1122 2024-09-16T01:53:08.829 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260417+0000", 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.830 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.260419+0000", 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.2189999999999998e-06 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287678+0000", 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.027259228999999999 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287694+0000", 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.6158e-05 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.831 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287702+0000", 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.3919999999999997e-06 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287781+0000", 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.9160999999999996e-05 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.287939+0000", 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00015805499999999999 2024-09-16T01:53:08.832 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.288103+0000", 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00016379 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.292525+0000", 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044224219999999996 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319137+0000", 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.026611804999999999 2024-09-16T01:53:08.833 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319169+0000", 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2194999999999999e-05 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.319212+0000", 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3275000000000002e-05 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.834 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1191 3.1b 3:db9b5f95:::100000001f8.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e76)", 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 88.780065467, 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.074847492000000002, 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.835 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1191 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954807+0000", 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.836 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:39.954808+0000", 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.392e-06 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011909+0000", 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.057100862000000002 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011929+0000", 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.9639999999999998e-05 2024-09-16T01:53:08.837 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.011941+0000", 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2646999999999999e-05 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012011+0000", 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.9973000000000002e-05 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012119+0000", 2024-09-16T01:53:08.838 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010783800000000001 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.012224+0000", 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000104756 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.017207+0000", 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0049825620000000003 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.839 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029549+0000", 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.012342096 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029607+0000", 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.8808999999999999e-05 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:40.029654+0000", 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.6916999999999998e-05 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.840 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1532 3.2b 3:d402c155:::100000001f8.000001f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 84.453090900999996, 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.077126069000000005, 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.841 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1532 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281781+0000", 2024-09-16T01:53:08.842 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.281785+0000", 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.4529999999999998e-06 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332301+0000", 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.050515879 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.843 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332318+0000", 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.7164e-05 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332326+0000", 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.337e-06 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332387+0000", 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0817999999999999e-05 2024-09-16T01:53:08.844 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332506+0000", 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000118544 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.332630+0000", 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000124203 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.335980+0000", 2024-09-16T01:53:08.845 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0033502470000000002 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358819+0000", 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.022839176999999999 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358863+0000", 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3664000000000001e-05 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.846 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.358908+0000", 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.4583000000000003e-05 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1544 3.6 3:629272c8:::100000001f8.00000205:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 84.312180760999993, 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.059444029000000002, 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.847 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1544 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422692+0000", 2024-09-16T01:53:08.848 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.422693+0000", 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.2869999999999999e-06 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475446+0000", 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.052752935000000001 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.849 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475464+0000", 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.772e-05 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475472+0000", 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5750000000000001e-06 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475537+0000", 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.4683000000000004e-05 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.850 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475692+0000", 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000155434 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 7", 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475809+0000", 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000116537 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.480282+0000", 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0044729820000000003 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.851 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482014+0000", 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0017321280000000001 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482078+0000", 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.3806000000000005e-05 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.482136+0000", 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 5.7942000000000002e-05 2024-09-16T01:53:08.852 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1545 3.2b 3:d5134dfc:::100000001f8.00000206:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 84.259377817000001, 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.068122770999999999, 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.853 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1545 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.854 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.475495+0000", 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.5499999999999996e-07 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516145+0000", 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.040649061 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.855 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516159+0000", 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.4064e-05 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516166+0000", 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 7.1409999999999999e-06 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516215+0000", 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.9509000000000003e-05 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516318+0000", 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000102704 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 2", 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.516425+0000", 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.000106708 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.519871+0000", 2024-09-16T01:53:08.857 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0034465229999999999 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543537+0000", 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.023665591 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543580+0000", 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 4.3049000000000001e-05 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.858 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:44.543617+0000", 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.7465999999999997e-05 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "description": "osd_op(client.4619.0:1666 3.1b 3:d89a249d:::100000001f8.0000027f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected e81)", 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "initiated_at": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "age": 82.940119276999994, 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.060336470000000003, 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "continuous": false, 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "type_data": { 2024-09-16T01:53:08.859 INFO:teuthology.orchestra.run.smithi161.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "client_info": { 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "client": "client.4619", 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "client_addr": "172.21.15.146:0/3300930908", 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "tid": 1666 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "events": [ 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "initiated", 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "header_read", 2024-09-16T01:53:08.860 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794753+0000", 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "throttled", 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.794755+0000", 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 2.0940000000000002e-06 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "all_read", 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836083+0000", 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.041328125 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.861 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "dispatched", 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836094+0000", 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 1.0813e-05 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "queued_for_pg", 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836103+0000", 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 8.5939999999999994e-06 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "reached_pg", 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836163+0000", 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 6.0412999999999999e-05 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.862 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "started", 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836259+0000", 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 9.6267999999999994e-05 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "waiting for subops from 0", 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.836364+0000", 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.00010463600000000001 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "op_commit", 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.839474+0000", 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.0031098359999999999 2024-09-16T01:53:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "sub_op_commit_rec", 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855018+0000", 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 0.015543927000000001 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "commit_sent", 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855057+0000", 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.8985e-05 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "event": "done", 2024-09-16T01:53:08.864 INFO:teuthology.orchestra.run.smithi161.stdout: "time": "2024-09-16T01:51:45.855090+0000", 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout: "duration": 3.2778999999999999e-05 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-09-16T01:53:08.865 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:08.865 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T01:53:08.868 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.796+0000 7f6fd9600640 1 -- 172.21.15.146:0/4029541785 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v128) ==== 113+0+0 (secure 0 0 0) 0x7f6fcc05cb30 con 0x7f6fdc06fc50 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6fc006c330 msgr2=0x7f6fc006e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6fc006c330 0x7f6fc006e7f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0037000 tx=0x7f6fd0008030 comp rx=0 tx=0).stop 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fdc06fc50 msgr2=0x7f6fdc1448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fdc06fc50 0x7f6fdc1448d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00c470 tx=0x7f6fcc00c870 comp rx=0 tx=0).stop 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 shutdown_connections 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6fc006c330 0x7f6fc006e7f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6fdc070030 0x7f6fdc143850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fdc06fc50 0x7f6fdc1448d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 --2- 172.21.15.146:0/4029541785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fdc06f590 0x7f6fdc143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:08.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.799+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 >> 172.21.15.146:0/4029541785 conn(0x7f6fdc06e670 msgr2=0x7f6fdc072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:08.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.800+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 shutdown_connections 2024-09-16T01:53:08.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:08.800+0000 7f6fe4000640 1 -- 172.21.15.146:0/4029541785 wait complete. 2024-09-16T01:53:08.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T01:53:08.870 INFO:tasks.ceph.osd.0.smithi146.stderr:2024-09-16T01:53:08.866+0000 7fa890200640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 36108) UID: 0 2024-09-16T01:53:08.970 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:08.968+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:09.013 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:09.013 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:09.013 INFO:teuthology.orchestra.run.smithi161.stdout: "num_ops": 0 2024-09-16T01:53:09.013 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:09.021 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:53:09.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.067+0000 7fcdbd600640 1 Processor -- start 2024-09-16T01:53:09.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.068+0000 7fcdbd600640 1 -- start start 2024-09-16T01:53:09.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdbd600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb80c6bb0 0x7fcdb8158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdbd600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcdb8159110 0x7fcdb8162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdbd600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb81586d0 con 0x7fcdb8152a10 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdbd600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fcdb8159550 con 0x7fcdb8159110 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdbd600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb806de70 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdb7e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcdb8152a10 0x7fcdb80c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33844/0 (socket says 172.21.15.146:33844) 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdb7e00640 1 -- 172.21.15.146:0/3488104419 learned_addr learned my addr 172.21.15.146:0/3488104419 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/3488104419 >> v2:172.21.15.146:3300/0 conn(0x7fcdb8159110 0x7fcdb8162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.069+0000 7fcdb7400640 1 --2- 172.21.15.146:0/3488104419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb80c6bb0 0x7fcdb8158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.070+0000 7fcdb6a00640 1 -- 172.21.15.146:0/3488104419 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1938047108 0 0) 0x7fcdb81586d0 con 0x7fcdb8152a10 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.070+0000 7fcdbcc00640 1 -- 172.21.15.146:0/3488104419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb80c6bb0 msgr2=0x7fcdb8158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.070+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/3488104419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb80c6bb0 0x7fcdb8158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.070+0000 7fcdbcc00640 1 -- 172.21.15.146:0/3488104419 >> v1:172.21.15.146:6789/0 conn(0x7fcdb8152a10 legacy=0x7fcdb80c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.070+0000 7fcdbcc00640 1 -- 172.21.15.146:0/3488104419 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb806e150 con 0x7fcdb8159110 2024-09-16T01:53:09.072 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:09.070+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:09.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.071+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/3488104419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb8159110 0x7fcdb8162fb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009d90 tx=0x7fcdb002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0ec86098d0f204 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.071+0000 7fcdb6a00640 1 -- 172.21.15.146:0/3488104419 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0007740 con 0x7fcdb8159110 2024-09-16T01:53:09.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.071+0000 7fcdb6a00640 1 -- 172.21.15.146:0/3488104419 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb00078e0 con 0x7fcdb8159110 2024-09-16T01:53:09.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.071+0000 7fcdb6a00640 1 -- 172.21.15.146:0/3488104419 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0007cb0 con 0x7fcdb8159110 2024-09-16T01:53:09.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 -- 172.21.15.146:0/3488104419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb8159110 msgr2=0x7fcdb8162fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 --2- 172.21.15.146:0/3488104419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb8159110 0x7fcdb8162fb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009d90 tx=0x7fcdb002a660 comp rx=0 tx=0).stop 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 -- 172.21.15.146:0/3488104419 shutdown_connections 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 --2- 172.21.15.146:0/3488104419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb8159110 0x7fcdb8162fb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 --2- 172.21.15.146:0/3488104419 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb80c6bb0 0x7fcdb8158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 -- 172.21.15.146:0/3488104419 >> 172.21.15.146:0/3488104419 conn(0x7fcdb80cc1b0 msgr2=0x7fcdb80cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 -- 172.21.15.146:0/3488104419 shutdown_connections 2024-09-16T01:53:09.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.072+0000 7fcdbd600640 1 -- 172.21.15.146:0/3488104419 wait complete. 2024-09-16T01:53:09.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.074+0000 7fcdbd600640 1 Processor -- start 2024-09-16T01:53:09.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.074+0000 7fcdbd600640 1 -- start start 2024-09-16T01:53:09.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.074+0000 7efd4f200640 1 Processor -- start 2024-09-16T01:53:09.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.074+0000 7efd4f200640 1 -- start start 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbd600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 0x7fcdb813ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbd600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb8152a10 0x7fcdb81e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbd600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb8159110 0x7fcdb81e3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbd600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb806ec40 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbd600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb806e960 con 0x7fcdb8152a10 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbd600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb8152df0 con 0x7fcdb8159110 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 0x7fcdb813ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdbcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb8159110 0x7fcdb81e3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb8152a10 0x7fcdb81e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 0x7fcdb813ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49456/0 (socket says 172.21.15.146:49456) 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 -- 172.21.15.146:0/2365664154 learned_addr learned my addr 172.21.15.146:0/2365664154 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb8159110 msgr2=0x7fcdb81e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcdb8159110 0x7fcdb81e3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb8152a10 msgr2=0x7fcdb81e3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdb8152a10 0x7fcdb81e3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.075+0000 7fcdb7e00640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb81e3610 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500cc260 0x7efd5015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd500b6150 con 0x7efd500d0720 2024-09-16T01:53:09.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4f200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd5015a940 con 0x7efd500d0340 2024-09-16T01:53:09.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efd500cb020 con 0x7efd500cc260 2024-09-16T01:53:09.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 0x7fcdb813ffe0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcda800c500 tx=0x7fcda800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4de00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efd500d0340 0x7efd500c6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55398/0 (socket says 172.21.15.146:55398) 2024-09-16T01:53:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4de00640 1 -- 172.21.15.146:0/3404503137 learned_addr learned my addr 172.21.15.146:0/3404503137 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4e800640 1 --2- 172.21.15.146:0/3404503137 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500cc260 0x7efd5015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8018070 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda8007580 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8004a20 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb8140520 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4e800640 1 -- 172.21.15.146:0/3404503137 >> v1:172.21.15.161:6789/0 conn(0x7efd500d0340 legacy=0x7efd500c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4e800640 1 -- 172.21.15.146:0/3404503137 >> v1:172.21.15.146:6789/0 conn(0x7efd500d0720 legacy=0x7efd501630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4e800640 1 -- 172.21.15.146:0/3404503137 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd500cb270 con 0x7efd500cc260 2024-09-16T01:53:09.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7efd4ca00640 1 -- 172.21.15.146:0/3404503137 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014081704 0 0) 0x7efd5015a940 con 0x7efd500d0340 2024-09-16T01:53:09.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.076+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81e21f0 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.078+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fcdb806db30 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.078+0000 7efd4e800640 1 --2- 172.21.15.146:0/3404503137 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500cc260 0x7efd5015b3e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efd48009dc0 tx=0x7efd4802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f581982c48589054 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.078+0000 7efd4ca00640 1 -- 172.21.15.146:0/3404503137 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48007740 con 0x7efd500cc260 2024-09-16T01:53:09.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.078+0000 7efd4ca00640 1 -- 172.21.15.146:0/3404503137 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd480078e0 con 0x7efd500cc260 2024-09-16T01:53:09.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.078+0000 7efd4ca00640 1 -- 172.21.15.146:0/3404503137 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48007cb0 con 0x7efd500cc260 2024-09-16T01:53:09.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.078+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcda801b070 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.079+0000 7efd4f200640 1 -- 172.21.15.146:0/3404503137 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500cc260 msgr2=0x7efd5015b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.079+0000 7efd4f200640 1 --2- 172.21.15.146:0/3404503137 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500cc260 0x7efd5015b3e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efd48009dc0 tx=0x7efd4802a6d0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.079+0000 7fcdafe00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcd9806c330 0x7fcd9806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.079+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5894+0+0 (secure 0 0 0) 0x7fcda808a800 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7fcdb7400640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcd9806c330 0x7fcd9806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7efd4f200640 1 -- 172.21.15.146:0/3404503137 shutdown_connections 2024-09-16T01:53:09.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7efd4f200640 1 --2- 172.21.15.146:0/3404503137 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500cc260 0x7efd5015b3e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7efd4f200640 1 -- 172.21.15.146:0/3404503137 >> 172.21.15.146:0/3404503137 conn(0x7efd5006e0d0 msgr2=0x7efd5006e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:09.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v128) ==== 54+0+16 (secure 0 0 0) 0x7fcdb81e3610 con 0x7fcdb80c6bb0 2024-09-16T01:53:09.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7efd4f200640 1 -- 172.21.15.146:0/3404503137 shutdown_connections 2024-09-16T01:53:09.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7efd4f200640 1 -- 172.21.15.146:0/3404503137 wait complete. 2024-09-16T01:53:09.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fcdb80cb100 0x7fcdb8145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdb80c4860 con 0x7fcdb80cb100 2024-09-16T01:53:09.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fcdb80cb100 0x7fcdb8145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.080+0000 7fcdb7400640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcd9806c330 0x7fcd9806e7f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcdb81459a0 tx=0x7fcda4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.081+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fcdb80cb100 0x7fcdb8145e20 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.081+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb0007640 con 0x7fcdb80cb100 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.082+0000 7efd4f200640 1 Processor -- start 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.082+0000 7efd4f200640 1 -- start start 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd500cc260 0x7efd500cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 0x7efd5014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd500d0720 0x7efd5014d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efd500cb550 con 0x7efd500d0720 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd5015aec0 con 0x7efd500cc260 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efd500c6a10 con 0x7efd500d0340 2024-09-16T01:53:09.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 0x7efd5014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 0x7efd5014e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43294/0 (socket says 172.21.15.146:43294) 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 -- 172.21.15.146:0/1968477328 learned_addr learned my addr 172.21.15.146:0/1968477328 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4de00640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd500cc260 0x7efd500cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4e800640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd500d0720 0x7efd5014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 -- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd500cc260 msgr2=0x7efd500cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd500cc260 0x7efd500cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 -- 172.21.15.146:0/1968477328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd500d0720 msgr2=0x7efd5014d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd500d0720 0x7efd5014d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4d400640 1 -- 172.21.15.146:0/1968477328 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd501f3640 con 0x7efd500d0340 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.083+0000 7efd4e800640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd500d0720 0x7efd5014d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.084+0000 7efd4d400640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 0x7efd5014e900 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00c3c0 tx=0x7efd3c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.084+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c017070 con 0x7efd500d0340 2024-09-16T01:53:09.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.084+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3c0052d0 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.084+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd501f3920 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.084+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3c003830 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.084+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd501f3ee0 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.085+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd500c7730 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.086+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efd3c01a070 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.087+0000 7efd46000640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efd3006c2b0 0x7efd3006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.087+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(128..128 src has 1..128) ==== 5894+0+0 (secure 0 0 0) 0x7efd3c08a810 con 0x7efd500d0340 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.087+0000 7efd4de00640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efd3006c2b0 0x7efd3006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.088+0000 7efd4de00640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efd3006c2b0 0x7efd3006e770 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efd40000f60 tx=0x7efd40007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.091+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd3c0587d0 con 0x7efd500d0340 2024-09-16T01:53:09.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.097+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb806e3c0 con 0x7fcdb80cb100 2024-09-16T01:53:09.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.099+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb806e3c0 con 0x7fcdb80cb100 2024-09-16T01:53:09.100 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0: {} 2024-09-16T01:53:09.101 INFO:teuthology.orchestra.run.smithi146.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T01:53:09.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.099+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fcdb81e46e0 0x7fcdb8157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.099+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fcdb8150920 con 0x7fcdb81e46e0 2024-09-16T01:53:09.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.099+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fcdb81e46e0 0x7fcdb8157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.100+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fcdb81e46e0 0x7fcdb8157960 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.101+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb8150920 con 0x7fcdb81e46e0 2024-09-16T01:53:09.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.115+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb814ca40 con 0x7fcdb81e46e0 2024-09-16T01:53:09.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.117+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb814ca40 con 0x7fcdb81e46e0 2024-09-16T01:53:09.118 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1: {} 2024-09-16T01:53:09.119 INFO:teuthology.orchestra.run.smithi146.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T01:53:09.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.117+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fcdb80ca550 0x7fcdb80ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.117+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fcdb80cae90 con 0x7fcdb80ca550 2024-09-16T01:53:09.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.118+0000 7fcdb7400640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fcdb80ca550 0x7fcdb80ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.118+0000 7fcdb7400640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fcdb80ca550 0x7fcdb80ca950 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.119+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcda401b040 con 0x7fcdb80ca550 2024-09-16T01:53:09.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.133+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb8062880 con 0x7fcdb80ca550 2024-09-16T01:53:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.135+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb8062880 con 0x7fcdb80ca550 2024-09-16T01:53:09.136 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2: {} 2024-09-16T01:53:09.136 INFO:teuthology.orchestra.run.smithi146.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T01:53:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.135+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fcdb8063360 0x7fcdb81e4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.135+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fcdb80c9830 con 0x7fcdb8063360 2024-09-16T01:53:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.135+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fcdb8063360 0x7fcdb81e4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.136+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fcdb8063360 0x7fcdb81e4c30 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.137+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb80c9830 con 0x7fcdb8063360 2024-09-16T01:53:09.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.151+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb81e5440 con 0x7fcdb8063360 2024-09-16T01:53:09.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.152+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb81e5440 con 0x7fcdb8063360 2024-09-16T01:53:09.153 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3: {} 2024-09-16T01:53:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T01:53:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.152+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fcdb802c640 0x7fcdb802caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.152+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fcdb80c9830 con 0x7fcdb802c640 2024-09-16T01:53:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.153+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fcdb802c640 0x7fcdb802caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.153+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fcdb802c640 0x7fcdb802caa0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.154+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb80c9830 con 0x7fcdb802c640 2024-09-16T01:53:09.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.168+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb80c9550 con 0x7fcdb802c640 2024-09-16T01:53:09.169 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:09.169+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:09.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.171+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb80c9550 con 0x7fcdb802c640 2024-09-16T01:53:09.172 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4: {} 2024-09-16T01:53:09.172 INFO:teuthology.orchestra.run.smithi146.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T01:53:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.171+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fcdb802d4f0 0x7fcdb8207050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.171+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fcdb81506c0 con 0x7fcdb802d4f0 2024-09-16T01:53:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.171+0000 7fcdb7400640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fcdb802d4f0 0x7fcdb8207050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.172+0000 7fcdb7400640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fcdb802d4f0 0x7fcdb8207050 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.172+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcda4028e50 con 0x7fcdb802d4f0 2024-09-16T01:53:09.181 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-09-16T01:53:09.181 INFO:teuthology.orchestra.run.smithi161.stdout: "ops": [], 2024-09-16T01:53:09.181 INFO:teuthology.orchestra.run.smithi161.stdout: "complaint_time": 180, 2024-09-16T01:53:09.181 INFO:teuthology.orchestra.run.smithi161.stdout: "num_blocked_ops": 0 2024-09-16T01:53:09.181 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-09-16T01:53:09.187 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:53:09.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.186+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb80c9860 con 0x7fcdb802d4f0 2024-09-16T01:53:09.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.188+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb80c9860 con 0x7fcdb802d4f0 2024-09-16T01:53:09.189 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5: {} 2024-09-16T01:53:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T01:53:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.188+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fcdb802d970 0x7fcdb8209ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.188+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fcdb820ca60 con 0x7fcdb802d970 2024-09-16T01:53:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.188+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fcdb802d970 0x7fcdb8209ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.189+0000 7fcdbcc00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fcdb802d970 0x7fcdb8209ab0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.190+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb820ca60 con 0x7fcdb802d970 2024-09-16T01:53:09.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.203+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb80008d0 con 0x7fcdb802d970 2024-09-16T01:53:09.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.205+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb80008d0 con 0x7fcdb802d970 2024-09-16T01:53:09.206 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6: {} 2024-09-16T01:53:09.207 INFO:teuthology.orchestra.run.smithi146.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T01:53:09.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.205+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fcdb820a5a0 0x7fcdb820aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.205+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fcdb820c6d0 con 0x7fcdb820a5a0 2024-09-16T01:53:09.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.205+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fcdb820a5a0 0x7fcdb820aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.206+0000 7fcdb7e00640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fcdb820a5a0 0x7fcdb820aa00 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.206+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb820c6d0 con 0x7fcdb820a5a0 2024-09-16T01:53:09.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.220+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcdb820af40 con 0x7fcdb820a5a0 2024-09-16T01:53:09.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdafe00640 1 -- 172.21.15.146:0/2365664154 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcdb820af40 con 0x7fcdb820a5a0 2024-09-16T01:53:09.223 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7: {} 2024-09-16T01:53:09.225 INFO:teuthology.orchestra.run.smithi146.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T01:53:09.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fcdb80cb100 msgr2=0x7fcdb8145e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fcdb80cb100 0x7fcdb8145e20 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fcdb81e46e0 msgr2=0x7fcdb8157960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7fcdb81e46e0 0x7fcdb8157960 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fcdb80ca550 msgr2=0x7fcdb80ca950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7fcdb80ca550 0x7fcdb80ca950 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fcdb8063360 msgr2=0x7fcdb81e4c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7fcdb8063360 0x7fcdb81e4c30 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fcdb802c640 msgr2=0x7fcdb802caa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fcdb802c640 0x7fcdb802caa0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fcdb802d4f0 msgr2=0x7fcdb8207050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7fcdb802d4f0 0x7fcdb8207050 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbcc00640 1 -- 172.21.15.146:0/2365664154 reap_dead start 2024-09-16T01:53:09.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fcdb802d970 msgr2=0x7fcdb8209ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fcdb802d970 0x7fcdb8209ab0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.222+0000 7fcdbcc00640 1 -- 172.21.15.146:0/2365664154 reap_dead start 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbcc00640 1 -- 172.21.15.146:0/2365664154 reap_dead start 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbcc00640 1 -- 172.21.15.146:0/2365664154 reap_dead start 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fcdb820a5a0 msgr2=0x7fcdb820aa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fcdb820a5a0 0x7fcdb820aa00 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcd9806c330 msgr2=0x7fcd9806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcd9806c330 0x7fcd9806e7f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcdb81459a0 tx=0x7fcda4009190 comp rx=0 tx=0).stop 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 msgr2=0x7fcdb813ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 0x7fcdb813ffe0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcda800c500 tx=0x7fcda800c9c0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 shutdown_connections 2024-09-16T01:53:09.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fcdb802d970 0x7fcdb8209ab0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7fcdb820a5a0 0x7fcdb820aa00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcd9806c330 0x7fcd9806e7f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 --2- 172.21.15.146:0/2365664154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdb80c6bb0 0x7fcdb813ffe0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 >> 172.21.15.146:0/2365664154 conn(0x7fcdb80cc1b0 msgr2=0x7fcdb8150ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:09.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 shutdown_connections 2024-09-16T01:53:09.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.223+0000 7fcdbd600640 1 -- 172.21.15.146:0/2365664154 wait complete. 2024-09-16T01:53:09.280 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:09.279+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:09.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.288+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7efd500c6480 con 0x7efd500d0340 2024-09-16T01:53:09.291 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:09.292 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:09.300 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:53:09.312 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:53:09.312 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:53:09.313 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-09-16T01:53:09.313 INFO:tasks.workunit.client.0.smithi146.stdout:iozone test complete. 2024-09-16T01:53:09.315 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T01:53:09.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T01:53:09.371 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:09.369+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:09.405 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:09.405 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:09.414 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:53:09.415 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-09-16T01:53:09.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T01:53:09.472 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:09.471+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:09.559 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:09.559 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:09.568 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:53:09.573 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:09.572+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:09.671 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:09.672 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:09.674 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-16T01:53:09.673+0000 7fcaf0c00640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 36119) UID: 0 2024-09-16T01:53:09.680 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:53:09.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T01:53:09.775 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:09.774+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:09.784 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:09.784 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:09.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:53:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.802+0000 7efd46000640 1 -- 172.21.15.146:0/1968477328 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v129) ==== 113+0+0 (secure 0 0 0) 0x7efd3c05cad0 con 0x7efd500d0340 2024-09-16T01:53:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T01:53:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efd3006c2b0 msgr2=0x7efd3006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efd3006c2b0 0x7efd3006e770 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efd40000f60 tx=0x7efd40007e40 comp rx=0 tx=0).stop 2024-09-16T01:53:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 msgr2=0x7efd5014e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 0x7efd5014e900 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd3c00c3c0 tx=0x7efd3c00c880 comp rx=0 tx=0).stop 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 shutdown_connections 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efd3006c2b0 0x7efd3006e770 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd500d0720 0x7efd5014d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd500d0340 0x7efd5014e900 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 --2- 172.21.15.146:0/1968477328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd500cc260 0x7efd500cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 >> 172.21.15.146:0/1968477328 conn(0x7efd5006e0d0 msgr2=0x7efd500c95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 shutdown_connections 2024-09-16T01:53:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.805+0000 7efd4f200640 1 -- 172.21.15.146:0/1968477328 wait complete. 2024-09-16T01:53:09.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T01:53:09.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.851+0000 7f572be00640 1 Processor -- start 2024-09-16T01:53:09.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.851+0000 7f572be00640 1 -- start start 2024-09-16T01:53:09.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572be00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f572c0c6b70 0x7f572c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0cc040 0x7f572c0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572c06d140 con 0x7f572c0c75a0 2024-09-16T01:53:09.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572be00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f572c15b880 con 0x7f572c0c6b70 2024-09-16T01:53:09.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572be00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f572c0cab20 con 0x7f572c0cc040 2024-09-16T01:53:09.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572aa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f572c0c6b70 0x7f572c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572b400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0cc040 0x7f572c0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572a000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f572c0c75a0 0x7f572c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33872/0 (socket says 172.21.15.146:33872) 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.852+0000 7f572a000640 1 -- 172.21.15.146:0/407521457 learned_addr learned my addr 172.21.15.146:0/407521457 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.853+0000 7f572b400640 1 -- 172.21.15.146:0/407521457 >> v2:172.21.15.161:3300/0 conn(0x7f572c0c6b70 msgr2=0x7f572c15b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.853+0000 7f572b400640 1 --2- 172.21.15.146:0/407521457 >> v2:172.21.15.161:3300/0 conn(0x7f572c0c6b70 0x7f572c15b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.853+0000 7f572b400640 1 -- 172.21.15.146:0/407521457 >> v1:172.21.15.146:6789/0 conn(0x7f572c0c75a0 legacy=0x7f572c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.853+0000 7f572b400640 1 -- 172.21.15.146:0/407521457 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c06d420 con 0x7f572c0cc040 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.853+0000 7f572aa00640 1 --2- 172.21.15.146:0/407521457 >> v2:172.21.15.161:3300/0 conn(0x7f572c0c6b70 0x7f572c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.854+0000 7f572b400640 1 --2- 172.21.15.146:0/407521457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0cc040 0x7f572c0cc4c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5724009d90 tx=0x7f572402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be09b758e7a45ad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.854+0000 7f5729600640 1 -- 172.21.15.146:0/407521457 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724007740 con 0x7f572c0cc040 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.854+0000 7f5729600640 1 -- 172.21.15.146:0/407521457 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57240078e0 con 0x7f572c0cc040 2024-09-16T01:53:09.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.854+0000 7f5729600640 1 -- 172.21.15.146:0/407521457 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5724007cf0 con 0x7f572c0cc040 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 -- 172.21.15.146:0/407521457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0cc040 msgr2=0x7f572c0cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 --2- 172.21.15.146:0/407521457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0cc040 0x7f572c0cc4c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5724009d90 tx=0x7f572402a5f0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 -- 172.21.15.146:0/407521457 shutdown_connections 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 --2- 172.21.15.146:0/407521457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0cc040 0x7f572c0cc4c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 --2- 172.21.15.146:0/407521457 >> v2:172.21.15.161:3300/0 conn(0x7f572c0c6b70 0x7f572c15b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 -- 172.21.15.146:0/407521457 >> 172.21.15.146:0/407521457 conn(0x7f572c0ccc20 msgr2=0x7f572c0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 -- 172.21.15.146:0/407521457 shutdown_connections 2024-09-16T01:53:09.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.855+0000 7f572be00640 1 -- 172.21.15.146:0/407521457 wait complete. 2024-09-16T01:53:09.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.856+0000 7f572be00640 1 Processor -- start 2024-09-16T01:53:09.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.856+0000 7f572be00640 1 -- start start 2024-09-16T01:53:09.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.857+0000 7f572be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 0x7f572c0c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.857+0000 7f572be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572c0c75a0 0x7f572c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.857+0000 7f572be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f572c0cc040 0x7f572c1f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.857+0000 7f572be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0cb050 con 0x7f572c0c75a0 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.857+0000 7f572be00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f572c0cad70 con 0x7f572c0cc040 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.857+0000 7f572be00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f572c0cbc80 con 0x7f572c0c6b70 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 0x7f572c0c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f572c0cc040 0x7f572c1f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 0x7f572c0c25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43306/0 (socket says 172.21.15.146:43306) 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 -- 172.21.15.146:0/3230673835 learned_addr learned my addr 172.21.15.146:0/3230673835 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572a000640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572c0c75a0 0x7f572c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 -- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f572c0cc040 msgr2=0x7f572c1f48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f572c0cc040 0x7f572c1f48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 -- 172.21.15.146:0/3230673835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572c0c75a0 msgr2=0x7f572c0c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572c0c75a0 0x7f572c0c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 -- 172.21.15.146:0/3230673835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572c157950 con 0x7f572c0c6b70 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572b400640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f572c0cc040 0x7f572c1f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.858+0000 7f572aa00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 0x7f572c0c25c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f571c00e990 tx=0x7f571c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.859+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c017070 con 0x7f572c0c6b70 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.859+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571c00c9e0 con 0x7f572c0c6b70 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.859+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f572c157c30 con 0x7f572c0c6b70 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.859+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c00cdb0 con 0x7f572c0c6b70 2024-09-16T01:53:09.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.859+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f572c1581f0 con 0x7f572c0c6b70 2024-09-16T01:53:09.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.860+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572c06cd20 con 0x7f572c0c6b70 2024-09-16T01:53:09.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.861+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f571c01a070 con 0x7f572c0c6b70 2024-09-16T01:53:09.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.861+0000 7f5723400640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f570c06c2b0 0x7f570c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:09.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.862+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(129..129 src has 1..129) ==== 5894+0+0 (secure 0 0 0) 0x7f571c08b730 con 0x7f572c0c6b70 2024-09-16T01:53:09.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.862+0000 7f572a000640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f570c06c2b0 0x7f570c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:09.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.862+0000 7f572a000640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f570c06c2b0 0x7f570c06e770 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f572c0cf170 tx=0x7f5718007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:09.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:09.870+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f571c0596f0 con 0x7f572c0c6b70 2024-09-16T01:53:09.875 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-09-16T01:53:09.874+0000 7ff61557f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 37295) UID: 0 2024-09-16T01:53:09.902 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:09.902 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:09.911 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:53:09.977 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:09.976+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.016+0000 7fdddcc00640 1 Processor -- start 2024-09-16T01:53:10.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.016+0000 7fdddcc00640 1 -- start start 2024-09-16T01:53:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fdddcc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fddd8154ac0 0x7fddd815b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fdddcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd806d820 0x7fddd80d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fdddcc00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd8154ea0 con 0x7fddd80d0d00 2024-09-16T01:53:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fdddcc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fddd815b6d0 con 0x7fddd8154ac0 2024-09-16T01:53:10.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fdddcc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80c5340 con 0x7fddd806d820 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fddd7400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fddd8154ac0 0x7fddd815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fddd7e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fddd80d0d00 0x7fddd815bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55414/0 (socket says 172.21.15.146:55414) 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fddd6a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd806d820 0x7fddd80d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.017+0000 7fddd7e00640 1 -- 172.21.15.146:0/1035837183 learned_addr learned my addr 172.21.15.146:0/1035837183 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.018+0000 7fddd7400640 1 -- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd806d820 msgr2=0x7fddd80d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.018+0000 7fddd7400640 1 --2- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd806d820 0x7fddd80d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.018+0000 7fddd7400640 1 -- 172.21.15.146:0/1035837183 >> v1:172.21.15.161:6789/0 conn(0x7fddd80d0d00 legacy=0x7fddd815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.018+0000 7fddd7400640 1 -- 172.21.15.146:0/1035837183 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd80c5620 con 0x7fddd8154ac0 2024-09-16T01:53:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.018+0000 7fddd6a00640 1 --2- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd806d820 0x7fddd80d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.018+0000 7fddd6000640 1 -- 172.21.15.146:0/1035837183 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68778143 0 0) 0x7fddd8154ea0 con 0x7fddd80d0d00 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.019+0000 7fddd7400640 1 --2- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd8154ac0 0x7fddd815b190 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097a0 tx=0x7fddc802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b7a26f5735bf9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.019+0000 7fddd6000640 1 -- 172.21.15.146:0/1035837183 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8037070 con 0x7fddd8154ac0 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.019+0000 7fddd6000640 1 -- 172.21.15.146:0/1035837183 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc8032440 con 0x7fddd8154ac0 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.019+0000 7fddd6000640 1 -- 172.21.15.146:0/1035837183 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8032810 con 0x7fddd8154ac0 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 -- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd8154ac0 msgr2=0x7fddd815b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd8154ac0 0x7fddd815b190 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097a0 tx=0x7fddc802a210 comp rx=0 tx=0).stop 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 -- 172.21.15.146:0/1035837183 shutdown_connections 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd806d820 0x7fddd80d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1035837183 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd8154ac0 0x7fddd815b190 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 -- 172.21.15.146:0/1035837183 >> 172.21.15.146:0/1035837183 conn(0x7fddd806e400 msgr2=0x7fddd806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 -- 172.21.15.146:0/1035837183 shutdown_connections 2024-09-16T01:53:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.020+0000 7fdddcc00640 1 -- 172.21.15.146:0/1035837183 wait complete. 2024-09-16T01:53:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.021+0000 7fdddcc00640 1 Processor -- start 2024-09-16T01:53:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.021+0000 7fdddcc00640 1 -- start start 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fdddcc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 0x7fddd81f4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fdddcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd80d0d00 0x7fddd814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fdddcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd8154ac0 0x7fddd80cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fdddcc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fddd80c5ec0 con 0x7fddd80d0d00 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fdddcc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fddd80c5be0 con 0x7fddd806d820 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fdddcc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fddd80c61a0 con 0x7fddd8154ac0 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fddd7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 0x7fddd81f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fddd7e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd8154ac0 0x7fddd80cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fddd7400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 0x7fddd81f4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55888/0 (socket says 172.21.15.146:55888) 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fddd7400640 1 -- 172.21.15.146:0/1769174528 learned_addr learned my addr 172.21.15.146:0/1769174528 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.022+0000 7fddd6a00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd80d0d00 0x7fddd814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddd7400640 1 -- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd8154ac0 msgr2=0x7fddd80cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddd7400640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd8154ac0 0x7fddd80cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddd7400640 1 -- 172.21.15.146:0/1769174528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd80d0d00 msgr2=0x7fddd814ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddd7400640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd80d0d00 0x7fddd814ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddd7400640 1 -- 172.21.15.146:0/1769174528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd814ef60 con 0x7fddd806d820 2024-09-16T01:53:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddd7400640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 0x7fddd81f4520 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fddc8035040 tx=0x7fddc80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8037040 con 0x7fddd806d820 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc8030500 con 0x7fddd806d820 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.023+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc80308f0 con 0x7fddd806d820 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.024+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd81f05d0 con 0x7fddd806d820 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.024+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd81f0b70 con 0x7fddd806d820 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.025+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd80cad60 con 0x7fddd806d820 2024-09-16T01:53:10.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.026+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fddc8044020 con 0x7fddd806d820 2024-09-16T01:53:10.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.026+0000 7fddcf400640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddb806c330 0x7fddb806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.027+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5894+0+0 (secure 0 0 0) 0x7fddc80ab600 con 0x7fddd806d820 2024-09-16T01:53:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.027+0000 7fddd6a00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddb806c330 0x7fddb806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.027+0000 7fddd6a00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddb806c330 0x7fddb806e7f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009560 tx=0x7fddc4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.031+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fddd814ef60 con 0x7fddd806d820 2024-09-16T01:53:10.058 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:10.058 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:10.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T01:53:10.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.068+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f572c0c9200 con 0x7f572c0c6b70 2024-09-16T01:53:10.113 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:10.112+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:10.179 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-09-16T01:53:10.178+0000 7f5e7d5e2640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 37293) UID: 0 2024-09-16T01:53:10.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.223+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fddd80d0760 con 0x7fddd806d820 2024-09-16T01:53:10.228 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:10.228 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:10.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T01:53:10.280 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-09-16T01:53:10.279+0000 7f0c746cb640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 37297) UID: 0 2024-09-16T01:53:10.380 INFO:tasks.ceph.osd.1.smithi146.stderr:2024-09-16T01:53:10.379+0000 7fedd2800640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 36113) UID: 0 2024-09-16T01:53:10.442 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:10.443 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:10.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T01:53:10.499 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:10.497+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:10.543 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:10.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T01:53:10.584 INFO:tasks.ceph.osd.2.smithi146.stderr:2024-09-16T01:53:10.583+0000 7f2650600640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 36117) UID: 0 2024-09-16T01:53:10.647 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T01:53:10.647 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T01:53:10.661 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T01:53:10.662 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:10.662 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:10.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T01:53:10.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.809+0000 7fddcf400640 1 -- 172.21.15.146:0/1769174528 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v130) ==== 113+0+0 (secure 0 0 0) 0x7fddc8079640 con 0x7fddd806d820 2024-09-16T01:53:10.810 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T01:53:10.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.809+0000 7f5723400640 1 -- 172.21.15.146:0/3230673835 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v130) ==== 97+0+0 (secure 0 0 0) 0x7f571c05d9f0 con 0x7f572c0c6b70 2024-09-16T01:53:10.811 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is set 2024-09-16T01:53:10.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddb806c330 msgr2=0x7fddb806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddb806c330 0x7fddb806e7f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009560 tx=0x7fddc4009190 comp rx=0 tx=0).stop 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 msgr2=0x7fddd81f4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 0x7fddd81f4520 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fddc8035040 tx=0x7fddc80046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f570c06c2b0 msgr2=0x7f570c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f570c06c2b0 0x7f570c06e770 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f572c0cf170 tx=0x7f5718007e80 comp rx=0 tx=0).stop 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 msgr2=0x7f572c0c25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 0x7f572c0c25c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f571c00e990 tx=0x7f571c00ee50 comp rx=0 tx=0).stop 2024-09-16T01:53:10.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 shutdown_connections 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f570c06c2b0 0x7f570c06e770 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 shutdown_connections 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fddb806c330 0x7fddb806e7f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fddd8154ac0 0x7fddd80cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd80d0d00 0x7fddd814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 --2- 172.21.15.146:0/1769174528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fddd806d820 0x7fddd81f4520 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 >> 172.21.15.146:0/1769174528 conn(0x7fddd806e400 msgr2=0x7fddd80b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f572c0cc040 0x7f572c1f48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572c0c75a0 0x7f572c0c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 --2- 172.21.15.146:0/3230673835 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f572c0c6b70 0x7f572c0c25c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.811+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 >> 172.21.15.146:0/3230673835 conn(0x7f572c0ccc20 msgr2=0x7f572c15a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.812+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 shutdown_connections 2024-09-16T01:53:10.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.812+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 shutdown_connections 2024-09-16T01:53:10.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.812+0000 7f572be00640 1 -- 172.21.15.146:0/3230673835 wait complete. 2024-09-16T01:53:10.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:10.812+0000 7fdddcc00640 1 -- 172.21.15.146:0/1769174528 wait complete. 2024-09-16T01:53:10.830 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T01:53:10.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:10.881 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:10.881 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:10.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T01:53:11.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.030+0000 7f9217200640 1 Processor -- start 2024-09-16T01:53:11.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.030+0000 7f9217200640 1 -- start start 2024-09-16T01:53:11.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.031+0000 7f9217200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92100cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.031+0000 7f9217200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921015af00 con 0x7f92100c6290 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.031+0000 7f9217200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921015b1e0 con 0x7f921015aa70 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.031+0000 7f9217200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f921006dbd0 con 0x7f92100c5eb0 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9216800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f92100c6290 0x7f921006c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33886/0 (socket says 172.21.15.146:33886) 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9216800640 1 -- 172.21.15.146:0/3202107720 learned_addr learned my addr 172.21.15.146:0/3202107720 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9215e00640 1 --2- 172.21.15.146:0/3202107720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92100cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9214a00640 1 -- 172.21.15.146:0/3202107720 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299337741 0 0) 0x7f921015af00 con 0x7f92100c6290 2024-09-16T01:53:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9215e00640 1 -- 172.21.15.146:0/3202107720 >> v1:172.21.15.161:6789/0 conn(0x7f921015aa70 legacy=0x7f9210154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9215e00640 1 -- 172.21.15.146:0/3202107720 >> v1:172.21.15.146:6789/0 conn(0x7f92100c6290 legacy=0x7f921006c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.032+0000 7f9215e00640 1 -- 172.21.15.146:0/3202107720 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921015b4c0 con 0x7f92100c5eb0 2024-09-16T01:53:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.033+0000 7f9215e00640 1 --2- 172.21.15.146:0/3202107720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92100cac30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9204009750 tx=0x7f920402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ceed25a52c0d98 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.033+0000 7f9214a00640 1 -- 172.21.15.146:0/3202107720 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9204037070 con 0x7f92100c5eb0 2024-09-16T01:53:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.033+0000 7f9214a00640 1 -- 172.21.15.146:0/3202107720 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920402ae40 con 0x7f92100c5eb0 2024-09-16T01:53:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.033+0000 7f9214a00640 1 -- 172.21.15.146:0/3202107720 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9204032660 con 0x7f92100c5eb0 2024-09-16T01:53:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 -- 172.21.15.146:0/3202107720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 msgr2=0x7f92100cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 --2- 172.21.15.146:0/3202107720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92100cac30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9204009750 tx=0x7f920402a130 comp rx=0 tx=0).stop 2024-09-16T01:53:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 -- 172.21.15.146:0/3202107720 shutdown_connections 2024-09-16T01:53:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 --2- 172.21.15.146:0/3202107720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92100cac30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 -- 172.21.15.146:0/3202107720 >> 172.21.15.146:0/3202107720 conn(0x7f9210067b30 msgr2=0x7f9210067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 -- 172.21.15.146:0/3202107720 shutdown_connections 2024-09-16T01:53:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.034+0000 7f9217200640 1 -- 172.21.15.146:0/3202107720 wait complete. 2024-09-16T01:53:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.036+0000 7f9217200640 1 Processor -- start 2024-09-16T01:53:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.036+0000 7f9217200640 1 -- start start 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9217200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92101f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9217200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92100c6290 0x7f92100c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9217200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92100c74c0 0x7f921015e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9217200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9210155020 con 0x7f92100c6290 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9217200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f921006c370 con 0x7f92100c74c0 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9217200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f92101551a0 con 0x7f92100c5eb0 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9215e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92101f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9215e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92101f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43338/0 (socket says 172.21.15.146:43338) 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9215e00640 1 -- 172.21.15.146:0/3757046131 learned_addr learned my addr 172.21.15.146:0/3757046131 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9215400640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92100c6290 0x7f92100c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.037+0000 7f9216800640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92100c74c0 0x7f921015e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9215e00640 1 -- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92100c74c0 msgr2=0x7f921015e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9215e00640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92100c74c0 0x7f921015e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9215e00640 1 -- 172.21.15.146:0/3757046131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92100c6290 msgr2=0x7f92100c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9215e00640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92100c6290 0x7f92100c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9215e00640 1 -- 172.21.15.146:0/3757046131 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92100c8f30 con 0x7f92100c5eb0 2024-09-16T01:53:11.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9215e00640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92101f4bd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9204000c00 tx=0x7f92040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9204037040 con 0x7f92100c5eb0 2024-09-16T01:53:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9204004030 con 0x7f92100c5eb0 2024-09-16T01:53:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9204006e10 con 0x7f92100c5eb0 2024-09-16T01:53:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921014f800 con 0x7f92100c5eb0 2024-09-16T01:53:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.038+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92100c8c50 con 0x7f92100c5eb0 2024-09-16T01:53:11.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.040+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92100c56f0 con 0x7f92100c5eb0 2024-09-16T01:53:11.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.040+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f920403a040 con 0x7f92100c5eb0 2024-09-16T01:53:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.041+0000 7f920a000640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91f406c2b0 0x7f91f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.041+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(130..130 src has 1..130) ==== 5894+0+0 (secure 0 0 0) 0x7f92040abb80 con 0x7f92100c5eb0 2024-09-16T01:53:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.041+0000 7f9215400640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91f406c2b0 0x7f91f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.042+0000 7f9215400640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91f406c2b0 0x7f91f406e770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f921006d280 tx=0x7f9200009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.045+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9204042020 con 0x7f92100c5eb0 2024-09-16T01:53:11.099 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:11.099 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:11.112 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T01:53:11.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.229+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92100c68d0 con 0x7f91f406c2b0 2024-09-16T01:53:11.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.235+0000 7f920a000640 1 -- 172.21.15.146:0/3757046131 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254483 (secure 0 0 0) 0x7f92100c68d0 con 0x7f91f406c2b0 2024-09-16T01:53:11.239 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:11.240 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91f406c2b0 msgr2=0x7f91f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91f406c2b0 0x7f91f406e770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f921006d280 tx=0x7f9200009040 comp rx=0 tx=0).stop 2024-09-16T01:53:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 msgr2=0x7f92101f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92101f4bd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9204000c00 tx=0x7f92040046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 shutdown_connections 2024-09-16T01:53:11.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f91f406c2b0 0x7f91f406e770 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92100c74c0 0x7f921015e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92100c6290 0x7f92100c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 --2- 172.21.15.146:0/3757046131 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92100c5eb0 0x7f92101f4bd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 >> 172.21.15.146:0/3757046131 conn(0x7f9210067b30 msgr2=0x7f9210152670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 shutdown_connections 2024-09-16T01:53:11.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.239+0000 7f9217200640 1 -- 172.21.15.146:0/3757046131 wait complete. 2024-09-16T01:53:11.258 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T01:53:10.807318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4220448681,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":119,"num_read_kb":103,"num_write":8770,"num_write_kb":25168818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":9886222853,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9646,"ondisk_log_size":9646,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":857348,"kb_used_data":673852,"kb_used_omap":9,"kb_used_meta":182838,"kb_avail":565373692,"statfs":{"total":579820584960,"available":578942660608,"internally_reserved":0,"allocated":690024448,"data_stored":10886264508,"data_compressed":544301959,"data_compressed_allocated":680136704,"data_compressed_original":10881404928,"omap_allocated":9562,"internal_metadata":187226790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4877970032,"num_objects":-1163,"num_object_clones":0,"num_object_copies":-2326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1170,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.164107"},"pg_stats":[{"pgid":"3.3f","version":"125'120","reported_seq":418,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.513032+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:53:06.513032+0000","last_peered":"2024-09-16T01:53:06.513032+0000","last_clean":"2024-09-16T01:53:06.513032+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:53:06.513032+0000","last_undegraded":"2024-09-16T01:53:06.513032+0000","last_fullsized":"2024-09-16T01:53:06.513032+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966877+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:09.966877+0000","last_peered":"2024-09-16T01:53:09.966877+0000","last_clean":"2024-09-16T01:53:09.966877+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:09.966877+0000","last_undegraded":"2024-09-16T01:53:09.966877+0000","last_fullsized":"2024-09-16T01:53:09.966877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":648,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966676+0000","last_change":"2024-09-16T01:52:51.502920+0000","last_active":"2024-09-16T01:53:09.966676+0000","last_peered":"2024-09-16T01:53:09.966676+0000","last_clean":"2024-09-16T01:53:09.966676+0000","last_became_active":"2024-09-16T01:52:51.502433+0000","last_became_peered":"2024-09-16T01:52:51.502433+0000","last_unstale":"2024-09-16T01:53:09.966676+0000","last_undegraded":"2024-09-16T01:53:09.966676+0000","last_fullsized":"2024-09-16T01:53:09.966676+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"125'140","reported_seq":367,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648483+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:53:06.648483+0000","last_peered":"2024-09-16T01:53:06.648483+0000","last_clean":"2024-09-16T01:53:06.648483+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:53:06.648483+0000","last_undegraded":"2024-09-16T01:53:06.648483+0000","last_fullsized":"2024-09-16T01:53:06.648483+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"127'170","reported_seq":473,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.967092+0000","last_change":"2024-09-16T01:52:53.516521+0000","last_active":"2024-09-16T01:53:09.967092+0000","last_peered":"2024-09-16T01:53:09.967092+0000","last_clean":"2024-09-16T01:53:09.967092+0000","last_became_active":"2024-09-16T01:52:53.516052+0000","last_became_peered":"2024-09-16T01:52:53.516052+0000","last_unstale":"2024-09-16T01:53:09.967092+0000","last_undegraded":"2024-09-16T01:53:09.967092+0000","last_fullsized":"2024-09-16T01:53:09.967092+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"125'117","reported_seq":390,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.647898+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:53:06.647898+0000","last_peered":"2024-09-16T01:53:06.647898+0000","last_clean":"2024-09-16T01:53:06.647898+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:53:06.647898+0000","last_undegraded":"2024-09-16T01:53:06.647898+0000","last_fullsized":"2024-09-16T01:53:06.647898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"125'144","reported_seq":537,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.584662+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:53:06.584662+0000","last_peered":"2024-09-16T01:53:06.584662+0000","last_clean":"2024-09-16T01:53:06.584662+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:53:06.584662+0000","last_undegraded":"2024-09-16T01:53:06.584662+0000","last_fullsized":"2024-09-16T01:53:06.584662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":133,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":546,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966191+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:53:09.966191+0000","last_peered":"2024-09-16T01:53:09.966191+0000","last_clean":"2024-09-16T01:53:09.966191+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:53:09.966191+0000","last_undegraded":"2024-09-16T01:53:09.966191+0000","last_fullsized":"2024-09-16T01:53:09.966191+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"125'126","reported_seq":345,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.469510+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:06.469510+0000","last_peered":"2024-09-16T01:53:06.469510+0000","last_clean":"2024-09-16T01:53:06.469510+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:06.469510+0000","last_undegraded":"2024-09-16T01:53:06.469510+0000","last_fullsized":"2024-09-16T01:53:06.469510+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"125'134","reported_seq":459,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.647954+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:53:06.647954+0000","last_peered":"2024-09-16T01:53:06.647954+0000","last_clean":"2024-09-16T01:53:06.647954+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:53:06.647954+0000","last_undegraded":"2024-09-16T01:53:06.647954+0000","last_fullsized":"2024-09-16T01:53:06.647954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"125'150","reported_seq":498,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.541893+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:53:06.541893+0000","last_peered":"2024-09-16T01:53:06.541893+0000","last_clean":"2024-09-16T01:53:06.541893+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:53:06.541893+0000","last_undegraded":"2024-09-16T01:53:06.541893+0000","last_fullsized":"2024-09-16T01:53:06.541893+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"125'115","reported_seq":254,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.533684+0000","last_change":"2024-09-16T01:52:50.499570+0000","last_active":"2024-09-16T01:53:06.533684+0000","last_peered":"2024-09-16T01:53:06.533684+0000","last_clean":"2024-09-16T01:53:06.533684+0000","last_became_active":"2024-09-16T01:52:50.498997+0000","last_became_peered":"2024-09-16T01:52:50.498997+0000","last_unstale":"2024-09-16T01:53:06.533684+0000","last_undegraded":"2024-09-16T01:53:06.533684+0000","last_fullsized":"2024-09-16T01:53:06.533684+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":517,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802729+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:53:09.802729+0000","last_peered":"2024-09-16T01:53:09.802729+0000","last_clean":"2024-09-16T01:53:09.802729+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:53:09.802729+0000","last_undegraded":"2024-09-16T01:53:09.802729+0000","last_fullsized":"2024-09-16T01:53:09.802729+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":441,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802522+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:09.802522+0000","last_peered":"2024-09-16T01:53:09.802522+0000","last_clean":"2024-09-16T01:53:09.802522+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:09.802522+0000","last_undegraded":"2024-09-16T01:53:09.802522+0000","last_fullsized":"2024-09-16T01:53:09.802522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":499,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966150+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:53:09.966150+0000","last_peered":"2024-09-16T01:53:09.966150+0000","last_clean":"2024-09-16T01:53:09.966150+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:53:09.966150+0000","last_undegraded":"2024-09-16T01:53:09.966150+0000","last_fullsized":"2024-09-16T01:53:09.966150+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"125'126","reported_seq":344,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.550262+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:06.550262+0000","last_peered":"2024-09-16T01:53:06.550262+0000","last_clean":"2024-09-16T01:53:06.550262+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:06.550262+0000","last_undegraded":"2024-09-16T01:53:06.550262+0000","last_fullsized":"2024-09-16T01:53:06.550262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"125'117","reported_seq":432,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.532712+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:53:06.532712+0000","last_peered":"2024-09-16T01:53:06.532712+0000","last_clean":"2024-09-16T01:53:06.532712+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:53:06.532712+0000","last_undegraded":"2024-09-16T01:53:06.532712+0000","last_fullsized":"2024-09-16T01:53:06.532712+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"125'120","reported_seq":533,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648123+0000","last_change":"2024-09-16T01:52:49.497350+0000","last_active":"2024-09-16T01:53:06.648123+0000","last_peered":"2024-09-16T01:53:06.648123+0000","last_clean":"2024-09-16T01:53:06.648123+0000","last_became_active":"2024-09-16T01:52:49.496908+0000","last_became_peered":"2024-09-16T01:52:49.496908+0000","last_unstale":"2024-09-16T01:53:06.648123+0000","last_undegraded":"2024-09-16T01:53:06.648123+0000","last_fullsized":"2024-09-16T01:53:06.648123+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:51.945635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":658,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802879+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:53:09.802879+0000","last_peered":"2024-09-16T01:53:09.802879+0000","last_clean":"2024-09-16T01:53:09.802879+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:53:09.802879+0000","last_undegraded":"2024-09-16T01:53:09.802879+0000","last_fullsized":"2024-09-16T01:53:09.802879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"125'130","reported_seq":492,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.461960+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:53:06.461960+0000","last_peered":"2024-09-16T01:53:06.461960+0000","last_clean":"2024-09-16T01:53:06.461960+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:53:06.461960+0000","last_undegraded":"2024-09-16T01:53:06.461960+0000","last_fullsized":"2024-09-16T01:53:06.461960+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"125'103","reported_seq":361,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.654012+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:53:06.654012+0000","last_peered":"2024-09-16T01:53:06.654012+0000","last_clean":"2024-09-16T01:53:06.654012+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:53:06.654012+0000","last_undegraded":"2024-09-16T01:53:06.654012+0000","last_fullsized":"2024-09-16T01:53:06.654012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"125'145","reported_seq":610,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.570636+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:53:06.570636+0000","last_peered":"2024-09-16T01:53:06.570636+0000","last_clean":"2024-09-16T01:53:06.570636+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:53:06.570636+0000","last_undegraded":"2024-09-16T01:53:06.570636+0000","last_fullsized":"2024-09-16T01:53:06.570636+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":368,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966370+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:09.966370+0000","last_peered":"2024-09-16T01:53:09.966370+0000","last_clean":"2024-09-16T01:53:09.966370+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:09.966370+0000","last_undegraded":"2024-09-16T01:53:09.966370+0000","last_fullsized":"2024-09-16T01:53:09.966370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"125'152","reported_seq":370,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.567970+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:06.567970+0000","last_peered":"2024-09-16T01:53:06.567970+0000","last_clean":"2024-09-16T01:53:06.567970+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:06.567970+0000","last_undegraded":"2024-09-16T01:53:06.567970+0000","last_fullsized":"2024-09-16T01:53:06.567970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":145,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966682+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:09.966682+0000","last_peered":"2024-09-16T01:53:09.966682+0000","last_clean":"2024-09-16T01:53:09.966682+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:09.966682+0000","last_undegraded":"2024-09-16T01:53:09.966682+0000","last_fullsized":"2024-09-16T01:53:09.966682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"126'156","reported_seq":377,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648273+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:06.648273+0000","last_peered":"2024-09-16T01:53:06.648273+0000","last_clean":"2024-09-16T01:53:06.648273+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:06.648273+0000","last_undegraded":"2024-09-16T01:53:06.648273+0000","last_fullsized":"2024-09-16T01:53:06.648273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":149,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":617,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803203+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:09.803203+0000","last_peered":"2024-09-16T01:53:09.803203+0000","last_clean":"2024-09-16T01:53:09.803203+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:09.803203+0000","last_undegraded":"2024-09-16T01:53:09.803203+0000","last_fullsized":"2024-09-16T01:53:09.803203+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":468,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802871+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:53:09.802871+0000","last_peered":"2024-09-16T01:53:09.802871+0000","last_clean":"2024-09-16T01:53:09.802871+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:53:09.802871+0000","last_undegraded":"2024-09-16T01:53:09.802871+0000","last_fullsized":"2024-09-16T01:53:09.802871+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":637,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966944+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:09.966944+0000","last_peered":"2024-09-16T01:53:09.966944+0000","last_clean":"2024-09-16T01:53:09.966944+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:09.966944+0000","last_undegraded":"2024-09-16T01:53:09.966944+0000","last_fullsized":"2024-09-16T01:53:09.966944+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"125'110","reported_seq":328,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.480755+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:06.480755+0000","last_peered":"2024-09-16T01:53:06.480755+0000","last_clean":"2024-09-16T01:53:06.480755+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:06.480755+0000","last_undegraded":"2024-09-16T01:53:06.480755+0000","last_fullsized":"2024-09-16T01:53:06.480755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"125'117","reported_seq":528,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.538723+0000","last_change":"2024-09-16T01:52:47.475996+0000","last_active":"2024-09-16T01:53:06.538723+0000","last_peered":"2024-09-16T01:53:06.538723+0000","last_clean":"2024-09-16T01:53:06.538723+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:53:06.538723+0000","last_undegraded":"2024-09-16T01:53:06.538723+0000","last_fullsized":"2024-09-16T01:53:06.538723+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:05.418025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":511,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802184+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:53:09.802184+0000","last_peered":"2024-09-16T01:53:09.802184+0000","last_clean":"2024-09-16T01:53:09.802184+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:53:09.802184+0000","last_undegraded":"2024-09-16T01:53:09.802184+0000","last_fullsized":"2024-09-16T01:53:09.802184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"125'115","reported_seq":334,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.587077+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:06.587077+0000","last_peered":"2024-09-16T01:53:06.587077+0000","last_clean":"2024-09-16T01:53:06.587077+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:06.587077+0000","last_undegraded":"2024-09-16T01:53:06.587077+0000","last_fullsized":"2024-09-16T01:53:06.587077+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":233,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966999+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:09.966999+0000","last_peered":"2024-09-16T01:53:09.966999+0000","last_clean":"2024-09-16T01:53:09.966999+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:09.966999+0000","last_undegraded":"2024-09-16T01:53:09.966999+0000","last_fullsized":"2024-09-16T01:53:09.966999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"125'117","reported_seq":668,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.582183+0000","last_change":"2024-09-16T01:52:52.506119+0000","last_active":"2024-09-16T01:53:06.582183+0000","last_peered":"2024-09-16T01:53:06.582183+0000","last_clean":"2024-09-16T01:53:06.582183+0000","last_became_active":"2024-09-16T01:52:52.505498+0000","last_became_peered":"2024-09-16T01:52:52.505498+0000","last_unstale":"2024-09-16T01:53:06.582183+0000","last_undegraded":"2024-09-16T01:53:06.582183+0000","last_fullsized":"2024-09-16T01:53:06.582183+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:00.694119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":240,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967051+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:53:05.967051+0000","last_peered":"2024-09-16T01:53:05.967051+0000","last_clean":"2024-09-16T01:53:05.967051+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:53:05.967051+0000","last_undegraded":"2024-09-16T01:53:05.967051+0000","last_fullsized":"2024-09-16T01:53:05.967051+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802374+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:53:09.802374+0000","last_peered":"2024-09-16T01:53:09.802374+0000","last_clean":"2024-09-16T01:53:09.802374+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:53:09.802374+0000","last_undegraded":"2024-09-16T01:53:09.802374+0000","last_fullsized":"2024-09-16T01:53:09.802374+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":408,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966391+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:09.966391+0000","last_peered":"2024-09-16T01:53:09.966391+0000","last_clean":"2024-09-16T01:53:09.966391+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:09.966391+0000","last_undegraded":"2024-09-16T01:53:09.966391+0000","last_fullsized":"2024-09-16T01:53:09.966391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":223,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966489+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:05.966489+0000","last_peered":"2024-09-16T01:53:05.966489+0000","last_clean":"2024-09-16T01:53:05.966489+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:05.966489+0000","last_undegraded":"2024-09-16T01:53:05.966489+0000","last_fullsized":"2024-09-16T01:53:05.966489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"125'130","reported_seq":314,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.536902+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:53:06.536902+0000","last_peered":"2024-09-16T01:53:06.536902+0000","last_clean":"2024-09-16T01:53:06.536902+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:53:06.536902+0000","last_undegraded":"2024-09-16T01:53:06.536902+0000","last_fullsized":"2024-09-16T01:53:06.536902+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":252,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967743+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:53:05.967743+0000","last_peered":"2024-09-16T01:53:05.967743+0000","last_clean":"2024-09-16T01:53:05.967743+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:53:05.967743+0000","last_undegraded":"2024-09-16T01:53:05.967743+0000","last_fullsized":"2024-09-16T01:53:05.967743+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":481,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966553+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:09.966553+0000","last_peered":"2024-09-16T01:53:09.966553+0000","last_clean":"2024-09-16T01:53:09.966553+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:09.966553+0000","last_undegraded":"2024-09-16T01:53:09.966553+0000","last_fullsized":"2024-09-16T01:53:09.966553+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966831+0000","last_change":"2024-09-16T01:52:46.621488+0000","last_active":"2024-09-16T01:53:05.966831+0000","last_peered":"2024-09-16T01:53:05.966831+0000","last_clean":"2024-09-16T01:53:05.966831+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:53:05.966831+0000","last_undegraded":"2024-09-16T01:53:05.966831+0000","last_fullsized":"2024-09-16T01:53:05.966831+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966895+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:09.966895+0000","last_peered":"2024-09-16T01:53:09.966895+0000","last_clean":"2024-09-16T01:53:09.966895+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:09.966895+0000","last_undegraded":"2024-09-16T01:53:09.966895+0000","last_fullsized":"2024-09-16T01:53:09.966895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966535+0000","last_change":"2024-09-16T01:52:46.632501+0000","last_active":"2024-09-16T01:53:05.966535+0000","last_peered":"2024-09-16T01:53:05.966535+0000","last_clean":"2024-09-16T01:53:05.966535+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:53:05.966535+0000","last_undegraded":"2024-09-16T01:53:05.966535+0000","last_fullsized":"2024-09-16T01:53:05.966535+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"125'121","reported_seq":339,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.565619+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:06.565619+0000","last_peered":"2024-09-16T01:53:06.565619+0000","last_clean":"2024-09-16T01:53:06.565619+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:06.565619+0000","last_undegraded":"2024-09-16T01:53:06.565619+0000","last_fullsized":"2024-09-16T01:53:06.565619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"123'170","reported_seq":414,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648602+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:53:06.648602+0000","last_peered":"2024-09-16T01:53:06.648602+0000","last_clean":"2024-09-16T01:53:06.648602+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:53:06.648602+0000","last_undegraded":"2024-09-16T01:53:06.648602+0000","last_fullsized":"2024-09-16T01:53:06.648602+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381505,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"127'170","reported_seq":500,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803045+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:53:09.803045+0000","last_peered":"2024-09-16T01:53:09.803045+0000","last_clean":"2024-09-16T01:53:09.803045+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:53:09.803045+0000","last_undegraded":"2024-09-16T01:53:09.803045+0000","last_fullsized":"2024-09-16T01:53:09.803045+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":299,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966055+0000","last_change":"2024-09-16T01:52:46.615925+0000","last_active":"2024-09-16T01:53:05.966055+0000","last_peered":"2024-09-16T01:53:05.966055+0000","last_clean":"2024-09-16T01:53:05.966055+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:53:05.966055+0000","last_undegraded":"2024-09-16T01:53:05.966055+0000","last_fullsized":"2024-09-16T01:53:05.966055+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":340,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":395,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802526+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:53:09.802526+0000","last_peered":"2024-09-16T01:53:09.802526+0000","last_clean":"2024-09-16T01:53:09.802526+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:53:09.802526+0000","last_undegraded":"2024-09-16T01:53:09.802526+0000","last_fullsized":"2024-09-16T01:53:09.802526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":303,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967144+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:05.967144+0000","last_peered":"2024-09-16T01:53:05.967144+0000","last_clean":"2024-09-16T01:53:05.967144+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:05.967144+0000","last_undegraded":"2024-09-16T01:53:05.967144+0000","last_fullsized":"2024-09-16T01:53:05.967144+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"125'141","reported_seq":490,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.588752+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:53:06.588752+0000","last_peered":"2024-09-16T01:53:06.588752+0000","last_clean":"2024-09-16T01:53:06.588752+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:53:06.588752+0000","last_undegraded":"2024-09-16T01:53:06.588752+0000","last_fullsized":"2024-09-16T01:53:06.588752+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":97,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966243+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:53:05.966243+0000","last_peered":"2024-09-16T01:53:05.966243+0000","last_clean":"2024-09-16T01:53:05.966243+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:53:05.966243+0000","last_undegraded":"2024-09-16T01:53:05.966243+0000","last_fullsized":"2024-09-16T01:53:05.966243+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"125'153","reported_seq":372,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.570769+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:06.570769+0000","last_peered":"2024-09-16T01:53:06.570769+0000","last_clean":"2024-09-16T01:53:06.570769+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:06.570769+0000","last_undegraded":"2024-09-16T01:53:06.570769+0000","last_fullsized":"2024-09-16T01:53:06.570769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966313+0000","last_change":"2024-09-16T01:52:46.599623+0000","last_active":"2024-09-16T01:53:05.966313+0000","last_peered":"2024-09-16T01:53:05.966313+0000","last_clean":"2024-09-16T01:53:05.966313+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:53:05.966313+0000","last_undegraded":"2024-09-16T01:53:05.966313+0000","last_fullsized":"2024-09-16T01:53:05.966313+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:22.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":784,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"125'159","reported_seq":377,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.522182+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:06.522182+0000","last_peered":"2024-09-16T01:53:06.522182+0000","last_clean":"2024-09-16T01:53:06.522182+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:06.522182+0000","last_undegraded":"2024-09-16T01:53:06.522182+0000","last_fullsized":"2024-09-16T01:53:06.522182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":151,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":101,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966552+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:53:09.966552+0000","last_peered":"2024-09-16T01:53:09.966552+0000","last_clean":"2024-09-16T01:53:09.966552+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:53:09.966552+0000","last_undegraded":"2024-09-16T01:53:09.966552+0000","last_fullsized":"2024-09-16T01:53:09.966552+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"125'119","reported_seq":441,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648329+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:53:06.648329+0000","last_peered":"2024-09-16T01:53:06.648329+0000","last_clean":"2024-09-16T01:53:06.648329+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:53:06.648329+0000","last_undegraded":"2024-09-16T01:53:06.648329+0000","last_fullsized":"2024-09-16T01:53:06.648329+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":122,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966938+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:53:09.966938+0000","last_peered":"2024-09-16T01:53:09.966938+0000","last_clean":"2024-09-16T01:53:09.966938+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:53:09.966938+0000","last_undegraded":"2024-09-16T01:53:09.966938+0000","last_fullsized":"2024-09-16T01:53:09.966938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"125'112","reported_seq":427,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.504636+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:53:06.504636+0000","last_peered":"2024-09-16T01:53:06.504636+0000","last_clean":"2024-09-16T01:53:06.504636+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:53:06.504636+0000","last_undegraded":"2024-09-16T01:53:06.504636+0000","last_fullsized":"2024-09-16T01:53:06.504636+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":231,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966729+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:09.966729+0000","last_peered":"2024-09-16T01:53:09.966729+0000","last_clean":"2024-09-16T01:53:09.966729+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:09.966729+0000","last_undegraded":"2024-09-16T01:53:09.966729+0000","last_fullsized":"2024-09-16T01:53:09.966729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"125'116","reported_seq":662,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.574039+0000","last_change":"2024-09-16T01:52:50.499005+0000","last_active":"2024-09-16T01:53:06.574039+0000","last_peered":"2024-09-16T01:53:06.574039+0000","last_clean":"2024-09-16T01:53:06.574039+0000","last_became_active":"2024-09-16T01:52:50.498634+0000","last_became_peered":"2024-09-16T01:52:50.498634+0000","last_unstale":"2024-09-16T01:53:06.574039+0000","last_undegraded":"2024-09-16T01:53:06.574039+0000","last_fullsized":"2024-09-16T01:53:06.574039+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":247,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803062+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:53:09.803062+0000","last_peered":"2024-09-16T01:53:09.803062+0000","last_clean":"2024-09-16T01:53:09.803062+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:53:09.803062+0000","last_undegraded":"2024-09-16T01:53:09.803062+0000","last_fullsized":"2024-09-16T01:53:09.803062+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"125'128","reported_seq":346,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.503811+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:06.503811+0000","last_peered":"2024-09-16T01:53:06.503811+0000","last_clean":"2024-09-16T01:53:06.503811+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:06.503811+0000","last_undegraded":"2024-09-16T01:53:06.503811+0000","last_fullsized":"2024-09-16T01:53:06.503811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":378,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967363+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:05.967363+0000","last_peered":"2024-09-16T01:53:05.967363+0000","last_clean":"2024-09-16T01:53:05.967363+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:05.967363+0000","last_undegraded":"2024-09-16T01:53:05.967363+0000","last_fullsized":"2024-09-16T01:53:05.967363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":226,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.965877+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:05.965877+0000","last_peered":"2024-09-16T01:53:05.965877+0000","last_clean":"2024-09-16T01:53:05.965877+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:05.965877+0000","last_undegraded":"2024-09-16T01:53:05.965877+0000","last_fullsized":"2024-09-16T01:53:05.965877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"125'120","reported_seq":481,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.647781+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:53:06.647781+0000","last_peered":"2024-09-16T01:53:06.647781+0000","last_clean":"2024-09-16T01:53:06.647781+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:53:06.647781+0000","last_undegraded":"2024-09-16T01:53:06.647781+0000","last_fullsized":"2024-09-16T01:53:06.647781+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":225,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966606+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:05.966606+0000","last_peered":"2024-09-16T01:53:05.966606+0000","last_clean":"2024-09-16T01:53:05.966606+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:05.966606+0000","last_undegraded":"2024-09-16T01:53:05.966606+0000","last_fullsized":"2024-09-16T01:53:05.966606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802302+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:53:09.802302+0000","last_peered":"2024-09-16T01:53:09.802302+0000","last_clean":"2024-09-16T01:53:09.802302+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:53:09.802302+0000","last_undegraded":"2024-09-16T01:53:09.802302+0000","last_fullsized":"2024-09-16T01:53:09.802302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"125'10","reported_seq":291,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966269+0000","last_change":"2024-09-16T01:52:46.569921+0000","last_active":"2024-09-16T01:53:09.966269+0000","last_peered":"2024-09-16T01:53:09.966269+0000","last_clean":"2024-09-16T01:53:09.966269+0000","last_became_active":"2024-09-16T01:52:46.544102+0000","last_became_peered":"2024-09-16T01:52:46.544102+0000","last_unstale":"2024-09-16T01:53:09.966269+0000","last_undegraded":"2024-09-16T01:53:09.966269+0000","last_fullsized":"2024-09-16T01:53:09.966269+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:14.446115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":601,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966294+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:09.966294+0000","last_peered":"2024-09-16T01:53:09.966294+0000","last_clean":"2024-09-16T01:53:09.966294+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:09.966294+0000","last_undegraded":"2024-09-16T01:53:09.966294+0000","last_fullsized":"2024-09-16T01:53:09.966294+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":225,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.965877+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:05.965877+0000","last_peered":"2024-09-16T01:53:05.965877+0000","last_clean":"2024-09-16T01:53:05.965877+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:05.965877+0000","last_undegraded":"2024-09-16T01:53:05.965877+0000","last_fullsized":"2024-09-16T01:53:05.965877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"125'125","reported_seq":445,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.653796+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:53:06.653796+0000","last_peered":"2024-09-16T01:53:06.653796+0000","last_clean":"2024-09-16T01:53:06.653796+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:53:06.653796+0000","last_undegraded":"2024-09-16T01:53:06.653796+0000","last_fullsized":"2024-09-16T01:53:06.653796+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":230,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966737+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:09.966737+0000","last_peered":"2024-09-16T01:53:09.966737+0000","last_clean":"2024-09-16T01:53:09.966737+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:09.966737+0000","last_undegraded":"2024-09-16T01:53:09.966737+0000","last_fullsized":"2024-09-16T01:53:09.966737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"125'122","reported_seq":343,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648907+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:06.648907+0000","last_peered":"2024-09-16T01:53:06.648907+0000","last_clean":"2024-09-16T01:53:06.648907+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:06.648907+0000","last_undegraded":"2024-09-16T01:53:06.648907+0000","last_fullsized":"2024-09-16T01:53:06.648907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":54,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648714+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:53:06.648714+0000","last_peered":"2024-09-16T01:53:06.648714+0000","last_clean":"2024-09-16T01:53:06.648714+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:53:06.648714+0000","last_undegraded":"2024-09-16T01:53:06.648714+0000","last_fullsized":"2024-09-16T01:53:06.648714+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803012+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:53:09.803012+0000","last_peered":"2024-09-16T01:53:09.803012+0000","last_clean":"2024-09-16T01:53:09.803012+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:53:09.803012+0000","last_undegraded":"2024-09-16T01:53:09.803012+0000","last_fullsized":"2024-09-16T01:53:09.803012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"128'21","reported_seq":230,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966728+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:53:09.966728+0000","last_peered":"2024-09-16T01:53:09.966728+0000","last_clean":"2024-09-16T01:53:09.966728+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:53:09.966728+0000","last_undegraded":"2024-09-16T01:53:09.966728+0000","last_fullsized":"2024-09-16T01:53:09.966728+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":433,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803097+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:53:09.803097+0000","last_peered":"2024-09-16T01:53:09.803097+0000","last_clean":"2024-09-16T01:53:09.803097+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:53:09.803097+0000","last_undegraded":"2024-09-16T01:53:09.803097+0000","last_fullsized":"2024-09-16T01:53:09.803097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":242,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967226+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:53:05.967226+0000","last_peered":"2024-09-16T01:53:05.967226+0000","last_clean":"2024-09-16T01:53:05.967226+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:53:05.967226+0000","last_undegraded":"2024-09-16T01:53:05.967226+0000","last_fullsized":"2024-09-16T01:53:05.967226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"125'131","reported_seq":440,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.427408+0000","last_change":"2024-09-16T01:52:52.504293+0000","last_active":"2024-09-16T01:53:06.427408+0000","last_peered":"2024-09-16T01:53:06.427408+0000","last_clean":"2024-09-16T01:53:06.427408+0000","last_became_active":"2024-09-16T01:52:52.503886+0000","last_became_peered":"2024-09-16T01:52:52.503886+0000","last_unstale":"2024-09-16T01:53:06.427408+0000","last_undegraded":"2024-09-16T01:53:06.427408+0000","last_fullsized":"2024-09-16T01:53:06.427408+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:29.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":108,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648339+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:53:06.648339+0000","last_peered":"2024-09-16T01:53:06.648339+0000","last_clean":"2024-09-16T01:53:06.648339+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:53:06.648339+0000","last_undegraded":"2024-09-16T01:53:06.648339+0000","last_fullsized":"2024-09-16T01:53:06.648339+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"125'125","reported_seq":398,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.583343+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:53:06.583343+0000","last_peered":"2024-09-16T01:53:06.583343+0000","last_clean":"2024-09-16T01:53:06.583343+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:53:06.583343+0000","last_undegraded":"2024-09-16T01:53:06.583343+0000","last_fullsized":"2024-09-16T01:53:06.583343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":109,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802229+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:53:09.802229+0000","last_peered":"2024-09-16T01:53:09.802229+0000","last_clean":"2024-09-16T01:53:09.802229+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:53:09.802229+0000","last_undegraded":"2024-09-16T01:53:09.802229+0000","last_fullsized":"2024-09-16T01:53:09.802229+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"125'128","reported_seq":404,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648243+0000","last_change":"2024-09-16T01:52:46.782699+0000","last_active":"2024-09-16T01:53:06.648243+0000","last_peered":"2024-09-16T01:53:06.648243+0000","last_clean":"2024-09-16T01:53:06.648243+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:53:06.648243+0000","last_undegraded":"2024-09-16T01:53:06.648243+0000","last_fullsized":"2024-09-16T01:53:06.648243+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:30.580835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":232,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802620+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:53:09.802620+0000","last_peered":"2024-09-16T01:53:09.802620+0000","last_clean":"2024-09-16T01:53:09.802620+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:53:09.802620+0000","last_undegraded":"2024-09-16T01:53:09.802620+0000","last_fullsized":"2024-09-16T01:53:09.802620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":593,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802593+0000","last_change":"2024-09-16T01:52:51.501914+0000","last_active":"2024-09-16T01:53:09.802593+0000","last_peered":"2024-09-16T01:53:09.802593+0000","last_clean":"2024-09-16T01:53:09.802593+0000","last_became_active":"2024-09-16T01:52:51.501494+0000","last_became_peered":"2024-09-16T01:52:51.501494+0000","last_unstale":"2024-09-16T01:53:09.802593+0000","last_undegraded":"2024-09-16T01:53:09.802593+0000","last_fullsized":"2024-09-16T01:53:09.802593+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:29.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":225,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.965928+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:05.965928+0000","last_peered":"2024-09-16T01:53:05.965928+0000","last_clean":"2024-09-16T01:53:05.965928+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:05.965928+0000","last_undegraded":"2024-09-16T01:53:05.965928+0000","last_fullsized":"2024-09-16T01:53:05.965928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"125'105","reported_seq":323,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.535644+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:06.535644+0000","last_peered":"2024-09-16T01:53:06.535644+0000","last_clean":"2024-09-16T01:53:06.535644+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:06.535644+0000","last_undegraded":"2024-09-16T01:53:06.535644+0000","last_fullsized":"2024-09-16T01:53:06.535644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"125'138","reported_seq":475,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.389369+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:53:06.389369+0000","last_peered":"2024-09-16T01:53:06.389369+0000","last_clean":"2024-09-16T01:53:06.389369+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:53:06.389369+0000","last_undegraded":"2024-09-16T01:53:06.389369+0000","last_fullsized":"2024-09-16T01:53:06.389369+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":94,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967312+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:53:05.967312+0000","last_peered":"2024-09-16T01:53:05.967312+0000","last_clean":"2024-09-16T01:53:05.967312+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:53:05.967312+0000","last_undegraded":"2024-09-16T01:53:05.967312+0000","last_fullsized":"2024-09-16T01:53:05.967312+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"125'118","reported_seq":432,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.412329+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:53:06.412329+0000","last_peered":"2024-09-16T01:53:06.412329+0000","last_clean":"2024-09-16T01:53:06.412329+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:53:06.412329+0000","last_undegraded":"2024-09-16T01:53:06.412329+0000","last_fullsized":"2024-09-16T01:53:06.412329+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":62,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967426+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:53:05.967426+0000","last_peered":"2024-09-16T01:53:05.967426+0000","last_clean":"2024-09-16T01:53:05.967426+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:53:05.967426+0000","last_undegraded":"2024-09-16T01:53:05.967426+0000","last_fullsized":"2024-09-16T01:53:05.967426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966833+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:09.966833+0000","last_peered":"2024-09-16T01:53:09.966833+0000","last_clean":"2024-09-16T01:53:09.966833+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:09.966833+0000","last_undegraded":"2024-09-16T01:53:09.966833+0000","last_fullsized":"2024-09-16T01:53:09.966833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.967035+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:09.967035+0000","last_peered":"2024-09-16T01:53:09.967035+0000","last_clean":"2024-09-16T01:53:09.967035+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:09.967035+0000","last_undegraded":"2024-09-16T01:53:09.967035+0000","last_fullsized":"2024-09-16T01:53:09.967035+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":403,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966931+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:09.966931+0000","last_peered":"2024-09-16T01:53:09.966931+0000","last_clean":"2024-09-16T01:53:09.966931+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:09.966931+0000","last_undegraded":"2024-09-16T01:53:09.966931+0000","last_fullsized":"2024-09-16T01:53:09.966931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966921+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:53:05.966921+0000","last_peered":"2024-09-16T01:53:05.966921+0000","last_clean":"2024-09-16T01:53:05.966921+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:53:05.966921+0000","last_undegraded":"2024-09-16T01:53:05.966921+0000","last_fullsized":"2024-09-16T01:53:05.966921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":4219469824,"num_objects":1006,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":559415296,"data_stored":8950644736,"data_compressed":445052032,"data_compressed_allocated":559415296,"data_compressed_original":8950644736,"omap_allocated":0,"internal_metadata":0},"log_size":9163,"ondisk_log_size":9163,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":388489,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":401,"num_write_kb":578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":248325,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118208,"data_stored":1006188,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509496,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323756,"kb_used_data":292568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048084,"statfs":{"total":96636764160,"available":96305238016,"internally_reserved":0,"allocated":299589632,"data_stored":4769624253,"data_compressed":238533504,"data_compressed_allocated":298057728,"data_compressed_original":4768923648,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509496,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235292,"kb_used_data":203844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94136548,"statfs":{"total":96636764160,"available":96395825152,"internally_reserved":0,"allocated":208736256,"data_stored":3310391576,"data_compressed":165525888,"data_compressed_allocated":206831616,"data_compressed_original":3309305856,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509495,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200040,"kb_used_data":172752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94171800,"statfs":{"total":96636764160,"available":96431923200,"internally_reserved":0,"allocated":176898048,"data_stored":2803471009,"data_compressed":140210147,"data_compressed_allocated":175202304,"data_compressed_original":2802601984,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509496,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1576960,"data_stored":728281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542200,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":729708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1642496,"data_stored":1319681,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574904,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":383053,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":383204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130809856,"data_stored":2092957696,"data_compressed":104686208,"data_compressed_allocated":130809856,"data_compressed_original":2092957696,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171704320,"data_stored":2747269120,"data_compressed":137413760,"data_compressed_allocated":171704320,"data_compressed_original":2747269120,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":197394432,"data_stored":3158310912,"data_compressed":157973376,"data_compressed_allocated":197394432,"data_compressed_original":3158310912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:11.262 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:11.275 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:11.275 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:11.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T01:53:11.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.430+0000 7f409d000640 1 Processor -- start 2024-09-16T01:53:11.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.430+0000 7f409d000640 1 -- start start 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f409d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40980c6bb0 0x7f40980c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f409d000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f40980c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f409d000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4098072f50 con 0x7f4098152850 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f409d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4098073210 con 0x7f40980c6bb0 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f409d000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f409815cb50 con 0x7f40980c58c0 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f4095e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4098152850 0x7f4098158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:55436/0 (socket says 172.21.15.146:55436) 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f4095e00640 1 -- 172.21.15.146:0/3656642227 learned_addr learned my addr 172.21.15.146:0/3656642227 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f408d400640 1 --2- 172.21.15.146:0/3656642227 >> v2:172.21.15.146:3300/0 conn(0x7f40980c6bb0 0x7f40980c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f4096800640 1 --2- 172.21.15.146:0/3656642227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f40980c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f4095400640 1 -- 172.21.15.146:0/3656642227 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992904981 0 0) 0x7f4098072f50 con 0x7f4098152850 2024-09-16T01:53:11.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f4095400640 1 -- 172.21.15.146:0/3656642227 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f407c003520 con 0x7f4098152850 2024-09-16T01:53:11.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f408d400640 1 -- 172.21.15.146:0/3656642227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 msgr2=0x7f40980c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:11.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f408d400640 1 --2- 172.21.15.146:0/3656642227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f40980c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f408d400640 1 -- 172.21.15.146:0/3656642227 >> v1:172.21.15.161:6789/0 conn(0x7f4098152850 legacy=0x7f4098158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.432+0000 7f408d400640 1 -- 172.21.15.146:0/3656642227 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409815bd90 con 0x7f40980c6bb0 2024-09-16T01:53:11.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.433+0000 7f408d400640 1 --2- 172.21.15.146:0/3656642227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f40980c9700 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f40800097a0 tx=0x7f408002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5900f2f5b69b7f01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f4095400640 1 -- 172.21.15.146:0/3656642227 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080037070 con 0x7f40980c6bb0 2024-09-16T01:53:11.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f4095400640 1 -- 172.21.15.146:0/3656642227 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4080032440 con 0x7f40980c6bb0 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f4095400640 1 -- 172.21.15.146:0/3656642227 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4080032810 con 0x7f40980c6bb0 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 -- 172.21.15.146:0/3656642227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 msgr2=0x7f40980c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 --2- 172.21.15.146:0/3656642227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f40980c9700 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f40800097a0 tx=0x7f408002a180 comp rx=0 tx=0).stop 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 -- 172.21.15.146:0/3656642227 shutdown_connections 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 --2- 172.21.15.146:0/3656642227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f40980c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 --2- 172.21.15.146:0/3656642227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f40980c9700 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 -- 172.21.15.146:0/3656642227 >> 172.21.15.146:0/3656642227 conn(0x7f409806f3c0 msgr2=0x7f409806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 -- 172.21.15.146:0/3656642227 shutdown_connections 2024-09-16T01:53:11.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.434+0000 7f409d000640 1 -- 172.21.15.146:0/3656642227 wait complete. 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.436+0000 7f409d000640 1 Processor -- start 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.436+0000 7f409d000640 1 -- start start 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f409d000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f4098147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f409d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f409814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f409d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4098152850 0x7f409814d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f409d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f409815c630 con 0x7f40980c6bb0 2024-09-16T01:53:11.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f409d000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f409815c350 con 0x7f4098152850 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f409d000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4098070200 con 0x7f40980c58c0 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f4095e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f4098147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f4096800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4098152850 0x7f409814d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f4095e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f4098147640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43346/0 (socket says 172.21.15.146:43346) 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f408d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f409814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.437+0000 7f4095e00640 1 -- 172.21.15.146:0/3758566682 learned_addr learned my addr 172.21.15.146:0/3758566682 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f4095e00640 1 -- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4098152850 msgr2=0x7f409814d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f4095e00640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4098152850 0x7f409814d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f4095e00640 1 -- 172.21.15.146:0/3758566682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 msgr2=0x7f409814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f4095e00640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f409814caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f4095e00640 1 -- 172.21.15.146:0/3758566682 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40980c3930 con 0x7f40980c58c0 2024-09-16T01:53:11.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f4095e00640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f4098147640 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f408800c3c0 tx=0x7f408800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088017070 con 0x7f40980c58c0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4088005ce0 con 0x7f40980c58c0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40980ca0e0 con 0x7f40980c58c0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880072f0 con 0x7f40980c58c0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.438+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40980ca6a0 con 0x7f40980c58c0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.439+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40980718a0 con 0x7f40980c58c0 2024-09-16T01:53:11.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.441+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f408801a070 con 0x7f40980c58c0 2024-09-16T01:53:11.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.441+0000 7f408ea00640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f407406c2e0 0x7f407406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.442+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(130..130 src has 1..130) ==== 5894+0+0 (secure 0 0 0) 0x7f408808a840 con 0x7f40980c58c0 2024-09-16T01:53:11.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.442+0000 7f408d400640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f407406c2e0 0x7f407406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.442+0000 7f408d400640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f407406c2e0 0x7f407406e7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4080034000 tx=0x7f408002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.446+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4088058800 con 0x7f40980c58c0 2024-09-16T01:53:11.491 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:11.491 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:11.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T01:53:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.630+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4098150fe0 con 0x7f407406c2e0 2024-09-16T01:53:11.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.636+0000 7f408ea00640 1 -- 172.21.15.146:0/3758566682 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254483 (secure 0 0 0) 0x7f4098150fe0 con 0x7f407406c2e0 2024-09-16T01:53:11.641 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f407406c2e0 msgr2=0x7f407406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f407406c2e0 0x7f407406e7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4080034000 tx=0x7f408002e000 comp rx=0 tx=0).stop 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 msgr2=0x7f4098147640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f4098147640 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f408800c3c0 tx=0x7f408800c880 comp rx=0 tx=0).stop 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 shutdown_connections 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f407406c2e0 0x7f407406e7a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4098152850 0x7f409814d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40980c6bb0 0x7f409814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 --2- 172.21.15.146:0/3758566682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40980c58c0 0x7f4098147640 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 >> 172.21.15.146:0/3758566682 conn(0x7f409806f3c0 msgr2=0x7f40980c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 shutdown_connections 2024-09-16T01:53:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.641+0000 7f409d000640 1 -- 172.21.15.146:0/3758566682 wait complete. 2024-09-16T01:53:11.660 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T01:53:10.807318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4220448681,"num_objects":1107,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":119,"num_read_kb":103,"num_write":8770,"num_write_kb":25168818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":9886222853,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9646,"ondisk_log_size":9646,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":857348,"kb_used_data":673852,"kb_used_omap":9,"kb_used_meta":182838,"kb_avail":565373692,"statfs":{"total":579820584960,"available":578942660608,"internally_reserved":0,"allocated":690024448,"data_stored":10886264508,"data_compressed":544301959,"data_compressed_allocated":680136704,"data_compressed_original":10881404928,"omap_allocated":9562,"internal_metadata":187226790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4877970032,"num_objects":-1163,"num_object_clones":0,"num_object_copies":-2326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1170,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.164107"},"pg_stats":[{"pgid":"3.3f","version":"125'120","reported_seq":418,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.513032+0000","last_change":"2024-09-16T01:52:11.751139+0000","last_active":"2024-09-16T01:53:06.513032+0000","last_peered":"2024-09-16T01:53:06.513032+0000","last_clean":"2024-09-16T01:53:06.513032+0000","last_became_active":"2024-09-16T01:52:09.417184+0000","last_became_peered":"2024-09-16T01:52:09.417184+0000","last_unstale":"2024-09-16T01:53:06.513032+0000","last_undegraded":"2024-09-16T01:53:06.513032+0000","last_fullsized":"2024-09-16T01:53:06.513032+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:40.707174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966877+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:09.966877+0000","last_peered":"2024-09-16T01:53:09.966877+0000","last_clean":"2024-09-16T01:53:09.966877+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:09.966877+0000","last_undegraded":"2024-09-16T01:53:09.966877+0000","last_fullsized":"2024-09-16T01:53:09.966877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":648,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966676+0000","last_change":"2024-09-16T01:52:51.502920+0000","last_active":"2024-09-16T01:53:09.966676+0000","last_peered":"2024-09-16T01:53:09.966676+0000","last_clean":"2024-09-16T01:53:09.966676+0000","last_became_active":"2024-09-16T01:52:51.502433+0000","last_became_peered":"2024-09-16T01:52:51.502433+0000","last_unstale":"2024-09-16T01:53:09.966676+0000","last_undegraded":"2024-09-16T01:53:09.966676+0000","last_fullsized":"2024-09-16T01:53:09.966676+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"125'140","reported_seq":367,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648483+0000","last_change":"2024-09-16T01:50:11.401907+0000","last_active":"2024-09-16T01:53:06.648483+0000","last_peered":"2024-09-16T01:53:06.648483+0000","last_clean":"2024-09-16T01:53:06.648483+0000","last_became_active":"2024-09-16T01:50:11.400970+0000","last_became_peered":"2024-09-16T01:50:11.400970+0000","last_unstale":"2024-09-16T01:53:06.648483+0000","last_undegraded":"2024-09-16T01:53:06.648483+0000","last_fullsized":"2024-09-16T01:53:06.648483+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:30:39.220739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"127'170","reported_seq":473,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.967092+0000","last_change":"2024-09-16T01:52:53.516521+0000","last_active":"2024-09-16T01:53:09.967092+0000","last_peered":"2024-09-16T01:53:09.967092+0000","last_clean":"2024-09-16T01:53:09.967092+0000","last_became_active":"2024-09-16T01:52:53.516052+0000","last_became_peered":"2024-09-16T01:52:53.516052+0000","last_unstale":"2024-09-16T01:53:09.967092+0000","last_undegraded":"2024-09-16T01:53:09.967092+0000","last_fullsized":"2024-09-16T01:53:09.967092+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3a","version":"125'117","reported_seq":390,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.647898+0000","last_change":"2024-09-16T01:51:51.081263+0000","last_active":"2024-09-16T01:53:06.647898+0000","last_peered":"2024-09-16T01:53:06.647898+0000","last_clean":"2024-09-16T01:53:06.647898+0000","last_became_active":"2024-09-16T01:51:49.173535+0000","last_became_peered":"2024-09-16T01:51:49.173535+0000","last_unstale":"2024-09-16T01:53:06.647898+0000","last_undegraded":"2024-09-16T01:53:06.647898+0000","last_fullsized":"2024-09-16T01:53:06.647898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:18.535661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"125'144","reported_seq":537,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.584662+0000","last_change":"2024-09-16T01:52:13.340248+0000","last_active":"2024-09-16T01:53:06.584662+0000","last_peered":"2024-09-16T01:53:06.584662+0000","last_clean":"2024-09-16T01:53:06.584662+0000","last_became_active":"2024-09-16T01:52:09.428262+0000","last_became_peered":"2024-09-16T01:52:09.428262+0000","last_unstale":"2024-09-16T01:53:06.584662+0000","last_undegraded":"2024-09-16T01:53:06.584662+0000","last_fullsized":"2024-09-16T01:53:06.584662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:25.640721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":133,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":546,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966191+0000","last_change":"2024-09-16T01:52:10.428694+0000","last_active":"2024-09-16T01:53:09.966191+0000","last_peered":"2024-09-16T01:53:09.966191+0000","last_clean":"2024-09-16T01:53:09.966191+0000","last_became_active":"2024-09-16T01:52:09.402181+0000","last_became_peered":"2024-09-16T01:52:09.402181+0000","last_unstale":"2024-09-16T01:53:09.966191+0000","last_undegraded":"2024-09-16T01:53:09.966191+0000","last_fullsized":"2024-09-16T01:53:09.966191+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"125'126","reported_seq":345,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.469510+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:06.469510+0000","last_peered":"2024-09-16T01:53:06.469510+0000","last_clean":"2024-09-16T01:53:06.469510+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:06.469510+0000","last_undegraded":"2024-09-16T01:53:06.469510+0000","last_fullsized":"2024-09-16T01:53:06.469510+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"125'134","reported_seq":459,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.647954+0000","last_change":"2024-09-16T01:52:18.209572+0000","last_active":"2024-09-16T01:53:06.647954+0000","last_peered":"2024-09-16T01:53:06.647954+0000","last_clean":"2024-09-16T01:53:06.647954+0000","last_became_active":"2024-09-16T01:52:09.375277+0000","last_became_peered":"2024-09-16T01:52:09.375277+0000","last_unstale":"2024-09-16T01:53:06.647954+0000","last_undegraded":"2024-09-16T01:53:06.647954+0000","last_fullsized":"2024-09-16T01:53:06.647954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.114190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.35","version":"125'150","reported_seq":498,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.541893+0000","last_change":"2024-09-16T01:52:14.210732+0000","last_active":"2024-09-16T01:53:06.541893+0000","last_peered":"2024-09-16T01:53:06.541893+0000","last_clean":"2024-09-16T01:53:06.541893+0000","last_became_active":"2024-09-16T01:52:09.419847+0000","last_became_peered":"2024-09-16T01:52:09.419847+0000","last_unstale":"2024-09-16T01:53:06.541893+0000","last_undegraded":"2024-09-16T01:53:06.541893+0000","last_fullsized":"2024-09-16T01:53:06.541893+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:48.934137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"125'115","reported_seq":254,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.533684+0000","last_change":"2024-09-16T01:52:50.499570+0000","last_active":"2024-09-16T01:53:06.533684+0000","last_peered":"2024-09-16T01:53:06.533684+0000","last_clean":"2024-09-16T01:53:06.533684+0000","last_became_active":"2024-09-16T01:52:50.498997+0000","last_became_peered":"2024-09-16T01:52:50.498997+0000","last_unstale":"2024-09-16T01:53:06.533684+0000","last_undegraded":"2024-09-16T01:53:06.533684+0000","last_fullsized":"2024-09-16T01:53:06.533684+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":517,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802729+0000","last_change":"2024-09-16T01:52:13.149233+0000","last_active":"2024-09-16T01:53:09.802729+0000","last_peered":"2024-09-16T01:53:09.802729+0000","last_clean":"2024-09-16T01:53:09.802729+0000","last_became_active":"2024-09-16T01:52:09.382431+0000","last_became_peered":"2024-09-16T01:52:09.382431+0000","last_unstale":"2024-09-16T01:53:09.802729+0000","last_undegraded":"2024-09-16T01:53:09.802729+0000","last_fullsized":"2024-09-16T01:53:09.802729+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:00.974199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":441,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802522+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:09.802522+0000","last_peered":"2024-09-16T01:53:09.802522+0000","last_clean":"2024-09-16T01:53:09.802522+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:09.802522+0000","last_undegraded":"2024-09-16T01:53:09.802522+0000","last_fullsized":"2024-09-16T01:53:09.802522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":499,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966150+0000","last_change":"2024-09-16T01:52:11.853926+0000","last_active":"2024-09-16T01:53:09.966150+0000","last_peered":"2024-09-16T01:53:09.966150+0000","last_clean":"2024-09-16T01:53:09.966150+0000","last_became_active":"2024-09-16T01:52:09.424407+0000","last_became_peered":"2024-09-16T01:52:09.424407+0000","last_unstale":"2024-09-16T01:53:09.966150+0000","last_undegraded":"2024-09-16T01:53:09.966150+0000","last_fullsized":"2024-09-16T01:53:09.966150+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:56.339192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"125'126","reported_seq":344,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.550262+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:06.550262+0000","last_peered":"2024-09-16T01:53:06.550262+0000","last_clean":"2024-09-16T01:53:06.550262+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:06.550262+0000","last_undegraded":"2024-09-16T01:53:06.550262+0000","last_fullsized":"2024-09-16T01:53:06.550262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"125'117","reported_seq":432,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.532712+0000","last_change":"2024-09-16T01:52:12.447025+0000","last_active":"2024-09-16T01:53:06.532712+0000","last_peered":"2024-09-16T01:53:06.532712+0000","last_clean":"2024-09-16T01:53:06.532712+0000","last_became_active":"2024-09-16T01:52:09.417711+0000","last_became_peered":"2024-09-16T01:52:09.417711+0000","last_unstale":"2024-09-16T01:53:06.532712+0000","last_undegraded":"2024-09-16T01:53:06.532712+0000","last_fullsized":"2024-09-16T01:53:06.532712+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:24.192522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"125'120","reported_seq":533,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648123+0000","last_change":"2024-09-16T01:52:49.497350+0000","last_active":"2024-09-16T01:53:06.648123+0000","last_peered":"2024-09-16T01:53:06.648123+0000","last_clean":"2024-09-16T01:53:06.648123+0000","last_became_active":"2024-09-16T01:52:49.496908+0000","last_became_peered":"2024-09-16T01:52:49.496908+0000","last_unstale":"2024-09-16T01:53:06.648123+0000","last_undegraded":"2024-09-16T01:53:06.648123+0000","last_fullsized":"2024-09-16T01:53:06.648123+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:51.945635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":658,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802879+0000","last_change":"2024-09-16T01:52:47.538124+0000","last_active":"2024-09-16T01:53:09.802879+0000","last_peered":"2024-09-16T01:53:09.802879+0000","last_clean":"2024-09-16T01:53:09.802879+0000","last_became_active":"2024-09-16T01:52:47.537602+0000","last_became_peered":"2024-09-16T01:52:47.537602+0000","last_unstale":"2024-09-16T01:53:09.802879+0000","last_undegraded":"2024-09-16T01:53:09.802879+0000","last_fullsized":"2024-09-16T01:53:09.802879+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:59.888922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"125'130","reported_seq":492,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.461960+0000","last_change":"2024-09-16T01:52:11.414438+0000","last_active":"2024-09-16T01:53:06.461960+0000","last_peered":"2024-09-16T01:53:06.461960+0000","last_clean":"2024-09-16T01:53:06.461960+0000","last_became_active":"2024-09-16T01:52:09.386408+0000","last_became_peered":"2024-09-16T01:52:09.386408+0000","last_unstale":"2024-09-16T01:53:06.461960+0000","last_undegraded":"2024-09-16T01:53:06.461960+0000","last_fullsized":"2024-09-16T01:53:06.461960+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:05.694307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":124,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"125'103","reported_seq":361,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.654012+0000","last_change":"2024-09-16T01:51:50.490795+0000","last_active":"2024-09-16T01:53:06.654012+0000","last_peered":"2024-09-16T01:53:06.654012+0000","last_clean":"2024-09-16T01:53:06.654012+0000","last_became_active":"2024-09-16T01:51:49.174512+0000","last_became_peered":"2024-09-16T01:51:49.174512+0000","last_unstale":"2024-09-16T01:53:06.654012+0000","last_undegraded":"2024-09-16T01:53:06.654012+0000","last_fullsized":"2024-09-16T01:53:06.654012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:01.568595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"125'145","reported_seq":610,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.570636+0000","last_change":"2024-09-16T01:52:44.525723+0000","last_active":"2024-09-16T01:53:06.570636+0000","last_peered":"2024-09-16T01:53:06.570636+0000","last_clean":"2024-09-16T01:53:06.570636+0000","last_became_active":"2024-09-16T01:52:44.521743+0000","last_became_peered":"2024-09-16T01:52:44.521743+0000","last_unstale":"2024-09-16T01:53:06.570636+0000","last_undegraded":"2024-09-16T01:53:06.570636+0000","last_fullsized":"2024-09-16T01:53:06.570636+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":368,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966370+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:09.966370+0000","last_peered":"2024-09-16T01:53:09.966370+0000","last_clean":"2024-09-16T01:53:09.966370+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:09.966370+0000","last_undegraded":"2024-09-16T01:53:09.966370+0000","last_fullsized":"2024-09-16T01:53:09.966370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"125'152","reported_seq":370,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.567970+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:06.567970+0000","last_peered":"2024-09-16T01:53:06.567970+0000","last_clean":"2024-09-16T01:53:06.567970+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:06.567970+0000","last_undegraded":"2024-09-16T01:53:06.567970+0000","last_fullsized":"2024-09-16T01:53:06.567970+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":145,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966682+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:09.966682+0000","last_peered":"2024-09-16T01:53:09.966682+0000","last_clean":"2024-09-16T01:53:09.966682+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:09.966682+0000","last_undegraded":"2024-09-16T01:53:09.966682+0000","last_fullsized":"2024-09-16T01:53:09.966682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"126'156","reported_seq":377,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648273+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:06.648273+0000","last_peered":"2024-09-16T01:53:06.648273+0000","last_clean":"2024-09-16T01:53:06.648273+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:06.648273+0000","last_undegraded":"2024-09-16T01:53:06.648273+0000","last_fullsized":"2024-09-16T01:53:06.648273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":149,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":617,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803203+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:09.803203+0000","last_peered":"2024-09-16T01:53:09.803203+0000","last_clean":"2024-09-16T01:53:09.803203+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:09.803203+0000","last_undegraded":"2024-09-16T01:53:09.803203+0000","last_fullsized":"2024-09-16T01:53:09.803203+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":468,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802871+0000","last_change":"2024-09-16T01:52:11.110169+0000","last_active":"2024-09-16T01:53:09.802871+0000","last_peered":"2024-09-16T01:53:09.802871+0000","last_clean":"2024-09-16T01:53:09.802871+0000","last_became_active":"2024-09-16T01:52:09.424359+0000","last_became_peered":"2024-09-16T01:52:09.424359+0000","last_unstale":"2024-09-16T01:53:09.802871+0000","last_undegraded":"2024-09-16T01:53:09.802871+0000","last_fullsized":"2024-09-16T01:53:09.802871+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:43.961015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":637,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966944+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:09.966944+0000","last_peered":"2024-09-16T01:53:09.966944+0000","last_clean":"2024-09-16T01:53:09.966944+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:09.966944+0000","last_undegraded":"2024-09-16T01:53:09.966944+0000","last_fullsized":"2024-09-16T01:53:09.966944+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"125'110","reported_seq":328,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.480755+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:06.480755+0000","last_peered":"2024-09-16T01:53:06.480755+0000","last_clean":"2024-09-16T01:53:06.480755+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:06.480755+0000","last_undegraded":"2024-09-16T01:53:06.480755+0000","last_fullsized":"2024-09-16T01:53:06.480755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"125'117","reported_seq":528,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.538723+0000","last_change":"2024-09-16T01:52:47.475996+0000","last_active":"2024-09-16T01:53:06.538723+0000","last_peered":"2024-09-16T01:53:06.538723+0000","last_clean":"2024-09-16T01:53:06.538723+0000","last_became_active":"2024-09-16T01:52:46.555530+0000","last_became_peered":"2024-09-16T01:52:46.555530+0000","last_unstale":"2024-09-16T01:53:06.538723+0000","last_undegraded":"2024-09-16T01:53:06.538723+0000","last_fullsized":"2024-09-16T01:53:06.538723+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:05.418025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":511,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802184+0000","last_change":"2024-09-16T01:52:44.499077+0000","last_active":"2024-09-16T01:53:09.802184+0000","last_peered":"2024-09-16T01:53:09.802184+0000","last_clean":"2024-09-16T01:53:09.802184+0000","last_became_active":"2024-09-16T01:52:44.498634+0000","last_became_peered":"2024-09-16T01:52:44.498634+0000","last_unstale":"2024-09-16T01:53:09.802184+0000","last_undegraded":"2024-09-16T01:53:09.802184+0000","last_fullsized":"2024-09-16T01:53:09.802184+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"125'115","reported_seq":334,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.587077+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:06.587077+0000","last_peered":"2024-09-16T01:53:06.587077+0000","last_clean":"2024-09-16T01:53:06.587077+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:06.587077+0000","last_undegraded":"2024-09-16T01:53:06.587077+0000","last_fullsized":"2024-09-16T01:53:06.587077+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":233,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966999+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:09.966999+0000","last_peered":"2024-09-16T01:53:09.966999+0000","last_clean":"2024-09-16T01:53:09.966999+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:09.966999+0000","last_undegraded":"2024-09-16T01:53:09.966999+0000","last_fullsized":"2024-09-16T01:53:09.966999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"125'117","reported_seq":668,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.582183+0000","last_change":"2024-09-16T01:52:52.506119+0000","last_active":"2024-09-16T01:53:06.582183+0000","last_peered":"2024-09-16T01:53:06.582183+0000","last_clean":"2024-09-16T01:53:06.582183+0000","last_became_active":"2024-09-16T01:52:52.505498+0000","last_became_peered":"2024-09-16T01:52:52.505498+0000","last_unstale":"2024-09-16T01:53:06.582183+0000","last_undegraded":"2024-09-16T01:53:06.582183+0000","last_fullsized":"2024-09-16T01:53:06.582183+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:00.694119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":240,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967051+0000","last_change":"2024-09-16T01:52:09.449049+0000","last_active":"2024-09-16T01:53:05.967051+0000","last_peered":"2024-09-16T01:53:05.967051+0000","last_clean":"2024-09-16T01:53:05.967051+0000","last_became_active":"2024-09-16T01:52:09.417204+0000","last_became_peered":"2024-09-16T01:52:09.417204+0000","last_unstale":"2024-09-16T01:53:05.967051+0000","last_undegraded":"2024-09-16T01:53:05.967051+0000","last_fullsized":"2024-09-16T01:53:05.967051+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802374+0000","last_change":"2024-09-16T01:52:46.550269+0000","last_active":"2024-09-16T01:53:09.802374+0000","last_peered":"2024-09-16T01:53:09.802374+0000","last_clean":"2024-09-16T01:53:09.802374+0000","last_became_active":"2024-09-16T01:52:46.543158+0000","last_became_peered":"2024-09-16T01:52:46.543158+0000","last_unstale":"2024-09-16T01:53:09.802374+0000","last_undegraded":"2024-09-16T01:53:09.802374+0000","last_fullsized":"2024-09-16T01:53:09.802374+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":408,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966391+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:09.966391+0000","last_peered":"2024-09-16T01:53:09.966391+0000","last_clean":"2024-09-16T01:53:09.966391+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:09.966391+0000","last_undegraded":"2024-09-16T01:53:09.966391+0000","last_fullsized":"2024-09-16T01:53:09.966391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":223,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966489+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:05.966489+0000","last_peered":"2024-09-16T01:53:05.966489+0000","last_clean":"2024-09-16T01:53:05.966489+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:05.966489+0000","last_undegraded":"2024-09-16T01:53:05.966489+0000","last_fullsized":"2024-09-16T01:53:05.966489+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"125'130","reported_seq":314,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.536902+0000","last_change":"2024-09-16T01:50:11.402665+0000","last_active":"2024-09-16T01:53:06.536902+0000","last_peered":"2024-09-16T01:53:06.536902+0000","last_clean":"2024-09-16T01:53:06.536902+0000","last_became_active":"2024-09-16T01:50:11.402336+0000","last_became_peered":"2024-09-16T01:50:11.402336+0000","last_unstale":"2024-09-16T01:53:06.536902+0000","last_undegraded":"2024-09-16T01:53:06.536902+0000","last_fullsized":"2024-09-16T01:53:06.536902+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":252,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967743+0000","last_change":"2024-09-16T01:52:09.414427+0000","last_active":"2024-09-16T01:53:05.967743+0000","last_peered":"2024-09-16T01:53:05.967743+0000","last_clean":"2024-09-16T01:53:05.967743+0000","last_became_active":"2024-09-16T01:52:09.394085+0000","last_became_peered":"2024-09-16T01:52:09.394085+0000","last_unstale":"2024-09-16T01:53:05.967743+0000","last_undegraded":"2024-09-16T01:53:05.967743+0000","last_fullsized":"2024-09-16T01:53:05.967743+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:12.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":481,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966553+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:09.966553+0000","last_peered":"2024-09-16T01:53:09.966553+0000","last_clean":"2024-09-16T01:53:09.966553+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:09.966553+0000","last_undegraded":"2024-09-16T01:53:09.966553+0000","last_fullsized":"2024-09-16T01:53:09.966553+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966831+0000","last_change":"2024-09-16T01:52:46.621488+0000","last_active":"2024-09-16T01:53:05.966831+0000","last_peered":"2024-09-16T01:53:05.966831+0000","last_clean":"2024-09-16T01:53:05.966831+0000","last_became_active":"2024-09-16T01:52:46.557896+0000","last_became_peered":"2024-09-16T01:52:46.557896+0000","last_unstale":"2024-09-16T01:53:05.966831+0000","last_undegraded":"2024-09-16T01:53:05.966831+0000","last_fullsized":"2024-09-16T01:53:05.966831+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966895+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:09.966895+0000","last_peered":"2024-09-16T01:53:09.966895+0000","last_clean":"2024-09-16T01:53:09.966895+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:09.966895+0000","last_undegraded":"2024-09-16T01:53:09.966895+0000","last_fullsized":"2024-09-16T01:53:09.966895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"74'8","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966535+0000","last_change":"2024-09-16T01:52:46.632501+0000","last_active":"2024-09-16T01:53:05.966535+0000","last_peered":"2024-09-16T01:53:05.966535+0000","last_clean":"2024-09-16T01:53:05.966535+0000","last_became_active":"2024-09-16T01:52:46.562268+0000","last_became_peered":"2024-09-16T01:52:46.562268+0000","last_unstale":"2024-09-16T01:53:05.966535+0000","last_undegraded":"2024-09-16T01:53:05.966535+0000","last_fullsized":"2024-09-16T01:53:05.966535+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"125'121","reported_seq":339,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.565619+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:06.565619+0000","last_peered":"2024-09-16T01:53:06.565619+0000","last_clean":"2024-09-16T01:53:06.565619+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:06.565619+0000","last_undegraded":"2024-09-16T01:53:06.565619+0000","last_fullsized":"2024-09-16T01:53:06.565619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"123'170","reported_seq":414,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648602+0000","last_change":"2024-09-16T01:51:52.243466+0000","last_active":"2024-09-16T01:53:06.648602+0000","last_peered":"2024-09-16T01:53:06.648602+0000","last_clean":"2024-09-16T01:53:06.648602+0000","last_became_active":"2024-09-16T01:51:52.243093+0000","last_became_peered":"2024-09-16T01:51:52.243093+0000","last_unstale":"2024-09-16T01:53:06.648602+0000","last_undegraded":"2024-09-16T01:53:06.648602+0000","last_fullsized":"2024-09-16T01:53:06.648602+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:48.180559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381505,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":237575,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"127'170","reported_seq":500,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803045+0000","last_change":"2024-09-16T01:52:12.431002+0000","last_active":"2024-09-16T01:53:09.803045+0000","last_peered":"2024-09-16T01:53:09.803045+0000","last_clean":"2024-09-16T01:53:09.803045+0000","last_became_active":"2024-09-16T01:52:09.424330+0000","last_became_peered":"2024-09-16T01:52:09.424330+0000","last_unstale":"2024-09-16T01:53:09.803045+0000","last_undegraded":"2024-09-16T01:53:09.803045+0000","last_fullsized":"2024-09-16T01:53:09.803045+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:35:36.746898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":299,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966055+0000","last_change":"2024-09-16T01:52:46.615925+0000","last_active":"2024-09-16T01:53:05.966055+0000","last_peered":"2024-09-16T01:53:05.966055+0000","last_clean":"2024-09-16T01:53:05.966055+0000","last_became_active":"2024-09-16T01:52:46.558348+0000","last_became_peered":"2024-09-16T01:52:46.558348+0000","last_unstale":"2024-09-16T01:53:05.966055+0000","last_undegraded":"2024-09-16T01:53:05.966055+0000","last_fullsized":"2024-09-16T01:53:05.966055+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":340,"num_keys_recovered":50,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":395,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802526+0000","last_change":"2024-09-16T01:48:58.828267+0000","last_active":"2024-09-16T01:53:09.802526+0000","last_peered":"2024-09-16T01:53:09.802526+0000","last_clean":"2024-09-16T01:53:09.802526+0000","last_became_active":"2024-09-16T01:48:58.827123+0000","last_became_peered":"2024-09-16T01:48:58.827123+0000","last_unstale":"2024-09-16T01:53:09.802526+0000","last_undegraded":"2024-09-16T01:53:09.802526+0000","last_fullsized":"2024-09-16T01:53:09.802526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:01.237871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":303,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967144+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:05.967144+0000","last_peered":"2024-09-16T01:53:05.967144+0000","last_clean":"2024-09-16T01:53:05.967144+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:05.967144+0000","last_undegraded":"2024-09-16T01:53:05.967144+0000","last_fullsized":"2024-09-16T01:53:05.967144+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"125'141","reported_seq":490,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.588752+0000","last_change":"2024-09-16T01:52:11.052533+0000","last_active":"2024-09-16T01:53:06.588752+0000","last_peered":"2024-09-16T01:53:06.588752+0000","last_clean":"2024-09-16T01:53:06.588752+0000","last_became_active":"2024-09-16T01:52:09.417164+0000","last_became_peered":"2024-09-16T01:52:09.417164+0000","last_unstale":"2024-09-16T01:53:06.588752+0000","last_undegraded":"2024-09-16T01:53:06.588752+0000","last_fullsized":"2024-09-16T01:53:06.588752+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:54.181674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":97,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966243+0000","last_change":"2024-09-16T01:52:09.440837+0000","last_active":"2024-09-16T01:53:05.966243+0000","last_peered":"2024-09-16T01:53:05.966243+0000","last_clean":"2024-09-16T01:53:05.966243+0000","last_became_active":"2024-09-16T01:52:09.416876+0000","last_became_peered":"2024-09-16T01:52:09.416876+0000","last_unstale":"2024-09-16T01:53:05.966243+0000","last_undegraded":"2024-09-16T01:53:05.966243+0000","last_fullsized":"2024-09-16T01:53:05.966243+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:15.254646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"125'153","reported_seq":372,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.570769+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:06.570769+0000","last_peered":"2024-09-16T01:53:06.570769+0000","last_clean":"2024-09-16T01:53:06.570769+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:06.570769+0000","last_undegraded":"2024-09-16T01:53:06.570769+0000","last_fullsized":"2024-09-16T01:53:06.570769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'17","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966313+0000","last_change":"2024-09-16T01:52:46.599623+0000","last_active":"2024-09-16T01:53:05.966313+0000","last_peered":"2024-09-16T01:53:05.966313+0000","last_clean":"2024-09-16T01:53:05.966313+0000","last_became_active":"2024-09-16T01:52:46.553898+0000","last_became_peered":"2024-09-16T01:52:46.553898+0000","last_unstale":"2024-09-16T01:53:05.966313+0000","last_undegraded":"2024-09-16T01:53:05.966313+0000","last_fullsized":"2024-09-16T01:53:05.966313+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:22.428578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":784,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"125'159","reported_seq":377,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.522182+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:06.522182+0000","last_peered":"2024-09-16T01:53:06.522182+0000","last_clean":"2024-09-16T01:53:06.522182+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:06.522182+0000","last_undegraded":"2024-09-16T01:53:06.522182+0000","last_fullsized":"2024-09-16T01:53:06.522182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134217728,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":151,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":101,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966552+0000","last_change":"2024-09-16T01:52:09.399751+0000","last_active":"2024-09-16T01:53:09.966552+0000","last_peered":"2024-09-16T01:53:09.966552+0000","last_clean":"2024-09-16T01:53:09.966552+0000","last_became_active":"2024-09-16T01:52:09.393569+0000","last_became_peered":"2024-09-16T01:52:09.393569+0000","last_unstale":"2024-09-16T01:53:09.966552+0000","last_undegraded":"2024-09-16T01:53:09.966552+0000","last_fullsized":"2024-09-16T01:53:09.966552+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"125'119","reported_seq":441,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648329+0000","last_change":"2024-09-16T01:52:10.126501+0000","last_active":"2024-09-16T01:53:06.648329+0000","last_peered":"2024-09-16T01:53:06.648329+0000","last_clean":"2024-09-16T01:53:06.648329+0000","last_became_active":"2024-09-16T01:52:09.381243+0000","last_became_peered":"2024-09-16T01:52:09.381243+0000","last_unstale":"2024-09-16T01:53:06.648329+0000","last_undegraded":"2024-09-16T01:53:06.648329+0000","last_fullsized":"2024-09-16T01:53:06.648329+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:22.944535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":122,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966938+0000","last_change":"2024-09-16T01:51:49.216615+0000","last_active":"2024-09-16T01:53:09.966938+0000","last_peered":"2024-09-16T01:53:09.966938+0000","last_clean":"2024-09-16T01:53:09.966938+0000","last_became_active":"2024-09-16T01:51:49.162754+0000","last_became_peered":"2024-09-16T01:51:49.162754+0000","last_unstale":"2024-09-16T01:53:09.966938+0000","last_undegraded":"2024-09-16T01:53:09.966938+0000","last_fullsized":"2024-09-16T01:53:09.966938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"125'112","reported_seq":427,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.504636+0000","last_change":"2024-09-16T01:52:14.062053+0000","last_active":"2024-09-16T01:53:06.504636+0000","last_peered":"2024-09-16T01:53:06.504636+0000","last_clean":"2024-09-16T01:53:06.504636+0000","last_became_active":"2024-09-16T01:52:09.391866+0000","last_became_peered":"2024-09-16T01:52:09.391866+0000","last_unstale":"2024-09-16T01:53:06.504636+0000","last_undegraded":"2024-09-16T01:53:06.504636+0000","last_fullsized":"2024-09-16T01:53:06.504636+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:54.103068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":231,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966729+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:09.966729+0000","last_peered":"2024-09-16T01:53:09.966729+0000","last_clean":"2024-09-16T01:53:09.966729+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:09.966729+0000","last_undegraded":"2024-09-16T01:53:09.966729+0000","last_fullsized":"2024-09-16T01:53:09.966729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"125'116","reported_seq":662,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.574039+0000","last_change":"2024-09-16T01:52:50.499005+0000","last_active":"2024-09-16T01:53:06.574039+0000","last_peered":"2024-09-16T01:53:06.574039+0000","last_clean":"2024-09-16T01:53:06.574039+0000","last_became_active":"2024-09-16T01:52:50.498634+0000","last_became_peered":"2024-09-16T01:52:50.498634+0000","last_unstale":"2024-09-16T01:53:06.574039+0000","last_undegraded":"2024-09-16T01:53:06.574039+0000","last_fullsized":"2024-09-16T01:53:06.574039+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:04:46.802828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":247,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803062+0000","last_change":"2024-09-16T01:52:09.391770+0000","last_active":"2024-09-16T01:53:09.803062+0000","last_peered":"2024-09-16T01:53:09.803062+0000","last_clean":"2024-09-16T01:53:09.803062+0000","last_became_active":"2024-09-16T01:52:09.378550+0000","last_became_peered":"2024-09-16T01:52:09.378550+0000","last_unstale":"2024-09-16T01:53:09.803062+0000","last_undegraded":"2024-09-16T01:53:09.803062+0000","last_fullsized":"2024-09-16T01:53:09.803062+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:21.522042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"125'128","reported_seq":346,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.503811+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:06.503811+0000","last_peered":"2024-09-16T01:53:06.503811+0000","last_clean":"2024-09-16T01:53:06.503811+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:06.503811+0000","last_undegraded":"2024-09-16T01:53:06.503811+0000","last_fullsized":"2024-09-16T01:53:06.503811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":378,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967363+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:05.967363+0000","last_peered":"2024-09-16T01:53:05.967363+0000","last_clean":"2024-09-16T01:53:05.967363+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:05.967363+0000","last_undegraded":"2024-09-16T01:53:05.967363+0000","last_fullsized":"2024-09-16T01:53:05.967363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":226,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.965877+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:05.965877+0000","last_peered":"2024-09-16T01:53:05.965877+0000","last_clean":"2024-09-16T01:53:05.965877+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:05.965877+0000","last_undegraded":"2024-09-16T01:53:05.965877+0000","last_fullsized":"2024-09-16T01:53:05.965877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"125'120","reported_seq":481,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.647781+0000","last_change":"2024-09-16T01:52:12.246679+0000","last_active":"2024-09-16T01:53:06.647781+0000","last_peered":"2024-09-16T01:53:06.647781+0000","last_clean":"2024-09-16T01:53:06.647781+0000","last_became_active":"2024-09-16T01:52:09.398089+0000","last_became_peered":"2024-09-16T01:52:09.398089+0000","last_unstale":"2024-09-16T01:53:06.647781+0000","last_undegraded":"2024-09-16T01:53:06.647781+0000","last_fullsized":"2024-09-16T01:53:06.647781+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:49:59.959475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":225,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966606+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:05.966606+0000","last_peered":"2024-09-16T01:53:05.966606+0000","last_clean":"2024-09-16T01:53:05.966606+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:05.966606+0000","last_undegraded":"2024-09-16T01:53:05.966606+0000","last_fullsized":"2024-09-16T01:53:05.966606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802302+0000","last_change":"2024-09-16T01:48:58.832556+0000","last_active":"2024-09-16T01:53:09.802302+0000","last_peered":"2024-09-16T01:53:09.802302+0000","last_clean":"2024-09-16T01:53:09.802302+0000","last_became_active":"2024-09-16T01:48:58.832276+0000","last_became_peered":"2024-09-16T01:48:58.832276+0000","last_unstale":"2024-09-16T01:53:09.802302+0000","last_undegraded":"2024-09-16T01:53:09.802302+0000","last_fullsized":"2024-09-16T01:53:09.802302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"125'10","reported_seq":291,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966269+0000","last_change":"2024-09-16T01:52:46.569921+0000","last_active":"2024-09-16T01:53:09.966269+0000","last_peered":"2024-09-16T01:53:09.966269+0000","last_clean":"2024-09-16T01:53:09.966269+0000","last_became_active":"2024-09-16T01:52:46.544102+0000","last_became_peered":"2024-09-16T01:52:46.544102+0000","last_unstale":"2024-09-16T01:53:09.966269+0000","last_undegraded":"2024-09-16T01:53:09.966269+0000","last_fullsized":"2024-09-16T01:53:09.966269+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:14.446115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":601,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966294+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:09.966294+0000","last_peered":"2024-09-16T01:53:09.966294+0000","last_clean":"2024-09-16T01:53:09.966294+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:09.966294+0000","last_undegraded":"2024-09-16T01:53:09.966294+0000","last_fullsized":"2024-09-16T01:53:09.966294+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":225,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.965877+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:05.965877+0000","last_peered":"2024-09-16T01:53:05.965877+0000","last_clean":"2024-09-16T01:53:05.965877+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:05.965877+0000","last_undegraded":"2024-09-16T01:53:05.965877+0000","last_fullsized":"2024-09-16T01:53:05.965877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"125'125","reported_seq":445,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.653796+0000","last_change":"2024-09-16T01:52:15.362654+0000","last_active":"2024-09-16T01:53:06.653796+0000","last_peered":"2024-09-16T01:53:06.653796+0000","last_clean":"2024-09-16T01:53:06.653796+0000","last_became_active":"2024-09-16T01:52:09.424947+0000","last_became_peered":"2024-09-16T01:52:09.424947+0000","last_unstale":"2024-09-16T01:53:06.653796+0000","last_undegraded":"2024-09-16T01:53:06.653796+0000","last_fullsized":"2024-09-16T01:53:06.653796+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:25.640975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":230,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966737+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:09.966737+0000","last_peered":"2024-09-16T01:53:09.966737+0000","last_clean":"2024-09-16T01:53:09.966737+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:09.966737+0000","last_undegraded":"2024-09-16T01:53:09.966737+0000","last_fullsized":"2024-09-16T01:53:09.966737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"125'122","reported_seq":343,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648907+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:06.648907+0000","last_peered":"2024-09-16T01:53:06.648907+0000","last_clean":"2024-09-16T01:53:06.648907+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:06.648907+0000","last_undegraded":"2024-09-16T01:53:06.648907+0000","last_fullsized":"2024-09-16T01:53:06.648907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":54,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648714+0000","last_change":"2024-09-16T01:52:41.777809+0000","last_active":"2024-09-16T01:53:06.648714+0000","last_peered":"2024-09-16T01:53:06.648714+0000","last_clean":"2024-09-16T01:53:06.648714+0000","last_became_active":"2024-09-16T01:52:41.726385+0000","last_became_peered":"2024-09-16T01:52:41.726385+0000","last_unstale":"2024-09-16T01:53:06.648714+0000","last_undegraded":"2024-09-16T01:53:06.648714+0000","last_fullsized":"2024-09-16T01:53:06.648714+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:28.164668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803012+0000","last_change":"2024-09-16T01:48:58.818792+0000","last_active":"2024-09-16T01:53:09.803012+0000","last_peered":"2024-09-16T01:53:09.803012+0000","last_clean":"2024-09-16T01:53:09.803012+0000","last_became_active":"2024-09-16T01:48:58.818508+0000","last_became_peered":"2024-09-16T01:48:58.818508+0000","last_unstale":"2024-09-16T01:53:09.803012+0000","last_undegraded":"2024-09-16T01:53:09.803012+0000","last_fullsized":"2024-09-16T01:53:09.803012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:47.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"128'21","reported_seq":230,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966728+0000","last_change":"2024-09-16T01:52:09.440148+0000","last_active":"2024-09-16T01:53:09.966728+0000","last_peered":"2024-09-16T01:53:09.966728+0000","last_clean":"2024-09-16T01:53:09.966728+0000","last_became_active":"2024-09-16T01:52:09.417961+0000","last_became_peered":"2024-09-16T01:52:09.417961+0000","last_unstale":"2024-09-16T01:53:09.966728+0000","last_undegraded":"2024-09-16T01:53:09.966728+0000","last_fullsized":"2024-09-16T01:53:09.966728+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:30.532418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":4,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":433,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.803097+0000","last_change":"2024-09-16T01:51:49.663819+0000","last_active":"2024-09-16T01:53:09.803097+0000","last_peered":"2024-09-16T01:53:09.803097+0000","last_clean":"2024-09-16T01:53:09.803097+0000","last_became_active":"2024-09-16T01:51:49.154517+0000","last_became_peered":"2024-09-16T01:51:49.154517+0000","last_unstale":"2024-09-16T01:53:09.803097+0000","last_undegraded":"2024-09-16T01:53:09.803097+0000","last_fullsized":"2024-09-16T01:53:09.803097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:19:59.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":242,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967226+0000","last_change":"2024-09-16T01:52:09.455908+0000","last_active":"2024-09-16T01:53:05.967226+0000","last_peered":"2024-09-16T01:53:05.967226+0000","last_clean":"2024-09-16T01:53:05.967226+0000","last_became_active":"2024-09-16T01:52:09.418237+0000","last_became_peered":"2024-09-16T01:52:09.418237+0000","last_unstale":"2024-09-16T01:53:05.967226+0000","last_undegraded":"2024-09-16T01:53:05.967226+0000","last_fullsized":"2024-09-16T01:53:05.967226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"125'131","reported_seq":440,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.427408+0000","last_change":"2024-09-16T01:52:52.504293+0000","last_active":"2024-09-16T01:53:06.427408+0000","last_peered":"2024-09-16T01:53:06.427408+0000","last_clean":"2024-09-16T01:53:06.427408+0000","last_became_active":"2024-09-16T01:52:52.503886+0000","last_became_peered":"2024-09-16T01:52:52.503886+0000","last_unstale":"2024-09-16T01:53:06.427408+0000","last_undegraded":"2024-09-16T01:53:06.427408+0000","last_fullsized":"2024-09-16T01:53:06.427408+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:29.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":108,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648339+0000","last_change":"2024-09-16T01:52:46.618081+0000","last_active":"2024-09-16T01:53:06.648339+0000","last_peered":"2024-09-16T01:53:06.648339+0000","last_clean":"2024-09-16T01:53:06.648339+0000","last_became_active":"2024-09-16T01:52:46.571905+0000","last_became_peered":"2024-09-16T01:52:46.571905+0000","last_unstale":"2024-09-16T01:53:06.648339+0000","last_undegraded":"2024-09-16T01:53:06.648339+0000","last_fullsized":"2024-09-16T01:53:06.648339+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:14.315648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"125'125","reported_seq":398,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.583343+0000","last_change":"2024-09-16T01:51:51.047714+0000","last_active":"2024-09-16T01:53:06.583343+0000","last_peered":"2024-09-16T01:53:06.583343+0000","last_clean":"2024-09-16T01:53:06.583343+0000","last_became_active":"2024-09-16T01:51:49.216665+0000","last_became_peered":"2024-09-16T01:51:49.216665+0000","last_unstale":"2024-09-16T01:53:06.583343+0000","last_undegraded":"2024-09-16T01:53:06.583343+0000","last_fullsized":"2024-09-16T01:53:06.583343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:04:37.692978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":109,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802229+0000","last_change":"2024-09-16T01:52:09.403261+0000","last_active":"2024-09-16T01:53:09.802229+0000","last_peered":"2024-09-16T01:53:09.802229+0000","last_clean":"2024-09-16T01:53:09.802229+0000","last_became_active":"2024-09-16T01:52:09.378607+0000","last_became_peered":"2024-09-16T01:52:09.378607+0000","last_unstale":"2024-09-16T01:53:09.802229+0000","last_undegraded":"2024-09-16T01:53:09.802229+0000","last_fullsized":"2024-09-16T01:53:09.802229+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:33.032076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"125'128","reported_seq":404,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.648243+0000","last_change":"2024-09-16T01:52:46.782699+0000","last_active":"2024-09-16T01:53:06.648243+0000","last_peered":"2024-09-16T01:53:06.648243+0000","last_clean":"2024-09-16T01:53:06.648243+0000","last_became_active":"2024-09-16T01:52:46.542609+0000","last_became_peered":"2024-09-16T01:52:46.542609+0000","last_unstale":"2024-09-16T01:53:06.648243+0000","last_undegraded":"2024-09-16T01:53:06.648243+0000","last_fullsized":"2024-09-16T01:53:06.648243+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:30.580835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":232,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802620+0000","last_change":"2024-09-16T01:48:57.803908+0000","last_active":"2024-09-16T01:53:09.802620+0000","last_peered":"2024-09-16T01:53:09.802620+0000","last_clean":"2024-09-16T01:53:09.802620+0000","last_became_active":"2024-09-16T01:48:57.803630+0000","last_became_peered":"2024-09-16T01:48:57.803630+0000","last_unstale":"2024-09-16T01:53:09.802620+0000","last_undegraded":"2024-09-16T01:53:09.802620+0000","last_fullsized":"2024-09-16T01:53:09.802620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:54.816690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":593,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.802593+0000","last_change":"2024-09-16T01:52:51.501914+0000","last_active":"2024-09-16T01:53:09.802593+0000","last_peered":"2024-09-16T01:53:09.802593+0000","last_clean":"2024-09-16T01:53:09.802593+0000","last_became_active":"2024-09-16T01:52:51.501494+0000","last_became_peered":"2024-09-16T01:52:51.501494+0000","last_unstale":"2024-09-16T01:53:09.802593+0000","last_undegraded":"2024-09-16T01:53:09.802593+0000","last_fullsized":"2024-09-16T01:53:09.802593+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:29.019903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":225,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.965928+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:05.965928+0000","last_peered":"2024-09-16T01:53:05.965928+0000","last_clean":"2024-09-16T01:53:05.965928+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:05.965928+0000","last_undegraded":"2024-09-16T01:53:05.965928+0000","last_fullsized":"2024-09-16T01:53:05.965928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"125'105","reported_seq":323,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.535644+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:06.535644+0000","last_peered":"2024-09-16T01:53:06.535644+0000","last_clean":"2024-09-16T01:53:06.535644+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:06.535644+0000","last_undegraded":"2024-09-16T01:53:06.535644+0000","last_fullsized":"2024-09-16T01:53:06.535644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"125'138","reported_seq":475,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.389369+0000","last_change":"2024-09-16T01:52:13.375623+0000","last_active":"2024-09-16T01:53:06.389369+0000","last_peered":"2024-09-16T01:53:06.389369+0000","last_clean":"2024-09-16T01:53:06.389369+0000","last_became_active":"2024-09-16T01:52:09.419290+0000","last_became_peered":"2024-09-16T01:52:09.419290+0000","last_unstale":"2024-09-16T01:53:06.389369+0000","last_undegraded":"2024-09-16T01:53:06.389369+0000","last_fullsized":"2024-09-16T01:53:06.389369+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:28:50.497332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":94,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967312+0000","last_change":"2024-09-16T01:52:09.455363+0000","last_active":"2024-09-16T01:53:05.967312+0000","last_peered":"2024-09-16T01:53:05.967312+0000","last_clean":"2024-09-16T01:53:05.967312+0000","last_became_active":"2024-09-16T01:52:09.429254+0000","last_became_peered":"2024-09-16T01:52:09.429254+0000","last_unstale":"2024-09-16T01:53:05.967312+0000","last_undegraded":"2024-09-16T01:53:05.967312+0000","last_fullsized":"2024-09-16T01:53:05.967312+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:25.911449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"125'118","reported_seq":432,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:06.412329+0000","last_change":"2024-09-16T01:52:10.700803+0000","last_active":"2024-09-16T01:53:06.412329+0000","last_peered":"2024-09-16T01:53:06.412329+0000","last_clean":"2024-09-16T01:53:06.412329+0000","last_became_active":"2024-09-16T01:52:09.395385+0000","last_became_peered":"2024-09-16T01:52:09.395385+0000","last_unstale":"2024-09-16T01:53:06.412329+0000","last_undegraded":"2024-09-16T01:53:06.412329+0000","last_fullsized":"2024-09-16T01:53:06.412329+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:02:02.823534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":62,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.967426+0000","last_change":"2024-09-16T01:52:40.776372+0000","last_active":"2024-09-16T01:53:05.967426+0000","last_peered":"2024-09-16T01:53:05.967426+0000","last_clean":"2024-09-16T01:53:05.967426+0000","last_became_active":"2024-09-16T01:52:40.724575+0000","last_became_peered":"2024-09-16T01:52:40.724575+0000","last_unstale":"2024-09-16T01:53:05.967426+0000","last_undegraded":"2024-09-16T01:53:05.967426+0000","last_fullsized":"2024-09-16T01:53:05.967426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966833+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:09.966833+0000","last_peered":"2024-09-16T01:53:09.966833+0000","last_clean":"2024-09-16T01:53:09.966833+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:09.966833+0000","last_undegraded":"2024-09-16T01:53:09.966833+0000","last_fullsized":"2024-09-16T01:53:09.966833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.967035+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:09.967035+0000","last_peered":"2024-09-16T01:53:09.967035+0000","last_clean":"2024-09-16T01:53:09.967035+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:09.967035+0000","last_undegraded":"2024-09-16T01:53:09.967035+0000","last_fullsized":"2024-09-16T01:53:09.967035+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":403,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T01:53:09.966931+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:09.966931+0000","last_peered":"2024-09-16T01:53:09.966931+0000","last_clean":"2024-09-16T01:53:09.966931+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:09.966931+0000","last_undegraded":"2024-09-16T01:53:09.966931+0000","last_fullsized":"2024-09-16T01:53:09.966931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T01:53:05.966921+0000","last_change":"2024-09-16T01:51:49.410995+0000","last_active":"2024-09-16T01:53:05.966921+0000","last_peered":"2024-09-16T01:53:05.966921+0000","last_clean":"2024-09-16T01:53:05.966921+0000","last_became_active":"2024-09-16T01:51:49.187026+0000","last_became_peered":"2024-09-16T01:51:49.187026+0000","last_unstale":"2024-09-16T01:53:05.966921+0000","last_undegraded":"2024-09-16T01:53:05.966921+0000","last_fullsized":"2024-09-16T01:53:05.966921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:29:37.986084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":4219469824,"num_objects":1006,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2984,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":559415296,"data_stored":8950644736,"data_compressed":445052032,"data_compressed_allocated":559415296,"data_compressed_original":8950644736,"omap_allocated":0,"internal_metadata":0},"log_size":9163,"ondisk_log_size":9163,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":388489,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":401,"num_write_kb":578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":248325,"num_keys_recovered":145,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1118208,"data_stored":1006188,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509496,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323756,"kb_used_data":292568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048084,"statfs":{"total":96636764160,"available":96305238016,"internally_reserved":0,"allocated":299589632,"data_stored":4769624253,"data_compressed":238533504,"data_compressed_allocated":298057728,"data_compressed_original":4768923648,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509496,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":235292,"kb_used_data":203844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94136548,"statfs":{"total":96636764160,"available":96395825152,"internally_reserved":0,"allocated":208736256,"data_stored":3310391576,"data_compressed":165525888,"data_compressed_allocated":206831616,"data_compressed_original":3309305856,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509495,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200040,"kb_used_data":172752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94171800,"statfs":{"total":96636764160,"available":96431923200,"internally_reserved":0,"allocated":176898048,"data_stored":2803471009,"data_compressed":140210147,"data_compressed_allocated":175202304,"data_compressed_original":2802601984,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":64424509496,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542200,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1576960,"data_stored":728281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542200,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":729708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1642496,"data_stored":1319681,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574904,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":383053,"data_compressed":36671,"data_compressed_allocated":45056,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":383204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":226229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21233664,"data_stored":339738624,"data_compressed":16049664,"data_compressed_allocated":21233664,"data_compressed_original":339738624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130809856,"data_stored":2092957696,"data_compressed":104686208,"data_compressed_allocated":130809856,"data_compressed_original":2092957696,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171704320,"data_stored":2747269120,"data_compressed":137413760,"data_compressed_allocated":171704320,"data_compressed_original":2747269120,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":197394432,"data_stored":3158310912,"data_compressed":157973376,"data_compressed_allocated":197394432,"data_compressed_original":3158310912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38273024,"data_stored":612368384,"data_compressed":28929024,"data_compressed_allocated":38273024,"data_compressed_original":612368384,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:11.665 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T01:53:11.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T01:53:11.665 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T01:53:11.665 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-16T01:53:11.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0 2024-09-16T01:53:11.670 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:11.671 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:11.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T01:53:11.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.830+0000 7fa440200640 1 Processor -- start 2024-09-16T01:53:11.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.830+0000 7fa440200640 1 -- start start 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa440200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa43815c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa440200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa438155000 0x7fa43815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa440200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4380b31e0 con 0x7fa4380d0dc0 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa440200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa4380b4110 con 0x7fa438155000 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa440200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa4380b43f0 con 0x7fa4380d09e0 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa43ce00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa4380d0dc0 0x7fa4381548f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:33896/0 (socket says 172.21.15.146:33896) 2024-09-16T01:53:11.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa43ce00640 1 -- 172.21.15.146:0/1621860525 learned_addr learned my addr 172.21.15.146:0/1621860525 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:11.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa43d800640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa43815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.831+0000 7fa43e200640 1 --2- 172.21.15.146:0/1621860525 >> v2:172.21.15.161:3300/0 conn(0x7fa438155000 0x7fa43815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa437e00640 1 -- 172.21.15.146:0/1621860525 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2401564387 0 0) 0x7fa4380b31e0 con 0x7fa4380d0dc0 2024-09-16T01:53:11.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa437e00640 1 -- 172.21.15.146:0/1621860525 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa420003520 con 0x7fa4380d0dc0 2024-09-16T01:53:11.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa43e200640 1 -- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 msgr2=0x7fa43815c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:11.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa43e200640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa43815c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa43e200640 1 -- 172.21.15.146:0/1621860525 >> v1:172.21.15.146:6789/0 conn(0x7fa4380d0dc0 legacy=0x7fa4381548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa43e200640 1 -- 172.21.15.146:0/1621860525 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4380b46d0 con 0x7fa438155000 2024-09-16T01:53:11.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.832+0000 7fa43d800640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa43815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.833+0000 7fa43e200640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa438155000 0x7fa43815c5d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa4300097a0 tx=0x7fa43002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be10878fe628afc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.833+0000 7fa437e00640 1 -- 172.21.15.146:0/1621860525 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430037070 con 0x7fa438155000 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.833+0000 7fa437e00640 1 -- 172.21.15.146:0/1621860525 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa430032440 con 0x7fa438155000 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.833+0000 7fa437e00640 1 -- 172.21.15.146:0/1621860525 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430032810 con 0x7fa438155000 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 -- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa438155000 msgr2=0x7fa43815c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa438155000 0x7fa43815c5d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa4300097a0 tx=0x7fa43002a210 comp rx=0 tx=0).stop 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 -- 172.21.15.146:0/1621860525 shutdown_connections 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa438155000 0x7fa43815c5d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 --2- 172.21.15.146:0/1621860525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa43815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 -- 172.21.15.146:0/1621860525 >> 172.21.15.146:0/1621860525 conn(0x7fa43806c920 msgr2=0x7fa43806cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:11.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 -- 172.21.15.146:0/1621860525 shutdown_connections 2024-09-16T01:53:11.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.834+0000 7fa440200640 1 -- 172.21.15.146:0/1621860525 wait complete. 2024-09-16T01:53:11.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.835+0000 7fa440200640 1 Processor -- start 2024-09-16T01:53:11.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 -- start start 2024-09-16T01:53:11.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa4380c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4380d0dc0 0x7fa4380c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 0x7fa43814c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4380b4f70 con 0x7fa438155000 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa4380b4c90 con 0x7fa4380d0dc0 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.836+0000 7fa440200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa43815adc0 con 0x7fa4380d09e0 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 0x7fa43814c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43d800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa4380c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43ce00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4380d0dc0 0x7fa4380c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43d800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa4380c6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43360/0 (socket says 172.21.15.146:43360) 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 0x7fa43814c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49532/0 (socket says 172.21.15.146:49532) 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43d800640 1 -- 172.21.15.146:0/2703238438 learned_addr learned my addr 172.21.15.146:0/2703238438 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:11.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 -- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 msgr2=0x7fa4380c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa4380c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 -- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4380d0dc0 msgr2=0x7fa4380c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4380d0dc0 0x7fa4380c4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.837+0000 7fa43e200640 1 -- 172.21.15.146:0/2703238438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4380ce510 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.838+0000 7fa43e200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 0x7fa43814c380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa4380cdf80 tx=0x7fa4300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.838+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430041030 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.838+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa43003d4e0 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.838+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430037040 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.838+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4381f0df0 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.838+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4380ccc10 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.839+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4381f11d0 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.840+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa43003d8d0 con 0x7fa438155000 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.841+0000 7fa435600640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa41806c2b0 0x7fa41806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:11.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.841+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5894+0+0 (secure 0 0 0) 0x7fa4300ab560 con 0x7fa438155000 2024-09-16T01:53:11.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.841+0000 7fa43d800640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa41806c2b0 0x7fa41806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:11.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.842+0000 7fa43d800640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa41806c2b0 0x7fa41806e770 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009bd0 tx=0x7fa42c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:11.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:11.849+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa43004b020 con 0x7fa438155000 2024-09-16T01:53:11.894 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:11.894 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:11.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T01:53:12.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.047+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} v 0) -- 0x7fa4381548f0 con 0x7fa438155000 2024-09-16T01:53:12.069 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.069 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T01:53:12.241 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.241 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.254 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T01:53:12.412 INFO:teuthology.orchestra.run.smithi146.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.412 INFO:teuthology.orchestra.run.smithi146.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.425 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T01:53:12.542 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.542 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.552 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T01:53:12.656 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.656 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.665 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T01:53:12.769 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.769 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.777 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T01:53:12.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.813+0000 7fa435600640 1 -- 172.21.15.146:0/2703238438 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]=0 set osd.6 primary-affinity to 0 (802) v131) ==== 129+0+0 (secure 0 0 0) 0x7fa430079650 con 0x7fa438155000 2024-09-16T01:53:12.814 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-16T01:53:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa41806c2b0 msgr2=0x7fa41806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa41806c2b0 0x7fa41806e770 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009bd0 tx=0x7fa42c00b040 comp rx=0 tx=0).stop 2024-09-16T01:53:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 msgr2=0x7fa43814c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 0x7fa43814c380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa4380cdf80 tx=0x7fa4300046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 shutdown_connections 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa41806c2b0 0x7fa41806e770 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa438155000 0x7fa43814c380 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4380d0dc0 0x7fa4380c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 --2- 172.21.15.146:0/2703238438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4380d09e0 0x7fa4380c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 >> 172.21.15.146:0/2703238438 conn(0x7fa43806c920 msgr2=0x7fa4380cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 shutdown_connections 2024-09-16T01:53:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.816+0000 7fa440200640 1 -- 172.21.15.146:0/2703238438 wait complete. 2024-09-16T01:53:12.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T01:53:12.885 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.886 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:12.894 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T01:53:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.988+0000 7fc878a00640 1 Processor -- start 2024-09-16T01:53:12.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.988+0000 7fc878a00640 1 -- start start 2024-09-16T01:53:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc878a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc874157d40 0x7fc8740c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc878a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8740c1150 con 0x7fc8741515f0 2024-09-16T01:53:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc878a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc874073350 con 0x7fc8741519d0 2024-09-16T01:53:12.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc878a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc874073630 con 0x7fc874157d40 2024-09-16T01:53:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc873400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc8741515f0 0x7fc8740c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:59900/0 (socket says 172.21.15.146:59900) 2024-09-16T01:53:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc873400640 1 -- 172.21.15.146:0/1762099775 learned_addr learned my addr 172.21.15.146:0/1762099775 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc872a00640 1 --1- 172.21.15.146:0/1762099775 >> v1:172.21.15.161:6789/0 conn(0x7fc8741519d0 0x7fc874157630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:46254/0 (socket says 172.21.15.146:46254) 2024-09-16T01:53:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.989+0000 7fc873e00640 1 --2- 172.21.15.146:0/1762099775 >> v2:172.21.15.146:3300/0 conn(0x7fc874157d40 0x7fc8740c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.990+0000 7fc872000640 1 -- 172.21.15.146:0/1762099775 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035147475 0 0) 0x7fc874073350 con 0x7fc8741519d0 2024-09-16T01:53:12.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.990+0000 7fc872000640 1 -- 172.21.15.146:0/1762099775 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc858005730 con 0x7fc8741519d0 2024-09-16T01:53:12.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.990+0000 7fc873e00640 1 -- 172.21.15.146:0/1762099775 >> v1:172.21.15.161:6789/0 conn(0x7fc8741519d0 legacy=0x7fc874157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.990+0000 7fc873e00640 1 -- 172.21.15.146:0/1762099775 >> v1:172.21.15.146:6789/0 conn(0x7fc8741515f0 legacy=0x7fc8740c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.990+0000 7fc872000640 1 -- 172.21.15.146:0/1762099775 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 248492772 0 0) 0x7fc8740c1150 con 0x7fc8741515f0 2024-09-16T01:53:12.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.990+0000 7fc873e00640 1 -- 172.21.15.146:0/1762099775 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8740c13f0 con 0x7fc874157d40 2024-09-16T01:53:12.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.991+0000 7fc873e00640 1 --2- 172.21.15.146:0/1762099775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc8740c8060 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009d90 tx=0x7fc86c02a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65694b97b220108f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.991+0000 7fc872000640 1 -- 172.21.15.146:0/1762099775 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c037070 con 0x7fc874157d40 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.991+0000 7fc872000640 1 -- 172.21.15.146:0/1762099775 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86c007740 con 0x7fc874157d40 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.991+0000 7fc872000640 1 -- 172.21.15.146:0/1762099775 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c007af0 con 0x7fc874157d40 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 -- 172.21.15.146:0/1762099775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 msgr2=0x7fc8740c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 --2- 172.21.15.146:0/1762099775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc8740c8060 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009d90 tx=0x7fc86c02a780 comp rx=0 tx=0).stop 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 -- 172.21.15.146:0/1762099775 shutdown_connections 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 --2- 172.21.15.146:0/1762099775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc8740c8060 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 -- 172.21.15.146:0/1762099775 >> 172.21.15.146:0/1762099775 conn(0x7fc874067950 msgr2=0x7fc874067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:12.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 -- 172.21.15.146:0/1762099775 shutdown_connections 2024-09-16T01:53:12.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.992+0000 7fc878a00640 1 -- 172.21.15.146:0/1762099775 wait complete. 2024-09-16T01:53:12.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.993+0000 7fc878a00640 1 Processor -- start 2024-09-16T01:53:12.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.994+0000 7fc878a00640 1 -- start start 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.994+0000 7fc878a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 0x7fc87406f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.994+0000 7fc878a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8741519d0 0x7fc87414a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.994+0000 7fc878a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc874143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.994+0000 7fc878a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc874074060 con 0x7fc874157d40 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc878a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc8740c19b0 con 0x7fc8741519d0 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc878a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc8740741e0 con 0x7fc8741515f0 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 0x7fc87406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 0x7fc87406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44172/0 (socket says 172.21.15.146:44172) 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 -- 172.21.15.146:0/3543769033 learned_addr learned my addr 172.21.15.146:0/3543769033 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc872a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8741519d0 0x7fc87414a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:12.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873e00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc874143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 -- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8741519d0 msgr2=0x7fc87414a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8741519d0 0x7fc87414a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 -- 172.21.15.146:0/3543769033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 msgr2=0x7fc874143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc874143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 -- 172.21.15.146:0/3543769033 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc874148440 con 0x7fc8741515f0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc872a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8741519d0 0x7fc87414a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873e00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc874143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.995+0000 7fc873400640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 0x7fc87406f600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c500 tx=0x7fc86400c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.996+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864018070 con 0x7fc8741515f0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.996+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc864007580 con 0x7fc8741515f0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.996+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc874148720 con 0x7fc8741515f0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.996+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864004a20 con 0x7fc8741515f0 2024-09-16T01:53:12.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.996+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8741e6530 con 0x7fc8741515f0 2024-09-16T01:53:12.999 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:12.999 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.997+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc874149f70 con 0x7fc8741515f0 2024-09-16T01:53:13.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.998+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc86401b070 con 0x7fc8741515f0 2024-09-16T01:53:13.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.999+0000 7fc86b400640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:13.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.999+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(131..131 src has 1..131) ==== 5894+0+0 (secure 0 0 0) 0x7fc86408a830 con 0x7fc8741515f0 2024-09-16T01:53:13.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:12.999+0000 7fc872a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:13.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.000+0000 7fc872a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc8740c6710 tx=0x7fc860007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:13.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.003+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc8640587f0 con 0x7fc8741515f0 2024-09-16T01:53:13.008 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T01:53:13.111 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.111 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.119 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T01:53:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.184+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fc8740b14d0 con 0x7fc8741515f0 2024-09-16T01:53:13.222 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.222 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T01:53:13.334 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.334 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.343 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T01:53:13.448 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.448 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.456 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T01:53:13.560 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.560 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.569 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T01:53:13.672 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.672 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.681 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T01:53:13.784 INFO:teuthology.orchestra.run.smithi161.stdout:ERROR: (22) Invalid argument 2024-09-16T01:53:13.784 INFO:teuthology.orchestra.run.smithi161.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T01:53:13.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.818+0000 7fc86b400640 1 -- 172.21.15.146:0/3543769033 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v132) ==== 91+0+0 (secure 0 0 0) 0x7fc86405caf0 con 0x7fc8741515f0 2024-09-16T01:53:13.819 INFO:teuthology.orchestra.run.smithi146.stderr:noscrub is unset 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 msgr2=0x7fc85406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc8740c6710 tx=0x7fc860007d30 comp rx=0 tx=0).stop 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 msgr2=0x7fc87406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 0x7fc87406f600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c500 tx=0x7fc86400c9c0 comp rx=0 tx=0).stop 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 shutdown_connections 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc85406c2b0 0x7fc85406e770 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:13.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc874157d40 0x7fc874143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:13.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8741519d0 0x7fc87414a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:13.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 --2- 172.21.15.146:0/3543769033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc8741515f0 0x7fc87406f600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:13.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 >> 172.21.15.146:0/3543769033 conn(0x7fc874067950 msgr2=0x7fc874074f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:13.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.824+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 shutdown_connections 2024-09-16T01:53:13.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:13.826+0000 7fc878a00640 1 -- 172.21.15.146:0/3543769033 wait complete. 2024-09-16T01:53:13.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T01:53:14.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.002+0000 7fa0cf200640 1 Processor -- start 2024-09-16T01:53:14.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.002+0000 7fa0cf200640 1 -- start start 2024-09-16T01:53:14.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.003+0000 7fa0cf200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0d0157d40 0x7fa0d00c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:14.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.003+0000 7fa0cf200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00c1150 con 0x7fa0d01515f0 2024-09-16T01:53:14.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.003+0000 7fa0cf200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d0073350 con 0x7fa0d01519d0 2024-09-16T01:53:14.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.003+0000 7fa0cf200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d0073630 con 0x7fa0d0157d40 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0cde00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa0d01515f0 0x7fa0d00c8a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:59908/0 (socket says 172.21.15.146:59908) 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0cde00640 1 -- 172.21.15.146:0/399764171 learned_addr learned my addr 172.21.15.146:0/399764171 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0ce800640 1 --2- 172.21.15.146:0/399764171 >> v2:172.21.15.161:3300/0 conn(0x7fa0d0157d40 0x7fa0d00c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0cd400640 1 -- 172.21.15.146:0/399764171 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398356062 0 0) 0x7fa0d00c1150 con 0x7fa0d01515f0 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0cd400640 1 -- 172.21.15.146:0/399764171 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0b4005730 con 0x7fa0d01515f0 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0cd400640 1 -- 172.21.15.146:0/399764171 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539334343 0 0) 0x7fa0d0073350 con 0x7fa0d01519d0 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0ce800640 1 -- 172.21.15.146:0/399764171 >> v1:172.21.15.161:6789/0 conn(0x7fa0d01519d0 legacy=0x7fa0d0157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0ce800640 1 -- 172.21.15.146:0/399764171 >> v1:172.21.15.146:6789/0 conn(0x7fa0d01515f0 legacy=0x7fa0d00c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.004+0000 7fa0ce800640 1 -- 172.21.15.146:0/399764171 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00c13f0 con 0x7fa0d0157d40 2024-09-16T01:53:14.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.005+0000 7fa0ce800640 1 --2- 172.21.15.146:0/399764171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d0157d40 0x7fa0d00c8060 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009d90 tx=0x7fa0c802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81a1cd5944dc373d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:14.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.005+0000 7fa0cd400640 1 -- 172.21.15.146:0/399764171 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8037070 con 0x7fa0d0157d40 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cd400640 1 -- 172.21.15.146:0/399764171 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c8007740 con 0x7fa0d0157d40 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cd400640 1 -- 172.21.15.146:0/399764171 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8007b10 con 0x7fa0d0157d40 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 -- 172.21.15.146:0/399764171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d0157d40 msgr2=0x7fa0d00c8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 --2- 172.21.15.146:0/399764171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d0157d40 0x7fa0d00c8060 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009d90 tx=0x7fa0c802a780 comp rx=0 tx=0).stop 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 -- 172.21.15.146:0/399764171 shutdown_connections 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 --2- 172.21.15.146:0/399764171 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d0157d40 0x7fa0d00c8060 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 -- 172.21.15.146:0/399764171 >> 172.21.15.146:0/399764171 conn(0x7fa0d0067950 msgr2=0x7fa0d0067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 -- 172.21.15.146:0/399764171 shutdown_connections 2024-09-16T01:53:14.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.006+0000 7fa0cf200640 1 -- 172.21.15.146:0/399764171 wait complete. 2024-09-16T01:53:14.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.008+0000 7fa0cf200640 1 Processor -- start 2024-09-16T01:53:14.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.008+0000 7fa0cf200640 1 -- start start 2024-09-16T01:53:14.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cf200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 0x7fa0d01ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:14.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0d01519d0 0x7fa0d01ece40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:14.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cf200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0d0157d40 0x7fa0d014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:14.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d0074060 con 0x7fa0d01519d0 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cf200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00c19b0 con 0x7fa0d01515f0 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cf200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d00741e0 con 0x7fa0d0157d40 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 0x7fa0d01ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 0x7fa0d01ec5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51334/0 (socket says 172.21.15.146:51334) 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 -- 172.21.15.146:0/3119131423 learned_addr learned my addr 172.21.15.146:0/3119131423 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0c5400640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0d01519d0 0x7fa0d01ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0ce800640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0d0157d40 0x7fa0d014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 -- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0d0157d40 msgr2=0x7fa0d014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0d0157d40 0x7fa0d014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 -- 172.21.15.146:0/3119131423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0d01519d0 msgr2=0x7fa0d01ece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0d01519d0 0x7fa0d01ece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.009+0000 7fa0cde00640 1 -- 172.21.15.146:0/3119131423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d014b450 con 0x7fa0d01515f0 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.010+0000 7fa0cde00640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 0x7fa0d01ec5c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c430 tx=0x7fa0c000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.010+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0017070 con 0x7fa0d01515f0 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.010+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c0005ce0 con 0x7fa0d01515f0 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.010+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00072f0 con 0x7fa0d01515f0 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.010+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01efaf0 con 0x7fa0d01515f0 2024-09-16T01:53:14.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.010+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d01ebdc0 con 0x7fa0d01515f0 2024-09-16T01:53:14.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.011+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d00723d0 con 0x7fa0d01515f0 2024-09-16T01:53:14.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.012+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa0c001a070 con 0x7fa0d01515f0 2024-09-16T01:53:14.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.013+0000 7fa0c6a00640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0ac06c2b0 0x7fa0ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:14.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.013+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5894+0+0 (secure 0 0 0) 0x7fa0c008a960 con 0x7fa0d01515f0 2024-09-16T01:53:14.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.013+0000 7fa0c5400640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0ac06c2b0 0x7fa0ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:14.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.013+0000 7fa0c5400640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0ac06c2b0 0x7fa0ac06e770 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00b16e0 tx=0x7fa0b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:14.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.017+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa0c0058930 con 0x7fa0d01515f0 2024-09-16T01:53:14.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.199+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fa0d0073bf0 con 0x7fa0d01515f0 2024-09-16T01:53:14.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.837+0000 7fa0c6a00640 1 -- 172.21.15.146:0/3119131423 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v133) ==== 101+0+0 (secure 0 0 0) 0x7fa0c005cc30 con 0x7fa0d01515f0 2024-09-16T01:53:14.838 INFO:teuthology.orchestra.run.smithi146.stderr:nodeep-scrub is unset 2024-09-16T01:53:14.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0ac06c2b0 msgr2=0x7fa0ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0ac06c2b0 0x7fa0ac06e770 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00b16e0 tx=0x7fa0b8009040 comp rx=0 tx=0).stop 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 msgr2=0x7fa0d01ec5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 0x7fa0d01ec5c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c430 tx=0x7fa0c000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 shutdown_connections 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa0ac06c2b0 0x7fa0ac06e770 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0d0157d40 0x7fa0d014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0d01519d0 0x7fa0d01ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 --2- 172.21.15.146:0/3119131423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0d01515f0 0x7fa0d01ec5c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.839+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 >> 172.21.15.146:0/3119131423 conn(0x7fa0d0067950 msgr2=0x7fa0d00c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:14.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.840+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 shutdown_connections 2024-09-16T01:53:14.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:14.840+0000 7fa0cf200640 1 -- 172.21.15.146:0/3119131423 wait complete. 2024-09-16T01:53:17.839 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T01:53:17.839 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-16T01:53:17.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-16T01:53:18.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.003+0000 7f0888600640 1 Processor -- start 2024-09-16T01:53:18.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.003+0000 7f0888600640 1 -- start start 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.004+0000 7f0888600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08800c8c00 0x7f08800c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0888600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08800c33c0 con 0x7f0880070950 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0888600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08800c36a0 con 0x7f08801528e0 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0888600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f08801596d0 con 0x7f08800c8c00 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0885200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0880070950 0x7f08800c93c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:59912/0 (socket says 172.21.15.146:59912) 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0885200640 1 -- 172.21.15.146:0/2213614653 learned_addr learned my addr 172.21.15.146:0/2213614653 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:18.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0886600640 1 --2- 172.21.15.146:0/2213614653 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08800c8c00 0x7f08800c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2882490934 0 0) 0x7f08800c33c0 con 0x7f0880070950 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0868003520 con 0x7f0880070950 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296590294 0 0) 0x7f08800c36a0 con 0x7f08801528e0 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08800c33c0 con 0x7f08801528e0 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0886600640 1 -- 172.21.15.146:0/2213614653 >> v1:172.21.15.161:6789/0 conn(0x7f08801528e0 legacy=0x7f0880158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0886600640 1 -- 172.21.15.146:0/2213614653 >> v1:172.21.15.146:6789/0 conn(0x7f0880070950 legacy=0x7f08800c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.005+0000 7f0886600640 1 -- 172.21.15.146:0/2213614653 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08801587c0 con 0x7f08800c8c00 2024-09-16T01:53:18.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.006+0000 7f0886600640 1 --2- 172.21.15.146:0/2213614653 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08800c8c00 0x7f08800c4fa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0878009dc0 tx=0x7f087802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54406b085165a6fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:18.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878007740 con 0x7f08800c8c00 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08780078e0 con 0x7f08800c8c00 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f087fe00640 1 -- 172.21.15.146:0/2213614653 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878007cb0 con 0x7f08800c8c00 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 -- 172.21.15.146:0/2213614653 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08800c8c00 msgr2=0x7f08800c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 --2- 172.21.15.146:0/2213614653 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08800c8c00 0x7f08800c4fa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0878009dc0 tx=0x7f087802a760 comp rx=0 tx=0).stop 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 -- 172.21.15.146:0/2213614653 shutdown_connections 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 --2- 172.21.15.146:0/2213614653 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08800c8c00 0x7f08800c4fa0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 -- 172.21.15.146:0/2213614653 >> 172.21.15.146:0/2213614653 conn(0x7f0880067950 msgr2=0x7f0880067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 -- 172.21.15.146:0/2213614653 shutdown_connections 2024-09-16T01:53:18.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.007+0000 7f0888600640 1 -- 172.21.15.146:0/2213614653 wait complete. 2024-09-16T01:53:18.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.009+0000 7f0888600640 1 Processor -- start 2024-09-16T01:53:18.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.009+0000 7f0888600640 1 -- start start 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0888600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 0x7f08801f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0888600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08800c8c00 0x7f088015ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0888600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08801528e0 0x7f08800c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0888600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f088015bca0 con 0x7f0880070950 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0888600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f08800c3980 con 0x7f08800c8c00 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0888600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f08800c6c30 con 0x7f08801528e0 2024-09-16T01:53:18.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0885c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 0x7f08801f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0885c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 0x7f08801f0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54986/0 (socket says 172.21.15.146:54986) 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0885c00640 1 -- 172.21.15.146:0/1207057065 learned_addr learned my addr 172.21.15.146:0/1207057065 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0885200640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08800c8c00 0x7f088015ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.010+0000 7f0886600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08801528e0 0x7f08800c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0885c00640 1 -- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08801528e0 msgr2=0x7f08800c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0885c00640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08801528e0 0x7f08800c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0885c00640 1 -- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08800c8c00 msgr2=0x7f088015ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0885c00640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08800c8c00 0x7f088015ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0885c00640 1 -- 172.21.15.146:0/1207057065 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08800c8310 con 0x7f0880070950 2024-09-16T01:53:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0885c00640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 0x7f08801f0b90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f087400c460 tx=0x7f087400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:18.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740052d0 con 0x7f0880070950 2024-09-16T01:53:18.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0874005490 con 0x7f0880070950 2024-09-16T01:53:18.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874012870 con 0x7f0880070950 2024-09-16T01:53:18.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08800c85b0 con 0x7f0880070950 2024-09-16T01:53:18.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.011+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08800cc0a0 con 0x7f0880070950 2024-09-16T01:53:18.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.012+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08800c4fa0 con 0x7f0880070950 2024-09-16T01:53:18.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.013+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0874012a10 con 0x7f0880070950 2024-09-16T01:53:18.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.014+0000 7f087d600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f086006c2b0 0x7f086006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:18.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.014+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5894+0+0 (secure 0 0 0) 0x7f087408afb0 con 0x7f0880070950 2024-09-16T01:53:18.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.014+0000 7f0885200640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f086006c2b0 0x7f086006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:18.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.015+0000 7f0885200640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f086006c2b0 0x7f086006e770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f086c0011e0 tx=0x7f086c007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:18.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.018+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0874059000 con 0x7f0880070950 2024-09-16T01:53:18.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.202+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd in", "ids": ["0"]} v 0) -- 0x7f0880067ed0 con 0x7f0880070950 2024-09-16T01:53:18.204 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:53:18.203+0000 7f49bee00640 -1 mon.a@0(leader).osd e133 definitely_dead 0 2024-09-16T01:53:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.841+0000 7f087d600640 1 -- 172.21.15.146:0/1207057065 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["0"]}]=0 marked in osd.0. v134) ==== 85+0+0 (secure 0 0 0) 0x7f087405d300 con 0x7f0880070950 2024-09-16T01:53:18.842 INFO:teuthology.orchestra.run.smithi146.stderr:marked in osd.0. 2024-09-16T01:53:18.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f086006c2b0 msgr2=0x7f086006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f086006c2b0 0x7f086006e770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f086c0011e0 tx=0x7f086c007d30 comp rx=0 tx=0).stop 2024-09-16T01:53:18.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 msgr2=0x7f08801f0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:18.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 0x7f08801f0b90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f087400c460 tx=0x7f087400c920 comp rx=0 tx=0).stop 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 shutdown_connections 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f086006c2b0 0x7f086006e770 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08801528e0 0x7f08800c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f08800c8c00 0x7f088015ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 --2- 172.21.15.146:0/1207057065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0880070950 0x7f08801f0b90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.848+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 >> 172.21.15.146:0/1207057065 conn(0x7f0880067950 msgr2=0x7f08800ca2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.849+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 shutdown_connections 2024-09-16T01:53:18.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:18.849+0000 7f0888600640 1 -- 172.21.15.146:0/1207057065 wait complete. 2024-09-16T01:53:18.880 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-16T01:53:18.880 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T01:53:18.880 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-16T01:53:18.880 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-09-16T01:53:19.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.036+0000 7f55c3200640 1 Processor -- start 2024-09-16T01:53:19.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.036+0000 7f55c3200640 1 -- start start 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.037+0000 7f55c3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc152850 0x7f55bc158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.037+0000 7f55c3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55bc158bb0 0x7f55bc0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.037+0000 7f55c3200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55bc0c6bb0 0x7f55bc0c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.037+0000 7f55c3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f55bc073210 con 0x7f55bc158bb0 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.037+0000 7f55c3200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f55bc072f50 con 0x7f55bc0c6bb0 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.037+0000 7f55c3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc15cb50 con 0x7f55bc152850 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55bc158bb0 0x7f55bc0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55bc158bb0 0x7f55bc0c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55002/0 (socket says 172.21.15.146:55002) 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 -- 172.21.15.146:0/2545549877 learned_addr learned my addr 172.21.15.146:0/2545549877 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55bbe00640 1 --2- 172.21.15.146:0/2545549877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc152850 0x7f55bc158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55c1200640 1 --2- 172.21.15.146:0/2545549877 >> v2:172.21.15.161:3300/0 conn(0x7f55bc0c6bb0 0x7f55bc0c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 -- 172.21.15.146:0/2545549877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc152850 msgr2=0x7f55bc158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:19.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 --2- 172.21.15.146:0/2545549877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc152850 0x7f55bc158670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 -- 172.21.15.146:0/2545549877 >> v2:172.21.15.161:3300/0 conn(0x7f55bc0c6bb0 msgr2=0x7f55bc0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:19.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 --2- 172.21.15.146:0/2545549877 >> v2:172.21.15.161:3300/0 conn(0x7f55bc0c6bb0 0x7f55bc0c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.038+0000 7f55b3400640 1 -- 172.21.15.146:0/2545549877 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc15bd90 con 0x7f55bc158bb0 2024-09-16T01:53:19.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.039+0000 7f55b3400640 1 --2- 172.21.15.146:0/2545549877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc158bb0 0x7f55bc0c58c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0c3330 tx=0x7f55a402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593b56ab28837a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.039+0000 7f55bb400640 1 -- 172.21.15.146:0/2545549877 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4037070 con 0x7f55bc158bb0 2024-09-16T01:53:19.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55bb400640 1 -- 172.21.15.146:0/2545549877 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a4032440 con 0x7f55bc158bb0 2024-09-16T01:53:19.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55bb400640 1 -- 172.21.15.146:0/2545549877 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4032810 con 0x7f55bc158bb0 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 -- 172.21.15.146:0/2545549877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc158bb0 msgr2=0x7f55bc0c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 --2- 172.21.15.146:0/2545549877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc158bb0 0x7f55bc0c58c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0c3330 tx=0x7f55a402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 -- 172.21.15.146:0/2545549877 shutdown_connections 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 --2- 172.21.15.146:0/2545549877 >> v2:172.21.15.161:3300/0 conn(0x7f55bc0c6bb0 0x7f55bc0c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 --2- 172.21.15.146:0/2545549877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc158bb0 0x7f55bc0c58c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 --2- 172.21.15.146:0/2545549877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc152850 0x7f55bc158670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 -- 172.21.15.146:0/2545549877 >> 172.21.15.146:0/2545549877 conn(0x7f55bc06f3c0 msgr2=0x7f55bc06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 -- 172.21.15.146:0/2545549877 shutdown_connections 2024-09-16T01:53:19.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.040+0000 7f55c3200640 1 -- 172.21.15.146:0/2545549877 wait complete. 2024-09-16T01:53:19.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.042+0000 7f55c3200640 1 Processor -- start 2024-09-16T01:53:19.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.042+0000 7f55c3200640 1 -- start start 2024-09-16T01:53:19.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55bc0c6bb0 0x7f55bc1f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 0x7f55bc0c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc158bb0 0x7f55bc0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc15c630 con 0x7f55bc152850 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc15c350 con 0x7f55bc0c6bb0 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc070200 con 0x7f55bc158bb0 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 0x7f55bc0c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 0x7f55bc0c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55004/0 (socket says 172.21.15.146:55004) 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 -- 172.21.15.146:0/288802444 learned_addr learned my addr 172.21.15.146:0/288802444 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55c1200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc158bb0 0x7f55bc0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55bbe00640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55bc0c6bb0 0x7f55bc1f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 -- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc158bb0 msgr2=0x7f55bc0c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc158bb0 0x7f55bc0c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 -- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55bc0c6bb0 msgr2=0x7f55bc1f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55bc0c6bb0 0x7f55bc1f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.043+0000 7f55b3400640 1 -- 172.21.15.146:0/288802444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc1f4310 con 0x7f55bc152850 2024-09-16T01:53:19.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.044+0000 7f55b3400640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 0x7f55bc0c41c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f55a402a220 tx=0x7f55a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.044+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4041070 con 0x7f55bc152850 2024-09-16T01:53:19.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.044+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a403c890 con 0x7f55bc152850 2024-09-16T01:53:19.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.044+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4037040 con 0x7f55bc152850 2024-09-16T01:53:19.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.044+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc0c2e70 con 0x7f55bc152850 2024-09-16T01:53:19.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.044+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc0c3430 con 0x7f55bc152850 2024-09-16T01:53:19.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.045+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55bc0718a0 con 0x7f55bc152850 2024-09-16T01:53:19.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.046+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f55a4044070 con 0x7f55bc152850 2024-09-16T01:53:19.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.047+0000 7f55b8c00640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f559806c2b0 0x7f559806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.047+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5894+0+0 (secure 0 0 0) 0x7f55a40abd60 con 0x7f55bc152850 2024-09-16T01:53:19.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.047+0000 7f55bbe00640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f559806c2b0 0x7f559806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.048+0000 7f55bbe00640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f559806c2b0 0x7f559806e770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009b40 tx=0x7f55ac009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.051+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f55a4079ed0 con 0x7f55bc152850 2024-09-16T01:53:19.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.235+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd in", "ids": ["4"]} v 0) -- 0x7f55bc06e100 con 0x7f55bc152850 2024-09-16T01:53:19.237 INFO:tasks.ceph.mon.a.smithi146.stderr:2024-09-16T01:53:19.236+0000 7f49bee00640 -1 mon.a@0(leader).osd e134 definitely_dead 0 2024-09-16T01:53:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.493+0000 7f55b8c00640 1 -- 172.21.15.146:0/288802444 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["4"]}]=0 marked in osd.4. v135) ==== 85+0+0 (secure 0 0 0) 0x7f55a407e1d0 con 0x7f55bc152850 2024-09-16T01:53:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:marked in osd.4. 2024-09-16T01:53:19.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.496+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f559806c2b0 msgr2=0x7f559806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.497+0000 7f55c3200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f559806c2b0 0x7f559806e770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009b40 tx=0x7f55ac009190 comp rx=0 tx=0).stop 2024-09-16T01:53:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.497+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 msgr2=0x7f55bc0c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.497+0000 7f55c3200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 0x7f55bc0c41c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f55a402a220 tx=0x7f55a40046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.498+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 shutdown_connections 2024-09-16T01:53:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.498+0000 7f55c3200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f559806c2b0 0x7f559806e770 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.498+0000 7f55c3200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55bc158bb0 0x7f55bc0c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.498+0000 7f55c3200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55bc152850 0x7f55bc0c41c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.498+0000 7f55c3200640 1 --2- 172.21.15.146:0/288802444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55bc0c6bb0 0x7f55bc1f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.498+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 >> 172.21.15.146:0/288802444 conn(0x7f55bc06f3c0 msgr2=0x7f55bc0c8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:19.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.499+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 shutdown_connections 2024-09-16T01:53:19.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.499+0000 7f55c3200640 1 -- 172.21.15.146:0/288802444 wait complete. 2024-09-16T01:53:19.532 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-16T01:53:19.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T01:53:19.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.690+0000 7f72ace00640 1 Processor -- start 2024-09-16T01:53:19.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.691+0000 7f72ace00640 1 -- start start 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72ace00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72a8152c30 0x7f72a80c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72ace00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72a80c6ab0 0x7f72a80c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72ace00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a80c4240 con 0x7f72a80c6e90 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72ace00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f72a80c4da0 con 0x7f72a80c6ab0 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72ace00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f72a80c4500 con 0x7f72a8152c30 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72a80c6ab0 0x7f72a80c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72a8152c30 0x7f72a80c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a6800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f72a80c6e90 0x7f72a80c7ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:46276/0 (socket says 172.21.15.146:46276) 2024-09-16T01:53:19.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a6800640 1 -- 172.21.15.146:0/167521336 learned_addr learned my addr 172.21.15.146:0/167521336 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:19.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5400640 1 -- 172.21.15.146:0/167521336 >> v2:172.21.15.161:3300/0 conn(0x7f72a8152c30 msgr2=0x7f72a80c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:19.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5400640 1 --2- 172.21.15.146:0/167521336 >> v2:172.21.15.161:3300/0 conn(0x7f72a8152c30 0x7f72a80c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5400640 1 -- 172.21.15.146:0/167521336 >> v1:172.21.15.161:6789/0 conn(0x7f72a80c6e90 legacy=0x7f72a80c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5400640 1 -- 172.21.15.146:0/167521336 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a80c47e0 con 0x7f72a80c6ab0 2024-09-16T01:53:19.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a4a00640 1 -- 172.21.15.146:0/167521336 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461113474 0 0) 0x7f72a80c4240 con 0x7f72a80c6e90 2024-09-16T01:53:19.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.692+0000 7f72a5e00640 1 --2- 172.21.15.146:0/167521336 >> v2:172.21.15.161:3300/0 conn(0x7f72a8152c30 0x7f72a80c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:19.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.693+0000 7f72a5400640 1 --2- 172.21.15.146:0/167521336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6ab0 0x7f72a80c8c40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f72a8071690 tx=0x7f729402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442bb74e937174b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72a4a00640 1 -- 172.21.15.146:0/167521336 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294037070 con 0x7f72a80c6ab0 2024-09-16T01:53:19.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72a4a00640 1 -- 172.21.15.146:0/167521336 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7294032440 con 0x7f72a80c6ab0 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72a4a00640 1 -- 172.21.15.146:0/167521336 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294032810 con 0x7f72a80c6ab0 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 -- 172.21.15.146:0/167521336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6ab0 msgr2=0x7f72a80c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 --2- 172.21.15.146:0/167521336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6ab0 0x7f72a80c8c40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f72a8071690 tx=0x7f729402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 -- 172.21.15.146:0/167521336 shutdown_connections 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 --2- 172.21.15.146:0/167521336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6ab0 0x7f72a80c8c40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 --2- 172.21.15.146:0/167521336 >> v2:172.21.15.161:3300/0 conn(0x7f72a8152c30 0x7f72a80c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 -- 172.21.15.146:0/167521336 >> 172.21.15.146:0/167521336 conn(0x7f72a806bc10 msgr2=0x7f72a806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 -- 172.21.15.146:0/167521336 shutdown_connections 2024-09-16T01:53:19.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.694+0000 7f72ace00640 1 -- 172.21.15.146:0/167521336 wait complete. 2024-09-16T01:53:19.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.696+0000 7f72ace00640 1 Processor -- start 2024-09-16T01:53:19.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.696+0000 7f72ace00640 1 -- start start 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72ace00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a80c6ab0 0x7f72a81f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72ace00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6e90 0x7f72a8149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72ace00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a8152c30 0x7f72a8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72ace00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f72a80c8700 con 0x7f72a80c6e90 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72ace00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72a80c6630 con 0x7f72a8152c30 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72ace00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f72a80c8880 con 0x7f72a80c6ab0 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72a5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6e90 0x7f72a8149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72a5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6e90 0x7f72a8149bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55008/0 (socket says 172.21.15.146:55008) 2024-09-16T01:53:19.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72a5400640 1 -- 172.21.15.146:0/130230711 learned_addr learned my addr 172.21.15.146:0/130230711 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a8152c30 0x7f72a8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.697+0000 7f72a5e00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a80c6ab0 0x7f72a81f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a5400640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a80c6ab0 msgr2=0x7f72a81f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a5400640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a80c6ab0 0x7f72a81f4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a5400640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a8152c30 msgr2=0x7f72a8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a5400640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a8152c30 0x7f72a8153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a5400640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a814be80 con 0x7f72a80c6e90 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a8152c30 0x7f72a8153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:19.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72a5400640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6e90 0x7f72a8149bc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7294000c00 tx=0x7f72940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294037050 con 0x7f72a80c6e90 2024-09-16T01:53:19.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7294038040 con 0x7f72a80c6e90 2024-09-16T01:53:19.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294004450 con 0x7f72a80c6e90 2024-09-16T01:53:19.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.698+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a814c160 con 0x7f72a80c6e90 2024-09-16T01:53:19.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.699+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a8153620 con 0x7f72a80c6e90 2024-09-16T01:53:19.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.699+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f72a80c68a0 con 0x7f72a80c6e90 2024-09-16T01:53:19.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.700+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f729403c6d0 con 0x7f72a80c6e90 2024-09-16T01:53:19.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.701+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f728806c2b0 0x7f728806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.701+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6452+0+0 (secure 0 0 0) 0x7f72940ac080 con 0x7f72a80c6e90 2024-09-16T01:53:19.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.701+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+1289 (secure 0 0 0) 0x7f72940ae6e0 con 0x7f72a80c6e90 2024-09-16T01:53:19.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.701+0000 7f72a5e00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f728806c2b0 0x7f728806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.702+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 0x7f72a81537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.702+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81f4ac0 con 0x7f72a80c9150 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.702+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 0x7f72a81537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.702+0000 7f72a5e00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f728806c2b0 0x7f728806e770 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7298009560 tx=0x7f7298009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.702+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f72a80c4240 con 0x7f72a80c9150 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 0x7f72a81537f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f72a0000f80 tx=0x7f72a000f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a000fc70 con 0x7f72a80c9150 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72a0018070 con 0x7f72a80c9150 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 msgr2=0x7f72a81537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 0x7f72a81537f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f72a0000f80 tx=0x7f72a000f5a0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 msgr2=0x7f72a81537f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:53:19.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.703+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c9150 0x7f72a81537f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.709+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 0x7f72a8148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.709+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81493d0 con 0x7f72a8148ab0 2024-09-16T01:53:19.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.710+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 0x7f72a8148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.710+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f72a8156100 con 0x7f72a8148ab0 2024-09-16T01:53:19.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.710+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 0x7f72a8148e90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f72a0000f80 tx=0x7f72a0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.710+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a002c0a0 con 0x7f72a8148ab0 2024-09-16T01:53:19.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.0 v2:172.21.15.146:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f72a0013ad0 con 0x7f72a8148ab0 2024-09-16T01:53:19.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 msgr2=0x7f72a8148e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 0x7f72a8148e90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f72a0000f80 tx=0x7f72a0001250 comp rx=0 tx=0).stop 2024-09-16T01:53:19.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 msgr2=0x7f72a8148e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:53:19.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a8148ab0 0x7f72a8148e90 secure :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f72a0000f80 tx=0x7f72a0001250 comp rx=0 tx=0).stop 2024-09-16T01:53:19.713 INFO:teuthology.orchestra.run.smithi146.stdout:mon.a: {} 2024-09-16T01:53:19.714 INFO:teuthology.orchestra.run.smithi146.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 0x7f72a81f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81533d0 con 0x7f72a81f05e0 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.712+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 0x7f72a81f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f72a8075960 con 0x7f72a81f05e0 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 0x7f72a81f09f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f72a0026000 tx=0x7f72a000f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0013e20 con 0x7f72a81f05e0 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72a0013e20 con 0x7f72a81f05e0 2024-09-16T01:53:19.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 msgr2=0x7f72a81f09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 0x7f72a81f09f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f72a0026000 tx=0x7f72a000f5a0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 msgr2=0x7f72a81f09f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:53:19.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 reap_dead start 2024-09-16T01:53:19.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.713+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f05e0 0x7f72a81f09f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.719+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 0x7f72a81f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.719+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81f1470 con 0x7f72a81f4580 2024-09-16T01:53:19.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.720+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 0x7f72a81f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.720+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f72a81547a0 con 0x7f72a81f4580 2024-09-16T01:53:19.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.721+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 0x7f72a81f0f30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f72a0013640 tx=0x7f72a002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.721+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a001bbe0 con 0x7f72a81f4580 2024-09-16T01:53:19.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f72a001bbe0 con 0x7f72a81f4580 2024-09-16T01:53:19.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 msgr2=0x7f72a81f0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 0x7f72a81f0f30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f72a0013640 tx=0x7f72a002a000 comp rx=0 tx=0).stop 2024-09-16T01:53:19.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 msgr2=0x7f72a81f0f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:53:19.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a81f4580 0x7f72a81f0f30 secure :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f72a0013640 tx=0x7f72a002a000 comp rx=0 tx=0).stop 2024-09-16T01:53:19.724 INFO:teuthology.orchestra.run.smithi146.stdout:mon.b: {} 2024-09-16T01:53:19.725 INFO:teuthology.orchestra.run.smithi146.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T01:53:19.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 0x7f72a80c0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f72a814a100 con 0x7f72a8153e20 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.723+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 0x7f72a80c0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f72a814ba20 con 0x7f72a8153e20 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 0x7f72a80c0090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f72a0013fd0 tx=0x7f72a000f5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a003c870 con 0x7f72a8153e20 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.2 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72a003c670 con 0x7f72a8153e20 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 msgr2=0x7f72a80c0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 0x7f72a80c0090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f72a0013fd0 tx=0x7f72a000f5a0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 msgr2=0x7f72a80c0090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:53:19.726 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.724+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8153e20 0x7f72a80c0090 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f72a0013fd0 tx=0x7f72a000f5a0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.730+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 0x7f72a80c05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.730+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f72a814bc30 con 0x7f72a8154200 2024-09-16T01:53:19.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.730+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 0x7f72a80c05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.731+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f72a82040c0 con 0x7f72a8154200 2024-09-16T01:53:19.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.731+0000 7f72a6800640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 0x7f72a80c05d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f72a00013b0 tx=0x7f72a0040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.731+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0058070 con 0x7f72a8154200 2024-09-16T01:53:19.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 <== mon.2 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f72a00541b0 con 0x7f72a8154200 2024-09-16T01:53:19.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 msgr2=0x7f72a80c05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 0x7f72a80c05d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f72a00013b0 tx=0x7f72a0040000 comp rx=0 tx=0).stop 2024-09-16T01:53:19.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f729e000640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 msgr2=0x7f72a80c05d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T01:53:19.734 INFO:teuthology.orchestra.run.smithi146.stdout:mon.c: {} 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f729e000640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72a8154200 0x7f72a80c05d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f728806c2b0 msgr2=0x7f728806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f728806c2b0 0x7f728806e770 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7298009560 tx=0x7f7298009190 comp rx=0 tx=0).stop 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6e90 msgr2=0x7f72a8149bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 --2- 172.21.15.146:0/130230711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72a80c6e90 0x7f72a8149bc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7294000c00 tx=0x7f72940046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72a6800640 1 -- 172.21.15.146:0/130230711 reap_dead start 2024-09-16T01:53:19.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 shutdown_connections 2024-09-16T01:53:19.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 >> 172.21.15.146:0/130230711 conn(0x7f72a806bc10 msgr2=0x7f72a8151db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:19.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 shutdown_connections 2024-09-16T01:53:19.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.733+0000 7f72ace00640 1 -- 172.21.15.146:0/130230711 wait complete. 2024-09-16T01:53:19.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T01:53:19.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.904+0000 7f026d600640 1 Processor -- start 2024-09-16T01:53:19.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.904+0000 7f026d600640 1 -- start start 2024-09-16T01:53:19.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.905+0000 7f026d600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02680c8d60 0x7f026806e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.905+0000 7f026d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02680c52a0 con 0x7f0268154be0 2024-09-16T01:53:19.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.905+0000 7f026d600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026806cec0 con 0x7f0268070d60 2024-09-16T01:53:19.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.905+0000 7f026d600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f026806ffe0 con 0x7f02680c8d60 2024-09-16T01:53:19.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.905+0000 7f0267e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0268154be0 0x7f026815dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:59914/0 (socket says 172.21.15.146:59914) 2024-09-16T01:53:19.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0267e00640 1 -- 172.21.15.146:0/225411796 learned_addr learned my addr 172.21.15.146:0/225411796 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:19.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0267400640 1 --2- 172.21.15.146:0/225411796 >> v2:172.21.15.146:3300/0 conn(0x7f02680c8d60 0x7f026806e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0266a00640 1 -- 172.21.15.146:0/225411796 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690147629 0 0) 0x7f02680c52a0 con 0x7f0268154be0 2024-09-16T01:53:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0266a00640 1 -- 172.21.15.146:0/225411796 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f024c003520 con 0x7f0268154be0 2024-09-16T01:53:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0266a00640 1 -- 172.21.15.146:0/225411796 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69769833 0 0) 0x7f026806cec0 con 0x7f0268070d60 2024-09-16T01:53:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0267400640 1 -- 172.21.15.146:0/225411796 >> v1:172.21.15.161:6789/0 conn(0x7f0268070d60 legacy=0x7f02680ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0267400640 1 -- 172.21.15.146:0/225411796 >> v1:172.21.15.146:6789/0 conn(0x7f0268154be0 legacy=0x7f026815dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.906+0000 7f0267400640 1 -- 172.21.15.146:0/225411796 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026806d1a0 con 0x7f02680c8d60 2024-09-16T01:53:19.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.907+0000 7f0267400640 1 --2- 172.21.15.146:0/225411796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02680c8d60 0x7f026806e960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f02540097a0 tx=0x7f025402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602a6159fc129b41 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.907+0000 7f0266a00640 1 -- 172.21.15.146:0/225411796 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254037070 con 0x7f02680c8d60 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.907+0000 7f0266a00640 1 -- 172.21.15.146:0/225411796 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0254032440 con 0x7f02680c8d60 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.907+0000 7f0266a00640 1 -- 172.21.15.146:0/225411796 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254032810 con 0x7f02680c8d60 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 -- 172.21.15.146:0/225411796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02680c8d60 msgr2=0x7f026806e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 --2- 172.21.15.146:0/225411796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02680c8d60 0x7f026806e960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f02540097a0 tx=0x7f025402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 -- 172.21.15.146:0/225411796 shutdown_connections 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 --2- 172.21.15.146:0/225411796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02680c8d60 0x7f026806e960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 -- 172.21.15.146:0/225411796 >> 172.21.15.146:0/225411796 conn(0x7f026806c240 msgr2=0x7f0268067950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 -- 172.21.15.146:0/225411796 shutdown_connections 2024-09-16T01:53:19.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.908+0000 7f026d600640 1 -- 172.21.15.146:0/225411796 wait complete. 2024-09-16T01:53:19.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.909+0000 7f026d600640 1 Processor -- start 2024-09-16T01:53:19.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 -- start start 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 0x7f026814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02680c8d60 0x7f026814bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0268154be0 0x7f02681e7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0268070750 con 0x7f0268070d60 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f02680702c0 con 0x7f02680c8d60 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.910+0000 7f026d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f02680ce950 con 0x7f0268154be0 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 0x7f026814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02680c8d60 0x7f026814bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 0x7f026814b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55048/0 (socket says 172.21.15.146:55048) 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f026cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0268154be0 0x7f02681e7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 -- 172.21.15.146:0/4171901887 learned_addr learned my addr 172.21.15.146:0/4171901887 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:19.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 -- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0268154be0 msgr2=0x7f02681e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0268154be0 0x7f02681e7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 -- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02680c8d60 msgr2=0x7f026814bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02680c8d60 0x7f026814bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267e00640 1 -- 172.21.15.146:0/4171901887 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0268154520 con 0x7f0268070d60 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.911+0000 7f0267400640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02680c8d60 0x7f026814bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.912+0000 7f0267e00640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 0x7f026814b000 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f025800c3c0 tx=0x7f025800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.912+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0258017070 con 0x7f0268070d60 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.912+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0258005ce0 con 0x7f0268070d60 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.912+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02681f16b0 con 0x7f0268070d60 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.912+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02580072f0 con 0x7f0268070d60 2024-09-16T01:53:19.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.912+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02681e5a80 con 0x7f0268070d60 2024-09-16T01:53:19.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.913+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f026806e960 con 0x7f0268070d60 2024-09-16T01:53:19.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.914+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f025801a070 con 0x7f0268070d60 2024-09-16T01:53:19.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.915+0000 7f025fe00640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f024806c2b0 0x7f024806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:19.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.915+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6452+0+0 (secure 0 0 0) 0x7f025808ab00 con 0x7f0268070d60 2024-09-16T01:53:19.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.915+0000 7f0267400640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f024806c2b0 0x7f024806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:19.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.916+0000 7f0267400640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f024806c2b0 0x7f024806e770 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0254034000 tx=0x7f025402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:19.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:19.920+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0268154520 con 0x7f0268070d60 2024-09-16T01:53:20.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.103+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f0268150d50 con 0x7f0268070d60 2024-09-16T01:53:20.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.495+0000 7f025fe00640 1 -- 172.21.15.146:0/4171901887 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v136) ==== 113+0+0 (secure 0 0 0) 0x7f02580588a0 con 0x7f0268070d60 2024-09-16T01:53:20.496 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T01:53:20.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.499+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f024806c2b0 msgr2=0x7f024806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:20.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.499+0000 7f026d600640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f024806c2b0 0x7f024806e770 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0254034000 tx=0x7f025402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:20.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.499+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 msgr2=0x7f026814b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:20.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.499+0000 7f026d600640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 0x7f026814b000 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f025800c3c0 tx=0x7f025800c880 comp rx=0 tx=0).stop 2024-09-16T01:53:20.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.500+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 shutdown_connections 2024-09-16T01:53:20.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.500+0000 7f026d600640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f024806c2b0 0x7f024806e770 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.500+0000 7f026d600640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0268154be0 0x7f02681e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.500+0000 7f026d600640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02680c8d60 0x7f026814bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.500+0000 7f026d600640 1 --2- 172.21.15.146:0/4171901887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0268070d60 0x7f026814b000 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.500+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 >> 172.21.15.146:0/4171901887 conn(0x7f026806c240 msgr2=0x7f02680ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:20.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.501+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 shutdown_connections 2024-09-16T01:53:20.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.501+0000 7f026d600640 1 -- 172.21.15.146:0/4171901887 wait complete. 2024-09-16T01:53:20.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T01:53:20.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.729+0000 7f6a7e000640 1 Processor -- start 2024-09-16T01:53:20.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.729+0000 7f6a7e000640 1 -- start start 2024-09-16T01:53:20.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a7e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a780c6cc0 0x7f6a7815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:20.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a7e000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6a7806fee0 0x7f6a7815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:20.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a7e000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a78158890 con 0x7f6a78152760 2024-09-16T01:53:20.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a7e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6a780c8530 con 0x7f6a780c6cc0 2024-09-16T01:53:20.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a7e000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6a78158b00 con 0x7f6a7806fee0 2024-09-16T01:53:20.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a77400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6a78152760 0x7f6a780c59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:46292/0 (socket says 172.21.15.146:46292) 2024-09-16T01:53:20.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a77400640 1 -- 172.21.15.146:0/4054928541 learned_addr learned my addr 172.21.15.146:0/4054928541 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:20.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a76a00640 1 --2- 172.21.15.146:0/4054928541 >> v2:172.21.15.146:3300/0 conn(0x7f6a780c6cc0 0x7f6a7815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:20.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a77e00640 1 --2- 172.21.15.146:0/4054928541 >> v2:172.21.15.161:3300/0 conn(0x7f6a7806fee0 0x7f6a7815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:20.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.731+0000 7f6a76000640 1 -- 172.21.15.146:0/4054928541 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184910129 0 0) 0x7f6a78158890 con 0x7f6a78152760 2024-09-16T01:53:20.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.732+0000 7f6a76000640 1 -- 172.21.15.146:0/4054928541 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a5c003520 con 0x7f6a78152760 2024-09-16T01:53:20.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.732+0000 7f6a77e00640 1 -- 172.21.15.146:0/4054928541 >> v2:172.21.15.146:3300/0 conn(0x7f6a780c6cc0 msgr2=0x7f6a7815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:20.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.732+0000 7f6a77e00640 1 --2- 172.21.15.146:0/4054928541 >> v2:172.21.15.146:3300/0 conn(0x7f6a780c6cc0 0x7f6a7815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.732+0000 7f6a77e00640 1 -- 172.21.15.146:0/4054928541 >> v1:172.21.15.161:6789/0 conn(0x7f6a78152760 legacy=0x7f6a780c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:20.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.732+0000 7f6a77e00640 1 -- 172.21.15.146:0/4054928541 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a78158de0 con 0x7f6a7806fee0 2024-09-16T01:53:20.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.733+0000 7f6a77e00640 1 --2- 172.21.15.146:0/4054928541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7806fee0 0x7f6a7815c2b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6a781598f0 tx=0x7f6a7002a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=485735218966d0fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.733+0000 7f6a76000640 1 -- 172.21.15.146:0/4054928541 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70007740 con 0x7f6a7806fee0 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.733+0000 7f6a76000640 1 -- 172.21.15.146:0/4054928541 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a700078e0 con 0x7f6a7806fee0 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.733+0000 7f6a76000640 1 -- 172.21.15.146:0/4054928541 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70007c90 con 0x7f6a7806fee0 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 -- 172.21.15.146:0/4054928541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7806fee0 msgr2=0x7f6a7815c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 --2- 172.21.15.146:0/4054928541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7806fee0 0x7f6a7815c2b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6a781598f0 tx=0x7f6a7002a6c0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 -- 172.21.15.146:0/4054928541 shutdown_connections 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 --2- 172.21.15.146:0/4054928541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a7806fee0 0x7f6a7815c2b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 --2- 172.21.15.146:0/4054928541 >> v2:172.21.15.146:3300/0 conn(0x7f6a780c6cc0 0x7f6a7815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 -- 172.21.15.146:0/4054928541 >> 172.21.15.146:0/4054928541 conn(0x7f6a7806e670 msgr2=0x7f6a7806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:20.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 -- 172.21.15.146:0/4054928541 shutdown_connections 2024-09-16T01:53:20.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.734+0000 7f6a7e000640 1 -- 172.21.15.146:0/4054928541 wait complete. 2024-09-16T01:53:20.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.736+0000 7f6a7e000640 1 Processor -- start 2024-09-16T01:53:20.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.736+0000 7f6a7e000640 1 -- start start 2024-09-16T01:53:20.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a7e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 0x7f6a780cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:20.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a7e000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a780c6cc0 0x7f6a7815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:20.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a7e000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a78152760 0x7f6a781557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:20.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a7e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a78152b40 con 0x7f6a7806fee0 2024-09-16T01:53:20.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a7e000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6a781593a0 con 0x7f6a78152760 2024-09-16T01:53:20.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a7e000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a78152cc0 con 0x7f6a780c6cc0 2024-09-16T01:53:20.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a77400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 0x7f6a780cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:20.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a77400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 0x7f6a780cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55074/0 (socket says 172.21.15.146:55074) 2024-09-16T01:53:20.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a77400640 1 -- 172.21.15.146:0/3029892733 learned_addr learned my addr 172.21.15.146:0/3029892733 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:20.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a76a00640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a780c6cc0 0x7f6a7815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:20.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.737+0000 7f6a77e00640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a78152760 0x7f6a781557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a77400640 1 -- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a780c6cc0 msgr2=0x7f6a7815b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a77400640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a780c6cc0 0x7f6a7815b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a77400640 1 -- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a78152760 msgr2=0x7f6a781557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a77400640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a78152760 0x7f6a781557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a77400640 1 -- 172.21.15.146:0/3029892733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a781f0ef0 con 0x7f6a7806fee0 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a77400640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 0x7f6a780cb180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c440 tx=0x7f6a6800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68018070 con 0x7f6a7806fee0 2024-09-16T01:53:20.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a680074e0 con 0x7f6a7806fee0 2024-09-16T01:53:20.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68004a20 con 0x7f6a7806fee0 2024-09-16T01:53:20.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a780c2790 con 0x7f6a7806fee0 2024-09-16T01:53:20.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.738+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a780c2d50 con 0x7f6a7806fee0 2024-09-16T01:53:20.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.740+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a780c7230 con 0x7f6a7806fee0 2024-09-16T01:53:20.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.740+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a6801b070 con 0x7f6a7806fee0 2024-09-16T01:53:20.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.741+0000 7f6a6f400640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a5806c2e0 0x7f6a5806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:20.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.742+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6693+0+0 (secure 0 0 0) 0x7f6a6808aae0 con 0x7f6a7806fee0 2024-09-16T01:53:20.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.742+0000 7f6a76a00640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a5806c2e0 0x7f6a5806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:20.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.742+0000 7f6a76a00640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a5806c2e0 0x7f6a5806e7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6a78150770 tx=0x7f6a64009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:20.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.748+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6a68058810 con 0x7f6a7806fee0 2024-09-16T01:53:20.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:20.981+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) -- 0x7f6a78072140 con 0x7f6a7806fee0 2024-09-16T01:53:21.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.499+0000 7f6a6f400640 1 -- 172.21.15.146:0/3029892733 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v137) ==== 133+0+0 (secure 0 0 0) 0x7f6a6805cb10 con 0x7f6a7806fee0 2024-09-16T01:53:21.500 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T01:53:21.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.503+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a5806c2e0 msgr2=0x7f6a5806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.503+0000 7f6a7e000640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a5806c2e0 0x7f6a5806e7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6a78150770 tx=0x7f6a64009190 comp rx=0 tx=0).stop 2024-09-16T01:53:21.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.503+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 msgr2=0x7f6a780cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.503+0000 7f6a7e000640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 0x7f6a780cb180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c440 tx=0x7f6a6800c900 comp rx=0 tx=0).stop 2024-09-16T01:53:21.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.504+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 shutdown_connections 2024-09-16T01:53:21.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.504+0000 7f6a7e000640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6a5806c2e0 0x7f6a5806e7a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.504+0000 7f6a7e000640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6a78152760 0x7f6a781557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.505+0000 7f6a7e000640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a780c6cc0 0x7f6a7815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.505+0000 7f6a7e000640 1 --2- 172.21.15.146:0/3029892733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a7806fee0 0x7f6a780cb180 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.505+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 >> 172.21.15.146:0/3029892733 conn(0x7f6a7806e670 msgr2=0x7f6a78151810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:21.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.505+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 shutdown_connections 2024-09-16T01:53:21.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.505+0000 7f6a7e000640 1 -- 172.21.15.146:0/3029892733 wait complete. 2024-09-16T01:53:21.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T01:53:21.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.730+0000 7f0683a00640 1 Processor -- start 2024-09-16T01:53:21.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.730+0000 7f0683a00640 1 -- start start 2024-09-16T01:53:21.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.731+0000 7f0683a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c15ed70 0x7f067c166440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:21.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.731+0000 7f0683a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067c15ba20 con 0x7f067c0cc0a0 2024-09-16T01:53:21.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.731+0000 7f0683a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067c15aea0 con 0x7f067c0cc480 2024-09-16T01:53:21.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.731+0000 7f0683a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f067c15e1d0 con 0x7f067c15ed70 2024-09-16T01:53:21.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f0681a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c15ed70 0x7f067c166440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:21.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f067be00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f067c0cc480 0x7f067c15e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:46300/0 (socket says 172.21.15.146:46300) 2024-09-16T01:53:21.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f067be00640 1 -- 172.21.15.146:0/602162364 learned_addr learned my addr 172.21.15.146:0/602162364 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:21.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f0681000640 1 --1- 172.21.15.146:0/602162364 >> v1:172.21.15.146:6789/0 conn(0x7f067c0cc0a0 0x7f067c0d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:59926/0 (socket says 172.21.15.146:59926) 2024-09-16T01:53:21.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f0681a00640 1 -- 172.21.15.146:0/602162364 >> v1:172.21.15.161:6789/0 conn(0x7f067c0cc480 legacy=0x7f067c15e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f0681a00640 1 -- 172.21.15.146:0/602162364 >> v1:172.21.15.146:6789/0 conn(0x7f067c0cc0a0 legacy=0x7f067c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f0681a00640 1 -- 172.21.15.146:0/602162364 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c15b180 con 0x7f067c15ed70 2024-09-16T01:53:21.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.732+0000 7f067b400640 1 -- 172.21.15.146:0/602162364 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308031518 0 0) 0x7f067c15aea0 con 0x7f067c0cc480 2024-09-16T01:53:21.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.733+0000 7f0681a00640 1 --2- 172.21.15.146:0/602162364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c15ed70 0x7f067c166440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0674009dc0 tx=0x7f067402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85f0cd1ff095002f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:21.734 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.734+0000 7f067b400640 1 -- 172.21.15.146:0/602162364 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674007740 con 0x7f067c15ed70 2024-09-16T01:53:21.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.734+0000 7f067b400640 1 -- 172.21.15.146:0/602162364 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740078e0 con 0x7f067c15ed70 2024-09-16T01:53:21.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.734+0000 7f067b400640 1 -- 172.21.15.146:0/602162364 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674007cb0 con 0x7f067c15ed70 2024-09-16T01:53:21.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.734+0000 7f0683a00640 1 -- 172.21.15.146:0/602162364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c15ed70 msgr2=0x7f067c166440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.734+0000 7f0683a00640 1 --2- 172.21.15.146:0/602162364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c15ed70 0x7f067c166440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0674009dc0 tx=0x7f067402a760 comp rx=0 tx=0).stop 2024-09-16T01:53:21.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.735+0000 7f0683a00640 1 -- 172.21.15.146:0/602162364 shutdown_connections 2024-09-16T01:53:21.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.735+0000 7f0683a00640 1 --2- 172.21.15.146:0/602162364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c15ed70 0x7f067c166440 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.735+0000 7f0683a00640 1 -- 172.21.15.146:0/602162364 >> 172.21.15.146:0/602162364 conn(0x7f067c06d560 msgr2=0x7f067c073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:21.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.735+0000 7f0683a00640 1 -- 172.21.15.146:0/602162364 shutdown_connections 2024-09-16T01:53:21.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.735+0000 7f0683a00640 1 -- 172.21.15.146:0/602162364 wait complete. 2024-09-16T01:53:21.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.737+0000 7f0683a00640 1 Processor -- start 2024-09-16T01:53:21.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.737+0000 7f0683a00640 1 -- start start 2024-09-16T01:53:21.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0683a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 0x7f067c1f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:21.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0683a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f067c0cc480 0x7f067c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:21.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0683a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f067c15ed70 0x7f067c154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:21.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0683a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f067c0cc8a0 con 0x7f067c0cc480 2024-09-16T01:53:21.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0683a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f067c15b740 con 0x7f067c15ed70 2024-09-16T01:53:21.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0683a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f067c0cbde0 con 0x7f067c0cc0a0 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0681000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 0x7f067c1f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0681000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 0x7f067c1f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:44286/0 (socket says 172.21.15.146:44286) 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0681000640 1 -- 172.21.15.146:0/1186983438 learned_addr learned my addr 172.21.15.146:0/1186983438 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.738+0000 7f0681a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f067c15ed70 0x7f067c154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f067be00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f067c0cc480 0x7f067c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0681000640 1 -- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f067c15ed70 msgr2=0x7f067c154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0681000640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f067c15ed70 0x7f067c154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0681000640 1 -- 172.21.15.146:0/1186983438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f067c0cc480 msgr2=0x7f067c15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0681000640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f067c0cc480 0x7f067c15d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:21.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0681000640 1 -- 172.21.15.146:0/1186983438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c153960 con 0x7f067c0cc0a0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0681000640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 0x7f067c1f0840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f067000c410 tx=0x7f067000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670017070 con 0x7f067c0cc0a0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.739+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06700074e0 con 0x7f067c0cc0a0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.740+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670005a10 con 0x7f067c0cc0a0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.740+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067c152b00 con 0x7f067c0cc0a0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.740+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f067c1530c0 con 0x7f067c0cc0a0 2024-09-16T01:53:21.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.741+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067c15aaf0 con 0x7f067c0cc0a0 2024-09-16T01:53:21.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.742+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f067001a070 con 0x7f067c0cc0a0 2024-09-16T01:53:21.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.743+0000 7f0678c00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f065c06c2b0 0x7f065c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:21.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.743+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(137..137 src has 1..137) ==== 6277+0+0 (secure 0 0 0) 0x7f067008a8d0 con 0x7f067c0cc0a0 2024-09-16T01:53:21.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.744+0000 7f067be00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f065c06c2b0 0x7f065c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:21.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.744+0000 7f067be00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f065c06c2b0 0x7f065c06e770 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f067c0cfe80 tx=0x7f0668007b90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:21.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.748+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0670058790 con 0x7f067c0cc0a0 2024-09-16T01:53:21.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:21.977+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f067c0cd810 con 0x7f067c0cc0a0 2024-09-16T01:53:22.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.505+0000 7f0678c00640 1 -- 172.21.15.146:0/1186983438 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v138) ==== 113+0+0 (secure 0 0 0) 0x7f067005ca90 con 0x7f067c0cc0a0 2024-09-16T01:53:22.507 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T01:53:22.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.510+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f065c06c2b0 msgr2=0x7f065c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:22.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.510+0000 7f0683a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f065c06c2b0 0x7f065c06e770 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f067c0cfe80 tx=0x7f0668007b90 comp rx=0 tx=0).stop 2024-09-16T01:53:22.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.510+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 msgr2=0x7f067c1f0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:22.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.510+0000 7f0683a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 0x7f067c1f0840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f067000c410 tx=0x7f067000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.513+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 shutdown_connections 2024-09-16T01:53:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.513+0000 7f0683a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f065c06c2b0 0x7f065c06e770 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.513+0000 7f0683a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f067c15ed70 0x7f067c154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.513+0000 7f0683a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f067c0cc480 0x7f067c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.513+0000 7f0683a00640 1 --2- 172.21.15.146:0/1186983438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f067c0cc0a0 0x7f067c1f0840 secure :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f067000c410 tx=0x7f067000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.513+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 >> 172.21.15.146:0/1186983438 conn(0x7f067c06d560 msgr2=0x7f067c073340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:22.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.514+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 shutdown_connections 2024-09-16T01:53:22.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.514+0000 7f0683a00640 1 -- 172.21.15.146:0/1186983438 wait complete. 2024-09-16T01:53:22.540 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T01:53:22.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.695+0000 7fbcd8e00640 1 Processor -- start 2024-09-16T01:53:22.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.695+0000 7fbcd8e00640 1 -- start start 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd8e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd40c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd8e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbcd415aa10 0x7fbcd4154730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd8e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd4071e60 con 0x7fbcd406f800 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd8e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd40c7680 con 0x7fbcd415aa10 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd8e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd40c7940 con 0x7fbcd406fbe0 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd1e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbcd406f800 0x7fbcd40c7dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:59934/0 (socket says 172.21.15.146:59934) 2024-09-16T01:53:22.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd1e00640 1 -- 172.21.15.146:0/2308357807 learned_addr learned my addr 172.21.15.146:0/2308357807 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd1400640 1 --2- 172.21.15.146:0/2308357807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd40c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd2800640 1 --2- 172.21.15.146:0/2308357807 >> v2:172.21.15.161:3300/0 conn(0x7fbcd415aa10 0x7fbcd4154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd0a00640 1 -- 172.21.15.146:0/2308357807 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4022248753 0 0) 0x7fbcd4071e60 con 0x7fbcd406f800 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd0a00640 1 -- 172.21.15.146:0/2308357807 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb8003520 con 0x7fbcd406f800 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd1400640 1 -- 172.21.15.146:0/2308357807 >> v2:172.21.15.161:3300/0 conn(0x7fbcd415aa10 msgr2=0x7fbcd4154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd1400640 1 --2- 172.21.15.146:0/2308357807 >> v2:172.21.15.161:3300/0 conn(0x7fbcd415aa10 0x7fbcd4154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.697+0000 7fbcd1400640 1 -- 172.21.15.146:0/2308357807 >> v1:172.21.15.146:6789/0 conn(0x7fbcd406f800 legacy=0x7fbcd40c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.698+0000 7fbcd1400640 1 -- 172.21.15.146:0/2308357807 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd4072120 con 0x7fbcd406fbe0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd1400640 1 --2- 172.21.15.146:0/2308357807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd40c96a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00097a0 tx=0x7fbcc002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=506e4a07afc495c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd0a00640 1 -- 172.21.15.146:0/2308357807 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0037070 con 0x7fbcd406fbe0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd0a00640 1 -- 172.21.15.146:0/2308357807 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc0032440 con 0x7fbcd406fbe0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd0a00640 1 -- 172.21.15.146:0/2308357807 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc00327f0 con 0x7fbcd406fbe0 2024-09-16T01:53:22.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 -- 172.21.15.146:0/2308357807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 msgr2=0x7fbcd40c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/2308357807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd40c96a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbcc00097a0 tx=0x7fbcc002a210 comp rx=0 tx=0).stop 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 -- 172.21.15.146:0/2308357807 shutdown_connections 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/2308357807 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd40c96a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/2308357807 >> v2:172.21.15.161:3300/0 conn(0x7fbcd415aa10 0x7fbcd4154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 -- 172.21.15.146:0/2308357807 >> 172.21.15.146:0/2308357807 conn(0x7fbcd406e670 msgr2=0x7fbcd406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.699+0000 7fbcd8e00640 1 -- 172.21.15.146:0/2308357807 shutdown_connections 2024-09-16T01:53:22.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.700+0000 7fbcd8e00640 1 -- 172.21.15.146:0/2308357807 wait complete. 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.701+0000 7fbcd8e00640 1 Processor -- start 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.701+0000 7fbcd8e00640 1 -- start start 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd8e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcd406f800 0x7fbcd41f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd8e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 0x7fbcd414a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd8e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd40c6da0 con 0x7fbcd415aa10 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd8e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd40726e0 con 0x7fbcd406f800 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd8e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd40c9be0 con 0x7fbcd406fbe0 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 0x7fbcd414a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:22.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 0x7fbcd414a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55096/0 (socket says 172.21.15.146:55096) 2024-09-16T01:53:22.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.702+0000 7fbcd2800640 1 -- 172.21.15.146:0/1978220741 learned_addr learned my addr 172.21.15.146:0/1978220741 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:22.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd1400640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:22.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd1e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcd406f800 0x7fbcd41f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:22.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd2800640 1 -- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 msgr2=0x7fbcd414d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:22.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd2800640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd414d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd2800640 1 -- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcd406f800 msgr2=0x7fbcd41f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd2800640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcd406f800 0x7fbcd41f43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd2800640 1 -- 172.21.15.146:0/1978220741 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd414a180 con 0x7fbcd415aa10 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd1400640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd414d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd2800640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 0x7fbcd414a940 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e7f0 tx=0x7fbccc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc017070 con 0x7fbcd415aa10 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbccc00c730 con 0x7fbcd415aa10 2024-09-16T01:53:22.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd4151950 con 0x7fbcd415aa10 2024-09-16T01:53:22.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.703+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc00cb20 con 0x7fbcd415aa10 2024-09-16T01:53:22.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.704+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd414a420 con 0x7fbcd415aa10 2024-09-16T01:53:22.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.705+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd40c7160 con 0x7fbcd415aa10 2024-09-16T01:53:22.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.705+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbccc01a070 con 0x7fbcd415aa10 2024-09-16T01:53:22.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.706+0000 7fbcca000640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcb406c2b0 0x7fbcb406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:22.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.706+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5857+0+0 (secure 0 0 0) 0x7fbccc08b6c0 con 0x7fbcd415aa10 2024-09-16T01:53:22.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.707+0000 7fbcd1e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcb406c2b0 0x7fbcb406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:22.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.707+0000 7fbcd1e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcb406c2b0 0x7fbcb406e770 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4005c00 tx=0x7fbcc4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:22.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.711+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbcd40c7160 con 0x7fbcd415aa10 2024-09-16T01:53:22.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:22.899+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7fbcd40c6320 con 0x7fbcd415aa10 2024-09-16T01:53:23.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.507+0000 7fbcca000640 1 -- 172.21.15.146:0/1978220741 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v139) ==== 133+0+0 (secure 0 0 0) 0x7fbccc0596a0 con 0x7fbcd415aa10 2024-09-16T01:53:23.508 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.510+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcb406c2b0 msgr2=0x7fbcb406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.510+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcb406c2b0 0x7fbcb406e770 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4005c00 tx=0x7fbcc4009040 comp rx=0 tx=0).stop 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.510+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 msgr2=0x7fbcd414a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.510+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 0x7fbcd414a940 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbccc00e7f0 tx=0x7fbccc00ecb0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.511+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 shutdown_connections 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.511+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcb406c2b0 0x7fbcb406e770 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.511+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcd415aa10 0x7fbcd414a940 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.511+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbcd406fbe0 0x7fbcd414d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.511+0000 7fbcd8e00640 1 --2- 172.21.15.146:0/1978220741 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbcd406f800 0x7fbcd41f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.511+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 >> 172.21.15.146:0/1978220741 conn(0x7fbcd406e670 msgr2=0x7fbcd4156c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:23.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.512+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 shutdown_connections 2024-09-16T01:53:23.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.512+0000 7fbcd8e00640 1 -- 172.21.15.146:0/1978220741 wait complete. 2024-09-16T01:53:23.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T01:53:23.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.696+0000 7faa0f000640 1 Processor -- start 2024-09-16T01:53:23.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.696+0000 7faa0f000640 1 -- start start 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa0815b190 0x7faa080d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0f000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa08154ea0 con 0x7faa08154ac0 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0f000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa080c5340 con 0x7faa080d0d00 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7faa080c5620 con 0x7faa0815b190 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0dc00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7faa08154ac0 0x7faa0806d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42904/0 (socket says 172.21.15.146:42904) 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0dc00640 1 -- 172.21.15.146:0/2972504787 learned_addr learned my addr 172.21.15.146:0/2972504787 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:23.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.697+0000 7faa0d200640 1 --2- 172.21.15.146:0/2972504787 >> v2:172.21.15.146:3300/0 conn(0x7faa0815b190 0x7faa080d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:23.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.698+0000 7faa0d200640 1 -- 172.21.15.146:0/2972504787 >> v1:172.21.15.161:6789/0 conn(0x7faa080d0d00 legacy=0x7faa0815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.698+0000 7faa0d200640 1 -- 172.21.15.146:0/2972504787 >> v1:172.21.15.146:6789/0 conn(0x7faa08154ac0 legacy=0x7faa0806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.698+0000 7faa0d200640 1 -- 172.21.15.146:0/2972504787 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa080c5900 con 0x7faa0815b190 2024-09-16T01:53:23.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.698+0000 7faa07e00640 1 -- 172.21.15.146:0/2972504787 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152701197 0 0) 0x7faa08154ea0 con 0x7faa08154ac0 2024-09-16T01:53:23.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.699+0000 7faa0d200640 1 --2- 172.21.15.146:0/2972504787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa0815b190 0x7faa080d07c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40097a0 tx=0x7fa9f402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2818dca8d01aaea2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:23.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.699+0000 7faa07e00640 1 -- 172.21.15.146:0/2972504787 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4037070 con 0x7faa0815b190 2024-09-16T01:53:23.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.699+0000 7faa07e00640 1 -- 172.21.15.146:0/2972504787 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f4032640 con 0x7faa0815b190 2024-09-16T01:53:23.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.699+0000 7faa07e00640 1 -- 172.21.15.146:0/2972504787 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4032a10 con 0x7faa0815b190 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 -- 172.21.15.146:0/2972504787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa0815b190 msgr2=0x7faa080d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 --2- 172.21.15.146:0/2972504787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa0815b190 0x7faa080d07c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40097a0 tx=0x7fa9f402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 -- 172.21.15.146:0/2972504787 shutdown_connections 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 --2- 172.21.15.146:0/2972504787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa0815b190 0x7faa080d07c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 -- 172.21.15.146:0/2972504787 >> 172.21.15.146:0/2972504787 conn(0x7faa0806e400 msgr2=0x7faa0806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 -- 172.21.15.146:0/2972504787 shutdown_connections 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.700+0000 7faa0f000640 1 -- 172.21.15.146:0/2972504787 wait complete. 2024-09-16T01:53:23.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.701+0000 7faa0f000640 1 Processor -- start 2024-09-16T01:53:23.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.701+0000 7faa0f000640 1 -- start start 2024-09-16T01:53:23.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.702+0000 7faa0f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 0x7faa0814dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:23.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.702+0000 7faa0f000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faa08154ac0 0x7faa0814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:23.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.702+0000 7faa0f000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa0815b190 0x7faa081e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:23.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.702+0000 7faa0f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faa080c61a0 con 0x7faa080d0d00 2024-09-16T01:53:23.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.702+0000 7faa0f000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faa080c5ec0 con 0x7faa0815b190 2024-09-16T01:53:23.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.702+0000 7faa0f000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7faa080cad60 con 0x7faa08154ac0 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 0x7faa0814dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa0815b190 0x7faa081e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0d200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faa08154ac0 0x7faa0814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 0x7faa0814dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37226/0 (socket says 172.21.15.146:37226) 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 -- 172.21.15.146:0/3761017816 learned_addr learned my addr 172.21.15.146:0/3761017816 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 -- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faa08154ac0 msgr2=0x7faa0814e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faa08154ac0 0x7faa0814e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 -- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa0815b190 msgr2=0x7faa081e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa0815b190 0x7faa081e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 -- 172.21.15.146:0/3761017816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa080ca160 con 0x7faa080d0d00 2024-09-16T01:53:23.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.703+0000 7faa0dc00640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 0x7faa0814dc50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00c4d0 tx=0x7fa9fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:23.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.704+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc018070 con 0x7faa080d0d00 2024-09-16T01:53:23.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.704+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9fc0074e0 con 0x7faa080d0d00 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.704+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc004a20 con 0x7faa080d0d00 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.704+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa081e3b70 con 0x7faa080d0d00 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.704+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa080c85d0 con 0x7faa080d0d00 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.705+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa080b34a0 con 0x7faa080d0d00 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.706+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa9fc004bc0 con 0x7faa080d0d00 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.706+0000 7faa05600640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa9e806c2e0 0x7fa9e806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:23.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.707+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5857+0+0 (secure 0 0 0) 0x7fa9fc08a5c0 con 0x7faa080d0d00 2024-09-16T01:53:23.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.707+0000 7faa0d200640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa9e806c2e0 0x7fa9e806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:23.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.707+0000 7faa0d200640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa9e806c2e0 0x7fa9e806e7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4034000 tx=0x7fa9f402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:23.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.711+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faa080b34a0 con 0x7faa080d0d00 2024-09-16T01:53:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:23.897+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7faa080c6d60 con 0x7faa080d0d00 2024-09-16T01:53:24.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.860+0000 7faa05600640 1 -- 172.21.15.146:0/3761017816 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v140) ==== 113+0+0 (secure 0 0 0) 0x7faa080c6d60 con 0x7faa080d0d00 2024-09-16T01:53:24.861 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.862+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa9e806c2e0 msgr2=0x7fa9e806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.862+0000 7faa0f000640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa9e806c2e0 0x7fa9e806e7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4034000 tx=0x7fa9f402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 msgr2=0x7faa0814dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 0x7faa0814dc50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00c4d0 tx=0x7fa9fc00c990 comp rx=0 tx=0).stop 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 shutdown_connections 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa9e806c2e0 0x7fa9e806e7a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa0815b190 0x7faa081e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faa08154ac0 0x7faa0814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 --2- 172.21.15.146:0/3761017816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa080d0d00 0x7faa0814dc50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:24.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 >> 172.21.15.146:0/3761017816 conn(0x7faa0806e400 msgr2=0x7faa08074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:24.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 shutdown_connections 2024-09-16T01:53:24.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:24.863+0000 7faa0f000640 1 -- 172.21.15.146:0/3761017816 wait complete. 2024-09-16T01:53:24.884 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T01:53:25.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.042+0000 7ff05f800640 1 Processor -- start 2024-09-16T01:53:25.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.043+0000 7ff05f800640 1 -- start start 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff05f800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff058152760 0x7ff0580c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff05f800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff058158890 con 0x7ff0580c6cc0 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff05f800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff058158b00 con 0x7ff05815c480 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff05f800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff0580c8530 con 0x7ff058152760 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff04fe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff0580c6cc0 0x7ff05815bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42912/0 (socket says 172.21.15.146:42912) 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff04fe00640 1 -- 172.21.15.146:0/2695221808 learned_addr learned my addr 172.21.15.146:0/2695221808 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:25.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff05ce00640 1 --2- 172.21.15.146:0/2695221808 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff058152760 0x7ff0580c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:25.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.044+0000 7ff057e00640 1 -- 172.21.15.146:0/2695221808 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2910705758 0 0) 0x7ff058158890 con 0x7ff0580c6cc0 2024-09-16T01:53:25.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.045+0000 7ff057e00640 1 -- 172.21.15.146:0/2695221808 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff03c003520 con 0x7ff0580c6cc0 2024-09-16T01:53:25.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.045+0000 7ff057e00640 1 -- 172.21.15.146:0/2695221808 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 849049824 0 0) 0x7ff058158b00 con 0x7ff05815c480 2024-09-16T01:53:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.045+0000 7ff05ce00640 1 -- 172.21.15.146:0/2695221808 >> v1:172.21.15.161:6789/0 conn(0x7ff05815c480 legacy=0x7ff058163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.045+0000 7ff05ce00640 1 -- 172.21.15.146:0/2695221808 >> v1:172.21.15.146:6789/0 conn(0x7ff0580c6cc0 legacy=0x7ff05815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.045+0000 7ff05ce00640 1 -- 172.21.15.146:0/2695221808 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff058158de0 con 0x7ff058152760 2024-09-16T01:53:25.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.046+0000 7ff05ce00640 1 --2- 172.21.15.146:0/2695221808 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff058152760 0x7ff0580c59d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff0480097d0 tx=0x7ff04802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=423c72f0944b958f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:25.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.046+0000 7ff057e00640 1 -- 172.21.15.146:0/2695221808 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048037070 con 0x7ff058152760 2024-09-16T01:53:25.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.046+0000 7ff057e00640 1 -- 172.21.15.146:0/2695221808 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff048032440 con 0x7ff058152760 2024-09-16T01:53:25.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.046+0000 7ff057e00640 1 -- 172.21.15.146:0/2695221808 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480327f0 con 0x7ff058152760 2024-09-16T01:53:25.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.046+0000 7ff05f800640 1 -- 172.21.15.146:0/2695221808 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff058152760 msgr2=0x7ff0580c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.046+0000 7ff05f800640 1 --2- 172.21.15.146:0/2695221808 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff058152760 0x7ff0580c59d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff0480097d0 tx=0x7ff04802a210 comp rx=0 tx=0).stop 2024-09-16T01:53:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.047+0000 7ff05f800640 1 -- 172.21.15.146:0/2695221808 shutdown_connections 2024-09-16T01:53:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.047+0000 7ff05f800640 1 --2- 172.21.15.146:0/2695221808 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff058152760 0x7ff0580c59d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.047+0000 7ff05f800640 1 -- 172.21.15.146:0/2695221808 >> 172.21.15.146:0/2695221808 conn(0x7ff05806e670 msgr2=0x7ff05806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.047+0000 7ff05f800640 1 -- 172.21.15.146:0/2695221808 shutdown_connections 2024-09-16T01:53:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.047+0000 7ff05f800640 1 -- 172.21.15.146:0/2695221808 wait complete. 2024-09-16T01:53:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.048+0000 7ff05f800640 1 Processor -- start 2024-09-16T01:53:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.048+0000 7ff05f800640 1 -- start start 2024-09-16T01:53:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05f800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 0x7ff05814e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05f800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff058152760 0x7ff058156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05f800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff05815c480 0x7ff058142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05f800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff058152b40 con 0x7ff0580c6cc0 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05f800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff0581593a0 con 0x7ff058152760 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05f800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff058152cc0 con 0x7ff05815c480 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 0x7ff05814e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff058152760 0x7ff058156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 0x7ff05814e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37240/0 (socket says 172.21.15.146:37240) 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05ce00640 1 -- 172.21.15.146:0/564552540 learned_addr learned my addr 172.21.15.146:0/564552540 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.049+0000 7ff05d800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff05815c480 0x7ff058142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05ce00640 1 -- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff05815c480 msgr2=0x7ff058142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05ce00640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff05815c480 0x7ff058142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05ce00640 1 -- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff058152760 msgr2=0x7ff058156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05ce00640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff058152760 0x7ff058156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05ce00640 1 -- 172.21.15.146:0/564552540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0581f1070 con 0x7ff0580c6cc0 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05d800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff05815c480 0x7ff058142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff05ce00640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 0x7ff05814e130 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff04802a250 tx=0x7ff0480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048037040 con 0x7ff0580c6cc0 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0480044c0 con 0x7ff0580c6cc0 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.050+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480325f0 con 0x7ff0580c6cc0 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.051+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0580c3960 con 0x7ff0580c6cc0 2024-09-16T01:53:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.051+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0580c3f20 con 0x7ff0580c6cc0 2024-09-16T01:53:25.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.052+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0580c72c0 con 0x7ff0580c6cc0 2024-09-16T01:53:25.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.053+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff048044020 con 0x7ff0580c6cc0 2024-09-16T01:53:25.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.054+0000 7ff055600640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff03406c2b0 0x7ff03406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:25.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.054+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5857+0+0 (secure 0 0 0) 0x7ff0480ab3c0 con 0x7ff0580c6cc0 2024-09-16T01:53:25.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.054+0000 7ff04fe00640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff03406c2b0 0x7ff03406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:25.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.055+0000 7ff04fe00640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff03406c2b0 0x7ff03406e770 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff05814dee0 tx=0x7ff040009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:25.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.059+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff04804b020 con 0x7ff0580c6cc0 2024-09-16T01:53:25.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.249+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7ff058072140 con 0x7ff0580c6cc0 2024-09-16T01:53:25.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.861+0000 7ff055600640 1 -- 172.21.15.146:0/564552540 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v141) ==== 133+0+0 (secure 0 0 0) 0x7ff0480794d0 con 0x7ff0580c6cc0 2024-09-16T01:53:25.863 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T01:53:25.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff03406c2b0 msgr2=0x7ff03406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff03406c2b0 0x7ff03406e770 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff05814dee0 tx=0x7ff040009040 comp rx=0 tx=0).stop 2024-09-16T01:53:25.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 msgr2=0x7ff05814e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:25.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 0x7ff05814e130 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff04802a250 tx=0x7ff0480046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 shutdown_connections 2024-09-16T01:53:25.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff03406c2b0 0x7ff03406e770 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff05815c480 0x7ff058142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff058152760 0x7ff058156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 --2- 172.21.15.146:0/564552540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0580c6cc0 0x7ff05814e130 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:25.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 >> 172.21.15.146:0/564552540 conn(0x7ff05806e670 msgr2=0x7ff0580719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:25.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 shutdown_connections 2024-09-16T01:53:25.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:25.865+0000 7ff05f800640 1 -- 172.21.15.146:0/564552540 wait complete. 2024-09-16T01:53:25.883 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T01:53:26.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.040+0000 7fd56cc00640 1 Processor -- start 2024-09-16T01:53:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.040+0000 7fd56cc00640 1 -- start start 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd56cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd56815aa10 0x7fd568154730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd56cc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd568071e60 con 0x7fd56806fbe0 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd56cc00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd568072120 con 0x7fd56806f800 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd56cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fd5680c7680 con 0x7fd56815aa10 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd565400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd56806fbe0 0x7fd5680c96a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42918/0 (socket says 172.21.15.146:42918) 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd565400640 1 -- 172.21.15.146:0/115510332 learned_addr learned my addr 172.21.15.146:0/115510332 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd566800640 1 --2- 172.21.15.146:0/115510332 >> v2:172.21.15.146:3300/0 conn(0x7fd56815aa10 0x7fd568154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd564a00640 1 -- 172.21.15.146:0/115510332 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949632673 0 0) 0x7fd568071e60 con 0x7fd56806fbe0 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd564a00640 1 -- 172.21.15.146:0/115510332 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd54c003520 con 0x7fd56806fbe0 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd564a00640 1 -- 172.21.15.146:0/115510332 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2291263464 0 0) 0x7fd568072120 con 0x7fd56806f800 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd566800640 1 -- 172.21.15.146:0/115510332 >> v1:172.21.15.161:6789/0 conn(0x7fd56806f800 legacy=0x7fd5680c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.042+0000 7fd566800640 1 -- 172.21.15.146:0/115510332 >> v1:172.21.15.146:6789/0 conn(0x7fd56806fbe0 legacy=0x7fd5680c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.043+0000 7fd566800640 1 -- 172.21.15.146:0/115510332 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd568072400 con 0x7fd56815aa10 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.043+0000 7fd566800640 1 --2- 172.21.15.146:0/115510332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd568154730 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd560009d90 tx=0x7fd56002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6cfd5689273a82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd564a00640 1 -- 172.21.15.146:0/115510332 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560037070 con 0x7fd56815aa10 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd564a00640 1 -- 172.21.15.146:0/115510332 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd560007740 con 0x7fd56815aa10 2024-09-16T01:53:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd564a00640 1 -- 172.21.15.146:0/115510332 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560007af0 con 0x7fd56815aa10 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd56cc00640 1 -- 172.21.15.146:0/115510332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 msgr2=0x7fd568154730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd56cc00640 1 --2- 172.21.15.146:0/115510332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd568154730 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd560009d90 tx=0x7fd56002a780 comp rx=0 tx=0).stop 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd56cc00640 1 -- 172.21.15.146:0/115510332 shutdown_connections 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd56cc00640 1 --2- 172.21.15.146:0/115510332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd568154730 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd56cc00640 1 -- 172.21.15.146:0/115510332 >> 172.21.15.146:0/115510332 conn(0x7fd56806e670 msgr2=0x7fd56806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.044+0000 7fd56cc00640 1 -- 172.21.15.146:0/115510332 shutdown_connections 2024-09-16T01:53:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.045+0000 7fd56cc00640 1 -- 172.21.15.146:0/115510332 wait complete. 2024-09-16T01:53:26.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.046+0000 7fd56cc00640 1 Processor -- start 2024-09-16T01:53:26.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.046+0000 7fd56cc00640 1 -- start start 2024-09-16T01:53:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd56cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd56806f800 0x7fd5680c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd56cc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd56806fbe0 0x7fd5680c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd56cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd5680bd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd56cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd5680c6da0 con 0x7fd56815aa10 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd56cc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5680c7940 con 0x7fd56806fbe0 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd56cc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd5680c6320 con 0x7fd56806f800 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd566800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd5680bd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd566800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd5680bd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37262/0 (socket says 172.21.15.146:37262) 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd566800640 1 -- 172.21.15.146:0/250191764 learned_addr learned my addr 172.21.15.146:0/250191764 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd565400640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd56806fbe0 0x7fd5680c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.047+0000 7fd565e00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd56806f800 0x7fd5680c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd566800640 1 -- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd56806f800 msgr2=0x7fd5680c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd566800640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd56806f800 0x7fd5680c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd566800640 1 -- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd56806fbe0 msgr2=0x7fd5680c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd566800640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd56806fbe0 0x7fd5680c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd566800640 1 -- 172.21.15.146:0/250191764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5680ca1d0 con 0x7fd56815aa10 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd566800640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd5680bd7b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd560009eb0 tx=0x7fd560004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560041030 con 0x7fd56815aa10 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd560003e10 con 0x7fd56815aa10 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560037040 con 0x7fd56815aa10 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.048+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd568151c10 con 0x7fd56815aa10 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.049+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5681f1140 con 0x7fd56815aa10 2024-09-16T01:53:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.050+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5680c7dd0 con 0x7fd56815aa10 2024-09-16T01:53:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.050+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd560044070 con 0x7fd56815aa10 2024-09-16T01:53:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.051+0000 7fd55e000640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd54806c2b0 0x7fd54806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.051+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5857+0+0 (secure 0 0 0) 0x7fd5600abd60 con 0x7fd56815aa10 2024-09-16T01:53:26.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.052+0000 7fd565e00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd54806c2b0 0x7fd54806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:26.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.052+0000 7fd565e00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd54806c2b0 0x7fd54806e770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd558000f60 tx=0x7fd558009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:26.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.056+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd56004b070 con 0x7fd56815aa10 2024-09-16T01:53:26.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.239+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7fd5680c3d00 con 0x7fd56815aa10 2024-09-16T01:53:26.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.863+0000 7fd55e000640 1 -- 172.21.15.146:0/250191764 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v142) ==== 113+0+0 (secure 0 0 0) 0x7fd560079e70 con 0x7fd56815aa10 2024-09-16T01:53:26.865 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T01:53:26.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.866+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd54806c2b0 msgr2=0x7fd54806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.866+0000 7fd56cc00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd54806c2b0 0x7fd54806e770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd558000f60 tx=0x7fd558009040 comp rx=0 tx=0).stop 2024-09-16T01:53:26.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.866+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 msgr2=0x7fd5680bd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:26.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.866+0000 7fd56cc00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd5680bd7b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd560009eb0 tx=0x7fd560004a40 comp rx=0 tx=0).stop 2024-09-16T01:53:26.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 shutdown_connections 2024-09-16T01:53:26.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd54806c2b0 0x7fd54806e770 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd56815aa10 0x7fd5680bd7b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd56806fbe0 0x7fd5680c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 --2- 172.21.15.146:0/250191764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd56806f800 0x7fd5680c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:26.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 >> 172.21.15.146:0/250191764 conn(0x7fd56806e670 msgr2=0x7fd5680c9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:26.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 shutdown_connections 2024-09-16T01:53:26.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:26.867+0000 7fd56cc00640 1 -- 172.21.15.146:0/250191764 wait complete. 2024-09-16T01:53:26.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T01:53:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.042+0000 7f0cb9800640 1 Processor -- start 2024-09-16T01:53:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.042+0000 7f0cb9800640 1 -- start start 2024-09-16T01:53:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.043+0000 7f0cb9800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0cb4154ac0 0x7f0cb415b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.043+0000 7f0cb9800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cb4154ea0 con 0x7f0cb415bc50 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.043+0000 7f0cb9800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cb40c5340 con 0x7f0cb406d820 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.043+0000 7f0cb9800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0cb415b6d0 con 0x7f0cb4154ac0 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.044+0000 7f0cb3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0cb4154ac0 0x7f0cb415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.044+0000 7f0cb8e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0cb415bc50 0x7f0cb415c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42928/0 (socket says 172.21.15.146:42928) 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.044+0000 7f0cb8e00640 1 -- 172.21.15.146:0/526995416 learned_addr learned my addr 172.21.15.146:0/526995416 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.044+0000 7f0cb3e00640 1 -- 172.21.15.146:0/526995416 >> v1:172.21.15.161:6789/0 conn(0x7f0cb406d820 legacy=0x7f0cb40d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.044+0000 7f0cb3e00640 1 -- 172.21.15.146:0/526995416 >> v1:172.21.15.146:6789/0 conn(0x7f0cb415bc50 legacy=0x7f0cb415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.044+0000 7f0cb3e00640 1 -- 172.21.15.146:0/526995416 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb40c5620 con 0x7f0cb4154ac0 2024-09-16T01:53:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.045+0000 7f0cb3e00640 1 --2- 172.21.15.146:0/526995416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb4154ac0 0x7f0cb415b190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40097a0 tx=0x7f0ca402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7079a34941b98b5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.045+0000 7f0cb2a00640 1 -- 172.21.15.146:0/526995416 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4037070 con 0x7f0cb4154ac0 2024-09-16T01:53:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.045+0000 7f0cb2a00640 1 -- 172.21.15.146:0/526995416 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca4032640 con 0x7f0cb4154ac0 2024-09-16T01:53:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.045+0000 7f0cb2a00640 1 -- 172.21.15.146:0/526995416 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4032a10 con 0x7f0cb4154ac0 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 -- 172.21.15.146:0/526995416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb4154ac0 msgr2=0x7f0cb415b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 --2- 172.21.15.146:0/526995416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb4154ac0 0x7f0cb415b190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40097a0 tx=0x7f0ca402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 -- 172.21.15.146:0/526995416 shutdown_connections 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 --2- 172.21.15.146:0/526995416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb4154ac0 0x7f0cb415b190 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 -- 172.21.15.146:0/526995416 >> 172.21.15.146:0/526995416 conn(0x7f0cb406e400 msgr2=0x7f0cb406e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 -- 172.21.15.146:0/526995416 shutdown_connections 2024-09-16T01:53:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.046+0000 7f0cb9800640 1 -- 172.21.15.146:0/526995416 wait complete. 2024-09-16T01:53:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.047+0000 7f0cb9800640 1 Processor -- start 2024-09-16T01:53:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.048+0000 7f0cb9800640 1 -- start start 2024-09-16T01:53:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.048+0000 7f0cb9800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cb406d820 0x7f0cb40cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.048+0000 7f0cb9800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0cb4154ac0 0x7f0cb40cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.048+0000 7f0cb9800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 0x7f0cb4149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb9800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb40c5ec0 con 0x7f0cb415bc50 2024-09-16T01:53:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb9800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb40c5be0 con 0x7f0cb406d820 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb9800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0cb40c61a0 con 0x7f0cb4154ac0 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 0x7f0cb4149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 0x7f0cb4149d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37288/0 (socket says 172.21.15.146:37288) 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 -- 172.21.15.146:0/695696570 learned_addr learned my addr 172.21.15.146:0/695696570 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb3e00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cb406d820 0x7f0cb40cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb3400640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0cb4154ac0 0x7f0cb40cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 -- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0cb4154ac0 msgr2=0x7f0cb40cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0cb4154ac0 0x7f0cb40cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 -- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cb406d820 msgr2=0x7f0cb40cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cb406d820 0x7f0cb40cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 -- 172.21.15.146:0/695696570 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb415dc20 con 0x7f0cb415bc50 2024-09-16T01:53:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.049+0000 7f0cb8e00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 0x7f0cb4149d30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00ca80 tx=0x7f0cac00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:27.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.050+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac017070 con 0x7f0cb415bc50 2024-09-16T01:53:27.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.050+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cac00ac30 con 0x7f0cb415bc50 2024-09-16T01:53:27.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.050+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac012600 con 0x7f0cb415bc50 2024-09-16T01:53:27.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.050+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb40cd9a0 con 0x7f0cb415bc50 2024-09-16T01:53:27.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.050+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb415d1b0 con 0x7f0cb415bc50 2024-09-16T01:53:27.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.051+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb40cad60 con 0x7f0cb415bc50 2024-09-16T01:53:27.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.052+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0cac01a070 con 0x7f0cb415bc50 2024-09-16T01:53:27.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.053+0000 7f0cabe00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0c9406c2b0 0x7f0c9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:27.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.053+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5857+0+0 (secure 0 0 0) 0x7f0cac08b130 con 0x7f0cb415bc50 2024-09-16T01:53:27.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.053+0000 7f0cb3e00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0c9406c2b0 0x7f0c9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:27.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.054+0000 7f0cb3e00640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0c9406c2b0 0x7f0c9406e770 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4034000 tx=0x7f0ca402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:27.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.057+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0cac00ae20 con 0x7f0cb415bc50 2024-09-16T01:53:27.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:53:27.086 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:53:27.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.250+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7f0cb4153960 con 0x7f0cb415bc50 2024-09-16T01:53:27.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.870+0000 7f0cabe00640 1 -- 172.21.15.146:0/695696570 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v143) ==== 133+0+0 (secure 0 0 0) 0x7f0cac0591a0 con 0x7f0cb415bc50 2024-09-16T01:53:27.871 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T01:53:27.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0c9406c2b0 msgr2=0x7f0c9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0c9406c2b0 0x7f0c9406e770 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4034000 tx=0x7f0ca402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 msgr2=0x7f0cb4149d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 0x7f0cb4149d30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00ca80 tx=0x7f0cac00cf40 comp rx=0 tx=0).stop 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 shutdown_connections 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0c9406c2b0 0x7f0c9406e770 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0cb415bc50 0x7f0cb4149d30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0cb4154ac0 0x7f0cb40cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 --2- 172.21.15.146:0/695696570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0cb406d820 0x7f0cb40cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 >> 172.21.15.146:0/695696570 conn(0x7f0cb406e400 msgr2=0x7f0cb40d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 shutdown_connections 2024-09-16T01:53:27.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:27.873+0000 7f0cb9800640 1 -- 172.21.15.146:0/695696570 wait complete. 2024-09-16T01:53:27.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T01:53:28.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.047+0000 7fc0b3400640 1 Processor -- start 2024-09-16T01:53:28.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.048+0000 7fc0b3400640 1 -- start start 2024-09-16T01:53:28.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0b3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:28.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0b3400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0ac158890 con 0x7fc0ac06fee0 2024-09-16T01:53:28.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0b3400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0ac158b00 con 0x7fc0ac152760 2024-09-16T01:53:28.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0b3400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ac0c8530 con 0x7fc0ac0c6cc0 2024-09-16T01:53:28.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0b1400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc0ac06fee0 0x7fc0ac163100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42932/0 (socket says 172.21.15.146:42932) 2024-09-16T01:53:28.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0b1400640 1 -- 172.21.15.146:0/3135987396 learned_addr learned my addr 172.21.15.146:0/3135987396 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:28.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0a3e00640 1 --2- 172.21.15.146:0/3135987396 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:28.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.050+0000 7fc0a3e00640 1 -- 172.21.15.146:0/3135987396 >> v1:172.21.15.161:6789/0 conn(0x7fc0ac152760 legacy=0x7fc0ac0c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.049+0000 7fc0abe00640 1 -- 172.21.15.146:0/3135987396 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339793412 0 0) 0x7fc0ac158b00 con 0x7fc0ac152760 2024-09-16T01:53:28.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.050+0000 7fc0a3e00640 1 -- 172.21.15.146:0/3135987396 >> v1:172.21.15.146:6789/0 conn(0x7fc0ac06fee0 legacy=0x7fc0ac163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.050+0000 7fc0a3e00640 1 -- 172.21.15.146:0/3135987396 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac158de0 con 0x7fc0ac0c6cc0 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.051+0000 7fc0a3e00640 1 --2- 172.21.15.146:0/3135987396 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac15bd70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc0940097d0 tx=0x7fc09402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7359e08bd5e44b09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.051+0000 7fc0abe00640 1 -- 172.21.15.146:0/3135987396 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc094037070 con 0x7fc0ac0c6cc0 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.051+0000 7fc0abe00640 1 -- 172.21.15.146:0/3135987396 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09402ae30 con 0x7fc0ac0c6cc0 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.051+0000 7fc0abe00640 1 -- 172.21.15.146:0/3135987396 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc094032640 con 0x7fc0ac0c6cc0 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.051+0000 7fc0b3400640 1 -- 172.21.15.146:0/3135987396 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 msgr2=0x7fc0ac15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.051+0000 7fc0b3400640 1 --2- 172.21.15.146:0/3135987396 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac15bd70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc0940097d0 tx=0x7fc09402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.052+0000 7fc0b3400640 1 -- 172.21.15.146:0/3135987396 shutdown_connections 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.052+0000 7fc0b3400640 1 --2- 172.21.15.146:0/3135987396 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac15bd70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.052+0000 7fc0b3400640 1 -- 172.21.15.146:0/3135987396 >> 172.21.15.146:0/3135987396 conn(0x7fc0ac06e670 msgr2=0x7fc0ac06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:28.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.052+0000 7fc0b3400640 1 -- 172.21.15.146:0/3135987396 shutdown_connections 2024-09-16T01:53:28.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.052+0000 7fc0b3400640 1 -- 172.21.15.146:0/3135987396 wait complete. 2024-09-16T01:53:28.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.053+0000 7fc0b3400640 1 Processor -- start 2024-09-16T01:53:28.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.053+0000 7fc0b3400640 1 -- start start 2024-09-16T01:53:28.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 0x7fc0ac1f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:28.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:28.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ac152760 0x7fc0ac14abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b3400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac152b40 con 0x7fc0ac06fee0 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b3400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac1593a0 con 0x7fc0ac152760 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b3400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ac152cc0 con 0x7fc0ac0c6cc0 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 0x7fc0ac1f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 0x7fc0ac1f3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37302/0 (socket says 172.21.15.146:37302) 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0b0a00640 1 -- 172.21.15.146:0/377245764 learned_addr learned my addr 172.21.15.146:0/377245764 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.054+0000 7fc0a3e00640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b1400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ac152760 0x7fc0ac14abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b0a00640 1 -- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 msgr2=0x7fc0ac0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b0a00640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b0a00640 1 -- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ac152760 msgr2=0x7fc0ac14abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b0a00640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ac152760 0x7fc0ac14abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b0a00640 1 -- 172.21.15.146:0/377245764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac1559a0 con 0x7fc0ac06fee0 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b1400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ac152760 0x7fc0ac14abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b0a00640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 0x7fc0ac1f3e90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c3c0 tx=0x7fc09c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c017070 con 0x7fc0ac06fee0 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09c0035f0 con 0x7fc0ac06fee0 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c0039e0 con 0x7fc0ac06fee0 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.055+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ac155c80 con 0x7fc0ac06fee0 2024-09-16T01:53:28.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.056+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ac1543d0 con 0x7fc0ac06fee0 2024-09-16T01:53:28.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.057+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ac0c7350 con 0x7fc0ac06fee0 2024-09-16T01:53:28.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.058+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc09c01a070 con 0x7fc0ac06fee0 2024-09-16T01:53:28.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.058+0000 7fc0a9600640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc08806c2e0 0x7fc08806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:28.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.059+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5857+0+0 (secure 0 0 0) 0x7fc09c08a6f0 con 0x7fc0ac06fee0 2024-09-16T01:53:28.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.059+0000 7fc0a3e00640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc08806c2e0 0x7fc08806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:28.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.060+0000 7fc0a3e00640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc08806c2e0 0x7fc08806e7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc094034000 tx=0x7fc09402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:28.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.063+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc09c058800 con 0x7fc0ac06fee0 2024-09-16T01:53:28.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.246+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7fc0ac153a80 con 0x7fc0ac06fee0 2024-09-16T01:53:28.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.877+0000 7fc0a9600640 1 -- 172.21.15.146:0/377245764 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v144) ==== 113+0+0 (secure 0 0 0) 0x7fc09c05cb00 con 0x7fc0ac06fee0 2024-09-16T01:53:28.878 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc08806c2e0 msgr2=0x7fc08806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc08806c2e0 0x7fc08806e7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc094034000 tx=0x7fc09402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 msgr2=0x7fc0ac1f3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 0x7fc0ac1f3e90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c3c0 tx=0x7fc09c00c880 comp rx=0 tx=0).stop 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 shutdown_connections 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc08806c2e0 0x7fc08806e7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0ac152760 0x7fc0ac14abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc0ac0c6cc0 0x7fc0ac0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 --2- 172.21.15.146:0/377245764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0ac06fee0 0x7fc0ac1f3e90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.880+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 >> 172.21.15.146:0/377245764 conn(0x7fc0ac06e670 msgr2=0x7fc0ac157520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:28.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.881+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 shutdown_connections 2024-09-16T01:53:28.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:28.881+0000 7fc0b3400640 1 -- 172.21.15.146:0/377245764 wait complete. 2024-09-16T01:53:28.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T01:53:29.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.059+0000 7ff504c00640 1 Processor -- start 2024-09-16T01:53:29.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.059+0000 7ff504c00640 1 -- start start 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff504c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff504c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff50006fee0 0x7ff50015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff504c00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff500158890 con 0x7ff500152760 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff504c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ff5000c8530 con 0x7ff50006fee0 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff504c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff500158b00 con 0x7ff5000c6cc0 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff4fe800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff50006fee0 0x7ff50015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.060+0000 7ff4fd400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fe800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff50006fee0 0x7ff50015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37318/0 (socket says 172.21.15.146:37318) 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fe800640 1 -- 172.21.15.146:0/4005948955 learned_addr learned my addr 172.21.15.146:0/4005948955 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:29.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fde00640 1 --1- 172.21.15.146:0/4005948955 >> v1:172.21.15.161:6789/0 conn(0x7ff500152760 0x7ff5000c59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44034/0 (socket says 172.21.15.146:44034) 2024-09-16T01:53:29.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fe800640 1 -- 172.21.15.146:0/4005948955 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 msgr2=0x7ff50015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:29.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fe800640 1 --2- 172.21.15.146:0/4005948955 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fe800640 1 -- 172.21.15.146:0/4005948955 >> v1:172.21.15.161:6789/0 conn(0x7ff500152760 legacy=0x7ff5000c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.061+0000 7ff4fe800640 1 -- 172.21.15.146:0/4005948955 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff500158de0 con 0x7ff50006fee0 2024-09-16T01:53:29.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.062+0000 7ff4fe800640 1 --2- 172.21.15.146:0/4005948955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff50006fee0 0x7ff50015c2b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009e50 tx=0x7ff4f802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f5fa05c20c1369 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.062+0000 7ff4fca00640 1 -- 172.21.15.146:0/4005948955 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8036070 con 0x7ff50006fee0 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.062+0000 7ff4fca00640 1 -- 172.21.15.146:0/4005948955 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f8031600 con 0x7ff50006fee0 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.062+0000 7ff4fca00640 1 -- 172.21.15.146:0/4005948955 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80319d0 con 0x7ff50006fee0 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 -- 172.21.15.146:0/4005948955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff50006fee0 msgr2=0x7ff50015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 --2- 172.21.15.146:0/4005948955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff50006fee0 0x7ff50015c2b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009e50 tx=0x7ff4f802a7a0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 -- 172.21.15.146:0/4005948955 shutdown_connections 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 --2- 172.21.15.146:0/4005948955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff50006fee0 0x7ff50015c2b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 --2- 172.21.15.146:0/4005948955 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 -- 172.21.15.146:0/4005948955 >> 172.21.15.146:0/4005948955 conn(0x7ff50006e670 msgr2=0x7ff50006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:29.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 -- 172.21.15.146:0/4005948955 shutdown_connections 2024-09-16T01:53:29.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.063+0000 7ff504c00640 1 -- 172.21.15.146:0/4005948955 wait complete. 2024-09-16T01:53:29.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.065+0000 7ff504c00640 1 Processor -- start 2024-09-16T01:53:29.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.065+0000 7ff504c00640 1 -- start start 2024-09-16T01:53:29.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.065+0000 7ff504c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff50006fee0 0x7ff500155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff504c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50014b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff504c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 0x7ff5001f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff504c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff500152b40 con 0x7ff500152760 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff504c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff5001593a0 con 0x7ff50006fee0 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff504c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff500152cc0 con 0x7ff5000c6cc0 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 0x7ff5001f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff50006fee0 0x7ff500155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff50006fee0 0x7ff500155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33504/0 (socket says 172.21.15.146:33504) 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 0x7ff5001f08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37334/0 (socket says 172.21.15.146:37334) 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fd400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50014b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fde00640 1 -- 172.21.15.146:0/3377595994 learned_addr learned my addr 172.21.15.146:0/3377595994 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 -- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 msgr2=0x7ff50014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50014b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 -- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff50006fee0 msgr2=0x7ff500155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff50006fee0 0x7ff500155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.066+0000 7ff4fe800640 1 -- 172.21.15.146:0/3377595994 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5001560c0 con 0x7ff500152760 2024-09-16T01:53:29.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.067+0000 7ff4fe800640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 0x7ff5001f08f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8030000 tx=0x7ff4f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:29.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.067+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8036050 con 0x7ff500152760 2024-09-16T01:53:29.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.067+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f8042070 con 0x7ff500152760 2024-09-16T01:53:29.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.067+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5001f3e20 con 0x7ff500152760 2024-09-16T01:53:29.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.067+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8009510 con 0x7ff500152760 2024-09-16T01:53:29.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.067+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5001f43a0 con 0x7ff500152760 2024-09-16T01:53:29.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.068+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5000c7230 con 0x7ff500152760 2024-09-16T01:53:29.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.069+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff4f8007b70 con 0x7ff500152760 2024-09-16T01:53:29.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.070+0000 7ff4f6000640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff4e006c2e0 0x7ff4e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.070+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 5857+0+0 (secure 0 0 0) 0x7ff4f80abb70 con 0x7ff500152760 2024-09-16T01:53:29.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.070+0000 7ff4fde00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff4e006c2e0 0x7ff4e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.071+0000 7ff4fde00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff4e006c2e0 0x7ff4e006e7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0004de0 tx=0x7ff4f0007f10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:29.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.074+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff4f804b020 con 0x7ff500152760 2024-09-16T01:53:29.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.262+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) -- 0x7ff500072140 con 0x7ff500152760 2024-09-16T01:53:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.494+0000 7ff4f6000640 1 -- 172.21.15.146:0/3377595994 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v145) ==== 133+0+0 (secure 0 0 0) 0x7ff4f8079ad0 con 0x7ff500152760 2024-09-16T01:53:29.496 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T01:53:29.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff4e006c2e0 msgr2=0x7ff4e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff4e006c2e0 0x7ff4e006e7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0004de0 tx=0x7ff4f0007f10 comp rx=0 tx=0).stop 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 msgr2=0x7ff5001f08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 0x7ff5001f08f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8030000 tx=0x7ff4f8004a40 comp rx=0 tx=0).stop 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 shutdown_connections 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff4e006c2e0 0x7ff4e006e7a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff500152760 0x7ff5001f08f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff5000c6cc0 0x7ff50014b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 --2- 172.21.15.146:0/3377595994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff50006fee0 0x7ff500155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 >> 172.21.15.146:0/3377595994 conn(0x7ff50006e670 msgr2=0x7ff500071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.498+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 shutdown_connections 2024-09-16T01:53:29.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.499+0000 7ff504c00640 1 -- 172.21.15.146:0/3377595994 wait complete. 2024-09-16T01:53:29.517 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T01:53:29.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.673+0000 7f122ec00640 1 Processor -- start 2024-09-16T01:53:29.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.673+0000 7f122ec00640 1 -- start start 2024-09-16T01:53:29.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.674+0000 7f122ec00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1228152760 0x7f12280c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.674+0000 7f122ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12280c6cc0 0x7f12280c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.674+0000 7f122ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1228158890 con 0x7f122806fee0 2024-09-16T01:53:29.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.674+0000 7f122ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f12280c8530 con 0x7f12280c6cc0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.674+0000 7f122ec00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1228158b00 con 0x7f1228152760 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f122cc00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f122806fee0 0x7f122815bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42942/0 (socket says 172.21.15.146:42942) 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1227400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12280c6cc0 0x7f12280c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f122cc00640 1 -- 172.21.15.146:0/3963190928 learned_addr learned my addr 172.21.15.146:0/3963190928 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1227e00640 1 --2- 172.21.15.146:0/3963190928 >> v2:172.21.15.161:3300/0 conn(0x7f1228152760 0x7f12280c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1227400640 1 -- 172.21.15.146:0/3963190928 >> v2:172.21.15.161:3300/0 conn(0x7f1228152760 msgr2=0x7f12280c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1227400640 1 --2- 172.21.15.146:0/3963190928 >> v2:172.21.15.161:3300/0 conn(0x7f1228152760 0x7f12280c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1227400640 1 -- 172.21.15.146:0/3963190928 >> v1:172.21.15.146:6789/0 conn(0x7f122806fee0 legacy=0x7f122815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1227400640 1 -- 172.21.15.146:0/3963190928 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1228158de0 con 0x7f12280c6cc0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.675+0000 7f1226a00640 1 -- 172.21.15.146:0/3963190928 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1598351342 0 0) 0x7f1228158890 con 0x7f122806fee0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.676+0000 7f1227400640 1 --2- 172.21.15.146:0/3963190928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280c6cc0 0x7f12280c5f10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f122806d830 tx=0x7f121402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1b8f84666b15e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.676+0000 7f1226a00640 1 -- 172.21.15.146:0/3963190928 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037070 con 0x7f12280c6cc0 2024-09-16T01:53:29.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.676+0000 7f1226a00640 1 -- 172.21.15.146:0/3963190928 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214032440 con 0x7f12280c6cc0 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.676+0000 7f1226a00640 1 -- 172.21.15.146:0/3963190928 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214032810 con 0x7f12280c6cc0 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 -- 172.21.15.146:0/3963190928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280c6cc0 msgr2=0x7f12280c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 --2- 172.21.15.146:0/3963190928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280c6cc0 0x7f12280c5f10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f122806d830 tx=0x7f121402a250 comp rx=0 tx=0).stop 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 -- 172.21.15.146:0/3963190928 shutdown_connections 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 --2- 172.21.15.146:0/3963190928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12280c6cc0 0x7f12280c5f10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 --2- 172.21.15.146:0/3963190928 >> v2:172.21.15.161:3300/0 conn(0x7f1228152760 0x7f12280c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 -- 172.21.15.146:0/3963190928 >> 172.21.15.146:0/3963190928 conn(0x7f122806e670 msgr2=0x7f122806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 -- 172.21.15.146:0/3963190928 shutdown_connections 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.677+0000 7f122ec00640 1 -- 172.21.15.146:0/3963190928 wait complete. 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.679+0000 7f122ec00640 1 Processor -- start 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.679+0000 7f122ec00640 1 -- start start 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.679+0000 7f122ec00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122806fee0 0x7f12280cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.679+0000 7f122ec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 0x7f1228149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 0x7f122814a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1228152b40 con 0x7f1228152760 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122ec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12281593a0 con 0x7f12280c6cc0 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122ec00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1228152cc0 con 0x7f122806fee0 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 0x7f122814a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 0x7f1228149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 0x7f122814a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37346/0 (socket says 172.21.15.146:37346) 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 0x7f1228149d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33518/0 (socket says 172.21.15.146:33518) 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122cc00640 1 -- 172.21.15.146:0/106196169 learned_addr learned my addr 172.21.15.146:0/106196169 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227e00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122806fee0 0x7f12280cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 -- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122806fee0 msgr2=0x7f12280cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122806fee0 0x7f12280cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 -- 172.21.15.146:0/106196169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 msgr2=0x7f122814a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 0x7f122814a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f1227400640 1 -- 172.21.15.146:0/106196169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12280caa50 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.680+0000 7f122cc00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 0x7f122814a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.681+0000 7f1227400640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 0x7f1228149d90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f12280cd2f0 tx=0x7f12140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.681+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037050 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.681+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214038040 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.681+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12280cacf0 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.681+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214004430 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.681+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12280cb2b0 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.682+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12280c7230 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.683+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1214003610 con 0x7f12280c6cc0 2024-09-16T01:53:29.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.684+0000 7f121fe00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120806c2b0 0x7f120806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:29.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.684+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5907+0+0 (secure 0 0 0) 0x7f12140abe10 con 0x7f12280c6cc0 2024-09-16T01:53:29.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.684+0000 7f1227e00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120806c2b0 0x7f120806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:29.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.685+0000 7f1227e00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120806c2b0 0x7f120806e770 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1218009aa0 tx=0x7f1218009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:29.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.689+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f12140af030 con 0x7f12280c6cc0 2024-09-16T01:53:29.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:29.872+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f1228072140 con 0x7f12280c6cc0 2024-09-16T01:53:30.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.501+0000 7f121fe00640 1 -- 172.21.15.146:0/106196169 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v146) ==== 113+0+0 (secure 0 0 0) 0x7f121407a380 con 0x7f12280c6cc0 2024-09-16T01:53:30.502 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.504+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120806c2b0 msgr2=0x7f120806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.504+0000 7f122ec00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120806c2b0 0x7f120806e770 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1218009aa0 tx=0x7f1218009190 comp rx=0 tx=0).stop 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.504+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 msgr2=0x7f1228149d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.504+0000 7f122ec00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 0x7f1228149d90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f12280cd2f0 tx=0x7f12140046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 shutdown_connections 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f120806c2b0 0x7f120806e770 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1228152760 0x7f122814a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12280c6cc0 0x7f1228149d90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 --2- 172.21.15.146:0/106196169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f122806fee0 0x7f12280cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 >> 172.21.15.146:0/106196169 conn(0x7f122806e670 msgr2=0x7f122814ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:30.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 shutdown_connections 2024-09-16T01:53:30.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.505+0000 7f122ec00640 1 -- 172.21.15.146:0/106196169 wait complete. 2024-09-16T01:53:30.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T01:53:30.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.689+0000 7f36dcc00640 1 Processor -- start 2024-09-16T01:53:30.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.690+0000 7f36dcc00640 1 -- start start 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36dcc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f36d80d0870 0x7f36d8154690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36dcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80c6d00 0x7f36d8154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36dcc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36d815e220 con 0x7f36d80d0490 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36dcc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f36d80c6160 con 0x7f36d80d0870 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36dcc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f36d815e500 con 0x7f36d80c6d00 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f36d80d0490 0x7f36d80c65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42956/0 (socket says 172.21.15.146:42956) 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7400640 1 -- 172.21.15.146:0/3706497669 learned_addr learned my addr 172.21.15.146:0/3706497669 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7e00640 1 --2- 172.21.15.146:0/3706497669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80c6d00 0x7f36d8154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36cea00640 1 --2- 172.21.15.146:0/3706497669 >> v2:172.21.15.161:3300/0 conn(0x7f36d80d0870 0x7f36d8154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:30.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d6a00640 1 -- 172.21.15.146:0/3706497669 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940998821 0 0) 0x7f36d815e220 con 0x7f36d80d0490 2024-09-16T01:53:30.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d6a00640 1 -- 172.21.15.146:0/3706497669 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36bc003520 con 0x7f36d80d0490 2024-09-16T01:53:30.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7e00640 1 -- 172.21.15.146:0/3706497669 >> v2:172.21.15.161:3300/0 conn(0x7f36d80d0870 msgr2=0x7f36d8154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:30.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7e00640 1 --2- 172.21.15.146:0/3706497669 >> v2:172.21.15.161:3300/0 conn(0x7f36d80d0870 0x7f36d8154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7e00640 1 -- 172.21.15.146:0/3706497669 >> v1:172.21.15.146:6789/0 conn(0x7f36d80d0490 legacy=0x7f36d80c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:30.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36d7e00640 1 -- 172.21.15.146:0/3706497669 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d815e7e0 con 0x7f36d80c6d00 2024-09-16T01:53:30.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.691+0000 7f36cea00640 1 --2- 172.21.15.146:0/3706497669 >> v2:172.21.15.161:3300/0 conn(0x7f36d80d0870 0x7f36d8154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.692+0000 7f36d7e00640 1 --2- 172.21.15.146:0/3706497669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80c6d00 0x7f36d8154bd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009d90 tx=0x7f36d002a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c29061a41fb67d78 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36d6a00640 1 -- 172.21.15.146:0/3706497669 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0007740 con 0x7f36d80c6d00 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36d6a00640 1 -- 172.21.15.146:0/3706497669 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d00078e0 con 0x7f36d80c6d00 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36d6a00640 1 -- 172.21.15.146:0/3706497669 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0007cb0 con 0x7f36d80c6d00 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 -- 172.21.15.146:0/3706497669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80c6d00 msgr2=0x7f36d8154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 --2- 172.21.15.146:0/3706497669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80c6d00 0x7f36d8154bd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009d90 tx=0x7f36d002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 -- 172.21.15.146:0/3706497669 shutdown_connections 2024-09-16T01:53:30.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 --2- 172.21.15.146:0/3706497669 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80c6d00 0x7f36d8154bd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 --2- 172.21.15.146:0/3706497669 >> v2:172.21.15.161:3300/0 conn(0x7f36d80d0870 0x7f36d8154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 -- 172.21.15.146:0/3706497669 >> 172.21.15.146:0/3706497669 conn(0x7f36d806d6f0 msgr2=0x7f36d806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:30.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 -- 172.21.15.146:0/3706497669 shutdown_connections 2024-09-16T01:53:30.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.693+0000 7f36dcc00640 1 -- 172.21.15.146:0/3706497669 wait complete. 2024-09-16T01:53:30.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.695+0000 7f36dcc00640 1 Processor -- start 2024-09-16T01:53:30.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.695+0000 7f36dcc00640 1 -- start start 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36dcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36d80c6d00 0x7f36d81e7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36dcc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 0x7f36d81f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36dcc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80d0870 0x7f36d814da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36dcc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f36d815dec0 con 0x7f36d80c6d00 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36dcc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f36d815eda0 con 0x7f36d80d0490 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36dcc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f36d80c51f0 con 0x7f36d80d0870 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36d7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36d80c6d00 0x7f36d81e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36cea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 0x7f36d81f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36d7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36d80c6d00 0x7f36d81e7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37350/0 (socket says 172.21.15.146:37350) 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36d7400640 1 -- 172.21.15.146:0/141314705 learned_addr learned my addr 172.21.15.146:0/141314705 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36cea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 0x7f36d81f3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:33534/0 (socket says 172.21.15.146:33534) 2024-09-16T01:53:30.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.696+0000 7f36d7e00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80d0870 0x7f36d814da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cea00640 1 -- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80d0870 msgr2=0x7f36d814da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cea00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80d0870 0x7f36d814da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cea00640 1 -- 172.21.15.146:0/141314705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36d80c6d00 msgr2=0x7f36d81e7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cea00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36d80c6d00 0x7f36d81e7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cea00640 1 -- 172.21.15.146:0/141314705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d80b5fe0 con 0x7f36d80d0490 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cea00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 0x7f36d81f3dc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f36c000e840 tx=0x7f36c000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0017070 con 0x7f36d80d0490 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36c000c940 con 0x7f36d80d0490 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c000cd30 con 0x7f36d80d0490 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d80b6280 con 0x7f36d80d0490 2024-09-16T01:53:30.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.697+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d81f4960 con 0x7f36d80d0490 2024-09-16T01:53:30.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.698+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d80c65f0 con 0x7f36d80d0490 2024-09-16T01:53:30.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.699+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f36c001a070 con 0x7f36d80d0490 2024-09-16T01:53:30.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.700+0000 7f36cfe00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f36b406c2b0 0x7f36b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:30.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.700+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5907+0+0 (secure 0 0 0) 0x7f36c008b6c0 con 0x7f36d80d0490 2024-09-16T01:53:30.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.700+0000 7f36d7400640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f36b406c2b0 0x7f36b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:30.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.701+0000 7f36d7400640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f36b406c2b0 0x7f36b406e770 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f36c8005880 tx=0x7f36c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:30.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.704+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f36c0059670 con 0x7f36d80d0490 2024-09-16T01:53:30.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:30.893+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7f36d81e7580 con 0x7f36d80d0490 2024-09-16T01:53:31.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.507+0000 7f36cfe00640 1 -- 172.21.15.146:0/141314705 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v147) ==== 133+0+0 (secure 0 0 0) 0x7f36c005d970 con 0x7f36d80d0490 2024-09-16T01:53:31.508 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.510+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f36b406c2b0 msgr2=0x7f36b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f36b406c2b0 0x7f36b406e770 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f36c8005880 tx=0x7f36c8009040 comp rx=0 tx=0).stop 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 msgr2=0x7f36d81f3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 0x7f36d81f3dc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f36c000e840 tx=0x7f36c000ed00 comp rx=0 tx=0).stop 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 shutdown_connections 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f36b406c2b0 0x7f36b406e770 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f36d80d0870 0x7f36d814da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36d80d0490 0x7f36d81f3dc0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 --2- 172.21.15.146:0/141314705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36d80c6d00 0x7f36d81e7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 >> 172.21.15.146:0/141314705 conn(0x7f36d806d6f0 msgr2=0x7f36d80ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:31.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 shutdown_connections 2024-09-16T01:53:31.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.511+0000 7f36dcc00640 1 -- 172.21.15.146:0/141314705 wait complete. 2024-09-16T01:53:31.529 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T01:53:31.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.686+0000 7f840fe00640 1 Processor -- start 2024-09-16T01:53:31.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.687+0000 7f840fe00640 1 -- start start 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8408152760 0x7f84080c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840fe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f840806fee0 0x7f840815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f84080c8530 con 0x7f8408152760 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840fe00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8408158890 con 0x7f840806fee0 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840fe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8408158b00 con 0x7f84080c6cc0 2024-09-16T01:53:31.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8408152760 0x7f84080c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:60440/0 (socket says 172.21.15.146:60440) 2024-09-16T01:53:31.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8408152760 0x7f84080c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37356/0 (socket says 172.21.15.146:37356) 2024-09-16T01:53:31.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840de00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f840806fee0 0x7f840815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.688+0000 7f840d400640 1 -- 172.21.15.146:0/2739450391 learned_addr learned my addr 172.21.15.146:0/2739450391 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.689+0000 7f840d400640 1 -- 172.21.15.146:0/2739450391 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 msgr2=0x7f84080c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.689+0000 7f840d400640 1 --2- 172.21.15.146:0/2739450391 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.689+0000 7f840d400640 1 -- 172.21.15.146:0/2739450391 >> v2:172.21.15.161:3300/0 conn(0x7f840806fee0 msgr2=0x7f840815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.689+0000 7f840d400640 1 --2- 172.21.15.146:0/2739450391 >> v2:172.21.15.161:3300/0 conn(0x7f840806fee0 0x7f840815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.689+0000 7f840d400640 1 -- 172.21.15.146:0/2739450391 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8408158de0 con 0x7f8408152760 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.690+0000 7f840d400640 1 --2- 172.21.15.146:0/2739450391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f84080c59d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f84080c9e60 tx=0x7f83fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893b4fd204ffe103 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.690+0000 7f8407e00640 1 -- 172.21.15.146:0/2739450391 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc037070 con 0x7f8408152760 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.690+0000 7f8407e00640 1 -- 172.21.15.146:0/2739450391 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83fc032440 con 0x7f8408152760 2024-09-16T01:53:31.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.690+0000 7f8407e00640 1 -- 172.21.15.146:0/2739450391 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc032810 con 0x7f8408152760 2024-09-16T01:53:31.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.690+0000 7f840fe00640 1 -- 172.21.15.146:0/2739450391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 msgr2=0x7f84080c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:31.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.690+0000 7f840fe00640 1 --2- 172.21.15.146:0/2739450391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f84080c59d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f84080c9e60 tx=0x7f83fc02a270 comp rx=0 tx=0).stop 2024-09-16T01:53:31.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 -- 172.21.15.146:0/2739450391 shutdown_connections 2024-09-16T01:53:31.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 --2- 172.21.15.146:0/2739450391 >> v2:172.21.15.161:3300/0 conn(0x7f840806fee0 0x7f840815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 --2- 172.21.15.146:0/2739450391 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 --2- 172.21.15.146:0/2739450391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f84080c59d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 -- 172.21.15.146:0/2739450391 >> 172.21.15.146:0/2739450391 conn(0x7f840806e670 msgr2=0x7f840806ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 -- 172.21.15.146:0/2739450391 shutdown_connections 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.691+0000 7f840fe00640 1 -- 172.21.15.146:0/2739450391 wait complete. 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.692+0000 7f840fe00640 1 Processor -- start 2024-09-16T01:53:31.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.692+0000 7f840fe00640 1 -- start start 2024-09-16T01:53:31.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840806fee0 0x7f8408158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f840814c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8408152b40 con 0x7f8408152760 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840fe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f84081593a0 con 0x7f840806fee0 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840fe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8408152cc0 con 0x7f84080c6cc0 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.693+0000 7f840de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f840814c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f840814c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37366/0 (socket says 172.21.15.146:37366) 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 -- 172.21.15.146:0/3758934431 learned_addr learned my addr 172.21.15.146:0/3758934431 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840d400640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840806fee0 0x7f8408158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840ca00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 -- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 msgr2=0x7f84080cc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:31.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080cc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 -- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840806fee0 msgr2=0x7f8408158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840806fee0 0x7f8408158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 -- 172.21.15.146:0/3758934431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84081576c0 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f840de00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f840814c400 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f840000eed0 tx=0x7f840000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400018020 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840000bd20 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.694+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400014640 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.695+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84081f43d0 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.695+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8408157c40 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.696+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84080c72c0 con 0x7f8408152760 2024-09-16T01:53:31.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.696+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8400005d10 con 0x7f8408152760 2024-09-16T01:53:31.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.697+0000 7f8405600640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83e806c2b0 0x7f83e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:31.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.697+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 5857+0+0 (secure 0 0 0) 0x7f840008bd80 con 0x7f8408152760 2024-09-16T01:53:31.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.698+0000 7f840d400640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83e806c2b0 0x7f83e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:31.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.698+0000 7f840d400640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83e806c2b0 0x7f83e806e770 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f83fc034000 tx=0x7f83fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:31.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.702+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8400022030 con 0x7f8408152760 2024-09-16T01:53:31.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:31.886+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f8408072140 con 0x7f8408152760 2024-09-16T01:53:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.504+0000 7f8405600640 1 -- 172.21.15.146:0/3758934431 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v148) ==== 113+0+0 (secure 0 0 0) 0x7f8400059cd0 con 0x7f8408152760 2024-09-16T01:53:32.506 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83e806c2b0 msgr2=0x7f83e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83e806c2b0 0x7f83e806e770 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f83fc034000 tx=0x7f83fc02e000 comp rx=0 tx=0).stop 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 msgr2=0x7f840814c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f840814c400 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f840000eed0 tx=0x7f840000b480 comp rx=0 tx=0).stop 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 shutdown_connections 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f83e806c2b0 0x7f83e806e770 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8408152760 0x7f840814c400 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f84080c6cc0 0x7f84080cc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 --2- 172.21.15.146:0/3758934431 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f840806fee0 0x7f8408158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 >> 172.21.15.146:0/3758934431 conn(0x7f840806e670 msgr2=0x7f84080c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:32.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 shutdown_connections 2024-09-16T01:53:32.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.508+0000 7f840fe00640 1 -- 172.21.15.146:0/3758934431 wait complete. 2024-09-16T01:53:32.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T01:53:32.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.687+0000 7fe45f200640 1 Processor -- start 2024-09-16T01:53:32.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.688+0000 7fe45f200640 1 -- start start 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe460154ac0 0x7fe46015b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45f200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe46015bc50 0x7fe46015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe460154ea0 con 0x7fe46006d820 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe46015b6d0 con 0x7fe460154ac0 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45f200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe46015c5c0 con 0x7fe46015bc50 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe455400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe46006d820 0x7fe4600d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:42970/0 (socket says 172.21.15.146:42970) 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe455400640 1 -- 172.21.15.146:0/416289780 learned_addr learned my addr 172.21.15.146:0/416289780 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:32.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45de00640 1 --2- 172.21.15.146:0/416289780 >> v2:172.21.15.146:3300/0 conn(0x7fe460154ac0 0x7fe46015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:32.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.689+0000 7fe45e800640 1 --2- 172.21.15.146:0/416289780 >> v2:172.21.15.161:3300/0 conn(0x7fe46015bc50 0x7fe46015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:32.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.690+0000 7fe45e800640 1 -- 172.21.15.146:0/416289780 >> v2:172.21.15.146:3300/0 conn(0x7fe460154ac0 msgr2=0x7fe46015b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:32.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.690+0000 7fe45e800640 1 --2- 172.21.15.146:0/416289780 >> v2:172.21.15.146:3300/0 conn(0x7fe460154ac0 0x7fe46015b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.690+0000 7fe45d400640 1 -- 172.21.15.146:0/416289780 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206317866 0 0) 0x7fe460154ea0 con 0x7fe46006d820 2024-09-16T01:53:32.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.690+0000 7fe45e800640 1 -- 172.21.15.146:0/416289780 >> v1:172.21.15.146:6789/0 conn(0x7fe46006d820 legacy=0x7fe4600d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:32.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.690+0000 7fe45e800640 1 -- 172.21.15.146:0/416289780 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4600c5340 con 0x7fe46015bc50 2024-09-16T01:53:32.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.690+0000 7fe45de00640 1 --2- 172.21.15.146:0/416289780 >> v2:172.21.15.146:3300/0 conn(0x7fe460154ac0 0x7fe46015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.691+0000 7fe45e800640 1 --2- 172.21.15.146:0/416289780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46015bc50 0x7fe46015c080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe46006bcc0 tx=0x7fe45802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=394e69dc30d72d52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.691+0000 7fe45d400640 1 -- 172.21.15.146:0/416289780 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458007740 con 0x7fe46015bc50 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.691+0000 7fe45d400640 1 -- 172.21.15.146:0/416289780 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4580078e0 con 0x7fe46015bc50 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.691+0000 7fe45d400640 1 -- 172.21.15.146:0/416289780 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458007cb0 con 0x7fe46015bc50 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.691+0000 7fe45f200640 1 -- 172.21.15.146:0/416289780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46015bc50 msgr2=0x7fe46015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.691+0000 7fe45f200640 1 --2- 172.21.15.146:0/416289780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46015bc50 0x7fe46015c080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe46006bcc0 tx=0x7fe45802a710 comp rx=0 tx=0).stop 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.692+0000 7fe45f200640 1 -- 172.21.15.146:0/416289780 shutdown_connections 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.692+0000 7fe45f200640 1 --2- 172.21.15.146:0/416289780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46015bc50 0x7fe46015c080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.692+0000 7fe45f200640 1 --2- 172.21.15.146:0/416289780 >> v2:172.21.15.146:3300/0 conn(0x7fe460154ac0 0x7fe46015b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.692+0000 7fe45f200640 1 -- 172.21.15.146:0/416289780 >> 172.21.15.146:0/416289780 conn(0x7fe46006e400 msgr2=0x7fe46006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:32.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.692+0000 7fe45f200640 1 -- 172.21.15.146:0/416289780 shutdown_connections 2024-09-16T01:53:32.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.692+0000 7fe45f200640 1 -- 172.21.15.146:0/416289780 wait complete. 2024-09-16T01:53:32.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.693+0000 7fe45f200640 1 Processor -- start 2024-09-16T01:53:32.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.693+0000 7fe45f200640 1 -- start start 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46006d820 0x7fe4600c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45f200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe460154ac0 0x7fe46014ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 0x7fe46015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4600c5be0 con 0x7fe46015bc50 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4600c5900 con 0x7fe46006d820 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45f200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe4600c5ec0 con 0x7fe460154ac0 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 0x7fe46015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:32.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46006d820 0x7fe4600c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 0x7fe46015e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37390/0 (socket says 172.21.15.146:37390) 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe45e800640 1 -- 172.21.15.146:0/2311945537 learned_addr learned my addr 172.21.15.146:0/2311945537 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.694+0000 7fe455400640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe460154ac0 0x7fe46014ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45e800640 1 -- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe460154ac0 msgr2=0x7fe46014ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45e800640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe460154ac0 0x7fe46014ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45e800640 1 -- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46006d820 msgr2=0x7fe4600c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45e800640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46006d820 0x7fe4600c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45e800640 1 -- 172.21.15.146:0/2311945537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4600c9ab0 con 0x7fe46015bc50 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45de00640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46006d820 0x7fe4600c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45e800640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 0x7fe46015e3b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe45802ad00 tx=0x7fe458004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45803f070 con 0x7fe46015bc50 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe458031bc0 con 0x7fe46015bc50 2024-09-16T01:53:32.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46015c9d0 con 0x7fe46015bc50 2024-09-16T01:53:32.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.695+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458009450 con 0x7fe46015bc50 2024-09-16T01:53:32.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.696+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe46015d8d0 con 0x7fe46015bc50 2024-09-16T01:53:32.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.697+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46015d6e0 con 0x7fe46015bc50 2024-09-16T01:53:32.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.697+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe458044460 con 0x7fe46015bc50 2024-09-16T01:53:32.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.698+0000 7fe456a00640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe43c06c2e0 0x7fe43c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:32.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.698+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 5857+0+0 (secure 0 0 0) 0x7fe4580acaa0 con 0x7fe46015bc50 2024-09-16T01:53:32.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.699+0000 7fe45de00640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe43c06c2e0 0x7fe43c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:32.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.700+0000 7fe45de00640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe43c06c2e0 0x7fe43c06e7a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe450009560 tx=0x7fe450009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:32.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.703+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe45807ab10 con 0x7fe46015bc50 2024-09-16T01:53:32.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:32.888+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7fe4600c61a0 con 0x7fe46015bc50 2024-09-16T01:53:33.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.512+0000 7fe456a00640 1 -- 172.21.15.146:0/2311945537 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v149) ==== 133+0+0 (secure 0 0 0) 0x7fe45807ee10 con 0x7fe46015bc50 2024-09-16T01:53:33.513 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T01:53:33.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe43c06c2e0 msgr2=0x7fe43c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe43c06c2e0 0x7fe43c06e7a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe450009560 tx=0x7fe450009190 comp rx=0 tx=0).stop 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 msgr2=0x7fe46015e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 0x7fe46015e3b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe45802ad00 tx=0x7fe458004a40 comp rx=0 tx=0).stop 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 shutdown_connections 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe43c06c2e0 0x7fe43c06e7a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe46015bc50 0x7fe46015e3b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe460154ac0 0x7fe46014ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 --2- 172.21.15.146:0/2311945537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe46006d820 0x7fe4600c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.515+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 >> 172.21.15.146:0/2311945537 conn(0x7fe46006e400 msgr2=0x7fe460151600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.516+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 shutdown_connections 2024-09-16T01:53:33.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.516+0000 7fe45f200640 1 -- 172.21.15.146:0/2311945537 wait complete. 2024-09-16T01:53:33.537 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T01:53:33.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.695+0000 7fb054800640 1 Processor -- start 2024-09-16T01:53:33.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.695+0000 7fb054800640 1 -- start start 2024-09-16T01:53:33.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.696+0000 7fb054800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb04c0c6cc0 0x7fb04c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:33.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.696+0000 7fb054800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c06fee0 0x7fb04c15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:33.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.696+0000 7fb054800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04c158890 con 0x7fb04c152760 2024-09-16T01:53:33.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.696+0000 7fb054800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb04c0c8530 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.696+0000 7fb054800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb04c158b00 con 0x7fb04c06fee0 2024-09-16T01:53:33.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb052800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c06fee0 0x7fb04c15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb051e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb04c152760 0x7fb04c0c59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44604/0 (socket says 172.21.15.146:44604) 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb051400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb04c0c6cc0 0x7fb04c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb051e00640 1 -- 172.21.15.146:0/3179676979 learned_addr learned my addr 172.21.15.146:0/3179676979 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb050a00640 1 -- 172.21.15.146:0/3179676979 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045723929 0 0) 0x7fb04c158890 con 0x7fb04c152760 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb052800640 1 -- 172.21.15.146:0/3179676979 >> v2:172.21.15.161:3300/0 conn(0x7fb04c0c6cc0 msgr2=0x7fb04c15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb052800640 1 --2- 172.21.15.146:0/3179676979 >> v2:172.21.15.161:3300/0 conn(0x7fb04c0c6cc0 0x7fb04c15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb052800640 1 -- 172.21.15.146:0/3179676979 >> v1:172.21.15.161:6789/0 conn(0x7fb04c152760 legacy=0x7fb04c0c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:33.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.697+0000 7fb052800640 1 -- 172.21.15.146:0/3179676979 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c158de0 con 0x7fb04c06fee0 2024-09-16T01:53:33.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.698+0000 7fb052800640 1 --2- 172.21.15.146:0/3179676979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c06fee0 0x7fb04c15c2b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb048009d90 tx=0x7fb04802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84db45af1c0d0568 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:33.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb050a00640 1 -- 172.21.15.146:0/3179676979 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048007740 con 0x7fb04c06fee0 2024-09-16T01:53:33.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb050a00640 1 -- 172.21.15.146:0/3179676979 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0480078e0 con 0x7fb04c06fee0 2024-09-16T01:53:33.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb050a00640 1 -- 172.21.15.146:0/3179676979 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048007cb0 con 0x7fb04c06fee0 2024-09-16T01:53:33.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 -- 172.21.15.146:0/3179676979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c06fee0 msgr2=0x7fb04c15c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 --2- 172.21.15.146:0/3179676979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c06fee0 0x7fb04c15c2b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb048009d90 tx=0x7fb04802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 -- 172.21.15.146:0/3179676979 shutdown_connections 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 --2- 172.21.15.146:0/3179676979 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c06fee0 0x7fb04c15c2b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 --2- 172.21.15.146:0/3179676979 >> v2:172.21.15.161:3300/0 conn(0x7fb04c0c6cc0 0x7fb04c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 -- 172.21.15.146:0/3179676979 >> 172.21.15.146:0/3179676979 conn(0x7fb04c06e670 msgr2=0x7fb04c06ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 -- 172.21.15.146:0/3179676979 shutdown_connections 2024-09-16T01:53:33.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.699+0000 7fb054800640 1 -- 172.21.15.146:0/3179676979 wait complete. 2024-09-16T01:53:33.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.701+0000 7fb054800640 1 Processor -- start 2024-09-16T01:53:33.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.701+0000 7fb054800640 1 -- start start 2024-09-16T01:53:33.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb054800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 0x7fb04c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:33.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb054800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 0x7fb04c145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb054800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb04c152760 0x7fb04c156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb054800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c152b40 con 0x7fb04c06fee0 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb054800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c1593a0 con 0x7fb04c152760 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb054800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb04c152cc0 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb051e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 0x7fb04c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb051400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 0x7fb04c145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb052800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb04c152760 0x7fb04c156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb051400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 0x7fb04c145dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39190/0 (socket says 172.21.15.146:39190) 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb051e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 0x7fb04c148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46932/0 (socket says 172.21.15.146:46932) 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.702+0000 7fb051400640 1 -- 172.21.15.146:0/367227470 learned_addr learned my addr 172.21.15.146:0/367227470 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051400640 1 -- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb04c152760 msgr2=0x7fb04c156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:33.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051400640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb04c152760 0x7fb04c156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051400640 1 -- 172.21.15.146:0/367227470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 msgr2=0x7fb04c148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051400640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 0x7fb04c148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051400640 1 -- 172.21.15.146:0/367227470 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c1485c0 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051e00640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 0x7fb04c148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb051400640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 0x7fb04c145dc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00e9b0 tx=0x7fb03c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c017070 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03c00c9a0 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04c146300 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c00cd90 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.703+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb04c1f1b40 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.704+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04c1f28d0 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.706+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb03c01a070 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.707+0000 7fb046000640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb03006c2b0 0x7fb03006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:33.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.707+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(149..149 src has 1..149) ==== 5857+0+0 (secure 0 0 0) 0x7fb03c08b8e0 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.707+0000 7fb051e00640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb03006c2b0 0x7fb03006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:33.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.708+0000 7fb051e00640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb03006c2b0 0x7fb03006e770 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb040000f60 tx=0x7fb040009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:33.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.711+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb03c059830 con 0x7fb04c0c6cc0 2024-09-16T01:53:33.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:33.897+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fb04c0c7230 con 0x7fb04c0c6cc0 2024-09-16T01:53:34.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.527+0000 7fb046000640 1 -- 172.21.15.146:0/367227470 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v150) ==== 113+0+0 (secure 0 0 0) 0x7fb03c05db30 con 0x7fb04c0c6cc0 2024-09-16T01:53:34.529 INFO:teuthology.orchestra.run.smithi146.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb03006c2b0 msgr2=0x7fb03006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb03006c2b0 0x7fb03006e770 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb040000f60 tx=0x7fb040009040 comp rx=0 tx=0).stop 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 msgr2=0x7fb04c145dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 0x7fb04c145dc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00e9b0 tx=0x7fb03c00ee70 comp rx=0 tx=0).stop 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 shutdown_connections 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fb03006c2b0 0x7fb03006e770 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb04c152760 0x7fb04c156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c0c6cc0 0x7fb04c145dc0 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00e9b0 tx=0x7fb03c00ee70 comp rx=0 tx=0).stop 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 --2- 172.21.15.146:0/367227470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb04c06fee0 0x7fb04c148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 >> 172.21.15.146:0/367227470 conn(0x7fb04c06e670 msgr2=0x7fb04c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:34.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 shutdown_connections 2024-09-16T01:53:34.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.532+0000 7fb054800640 1 -- 172.21.15.146:0/367227470 wait complete. 2024-09-16T01:53:34.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T01:53:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.708+0000 7f62eb200640 1 Processor -- start 2024-09-16T01:53:34.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.708+0000 7f62eb200640 1 -- start start 2024-09-16T01:53:34.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62eb200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec154a80 0x7f62ec072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:34.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62eb200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f62ec0717a0 0x7f62ec0c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:34.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62eb200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec157690 con 0x7f62ec071bc0 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62eb200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec1582a0 con 0x7f62ec0717a0 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62eb200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec157970 con 0x7f62ec154a80 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62ea800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f62ec071bc0 0x7f62ec06cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47094/0 (socket says 172.21.15.146:47094) 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.709+0000 7f62ea800640 1 -- 172.21.15.146:0/556070141 learned_addr learned my addr 172.21.15.146:0/556070141 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9e00640 1 --2- 172.21.15.146:0/556070141 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec154a80 0x7f62ec072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9400640 1 --2- 172.21.15.146:0/556070141 >> v2:172.21.15.161:3300/0 conn(0x7f62ec0717a0 0x7f62ec0c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e8a00640 1 -- 172.21.15.146:0/556070141 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677588539 0 0) 0x7f62ec157690 con 0x7f62ec071bc0 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e8a00640 1 -- 172.21.15.146:0/556070141 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62d0003520 con 0x7f62ec071bc0 2024-09-16T01:53:34.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9e00640 1 -- 172.21.15.146:0/556070141 >> v2:172.21.15.161:3300/0 conn(0x7f62ec0717a0 msgr2=0x7f62ec0c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9e00640 1 --2- 172.21.15.146:0/556070141 >> v2:172.21.15.161:3300/0 conn(0x7f62ec0717a0 0x7f62ec0c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9e00640 1 -- 172.21.15.146:0/556070141 >> v1:172.21.15.146:6789/0 conn(0x7f62ec071bc0 legacy=0x7f62ec06cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9e00640 1 -- 172.21.15.146:0/556070141 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec157c50 con 0x7f62ec154a80 2024-09-16T01:53:34.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.710+0000 7f62e9400640 1 --2- 172.21.15.146:0/556070141 >> v2:172.21.15.161:3300/0 conn(0x7f62ec0717a0 0x7f62ec0c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.711+0000 7f62e9e00640 1 --2- 172.21.15.146:0/556070141 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec154a80 0x7f62ec072180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097a0 tx=0x7f62dc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e4b908eacf8b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.711+0000 7f62e8a00640 1 -- 172.21.15.146:0/556070141 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc037070 con 0x7f62ec154a80 2024-09-16T01:53:34.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.711+0000 7f62e8a00640 1 -- 172.21.15.146:0/556070141 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc032440 con 0x7f62ec154a80 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.711+0000 7f62e8a00640 1 -- 172.21.15.146:0/556070141 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0327f0 con 0x7f62ec154a80 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 -- 172.21.15.146:0/556070141 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec154a80 msgr2=0x7f62ec072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 --2- 172.21.15.146:0/556070141 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec154a80 0x7f62ec072180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097a0 tx=0x7f62dc02a180 comp rx=0 tx=0).stop 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 -- 172.21.15.146:0/556070141 shutdown_connections 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 --2- 172.21.15.146:0/556070141 >> v2:172.21.15.161:3300/0 conn(0x7f62ec0717a0 0x7f62ec0c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 --2- 172.21.15.146:0/556070141 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec154a80 0x7f62ec072180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 -- 172.21.15.146:0/556070141 >> 172.21.15.146:0/556070141 conn(0x7f62ec06f710 msgr2=0x7f62ec06fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 -- 172.21.15.146:0/556070141 shutdown_connections 2024-09-16T01:53:34.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.712+0000 7f62eb200640 1 -- 172.21.15.146:0/556070141 wait complete. 2024-09-16T01:53:34.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 Processor -- start 2024-09-16T01:53:34.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 -- start start 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 0x7f62ec0cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62ec154a80 0x7f62ec156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec0c2850 0x7f62ec0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0c8630 con 0x7f62ec0717a0 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec154e60 con 0x7f62ec154a80 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.714+0000 7f62eb200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec0c87b0 con 0x7f62ec0c2850 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 0x7f62ec0cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 0x7f62ec0cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46948/0 (socket says 172.21.15.146:46948) 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 -- 172.21.15.146:0/3263553725 learned_addr learned my addr 172.21.15.146:0/3263553725 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62ea800640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec0c2850 0x7f62ec0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9400640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62ec154a80 0x7f62ec156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 -- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec0c2850 msgr2=0x7f62ec0af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec0c2850 0x7f62ec0af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 -- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62ec154a80 msgr2=0x7f62ec156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62ec154a80 0x7f62ec156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.715+0000 7f62e9e00640 1 -- 172.21.15.146:0/3263553725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec155ae0 con 0x7f62ec0717a0 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.716+0000 7f62e9e00640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 0x7f62ec0cb180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0098c0 tx=0x7f62dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.716+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc037040 con 0x7f62ec0717a0 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.716+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc0044c0 con 0x7f62ec0717a0 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.716+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0325f0 con 0x7f62ec0717a0 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.716+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec1f4cf0 con 0x7f62ec0717a0 2024-09-16T01:53:34.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.716+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1f4fd0 con 0x7f62ec0717a0 2024-09-16T01:53:34.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.717+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec073c70 con 0x7f62ec0717a0 2024-09-16T01:53:34.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.718+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f62dc044020 con 0x7f62ec0717a0 2024-09-16T01:53:34.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.719+0000 7f62e2000640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62cc06c2e0 0x7f62cc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:34.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.719+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5857+0+0 (secure 0 0 0) 0x7f62dc0ab5c0 con 0x7f62ec0717a0 2024-09-16T01:53:34.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.719+0000 7f62e9400640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62cc06c2e0 0x7f62cc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:34.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.719+0000 7f62e9400640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62cc06c2e0 0x7f62cc06e7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009560 tx=0x7f62d8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:34.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.723+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f62ec073c70 con 0x7f62ec0717a0 2024-09-16T01:53:34.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:34.908+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) -- 0x7f62ec073890 con 0x7f62ec0717a0 2024-09-16T01:53:35.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.530+0000 7f62e2000640 1 -- 172.21.15.146:0/3263553725 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v151) ==== 133+0+0 (secure 0 0 0) 0x7f62ec073890 con 0x7f62ec0717a0 2024-09-16T01:53:35.532 INFO:teuthology.orchestra.run.smithi146.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.533+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62cc06c2e0 msgr2=0x7f62cc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.533+0000 7f62eb200640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62cc06c2e0 0x7f62cc06e7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009560 tx=0x7f62d8009190 comp rx=0 tx=0).stop 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.533+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 msgr2=0x7f62ec0cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.533+0000 7f62eb200640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 0x7f62ec0cb180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0098c0 tx=0x7f62dc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 shutdown_connections 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f62cc06c2e0 0x7f62cc06e7a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f62ec0c2850 0x7f62ec0af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62ec154a80 0x7f62ec156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 --2- 172.21.15.146:0/3263553725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62ec0717a0 0x7f62ec0cb180 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 >> 172.21.15.146:0/3263553725 conn(0x7f62ec06f710 msgr2=0x7f62ec06d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:35.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 shutdown_connections 2024-09-16T01:53:35.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.534+0000 7f62eb200640 1 -- 172.21.15.146:0/3263553725 wait complete. 2024-09-16T01:53:35.553 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T01:53:35.553 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T01:53:35.553 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T01:53:35.553 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T01:53:35.553 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T01:53:35.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:53:35.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.706+0000 7f9489600640 1 Processor -- start 2024-09-16T01:53:35.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.706+0000 7f9489600640 1 -- start start 2024-09-16T01:53:35.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.707+0000 7f9489600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9484071b30 0x7f94840c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:35.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.707+0000 7f9489600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9484150bf0 0x7f9484151020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:35.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9489600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94841578a0 con 0x7f94840c98d0 2024-09-16T01:53:35.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9489600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f94840c7a80 con 0x7f9484071b30 2024-09-16T01:53:35.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9489600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9484072620 con 0x7f9484150bf0 2024-09-16T01:53:35.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9484071b30 0x7f94840c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:35.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f94840c98d0 0x7f9484071f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47108/0 (socket says 172.21.15.146:47108) 2024-09-16T01:53:35.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482000640 1 -- 172.21.15.146:0/1627069187 learned_addr learned my addr 172.21.15.146:0/1627069187 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:35.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9483400640 1 --2- 172.21.15.146:0/1627069187 >> v2:172.21.15.161:3300/0 conn(0x7f9484150bf0 0x7f9484151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:35.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482a00640 1 -- 172.21.15.146:0/1627069187 >> v2:172.21.15.161:3300/0 conn(0x7f9484150bf0 msgr2=0x7f9484151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:35.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482a00640 1 --2- 172.21.15.146:0/1627069187 >> v2:172.21.15.161:3300/0 conn(0x7f9484150bf0 0x7f9484151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482a00640 1 -- 172.21.15.146:0/1627069187 >> v1:172.21.15.146:6789/0 conn(0x7f94840c98d0 legacy=0x7f9484071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9482a00640 1 -- 172.21.15.146:0/1627069187 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9484072f30 con 0x7f9484071b30 2024-09-16T01:53:35.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.708+0000 7f9481600640 1 -- 172.21.15.146:0/1627069187 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039691102 0 0) 0x7f94841578a0 con 0x7f94840c98d0 2024-09-16T01:53:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.709+0000 7f9482a00640 1 --2- 172.21.15.146:0/1627069187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484071b30 0x7f94840c6dc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f948406ce40 tx=0x7f947402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aed7d44efc23adf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9481600640 1 -- 172.21.15.146:0/1627069187 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474037070 con 0x7f9484071b30 2024-09-16T01:53:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9481600640 1 -- 172.21.15.146:0/1627069187 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9474032440 con 0x7f9484071b30 2024-09-16T01:53:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9481600640 1 -- 172.21.15.146:0/1627069187 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474032810 con 0x7f9484071b30 2024-09-16T01:53:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 -- 172.21.15.146:0/1627069187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484071b30 msgr2=0x7f94840c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 --2- 172.21.15.146:0/1627069187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484071b30 0x7f94840c6dc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f948406ce40 tx=0x7f947402a230 comp rx=0 tx=0).stop 2024-09-16T01:53:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 -- 172.21.15.146:0/1627069187 shutdown_connections 2024-09-16T01:53:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 --2- 172.21.15.146:0/1627069187 >> v2:172.21.15.161:3300/0 conn(0x7f9484150bf0 0x7f9484151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 --2- 172.21.15.146:0/1627069187 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484071b30 0x7f94840c6dc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 -- 172.21.15.146:0/1627069187 >> 172.21.15.146:0/1627069187 conn(0x7f948406f3c0 msgr2=0x7f948406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 -- 172.21.15.146:0/1627069187 shutdown_connections 2024-09-16T01:53:35.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.710+0000 7f9489600640 1 -- 172.21.15.146:0/1627069187 wait complete. 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.712+0000 7f9489600640 1 Processor -- start 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.712+0000 7f9489600640 1 -- start start 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9489600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9484071b30 0x7f9484148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9489600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94840c98d0 0x7f9484149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9489600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 0x7f9484149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9489600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f94840c8cb0 con 0x7f9484150bf0 2024-09-16T01:53:35.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9489600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f94840731f0 con 0x7f9484071b30 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9489600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f94840c8e30 con 0x7f94840c98d0 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 0x7f9484149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 0x7f9484149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46956/0 (socket says 172.21.15.146:46956) 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 -- 172.21.15.146:0/21324100 learned_addr learned my addr 172.21.15.146:0/21324100 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9482a00640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9484071b30 0x7f9484148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9482000640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94840c98d0 0x7f9484149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 -- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94840c98d0 msgr2=0x7f9484149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94840c98d0 0x7f9484149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 -- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9484071b30 msgr2=0x7f9484148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9484071b30 0x7f9484148e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.713+0000 7f9483400640 1 -- 172.21.15.146:0/21324100 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94841539f0 con 0x7f9484150bf0 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.714+0000 7f9483400640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 0x7f9484149f40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f947c00efc0 tx=0x7f947c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:35.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.714+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c018020 con 0x7f9484150bf0 2024-09-16T01:53:35.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.714+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f947c014480 con 0x7f9484150bf0 2024-09-16T01:53:35.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.714+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9484153cd0 con 0x7f9484150bf0 2024-09-16T01:53:35.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.714+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c014870 con 0x7f9484150bf0 2024-09-16T01:53:35.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.714+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9484154290 con 0x7f9484150bf0 2024-09-16T01:53:35.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.715+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94840c9220 con 0x7f9484150bf0 2024-09-16T01:53:35.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.716+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f947c01b020 con 0x7f9484150bf0 2024-09-16T01:53:35.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.717+0000 7f947aa00640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f946406c2e0 0x7f946406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:35.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.717+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5857+0+0 (secure 0 0 0) 0x7f947c08c1a0 con 0x7f9484150bf0 2024-09-16T01:53:35.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.717+0000 7f9482a00640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f946406c2e0 0x7f946406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:35.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.718+0000 7f9482a00640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f946406c2e0 0x7f946406e7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9474034000 tx=0x7f947402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:35.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.721+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f947c022020 con 0x7f9484150bf0 2024-09-16T01:53:35.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.896+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94840733e0 con 0x7f9484150bf0 2024-09-16T01:53:35.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.898+0000 7f947aa00640 1 -- 172.21.15.146:0/21324100 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16939 (secure 0 0 0) 0x7f947c010040 con 0x7f9484150bf0 2024-09-16T01:53:35.899 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:35.899 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":151,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:53:35.529421+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:53:19.491406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":2,"to":5}]},{"pgid":"3.20","mappings":[{"from":2,"to":7}]},{"pgid":"3.2d","mappings":[{"from":3,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:53:35.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f946406c2e0 msgr2=0x7f946406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f946406c2e0 0x7f946406e7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9474034000 tx=0x7f947402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:35.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 msgr2=0x7f9484149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 0x7f9484149f40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f947c00efc0 tx=0x7f947c00c490 comp rx=0 tx=0).stop 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 shutdown_connections 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f946406c2e0 0x7f946406e7a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9484150bf0 0x7f9484149f40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94840c98d0 0x7f9484149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 --2- 172.21.15.146:0/21324100 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9484071b30 0x7f9484148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 >> 172.21.15.146:0/21324100 conn(0x7f948406f3c0 msgr2=0x7f94840c52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 shutdown_connections 2024-09-16T01:53:35.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:35.900+0000 7f9489600640 1 -- 172.21.15.146:0/21324100 wait complete. 2024-09-16T01:53:35.919 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T01:53:35.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:53:36.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.072+0000 7f55ff000640 1 Processor -- start 2024-09-16T01:53:36.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.072+0000 7f55ff000640 1 -- start start 2024-09-16T01:53:36.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55ff000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55f80c9700 0x7f55f8164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55ff000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55f8072f50 con 0x7f55f8152850 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55ff000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55f815cb50 con 0x7f55f80c6bb0 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55ff000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f55f8073210 con 0x7f55f80c9700 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55f7e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f55f8152850 0x7f55f8158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47114/0 (socket says 172.21.15.146:47114) 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55f7e00640 1 -- 172.21.15.146:0/1992931731 learned_addr learned my addr 172.21.15.146:0/1992931731 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55fd000640 1 --2- 172.21.15.146:0/1992931731 >> v2:172.21.15.161:3300/0 conn(0x7f55f80c9700 0x7f55f8164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55f6a00640 1 -- 172.21.15.146:0/1992931731 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3673017118 0 0) 0x7f55f815cb50 con 0x7f55f80c6bb0 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55f6a00640 1 -- 172.21.15.146:0/1992931731 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55dc003520 con 0x7f55f80c6bb0 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55f6a00640 1 -- 172.21.15.146:0/1992931731 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105730715 0 0) 0x7f55f8072f50 con 0x7f55f8152850 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55fd000640 1 -- 172.21.15.146:0/1992931731 >> v1:172.21.15.161:6789/0 conn(0x7f55f80c6bb0 legacy=0x7f55f80c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55fd000640 1 -- 172.21.15.146:0/1992931731 >> v1:172.21.15.146:6789/0 conn(0x7f55f8152850 legacy=0x7f55f8158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.074+0000 7f55fd000640 1 -- 172.21.15.146:0/1992931731 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f815bd90 con 0x7f55f80c9700 2024-09-16T01:53:36.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.075+0000 7f55fd000640 1 --2- 172.21.15.146:0/1992931731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c9700 0x7f55f8164000 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f55f00097a0 tx=0x7f55f002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f03bfff025a0c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55f6a00640 1 -- 172.21.15.146:0/1992931731 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0037070 con 0x7f55f80c9700 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55f6a00640 1 -- 172.21.15.146:0/1992931731 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f0032440 con 0x7f55f80c9700 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55f6a00640 1 -- 172.21.15.146:0/1992931731 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f00327f0 con 0x7f55f80c9700 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 -- 172.21.15.146:0/1992931731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c9700 msgr2=0x7f55f8164000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 --2- 172.21.15.146:0/1992931731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c9700 0x7f55f8164000 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f55f00097a0 tx=0x7f55f002a230 comp rx=0 tx=0).stop 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 -- 172.21.15.146:0/1992931731 shutdown_connections 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 --2- 172.21.15.146:0/1992931731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c9700 0x7f55f8164000 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 -- 172.21.15.146:0/1992931731 >> 172.21.15.146:0/1992931731 conn(0x7f55f806f3c0 msgr2=0x7f55f806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 -- 172.21.15.146:0/1992931731 shutdown_connections 2024-09-16T01:53:36.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.076+0000 7f55ff000640 1 -- 172.21.15.146:0/1992931731 wait complete. 2024-09-16T01:53:36.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.078+0000 7f55ff000640 1 Processor -- start 2024-09-16T01:53:36.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.078+0000 7f55ff000640 1 -- start start 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55ff000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c6bb0 0x7f55f80c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55ff000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55f80c9700 0x7f55f814cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55ff000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 0x7f55f8144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55ff000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f55f815c630 con 0x7f55f8152850 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55ff000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f55f815c350 con 0x7f55f80c6bb0 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55ff000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f55f8070200 con 0x7f55f80c9700 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55fd000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 0x7f55f8144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55fd000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 0x7f55f8144af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46966/0 (socket says 172.21.15.146:46966) 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55fd000640 1 -- 172.21.15.146:0/2585032688 learned_addr learned my addr 172.21.15.146:0/2585032688 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55f7e00640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c6bb0 0x7f55f80c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.079+0000 7f55f7400640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55f80c9700 0x7f55f814cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55f7e00640 1 -- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55f80c9700 msgr2=0x7f55f814cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55f7e00640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55f80c9700 0x7f55f814cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55f7e00640 1 -- 172.21.15.146:0/2585032688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 msgr2=0x7f55f8144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55f7e00640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 0x7f55f8144af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55f7e00640 1 -- 172.21.15.146:0/2585032688 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f81f4820 con 0x7f55f80c6bb0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55fd000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 0x7f55f8144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55f7e00640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c6bb0 0x7f55f80c4f30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c780 tx=0x7f55e800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8018070 con 0x7f55f80c6bb0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e800ce30 con 0x7f55f80c6bb0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f80c38a0 con 0x7f55f80c6bb0 2024-09-16T01:53:36.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e800a860 con 0x7f55f80c6bb0 2024-09-16T01:53:36.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.080+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f81f3d40 con 0x7f55f80c6bb0 2024-09-16T01:53:36.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.081+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f8073780 con 0x7f55f80c6bb0 2024-09-16T01:53:36.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.082+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f55e801b070 con 0x7f55f80c6bb0 2024-09-16T01:53:36.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.083+0000 7f55efe00640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f55d806c2b0 0x7f55d806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.083+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5857+0+0 (secure 0 0 0) 0x7f55e808a9c0 con 0x7f55f80c6bb0 2024-09-16T01:53:36.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.083+0000 7f55f7400640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f55d806c2b0 0x7f55d806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.084+0000 7f55f7400640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f55d806c2b0 0x7f55d806e770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f55f80c7d30 tx=0x7f55e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.088+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f55f8073780 con 0x7f55f80c6bb0 2024-09-16T01:53:36.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.266+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55f8150eb0 con 0x7f55f80c6bb0 2024-09-16T01:53:36.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.268+0000 7f55efe00640 1 -- 172.21.15.146:0/2585032688 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16939 (secure 0 0 0) 0x7f55e80589a0 con 0x7f55f80c6bb0 2024-09-16T01:53:36.269 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:36.269 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":151,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:53:35.529421+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:53:19.491406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":2,"to":5}]},{"pgid":"3.20","mappings":[{"from":2,"to":7}]},{"pgid":"3.2d","mappings":[{"from":3,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f55d806c2b0 msgr2=0x7f55d806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f55d806c2b0 0x7f55d806e770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f55f80c7d30 tx=0x7f55e4009040 comp rx=0 tx=0).stop 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c6bb0 msgr2=0x7f55f80c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c6bb0 0x7f55f80c4f30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c780 tx=0x7f55e800cc40 comp rx=0 tx=0).stop 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 shutdown_connections 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f55d806c2b0 0x7f55d806e770 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55f8152850 0x7f55f8144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f55f80c9700 0x7f55f814cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 --2- 172.21.15.146:0/2585032688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f55f80c6bb0 0x7f55f80c4f30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 >> 172.21.15.146:0/2585032688 conn(0x7f55f806f3c0 msgr2=0x7f55f8147dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 shutdown_connections 2024-09-16T01:53:36.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.270+0000 7f55ff000640 1 -- 172.21.15.146:0/2585032688 wait complete. 2024-09-16T01:53:36.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T01:53:36.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T01:53:36.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T01:53:36.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T01:53:36.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T01:53:36.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T01:53:36.289 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T01:53:36.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T01:53:36.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.569+0000 7fabd6400640 1 Processor -- start 2024-09-16T01:53:36.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.570+0000 7fabd6400640 1 -- start start 2024-09-16T01:53:36.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.570+0000 7f41e5800640 1 Processor -- start 2024-09-16T01:53:36.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.571+0000 7f41e5800640 1 -- start start 2024-09-16T01:53:36.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.572+0000 7fabd6400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01509a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.572+0000 7fabd6400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fabd00c70e0 0x7fabd0072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.572+0000 7fabd6400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fabd0072dd0 0x7fabd00727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.572+0000 7fabd6400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fabd0157580 con 0x7fabd00c70e0 2024-09-16T01:53:36.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.572+0000 7fabd6400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fabd0150ee0 con 0x7fabd0072dd0 2024-09-16T01:53:36.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.572+0000 7fabd6400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fabd0071ce0 con 0x7fabd006fee0 2024-09-16T01:53:36.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7f41e5800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e0154d70 0x7f41e015a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7f41e5800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f41e00c5910 0x7f41e00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7f41e5800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41e015b030 con 0x7f41e00c53c0 2024-09-16T01:53:36.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7f41e5800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f41e00cc3c0 con 0x7f41e00c5910 2024-09-16T01:53:36.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7f41e5800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f41e015b8d0 con 0x7f41e0154d70 2024-09-16T01:53:36.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7fabcfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fabd0072dd0 0x7fabd00727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7fabcfe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fabd0072dd0 0x7fabd00727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:59994/0 (socket says 172.21.15.146:59994) 2024-09-16T01:53:36.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7fabcfe00640 1 -- 172.21.15.146:0/4018711597 learned_addr learned my addr 172.21.15.146:0/4018711597 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7fabcf400640 1 --2- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.573+0000 7fabcea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fabd00c70e0 0x7fabd0072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7f41de000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f41e00c5910 0x7f41e00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7fabcfe00640 1 -- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 msgr2=0x7fabd01509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7fabcfe00640 1 --2- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7fabcfe00640 1 -- 172.21.15.146:0/4018711597 >> v2:172.21.15.146:3300/0 conn(0x7fabd00c70e0 msgr2=0x7fabd0072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7fabcfe00640 1 --2- 172.21.15.146:0/4018711597 >> v2:172.21.15.146:3300/0 conn(0x7fabd00c70e0 0x7fabd0072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7fabcfe00640 1 -- 172.21.15.146:0/4018711597 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd0156c50 con 0x7fabd0072dd0 2024-09-16T01:53:36.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7f41df400640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f41e00c53c0 0x7f41e00c6800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44612/0 (socket says 172.21.15.146:44612) 2024-09-16T01:53:36.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7f41df400640 1 -- 172.21.15.146:0/2855885510 learned_addr learned my addr 172.21.15.146:0/2855885510 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:36.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7f1f67000640 1 Processor -- start 2024-09-16T01:53:36.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.574+0000 7f1f67000640 1 -- start start 2024-09-16T01:53:36.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.575+0000 7f41de000640 1 -- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e0154d70 msgr2=0x7f41e015a910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:53:36.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.575+0000 7f41de000640 1 --2- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e0154d70 0x7f41e015a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.575+0000 7f41de000640 1 -- 172.21.15.146:0/2855885510 >> v1:172.21.15.161:6789/0 conn(0x7f41e00c53c0 legacy=0x7f41e00c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.575+0000 7f41de000640 1 -- 172.21.15.146:0/2855885510 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e015b310 con 0x7f41e00c5910 2024-09-16T01:53:36.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.575+0000 7f12ea400640 1 Processor -- start 2024-09-16T01:53:36.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.576+0000 7fabcfe00640 1 --2- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd0072dd0 0x7fabd00727d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fabd00698e0 tx=0x7fabb80093b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=178194779a6e6ace server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.576+0000 7f12ea400640 1 -- start start 2024-09-16T01:53:36.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.576+0000 7fabce000640 1 -- 172.21.15.146:0/4018711597 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8036070 con 0x7fabd0072dd0 2024-09-16T01:53:36.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.576+0000 7fabce000640 1 -- 172.21.15.146:0/4018711597 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb8009600 con 0x7fabd0072dd0 2024-09-16T01:53:36.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.576+0000 7fabce000640 1 -- 172.21.15.146:0/4018711597 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8009a30 con 0x7fabd0072dd0 2024-09-16T01:53:36.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f1f67000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1f60154d30 0x7f1f600c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f41de000640 1 --2- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e00c5910 0x7f41e00cb110 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097a0 tx=0x7f41cc02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39d8a9dc97c553a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f1f67000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f60155110 con 0x7f1f600c7a60 2024-09-16T01:53:36.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f1f67000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f60073140 con 0x7f1f600c7f30 2024-09-16T01:53:36.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f1f67000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1f600c6bb0 con 0x7f1f60154d30 2024-09-16T01:53:36.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7fe129800640 1 Processor -- start 2024-09-16T01:53:36.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f1f65000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1f600c7a60 0x7f1f6006dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47130/0 (socket says 172.21.15.146:47130) 2024-09-16T01:53:36.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7f1f65000640 1 -- 172.21.15.146:0/653522109 learned_addr learned my addr 172.21.15.146:0/653522109 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 -- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd0072dd0 msgr2=0x7fabd00727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 --2- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd0072dd0 0x7fabd00727d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fabd00698e0 tx=0x7fabb80093b0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.577+0000 7fe129800640 1 -- start start 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 -- 172.21.15.146:0/4018711597 shutdown_connections 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 --2- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd0072dd0 0x7fabd00727d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 --2- 172.21.15.146:0/4018711597 >> v2:172.21.15.146:3300/0 conn(0x7fabd00c70e0 0x7fabd0072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 --2- 172.21.15.146:0/4018711597 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7fabd6400640 1 -- 172.21.15.146:0/4018711597 >> 172.21.15.146:0/4018711597 conn(0x7fabd006f3c0 msgr2=0x7fabd006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f1f5ea00640 1 -- 172.21.15.146:0/653522109 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3017478708 0 0) 0x7f1f60073140 con 0x7f1f600c7f30 2024-09-16T01:53:36.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f1f5ea00640 1 -- 172.21.15.146:0/653522109 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f44003520 con 0x7f1f600c7f30 2024-09-16T01:53:36.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12ea400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12e40c70e0 0x7f12e406da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12ea400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12e406df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12ea400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e41569d0 con 0x7f12e406fee0 2024-09-16T01:53:36.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12ea400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f12e4157580 con 0x7f12e40c70e0 2024-09-16T01:53:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12ea400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f12e4071ce0 con 0x7f12e4150c10 2024-09-16T01:53:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fabd6400640 1 -- 172.21.15.146:0/4018711597 shutdown_connections 2024-09-16T01:53:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12e3400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f12e406fee0 0x7f12e4072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47146/0 (socket says 172.21.15.146:47146) 2024-09-16T01:53:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f12e3400640 1 -- 172.21.15.146:0/2465219962 learned_addr learned my addr 172.21.15.146:0/2465219962 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:36.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f1f5fe00640 1 --2- 172.21.15.146:0/653522109 >> v2:172.21.15.161:3300/0 conn(0x7f1f60154d30 0x7f1f600c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7f41dd600640 1 -- 172.21.15.146:0/2855885510 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc037070 con 0x7f41e00c5910 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7f41dd600640 1 -- 172.21.15.146:0/2855885510 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41cc032440 con 0x7f41e00c5910 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7f41dd600640 1 -- 172.21.15.146:0/2855885510 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc032810 con 0x7f41e00c5910 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fe129800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe124152a50 0x7fe1240c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fe129800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fe129800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1240ca5d0 con 0x7fe1240b0b90 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fe129800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fe1240c32c0 con 0x7fe124152a50 2024-09-16T01:53:36.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fe129800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe1240c35a0 con 0x7fe1240c9c20 2024-09-16T01:53:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686171576 0 0) 0x7f12d0001110 con 0x7f12e406fee0 2024-09-16T01:53:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12c8003520 con 0x7f12e406fee0 2024-09-16T01:53:36.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7fe128e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7fe128e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39270/0 (socket says 172.21.15.146:39270) 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7fe128e00640 1 -- 172.21.15.146:0/3741839691 learned_addr learned my addr 172.21.15.146:0/3741839691 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e3e00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12e406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1011494448 0 0) 0x7f12c8003520 con 0x7f12e406fee0 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e2a00640 1 --2- 172.21.15.146:0/2465219962 >> v2:172.21.15.146:3300/0 conn(0x7f12e40c70e0 0x7f12e406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 msgr2=0x7f12e406df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e2000640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12e406df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 >> v2:172.21.15.146:3300/0 conn(0x7f12e40c70e0 msgr2=0x7f12e406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7f12e2000640 1 --2- 172.21.15.146:0/2465219962 >> v2:172.21.15.146:3300/0 conn(0x7f12e40c70e0 0x7f12e406da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.580+0000 7fe123e00640 1 --2- 172.21.15.146:0/3741839691 >> v2:172.21.15.146:3300/0 conn(0x7fe124152a50 0x7fe1240c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7fe128e00640 1 -- 172.21.15.146:0/3741839691 >> v2:172.21.15.146:3300/0 conn(0x7fe124152a50 msgr2=0x7fe1240c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f1f5ea00640 1 -- 172.21.15.146:0/653522109 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3136435610 0 0) 0x7f1f44003520 con 0x7f1f600c7f30 2024-09-16T01:53:36.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f1f5fe00640 1 -- 172.21.15.146:0/653522109 >> v1:172.21.15.161:6789/0 conn(0x7f1f600c7f30 legacy=0x7f1f600c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 -- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e00c5910 msgr2=0x7f41e00cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 --2- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e00c5910 0x7f41e00cb110 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0097a0 tx=0x7f41cc02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 -- 172.21.15.146:0/2855885510 shutdown_connections 2024-09-16T01:53:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 --2- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e00c5910 0x7f41e00cb110 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 --2- 172.21.15.146:0/2855885510 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e0154d70 0x7f41e015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 -- 172.21.15.146:0/2855885510 >> 172.21.15.146:0/2855885510 conn(0x7f41e0067b30 msgr2=0x7f41e0067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e4156c50 con 0x7f12e406fee0 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 <== mon.0 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7f12d0001840 con 0x7f12e406fee0 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2a00640 1 --2- 172.21.15.146:0/2465219962 >> v2:172.21.15.146:3300/0 conn(0x7f12e40c70e0 0x7f12e406da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 >> v1:172.21.15.146:6789/0 conn(0x7f12e406fee0 legacy=0x7f12e4072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c80039b0 0x7f12c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f12e4157580 con 0x7f12c80039b0 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12e4071ce0 con 0x7f12c8004d40 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f12c800bb50 con 0x7f12e4150c10 2024-09-16T01:53:36.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e2a00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 -- 172.21.15.146:0/2855885510 shutdown_connections 2024-09-16T01:53:36.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f41e5800640 1 -- 172.21.15.146:0/2855885510 wait complete. 2024-09-16T01:53:36.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f12e3e00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3400640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c80039b0 0x7f12c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.578+0000 7f1f5fe00640 1 -- 172.21.15.146:0/653522109 >> v1:172.21.15.146:6789/0 conn(0x7f1f600c7a60 legacy=0x7f1f6006dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7f1f5fe00640 1 -- 172.21.15.146:0/653522109 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f600c6e00 con 0x7f1f60154d30 2024-09-16T01:53:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7fe128e00640 1 --2- 172.21.15.146:0/3741839691 >> v2:172.21.15.146:3300/0 conn(0x7fe124152a50 0x7fe1240c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7fe128e00640 1 -- 172.21.15.146:0/3741839691 >> v1:172.21.15.161:6789/0 conn(0x7fe1240b0b90 legacy=0x7fe12406f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7fe128e00640 1 -- 172.21.15.146:0/3741839691 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1240c3880 con 0x7fe1240c9c20 2024-09-16T01:53:36.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.579+0000 7fabd6400640 1 -- 172.21.15.146:0/4018711597 wait complete. 2024-09-16T01:53:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7fabd6400640 1 Processor -- start 2024-09-16T01:53:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.581+0000 7f09b4c00640 1 Processor -- start 2024-09-16T01:53:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f09b4c00640 1 -- start start 2024-09-16T01:53:36.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3e00640 1 -- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 msgr2=0x7f12c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3e00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f1f5fe00640 1 --2- 172.21.15.146:0/653522109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f60154d30 0x7f1f600c9570 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009720 tx=0x7f1f5002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c25088b16164689 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3e00640 1 -- 172.21.15.146:0/2465219962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c80039b0 msgr2=0x7f12c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3e00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c80039b0 0x7f12c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7fe123e00640 1 --2- 172.21.15.146:0/3741839691 >> v2:172.21.15.146:3300/0 conn(0x7fe124152a50 0x7fe1240c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3e00640 1 -- 172.21.15.146:0/2465219962 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e4156c50 con 0x7f12c8004d40 2024-09-16T01:53:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e2a00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e4150c10 0x7f12c8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7f12e3400640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c80039b0 0x7f12c8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.582+0000 7fabd6400640 1 -- start start 2024-09-16T01:53:36.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7f1f5ea00640 1 -- 172.21.15.146:0/653522109 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50037070 con 0x7f1f60154d30 2024-09-16T01:53:36.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7f1f5ea00640 1 -- 172.21.15.146:0/653522109 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f5002acf0 con 0x7f1f60154d30 2024-09-16T01:53:36.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7f1f5ea00640 1 -- 172.21.15.146:0/653522109 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c001670 con 0x7f1f60154d30 2024-09-16T01:53:36.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7fe128e00640 1 --2- 172.21.15.146:0/3741839691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240ca090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe114009870 tx=0x7fe11402a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1fc822b899d9d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7fe122a00640 1 -- 172.21.15.146:0/3741839691 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114037070 con 0x7fe1240c9c20 2024-09-16T01:53:36.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7fe122a00640 1 -- 172.21.15.146:0/3741839691 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe114032440 con 0x7fe1240c9c20 2024-09-16T01:53:36.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.583+0000 7fe122a00640 1 -- 172.21.15.146:0/3741839691 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114032850 con 0x7fe1240c9c20 2024-09-16T01:53:36.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f41e5800640 1 Processor -- start 2024-09-16T01:53:36.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 -- 172.21.15.146:0/653522109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f60154d30 msgr2=0x7f1f600c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 --2- 172.21.15.146:0/653522109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f60154d30 0x7f1f600c9570 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009720 tx=0x7f1f5002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 -- 172.21.15.146:0/653522109 shutdown_connections 2024-09-16T01:53:36.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 --2- 172.21.15.146:0/653522109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f60154d30 0x7f1f600c9570 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 -- 172.21.15.146:0/653522109 >> 172.21.15.146:0/653522109 conn(0x7f1f6006f3c0 msgr2=0x7f1f6006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 -- 172.21.15.146:0/653522109 shutdown_connections 2024-09-16T01:53:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f12e3e00640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12c80051c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00bd10 tx=0x7f12c8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=178e5ff4aa4275a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7fe129800640 1 -- 172.21.15.146:0/3741839691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 msgr2=0x7fe1240ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7fe129800640 1 --2- 172.21.15.146:0/3741839691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240ca090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe114009870 tx=0x7fe11402a2c0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f1f67000640 1 -- 172.21.15.146:0/653522109 wait complete. 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7fabd6400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01ece40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7fabd6400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd0072dd0 0x7fabd014c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7fabd6400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd00c70e0 0x7fabd0151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fabd6400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fabd00c9a90 con 0x7fabd0072dd0 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fabd6400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fabd0157210 con 0x7fabd00c70e0 2024-09-16T01:53:36.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fabd6400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fabd00718c0 con 0x7fabd006fee0 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc03b070 con 0x7f12c8004d40 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12dc036600 con 0x7f12c8004d40 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7f12e2000640 1 -- 172.21.15.146:0/2465219962 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00b8e0 con 0x7f12c8004d40 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fe129800640 1 -- 172.21.15.146:0/3741839691 shutdown_connections 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fe129800640 1 --2- 172.21.15.146:0/3741839691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240ca090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fe129800640 1 --2- 172.21.15.146:0/3741839691 >> v2:172.21.15.146:3300/0 conn(0x7fe124152a50 0x7fe1240c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fe129800640 1 -- 172.21.15.146:0/3741839691 >> 172.21.15.146:0/3741839691 conn(0x7fe1240738a0 msgr2=0x7fe124073c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7f09b4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 0x7f09a8005a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fe129800640 1 -- 172.21.15.146:0/3741839691 shutdown_connections 2024-09-16T01:53:36.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fabcf400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fabcf400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01ece40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39304/0 (socket says 172.21.15.146:39304) 2024-09-16T01:53:36.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fabcf400640 1 -- 172.21.15.146:0/3754752714 learned_addr learned my addr 172.21.15.146:0/3754752714 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7f09ade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 0x7f09a8005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.584+0000 7f41e5800640 1 -- start start 2024-09-16T01:53:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41e5800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e00c53c0 0x7f41e00ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41e5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e00c5910 0x7f41e00cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41e5800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e0154d70 0x7f41e014bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41e5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f41e00c6000 con 0x7f41e00c5910 2024-09-16T01:53:36.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f09ade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 0x7f09a8005a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39316/0 (socket says 172.21.15.146:39316) 2024-09-16T01:53:36.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f09ade00640 1 -- 172.21.15.146:0/1481127918 learned_addr learned my addr 172.21.15.146:0/1481127918 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7fabcf400640 1 -- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd00c70e0 msgr2=0x7fabd0151350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:53:36.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7fabcf400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabd00c70e0 0x7fabd0151350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41e5800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f41e00c6370 con 0x7f41e0154d70 2024-09-16T01:53:36.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41e5800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f41e00c6f10 con 0x7f41e00c53c0 2024-09-16T01:53:36.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7fabcf400640 1 -- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd0072dd0 msgr2=0x7fabd014c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7fabcf400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd0072dd0 0x7fabd014c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7fabcf400640 1 -- 172.21.15.146:0/3754752714 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd0158380 con 0x7fabd006fee0 2024-09-16T01:53:36.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f12ea400640 1 -- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 msgr2=0x7f12c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f12ea400640 1 --2- 172.21.15.146:0/2465219962 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12c80051c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00bd10 tx=0x7f12c8005830 comp rx=0 tx=0).stop 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09a80deaa0 con 0x7f09a804c0c0 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09a804ad10 con 0x7f09a800de10 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f09a8044c90 con 0x7f09a8045120 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7fabcf400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01ece40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fabc8011ae0 tx=0x7fabc800bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41dea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e00c53c0 0x7f41e00ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41dea00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e00c53c0 0x7f41e00ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39328/0 (socket says 172.21.15.146:39328) 2024-09-16T01:53:36.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41dea00640 1 -- 172.21.15.146:0/2962719061 learned_addr learned my addr 172.21.15.146:0/2962719061 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f41df400640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e0154d70 0x7f41e014bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f12e3e00640 1 -- 172.21.15.146:0/2465219962 reap_dead start 2024-09-16T01:53:36.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f12ea400640 1 -- 172.21.15.146:0/2465219962 shutdown_connections 2024-09-16T01:53:36.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.586+0000 7f12ea400640 1 -- 172.21.15.146:0/2465219962 >> 172.21.15.146:0/2465219962 conn(0x7f12e406f3c0 msgr2=0x7f12e406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7f12ea400640 1 -- 172.21.15.146:0/2465219962 shutdown_connections 2024-09-16T01:53:36.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.585+0000 7fe129800640 1 -- 172.21.15.146:0/3741839691 wait complete. 2024-09-16T01:53:36.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc801e070 con 0x7fabd006fee0 2024-09-16T01:53:36.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7f41de000640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e00c5910 0x7f41e00cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc800fb60 con 0x7fabd006fee0 2024-09-16T01:53:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabd0159680 con 0x7fabd006fee0 2024-09-16T01:53:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8019400 con 0x7fabd006fee0 2024-09-16T01:53:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f41dea00640 1 -- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e0154d70 msgr2=0x7f41e014bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f41dea00640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41e0154d70 0x7f41e014bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7f09aca00640 1 -- 172.21.15.146:0/1481127918 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3727339843 0 0) 0x7f09a804ad10 con 0x7f09a800de10 2024-09-16T01:53:36.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7f09aca00640 1 -- 172.21.15.146:0/1481127918 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0994003520 con 0x7f09a800de10 2024-09-16T01:53:36.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f09ade00640 1 -- 172.21.15.146:0/1481127918 >> v1:172.21.15.161:6789/0 conn(0x7f09a800de10 legacy=0x7f09a80ddc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f1f67000640 1 Processor -- start 2024-09-16T01:53:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f41dea00640 1 -- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e00c5910 msgr2=0x7f41e00cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f41dea00640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41e00c5910 0x7f41e00cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f1f67000640 1 -- start start 2024-09-16T01:53:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.589+0000 7f41dea00640 1 -- 172.21.15.146:0/2962719061 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e01e3c60 con 0x7f41e00c53c0 2024-09-16T01:53:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.589+0000 7fa340200640 1 Processor -- start 2024-09-16T01:53:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f09ade00640 1 -- 172.21.15.146:0/1481127918 >> v1:172.21.15.146:6789/0 conn(0x7f09a804c0c0 legacy=0x7f09a80de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.588+0000 7f09ade00640 1 -- 172.21.15.146:0/1481127918 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a804afd0 con 0x7f09a8045120 2024-09-16T01:53:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.589+0000 7f41dea00640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e00c53c0 0x7f41e00ca0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f41d800ba80 tx=0x7f41d800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.589+0000 7fa340200640 1 -- start start 2024-09-16T01:53:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f09ade00640 1 --2- 172.21.15.146:0/1481127918 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 0x7f09a8005a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009820 tx=0x7f09a002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5921cdf09028fb0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabd0158900 con 0x7fabd006fee0 2024-09-16T01:53:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.587+0000 7f12ea400640 1 -- 172.21.15.146:0/2465219962 wait complete. 2024-09-16T01:53:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f12ea400640 1 Processor -- start 2024-09-16T01:53:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7fe129800640 1 Processor -- start 2024-09-16T01:53:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f67000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f600c7f30 0x7f1f6014c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f67000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1f60154d30 0x7f1f6014cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f67000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f60156900 0x7f1f601f83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f67000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60072e00 con 0x7f1f60156900 2024-09-16T01:53:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f67000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60072920 con 0x7f1f600c7f30 2024-09-16T01:53:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f67000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6006e1f0 con 0x7f1f60154d30 2024-09-16T01:53:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8019070 con 0x7f41e00c53c0 2024-09-16T01:53:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7fe129800640 1 -- start start 2024-09-16T01:53:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d800cc90 con 0x7f41e00c53c0 2024-09-16T01:53:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8014680 con 0x7f41e00c53c0 2024-09-16T01:53:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f09aca00640 1 -- 172.21.15.146:0/1481127918 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0037070 con 0x7f09a8045120 2024-09-16T01:53:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f600c7f30 0x7f1f6014c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f600c7f30 0x7f1f6014c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60034/0 (socket says 172.21.15.146:60034) 2024-09-16T01:53:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.590+0000 7f1f5fe00640 1 -- 172.21.15.146:0/4245397584 learned_addr learned my addr 172.21.15.146:0/4245397584 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f1f65000640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f60156900 0x7f1f601f83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f1f5f400640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1f60154d30 0x7f1f6014cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f12ea400640 1 -- start start 2024-09-16T01:53:36.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f09aca00640 1 -- 172.21.15.146:0/1481127918 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a002ad40 con 0x7f09a8045120 2024-09-16T01:53:36.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faba8001030 con 0x7fabd006fee0 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f9d6f000640 1 Processor -- start 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f9d6f000640 1 -- start start 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7fe129800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1240b0b90 0x7fe1240cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7fe129800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7fe129800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe124152a50 0x7fe1240c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe129800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe1240c90f0 con 0x7fe124152a50 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe129800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe1240c9700 con 0x7fe1240b0b90 2024-09-16T01:53:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe129800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe12406fba0 con 0x7fe1240c9c20 2024-09-16T01:53:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7fa340200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa338154c60 0x7fa33815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa340200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 0x7fa3380cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa340200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33815b140 con 0x7fa3380cca50 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa340200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa33815b9e0 con 0x7fa338154c60 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa340200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa33815b420 con 0x7fa33815e9a0 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa33d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa338154c60 0x7fa33815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa33e200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa3380cca50 0x7fa3380c7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44638/0 (socket says 172.21.15.146:44638) 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa33e200640 1 -- 172.21.15.146:0/2326911581 learned_addr learned my addr 172.21.15.146:0/2326911581 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e01e3f20 con 0x7f41e00c53c0 2024-09-16T01:53:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e01e44e0 con 0x7f41e00c53c0 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f12ea400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12e40c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f12ea400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e41de9d0 0x7f12e414da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f12ea400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e414ea20 0x7f12e414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f12ea400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f12e40c9a90 con 0x7f12e414ea20 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fa33ce00640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 0x7fa3380cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f12ea400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f12e4157210 con 0x7f12c8004d40 2024-09-16T01:53:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f12ea400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f12e40718c0 con 0x7f12e41de9d0 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f1f5fe00640 1 -- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1f60154d30 msgr2=0x7f1f6014cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f1f5fe00640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1f60154d30 0x7f1f6014cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f1f5fe00640 1 -- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f60156900 msgr2=0x7f1f601f83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f1f5fe00640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f60156900 0x7f1f601f83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f1f5fe00640 1 -- 172.21.15.146:0/4245397584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f60158370 con 0x7f1f600c7f30 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe123e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1240b0b90 0x7fe1240cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe123400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe123400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240c6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39338/0 (socket says 172.21.15.146:39338) 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe123400640 1 -- 172.21.15.146:0/3338335471 learned_addr learned my addr 172.21.15.146:0/3338335471 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fe128e00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe124152a50 0x7fe1240c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fe123400640 1 -- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1240b0b90 msgr2=0x7fe1240cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fe123400640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe1240b0b90 0x7fe1240cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fe123400640 1 -- 172.21.15.146:0/3338335471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe124152a50 msgr2=0x7fe1240c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fe123400640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe124152a50 0x7fe1240c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fe123400640 1 -- 172.21.15.146:0/3338335471 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1240b13a0 con 0x7fe1240c9c20 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f9d6f000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9d68150c10 0x7f9d68072d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f9d6f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d680c9650 0x7f9d68071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.592+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fabc8021070 con 0x7fabd006fee0 2024-09-16T01:53:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fe123400640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240c6300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c410 tx=0x7fe11000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f9d6f000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d68073260 con 0x7f9d680718c0 2024-09-16T01:53:36.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f9d6f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9d681569d0 con 0x7f9d680c9650 2024-09-16T01:53:36.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41e015a910 con 0x7f41e00c53c0 2024-09-16T01:53:36.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7f9d6f000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9d6806dbd0 con 0x7f9d68150c10 2024-09-16T01:53:36.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f1f5f400640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1f60154d30 0x7f1f6014cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:36.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fabc7400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fabac06c3b0 0x7fabac06e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fabc808f1d0 con 0x7fabd006fee0 2024-09-16T01:53:36.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f12e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e414ea20 0x7f12e414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f12e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e414ea20 0x7f12e414eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47080/0 (socket says 172.21.15.146:47080) 2024-09-16T01:53:36.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f12e3e00640 1 -- 172.21.15.146:0/766606633 learned_addr learned my addr 172.21.15.146:0/766606633 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f12e2a00640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e41de9d0 0x7f12e414da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f1f5fe00640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f600c7f30 0x7f1f6014c690 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009840 tx=0x7f1f500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50037040 con 0x7f1f600c7f30 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f50004250 con 0x7f1f600c7f30 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50003610 con 0x7f1f600c7f30 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f1f67000640 1 -- 172.21.15.146:0/4245397584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6014d490 con 0x7f1f600c7f30 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f1f67000640 1 -- 172.21.15.146:0/4245397584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f60157dd0 con 0x7f1f600c7f30 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110017070 con 0x7fe1240c9c20 2024-09-16T01:53:36.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1100035f0 con 0x7fe1240c9c20 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1100039c0 con 0x7fe1240c9c20 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1240b1640 con 0x7fe1240c9c20 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.591+0000 7f09aca00640 1 -- 172.21.15.146:0/1481127918 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c001670 con 0x7f09a8045120 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 msgr2=0x7f09a8005a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f09b4c00640 1 --2- 172.21.15.146:0/1481127918 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 0x7f09a8005a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009820 tx=0x7f09a002a230 comp rx=0 tx=0).stop 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.593+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 87570598 0 0) 0x7fa33815b140 con 0x7fa3380cca50 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa320003520 con 0x7fa3380cca50 2024-09-16T01:53:36.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 851727381 0 0) 0x7fa320003520 con 0x7fa3380cca50 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 msgr2=0x7fa3380cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 0x7fa3380cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 >> v2:172.21.15.146:3300/0 conn(0x7fa338154c60 msgr2=0x7fa33815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 --2- 172.21.15.146:0/2326911581 >> v2:172.21.15.146:3300/0 conn(0x7fa338154c60 0x7fa33815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33815b700 con 0x7fa3380cca50 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3285484380 0 0) 0x7fa32c001670 con 0x7fa3380cca50 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 >> v1:172.21.15.161:6789/0 conn(0x7fa3380cca50 legacy=0x7fa3380c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3200039b0 0x7fa320003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 0x7fa320004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa320004d40 0x7fa3200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa33815b9e0 con 0x7fa320004d40 2024-09-16T01:53:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa33815b420 con 0x7fa3200039b0 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa32000b9c0 con 0x7fa33815e9a0 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f9d67e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9d68150c10 0x7f9d68072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f9d67e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9d68150c10 0x7f9d68072d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60046/0 (socket says 172.21.15.146:60046) 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.594+0000 7f9d67e00640 1 -- 172.21.15.146:0/2655195714 learned_addr learned my addr 172.21.15.146:0/2655195714 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f9d67e00640 1 -- 172.21.15.146:0/2655195714 >> v2:172.21.15.146:3300/0 conn(0x7f9d680c9650 msgr2=0x7f9d68071ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f9d67e00640 1 --2- 172.21.15.146:0/2655195714 >> v2:172.21.15.146:3300/0 conn(0x7f9d680c9650 0x7f9d68071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f9d67e00640 1 -- 172.21.15.146:0/2655195714 >> v1:172.21.15.161:6789/0 conn(0x7f9d680718c0 legacy=0x7f9d68159fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f9d67e00640 1 -- 172.21.15.146:0/2655195714 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6806deb0 con 0x7f9d68150c10 2024-09-16T01:53:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7fabc7400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fabac074790 0x7fabac076c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabc8012e50 con 0x7fabac074790 2024-09-16T01:53:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7fa33d800640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3200039b0 0x7fa320003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7fa33ce00640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 0x7fa320004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7f12e3400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12e40c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fa33d800640 1 -- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 msgr2=0x7fa320004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.595+0000 7fabcfe00640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fabac074790 0x7fabac076c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 shutdown_connections 2024-09-16T01:53:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f09b4c00640 1 --2- 172.21.15.146:0/1481127918 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a8045120 0x7f09a8005a30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 >> 172.21.15.146:0/1481127918 conn(0x7f09a804fc80 msgr2=0x7f09a8050090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f12e2a00640 1 -- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 msgr2=0x7f12e40c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f12e2a00640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12c8004d40 0x7f12e40c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f12e2a00640 1 -- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e414ea20 msgr2=0x7f12e414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f12e2a00640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e414ea20 0x7f12e414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f12e2a00640 1 -- 172.21.15.146:0/766606633 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e414fa20 con 0x7f12e41de9d0 2024-09-16T01:53:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1240cd9d0 con 0x7fe1240c9c20 2024-09-16T01:53:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 shutdown_connections 2024-09-16T01:53:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fa33d800640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33815e9a0 0x7fa320004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fa33d800640 1 -- 172.21.15.146:0/2326911581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa320004d40 msgr2=0x7fa3200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fa33d800640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa320004d40 0x7fa3200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fa33d800640 1 -- 172.21.15.146:0/2326911581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33815b700 con 0x7fa3200039b0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7fabc8096050 con 0x7fabd006fee0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7fabcea00640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fabac06c3b0 0x7fabac06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7fabcfe00640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fabac074790 0x7fabac076c30 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7f12e2a00640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e41de9d0 0x7f12e414da70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f12d400f570 tx=0x7f12d400fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4018020 con 0x7f12e41de9d0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d4004440 con 0x7f12e41de9d0 2024-09-16T01:53:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4003810 con 0x7f12e41de9d0 2024-09-16T01:53:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f1f67000640 1 -- 172.21.15.146:0/4245397584 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1f600718c0 con 0x7f1f600c7f30 2024-09-16T01:53:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe1240c5100 con 0x7fe1240c9c20 2024-09-16T01:53:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.596+0000 7f09b4c00640 1 -- 172.21.15.146:0/1481127918 wait complete. 2024-09-16T01:53:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.597+0000 7fa33d800640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3200039b0 0x7fa320003dd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa324009ac0 tx=0x7fa320005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52540ed0cfe29676 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.598+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e414fd00 con 0x7f12e41de9d0 2024-09-16T01:53:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.598+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e414d130 con 0x7f12e41de9d0 2024-09-16T01:53:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.598+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f41d8014820 con 0x7f41e00c53c0 2024-09-16T01:53:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7f09b4c00640 1 Processor -- start 2024-09-16T01:53:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe110003b60 con 0x7fe1240c9c20 2024-09-16T01:53:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fabc8012e50 con 0x7fabac074790 2024-09-16T01:53:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324036070 con 0x7fa3200039b0 2024-09-16T01:53:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa324031400 con 0x7fa3200039b0 2024-09-16T01:53:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7fa337e00640 1 -- 172.21.15.146:0/2326911581 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324009690 con 0x7fa3200039b0 2024-09-16T01:53:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7fabcea00640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fabac06c3b0 0x7fabac06e870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fabd0150df0 tx=0x7fabc0009500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f12e414f7d0 con 0x7f12e41de9d0 2024-09-16T01:53:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7f9d67e00640 1 --2- 172.21.15.146:0/2655195714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 0x7f9d68072d20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9d680cd270 tx=0x7f9d5802c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96747b7a98ddc6c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7fe11be00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe10406c2b0 0x7fe10406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fe11008a800 con 0x7fe1240c9c20 2024-09-16T01:53:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.599+0000 7f09b4c00640 1 -- start start 2024-09-16T01:53:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7f09b4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a800de10 0x7f09a80cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7f09b4c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a8045120 0x7f09a80d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09b4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a804c0c0 0x7f09a80c3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09b4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f09a80467a0 con 0x7f09a804c0c0 2024-09-16T01:53:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09b4c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09a80464c0 con 0x7f09a8045120 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09b4c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f09a8005f70 con 0x7f09a800de10 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09ade00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a800de10 0x7f09a80cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09ad400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a8045120 0x7f09a80d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09ad400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a8045120 0x7f09a80d79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60074/0 (socket says 172.21.15.146:60074) 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09ad400640 1 -- 172.21.15.146:0/3061523101 learned_addr learned my addr 172.21.15.146:0/3061523101 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f09ae800640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a804c0c0 0x7f09a80c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f9d66a00640 1 -- 172.21.15.146:0/2655195714 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58035070 con 0x7f9d68150c10 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1f500037b0 con 0x7f1f600c7f30 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7f41d6a00640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41c006c3b0 0x7f41c006e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.600+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f41d808b890 con 0x7f41e00c53c0 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f41de000640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41c006c3b0 0x7f41c006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f41d6a00640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f41c0074810 0x7f41c0076cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41d8021ec0 con 0x7f41c0074810 2024-09-16T01:53:36.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f41d808bc80 con 0x7f41e00c53c0 2024-09-16T01:53:36.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f9d66a00640 1 -- 172.21.15.146:0/2655195714 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d58031610 con 0x7f9d68150c10 2024-09-16T01:53:36.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f9d66a00640 1 -- 172.21.15.146:0/2655195714 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d580319e0 con 0x7f9d68150c10 2024-09-16T01:53:36.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7fa340200640 1 -- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3200039b0 msgr2=0x7fa320003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7fa340200640 1 --2- 172.21.15.146:0/2326911581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3200039b0 0x7fa320003dd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa324009ac0 tx=0x7fa320005830 comp rx=0 tx=0).stop 2024-09-16T01:53:36.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.601+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f12d401b020 con 0x7f12e41de9d0 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f1f57e00640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1f4006c300 0x7f1f4006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fe123e00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe10406c2b0 0x7fe10406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fe11be00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe104074710 0x7fe104076bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe110009e50 con 0x7fe104074710 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fa33e200640 1 -- 172.21.15.146:0/2326911581 reap_dead start 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fa340200640 1 -- 172.21.15.146:0/2326911581 shutdown_connections 2024-09-16T01:53:36.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fa340200640 1 -- 172.21.15.146:0/2326911581 >> 172.21.15.146:0/2326911581 conn(0x7fa33806bc10 msgr2=0x7fa33806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7fe110091050 con 0x7fe1240c9c20 2024-09-16T01:53:36.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f1f500abea0 con 0x7f1f600c7f30 2024-09-16T01:53:36.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f1f57e00640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1f40074700 0x7f1f40076ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f41df400640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f41c0074810 0x7f41c0076cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fa340200640 1 -- 172.21.15.146:0/2326911581 shutdown_connections 2024-09-16T01:53:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f9d6f000640 1 -- 172.21.15.146:0/2655195714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 msgr2=0x7f9d68072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f9d6f000640 1 --2- 172.21.15.146:0/2655195714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 0x7f9d68072d20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9d680cd270 tx=0x7f9d5802c9c0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f12db400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f12c406c300 0x7f12c406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f12d408b5e0 con 0x7f12e41de9d0 2024-09-16T01:53:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f50034ec0 con 0x7f1f40074700 2024-09-16T01:53:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f1f500ae4c0 con 0x7f1f600c7f30 2024-09-16T01:53:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f9d6f000640 1 -- 172.21.15.146:0/2655195714 shutdown_connections 2024-09-16T01:53:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7f1f5f400640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1f4006c300 0x7f1f4006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f1f65000640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1f40074700 0x7f1f40076ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f12db400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f12c4074760 0x7f12c4076c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12c40772d0 con 0x7f12c4074760 2024-09-16T01:53:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f12d4092050 con 0x7f12e41de9d0 2024-09-16T01:53:36.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f9d6f000640 1 --2- 172.21.15.146:0/2655195714 >> v2:172.21.15.146:3300/0 conn(0x7f9d680c9650 0x7f9d68071ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f9d6f000640 1 --2- 172.21.15.146:0/2655195714 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 0x7f9d68072d20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.602+0000 7fe128e00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe104074710 0x7fe104076bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7fe123e00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe10406c2b0 0x7fe10406e770 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe11c008ee0 tx=0x7fe11c008430 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f12e3400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f12c406c300 0x7f12c406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7fe128e00640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe104074710 0x7fe104076bb0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f9d6f000640 1 -- 172.21.15.146:0/2655195714 >> 172.21.15.146:0/2655195714 conn(0x7f9d6806f3c0 msgr2=0x7f9d6806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f9d6f000640 1 -- 172.21.15.146:0/2655195714 shutdown_connections 2024-09-16T01:53:36.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f41de000640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41c006c3b0 0x7f41c006e870 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f41cc034000 tx=0x7f41cc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f9d6f000640 1 -- 172.21.15.146:0/2655195714 wait complete. 2024-09-16T01:53:36.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f09ade00640 1 -- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a8045120 msgr2=0x7f09a80d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f09ade00640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a8045120 0x7f09a80d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f09ade00640 1 -- 172.21.15.146:0/3061523101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a804c0c0 msgr2=0x7f09a80c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f09ade00640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a804c0c0 0x7f09a80c3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f09ade00640 1 -- 172.21.15.146:0/3061523101 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a80d61b0 con 0x7f09a800de10 2024-09-16T01:53:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7f09ae800640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a804c0c0 0x7f09a80c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f09ade00640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a800de10 0x7f09a80cb060 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f09a0031000 tx=0x7f09a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.603+0000 7fa340200640 1 -- 172.21.15.146:0/2326911581 wait complete. 2024-09-16T01:53:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f12e3e00640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f12c4074760 0x7f12c4076c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f1f5f400640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1f4006c300 0x7f1f4006e7c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c001eb0 tx=0x7f1f4c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe110009e50 con 0x7fe104074710 2024-09-16T01:53:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0037040 con 0x7f09a800de10 2024-09-16T01:53:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a0004250 con 0x7f09a800de10 2024-09-16T01:53:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.604+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0003610 con 0x7f09a800de10 2024-09-16T01:53:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.605+0000 7f1f65000640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1f40074700 0x7f1f40076ba0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.605+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a80d5380 con 0x7f09a800de10 2024-09-16T01:53:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.605+0000 7f12e3400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f12c406c300 0x7f12c406e7c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f12d0001050 tx=0x7f12d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.605+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a802fec0 con 0x7f09a800de10 2024-09-16T01:53:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.605+0000 7f12e3e00640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f12c4074760 0x7f12c4076c00 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.606+0000 7fa340200640 1 Processor -- start 2024-09-16T01:53:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.606+0000 7f9d6f000640 1 Processor -- start 2024-09-16T01:53:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.606+0000 7fa340200640 1 -- start start 2024-09-16T01:53:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0988001030 con 0x7f09a800de10 2024-09-16T01:53:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.606+0000 7f41df400640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f41c0074810 0x7f41c0076cb0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f50034ec0 con 0x7f1f40074700 2024-09-16T01:53:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f12c40772d0 con 0x7f12c4074760 2024-09-16T01:53:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.606+0000 7f9d6f000640 1 -- start start 2024-09-16T01:53:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7fa340200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3381524f0 0x7fa33814dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7fa340200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3381528d0 0x7fa33814ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7fa340200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33814af60 0x7fa33814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7fa340200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa3380c8b10 con 0x7fa3381528d0 2024-09-16T01:53:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7fa340200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3380c7790 con 0x7fa3381524f0 2024-09-16T01:53:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.607+0000 7fa340200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa33815eda0 con 0x7fa33814af60 2024-09-16T01:53:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7f9d6f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d680718c0 0x7f9d681e44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7f9d6f000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680c9650 0x7f9d6814c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7fa33ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3381528d0 0x7fa33814ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7fa33d800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3381524f0 0x7fa33814dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7fa33d800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3381524f0 0x7fa33814dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60086/0 (socket says 172.21.15.146:60086) 2024-09-16T01:53:36.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7fa33d800640 1 -- 172.21.15.146:0/2941297789 learned_addr learned my addr 172.21.15.146:0/2941297789 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7fa33e200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33814af60 0x7fa33814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41d8021ec0 con 0x7f41c0074810 2024-09-16T01:53:36.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.608+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f09a003a040 con 0x7f09a800de10 2024-09-16T01:53:36.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d6f000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 0x7f9d681e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d6f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6806e750 con 0x7f9d680718c0 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d6f000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6806e470 con 0x7f9d68150c10 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d6f000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9d68072a10 con 0x7f9d680c9650 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7fa33e200640 1 -- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3381524f0 msgr2=0x7fa33814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7fa33e200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3381524f0 0x7fa33814dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7fa33e200640 1 -- 172.21.15.146:0/2941297789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3381528d0 msgr2=0x7fa33814ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7fa33e200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3381528d0 0x7fa33814ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7fa33e200640 1 -- 172.21.15.146:0/2941297789 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa338159000 con 0x7fa33814af60 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7f09a6000640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099006c3b0 0x7f099006e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f09a00ab820 con 0x7f09a800de10 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f09a6000640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f09900747b0 0x7f0990076c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09a003dea0 con 0x7f09900747b0 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f09a00b02a0 con 0x7f09a800de10 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f09ae800640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f09900747b0 0x7f0990076c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7fa33e200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33814af60 0x7fa33814b410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa338071460 tx=0x7fa32c00cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c004cb0 con 0x7fa33814af60 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa32c004e50 con 0x7fa33814af60 2024-09-16T01:53:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0072b0 con 0x7fa33814af60 2024-09-16T01:53:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.612+0000 7f09ae800640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f09900747b0 0x7f0990076c50 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.613+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09a003dea0 con 0x7f09900747b0 2024-09-16T01:53:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d67400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680c9650 0x7f9d6814c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d67400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680c9650 0x7f9d6814c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39404/0 (socket says 172.21.15.146:39404) 2024-09-16T01:53:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d67400640 1 -- 172.21.15.146:0/3652924095 learned_addr learned my addr 172.21.15.146:0/3652924095 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.613+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3381f4b00 con 0x7fa33814af60 2024-09-16T01:53:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.613+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3381592a0 con 0x7fa33814af60 2024-09-16T01:53:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.609+0000 7f9d67e00640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d680718c0 0x7f9d681e44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7f9d67e00640 1 -- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680c9650 msgr2=0x7f9d6814c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7f9d67e00640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d680c9650 0x7f9d6814c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.613+0000 7f09ad400640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099006c3b0 0x7f099006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7f9d67e00640 1 -- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 msgr2=0x7f9d681e3300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7f9d67e00640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d68150c10 0x7f9d681e3300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.610+0000 7f9d67e00640 1 -- 172.21.15.146:0/3652924095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d68151ec0 con 0x7f9d680718c0 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f9d67e00640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d680718c0 0x7f9d681e44c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9d58034000 tx=0x7f9d580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58035040 con 0x7f9d680718c0 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d58030b50 con 0x7f9d680718c0 2024-09-16T01:53:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.611+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58045480 con 0x7f9d680718c0 2024-09-16T01:53:36.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.612+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d68151be0 con 0x7f9d680718c0 2024-09-16T01:53:36.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.612+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6814db80 con 0x7f9d680718c0 2024-09-16T01:53:36.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.615+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d40001030 con 0x7f9d680718c0 2024-09-16T01:53:36.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.615+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa338155040 con 0x7fa33814af60 2024-09-16T01:53:36.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.616+0000 7f09ad400640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099006c3b0 0x7f099006e870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f09a80cd920 tx=0x7f099c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.617+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9d58038070 con 0x7f9d680718c0 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.619+0000 7f9d5fe00640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d4406c380 0x7f9d4406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.619+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f9d580ac3c0 con 0x7f9d680718c0 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.619+0000 7f9d5fe00640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9d44074760 0x7f9d44076c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.620+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d440772d0 con 0x7f9d44074760 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.620+0000 7f9d67400640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d4406c380 0x7f9d4406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.620+0000 7f9d6d000640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9d44074760 0x7f9d44076c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.621+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7f9d580b3050 con 0x7f9d680718c0 2024-09-16T01:53:36.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.621+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa32c007450 con 0x7fa33814af60 2024-09-16T01:53:36.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.622+0000 7f9d6d000640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9d44074760 0x7f9d44076c00 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.622+0000 7fa335600640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa31806c3b0 0x7fa31806e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.622+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fa32c08a820 con 0x7fa33814af60 2024-09-16T01:53:36.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.622+0000 7f9d67400640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d4406c380 0x7f9d4406e840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9d68154270 tx=0x7f9d50008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.623+0000 7fa335600640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa318074810 0x7fa318076cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:36.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.623+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa32c006e50 con 0x7fa318074810 2024-09-16T01:53:36.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.623+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=152) ==== 24+0+0 (secure 0 0 0) 0x7fa32c08f2a0 con 0x7fa33814af60 2024-09-16T01:53:36.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.623+0000 7fa33d800640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa31806c3b0 0x7fa31806e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.623+0000 7fa33ce00640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa318074810 0x7fa318076cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:36.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.623+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d440772d0 con 0x7f9d44074760 2024-09-16T01:53:36.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.624+0000 7fa33ce00640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa318074810 0x7fa318076cb0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.624+0000 7fa33d800640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa31806c3b0 0x7fa31806e870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa3240098e0 tx=0x7fa324008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:36.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.625+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa32c006e50 con 0x7fa318074810 2024-09-16T01:53:36.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.628+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe12406f8a0 con 0x7fe104074710 2024-09-16T01:53:36.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.628+0000 7fe11be00640 1 -- 172.21.15.146:0/3338335471 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe12406f8a0 con 0x7fe104074710 2024-09-16T01:53:36.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.631+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faba8002dd0 con 0x7fabac074790 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.631+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe104074710 msgr2=0x7fe104076bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.631+0000 7fe129800640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fe104074710 0x7fe104076bb0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe10406c2b0 msgr2=0x7fe10406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe129800640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe10406c2b0 0x7fe10406e770 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe11c008ee0 tx=0x7fe11c008430 comp rx=0 tx=0).stop 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 msgr2=0x7fe1240c6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe129800640 1 --2- 172.21.15.146:0/3338335471 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe1240c9c20 0x7fe1240c6300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c410 tx=0x7fe11000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe128e00640 1 -- 172.21.15.146:0/3338335471 reap_dead start 2024-09-16T01:53:36.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fabc7400640 1 -- 172.21.15.146:0/3754752714 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faba8002dd0 con 0x7fabac074790 2024-09-16T01:53:36.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 shutdown_connections 2024-09-16T01:53:36.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.632+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 >> 172.21.15.146:0/3338335471 conn(0x7fe1240738a0 msgr2=0x7fe12414d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.633+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 shutdown_connections 2024-09-16T01:53:36.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.633+0000 7fe129800640 1 -- 172.21.15.146:0/3338335471 wait complete. 2024-09-16T01:53:36.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.633+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fabac074790 msgr2=0x7fabac076c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.633+0000 7fabd6400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7fabac074790 0x7fabac076c30 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fabac06c3b0 msgr2=0x7fabac06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabd6400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fabac06c3b0 0x7fabac06e870 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fabd0150df0 tx=0x7fabc0009500 comp rx=0 tx=0).stop 2024-09-16T01:53:36.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 msgr2=0x7fabd01ece40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabd6400640 1 --2- 172.21.15.146:0/3754752714 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabd006fee0 0x7fabd01ece40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fabc8011ae0 tx=0x7fabc800bf30 comp rx=0 tx=0).stop 2024-09-16T01:53:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabcfe00640 1 -- 172.21.15.146:0/3754752714 reap_dead start 2024-09-16T01:53:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 shutdown_connections 2024-09-16T01:53:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 >> 172.21.15.146:0/3754752714 conn(0x7fabd006f3c0 msgr2=0x7fabd01562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.634+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f12e4072dd0 con 0x7f12c4074760 2024-09-16T01:53:36.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.635+0000 7f12db400640 1 -- 172.21.15.146:0/766606633 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f12e4072dd0 con 0x7f12c4074760 2024-09-16T01:53:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f12c4074760 msgr2=0x7f12c4076c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12ea400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f12c4074760 0x7f12c4076c00 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f12c406c300 msgr2=0x7f12c406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12ea400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f12c406c300 0x7f12c406e7c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f12d0001050 tx=0x7f12d0009040 comp rx=0 tx=0).stop 2024-09-16T01:53:36.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e41de9d0 msgr2=0x7f12e414da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12ea400640 1 --2- 172.21.15.146:0/766606633 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f12e41de9d0 0x7f12e414da70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f12d400f570 tx=0x7f12d400fa30 comp rx=0 tx=0).stop 2024-09-16T01:53:36.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.636+0000 7f12e3e00640 1 -- 172.21.15.146:0/766606633 reap_dead start 2024-09-16T01:53:36.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.637+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 shutdown_connections 2024-09-16T01:53:36.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.637+0000 7fabd6400640 1 -- 172.21.15.146:0/3754752714 wait complete. 2024-09-16T01:53:36.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.637+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 shutdown_connections 2024-09-16T01:53:36.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.637+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 >> 172.21.15.146:0/766606633 conn(0x7f12e406f3c0 msgr2=0x7f12e4155fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.637+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 shutdown_connections 2024-09-16T01:53:36.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.637+0000 7f12ea400640 1 -- 172.21.15.146:0/766606633 wait complete. 2024-09-16T01:53:36.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.642+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41e01e4d80 con 0x7f41c0074810 2024-09-16T01:53:36.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.642+0000 7f1f67000640 1 -- 172.21.15.146:0/4245397584 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1f600c9b60 con 0x7f1f40074700 2024-09-16T01:53:36.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.643+0000 7f41d6a00640 1 -- 172.21.15.146:0/2962719061 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41e01e4d80 con 0x7f41c0074810 2024-09-16T01:53:36.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.643+0000 7f1f57e00640 1 -- 172.21.15.146:0/4245397584 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1f600c9b60 con 0x7f1f40074700 2024-09-16T01:53:36.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.644+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f41c0074810 msgr2=0x7f41c0076cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.644+0000 7f41e5800640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f41c0074810 0x7f41c0076cb0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.644+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1f40074700 msgr2=0x7f1f40076ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.644+0000 7f1f55600640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7f1f40074700 0x7f1f40076ba0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0988002dd0 con 0x7f09900747b0 2024-09-16T01:53:36.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41c006c3b0 msgr2=0x7f41c006e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f41e5800640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f41c006c3b0 0x7f41c006e870 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f41cc034000 tx=0x7f41cc02e000 comp rx=0 tx=0).stop 2024-09-16T01:53:36.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e00c53c0 msgr2=0x7f41e00ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f41e5800640 1 --2- 172.21.15.146:0/2962719061 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41e00c53c0 0x7f41e00ca0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f41d800ba80 tx=0x7f41d800bf40 comp rx=0 tx=0).stop 2024-09-16T01:53:36.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1f4006c300 msgr2=0x7f1f4006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f1f55600640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1f4006c300 0x7f1f4006e7c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c001eb0 tx=0x7f1f4c009040 comp rx=0 tx=0).stop 2024-09-16T01:53:36.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f41df400640 1 -- 172.21.15.146:0/2962719061 reap_dead start 2024-09-16T01:53:36.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f600c7f30 msgr2=0x7f1f6014c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f1f55600640 1 --2- 172.21.15.146:0/4245397584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f600c7f30 0x7f1f6014c690 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009840 tx=0x7f1f500046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.645+0000 7f1f65000640 1 -- 172.21.15.146:0/4245397584 reap_dead start 2024-09-16T01:53:36.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 shutdown_connections 2024-09-16T01:53:36.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 >> 172.21.15.146:0/4245397584 conn(0x7f1f6006f3c0 msgr2=0x7f1f6006e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f09a6000640 1 -- 172.21.15.146:0/3061523101 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0988002dd0 con 0x7f09900747b0 2024-09-16T01:53:36.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 shutdown_connections 2024-09-16T01:53:36.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f1f55600640 1 -- 172.21.15.146:0/4245397584 wait complete. 2024-09-16T01:53:36.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f09900747b0 msgr2=0x7f0990076c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 shutdown_connections 2024-09-16T01:53:36.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 >> 172.21.15.146:0/2962719061 conn(0x7f41e0067b30 msgr2=0x7f41e00d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.646+0000 7f09b4c00640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f09900747b0 0x7f0990076c50 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.647+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 shutdown_connections 2024-09-16T01:53:36.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.647+0000 7f41e5800640 1 -- 172.21.15.146:0/2962719061 wait complete. 2024-09-16T01:53:36.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.647+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099006c3b0 msgr2=0x7f099006e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.647+0000 7f09b4c00640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f099006c3b0 0x7f099006e870 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f09a80cd920 tx=0x7f099c009040 comp rx=0 tx=0).stop 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.647+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a800de10 msgr2=0x7f09a80cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.647+0000 7f09b4c00640 1 --2- 172.21.15.146:0/3061523101 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09a800de10 0x7f09a80cb060 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f09a0031000 tx=0x7f09a00046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.648+0000 7f09ae800640 1 -- 172.21.15.146:0/3061523101 reap_dead start 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.648+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 shutdown_connections 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.648+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 >> 172.21.15.146:0/3061523101 conn(0x7f09a804fc80 msgr2=0x7f09a8035160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.648+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 shutdown_connections 2024-09-16T01:53:36.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.648+0000 7f09b4c00640 1 -- 172.21.15.146:0/3061523101 wait complete. 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.652+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d40002dd0 con 0x7f9d44074760 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.655+0000 7f9d5fe00640 1 -- 172.21.15.146:0/3652924095 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9d40002dd0 con 0x7f9d44074760 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.655+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9d44074760 msgr2=0x7f9d44076c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.655+0000 7f9d6f000640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7f9d44074760 0x7f9d44076c00 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.656+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d4406c380 msgr2=0x7f9d4406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.656+0000 7f9d6f000640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d4406c380 0x7f9d4406e840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9d68154270 tx=0x7f9d50008040 comp rx=0 tx=0).stop 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.656+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d680718c0 msgr2=0x7f9d681e44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.656+0000 7f9d6f000640 1 --2- 172.21.15.146:0/3652924095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d680718c0 0x7f9d681e44c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9d58034000 tx=0x7f9d580046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.656+0000 7f9d6d000640 1 -- 172.21.15.146:0/3652924095 reap_dead start 2024-09-16T01:53:36.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.657+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 shutdown_connections 2024-09-16T01:53:36.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.657+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 >> 172.21.15.146:0/3652924095 conn(0x7f9d6806f3c0 msgr2=0x7f9d681510b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.657+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 shutdown_connections 2024-09-16T01:53:36.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.657+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3380c3bb0 con 0x7fa318074810 2024-09-16T01:53:36.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.657+0000 7f9d6f000640 1 -- 172.21.15.146:0/3652924095 wait complete. 2024-09-16T01:53:36.690 INFO:teuthology.orchestra.run.smithi146.stdout:55834574910 2024-09-16T01:53:36.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T01:53:36.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.658+0000 7fa335600640 1 -- 172.21.15.146:0/2941297789 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa3380c3bb0 con 0x7fa318074810 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa318074810 msgr2=0x7fa318076cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa340200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fa318074810 0x7fa318076cb0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa31806c3b0 msgr2=0x7fa31806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa340200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa31806c3b0 0x7fa31806e870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa3240098e0 tx=0x7fa324008030 comp rx=0 tx=0).stop 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33814af60 msgr2=0x7fa33814b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa340200640 1 --2- 172.21.15.146:0/2941297789 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa33814af60 0x7fa33814b410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa338071460 tx=0x7fa32c00cab0 comp rx=0 tx=0).stop 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.659+0000 7fa33e200640 1 -- 172.21.15.146:0/2941297789 reap_dead start 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.662+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 shutdown_connections 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.662+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 >> 172.21.15.146:0/2941297789 conn(0x7fa33806bc10 msgr2=0x7fa33806f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.662+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 shutdown_connections 2024-09-16T01:53:36.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:36.662+0000 7fa340200640 1 -- 172.21.15.146:0/2941297789 wait complete. 2024-09-16T01:53:36.693 INFO:teuthology.orchestra.run.smithi146.stdout:64424509502 2024-09-16T01:53:36.693 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T01:53:36.694 INFO:teuthology.orchestra.run.smithi146.stdout:64424509503 2024-09-16T01:53:36.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T01:53:36.695 INFO:teuthology.orchestra.run.smithi146.stdout:64424509502 2024-09-16T01:53:36.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T01:53:36.696 INFO:teuthology.orchestra.run.smithi146.stdout:64424509503 2024-09-16T01:53:36.696 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T01:53:36.697 INFO:teuthology.orchestra.run.smithi146.stdout:60129542206 2024-09-16T01:53:36.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T01:53:36.698 INFO:teuthology.orchestra.run.smithi146.stdout:60129542206 2024-09-16T01:53:36.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T01:53:36.699 INFO:teuthology.orchestra.run.smithi146.stdout:60129542206 2024-09-16T01:53:36.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T01:53:37.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.000+0000 7f4032200640 1 Processor -- start 2024-09-16T01:53:37.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.001+0000 7fc5ba400640 1 Processor -- start 2024-09-16T01:53:37.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.001+0000 7fc5ba400640 1 -- start start 2024-09-16T01:53:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.000+0000 7f4032200640 1 -- start start 2024-09-16T01:53:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.002+0000 7f4032200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f402c0c53d0 0x7f402c166190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.002+0000 7f4032200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402c0cc600 con 0x7f402c154af0 2024-09-16T01:53:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.002+0000 7f4032200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402c0d02f0 con 0x7f402c0b3030 2024-09-16T01:53:37.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.002+0000 7f4032200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f402c0d0c20 con 0x7f402c0c53d0 2024-09-16T01:53:37.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.003+0000 7f402be00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f402c0b3030 0x7f402c06c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44660/0 (socket says 172.21.15.146:44660) 2024-09-16T01:53:37.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.003+0000 7f4031800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f402c0c53d0 0x7f402c166190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.003+0000 7f4031800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f402c0c53d0 0x7f402c166190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60092/0 (socket says 172.21.15.146:60092) 2024-09-16T01:53:37.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.003+0000 7f4031800640 1 -- 172.21.15.146:0/911452622 learned_addr learned my addr 172.21.15.146:0/911452622 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fa537e00640 1 Processor -- start 2024-09-16T01:53:37.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7f4031800640 1 -- 172.21.15.146:0/911452622 >> v1:172.21.15.161:6789/0 conn(0x7f402c0b3030 legacy=0x7f402c06c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.005+0000 7f4031800640 1 -- 172.21.15.146:0/911452622 >> v1:172.21.15.146:6789/0 conn(0x7f402c154af0 legacy=0x7f402c15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fc5ba400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc0c0 0x7fc5b415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fc5ba400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b415b1d0 con 0x7fc5b40cb2f0 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fc5ba400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b415b4b0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fc5ba400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b40cbd00 con 0x7fc5b40cc0c0 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fc5b3e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc5b40cc4a0 0x7fc5b4165430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44666/0 (socket says 172.21.15.146:44666) 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.004+0000 7fc5b3e00640 1 -- 172.21.15.146:0/2714094561 learned_addr learned my addr 172.21.15.146:0/2714094561 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.005+0000 7fc5b3400640 1 --2- 172.21.15.146:0/2714094561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc0c0 0x7fc5b415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.005+0000 7fa537e00640 1 -- start start 2024-09-16T01:53:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.005+0000 7f4031800640 1 -- 172.21.15.146:0/911452622 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c0d05d0 con 0x7f402c0c53d0 2024-09-16T01:53:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.006+0000 7fc5b2000640 1 -- 172.21.15.146:0/2714094561 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249945721 0 0) 0x7fc5b415b1d0 con 0x7fc5b40cb2f0 2024-09-16T01:53:37.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.006+0000 7fc5b2000640 1 -- 172.21.15.146:0/2714094561 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc598003520 con 0x7fc5b40cb2f0 2024-09-16T01:53:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa537e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa538158ea0 0x7fa5380c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7f4031800640 1 --2- 172.21.15.146:0/911452622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c166190 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f402c0d0da0 tx=0x7f402402a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d452f063d1001330 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa537e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa538073e90 0x7fa5380c15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa537e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa53806c890 con 0x7fa5380cc8b0 2024-09-16T01:53:37.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa537e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa53806cb70 con 0x7fa538073e90 2024-09-16T01:53:37.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa537e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa5380c70f0 con 0x7fa538158ea0 2024-09-16T01:53:37.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa536a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa538158ea0 0x7fa5380c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7f402b400640 1 -- 172.21.15.146:0/911452622 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024036070 con 0x7f402c0c53d0 2024-09-16T01:53:37.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fc5b3400640 1 -- 172.21.15.146:0/2714094561 >> v1:172.21.15.161:6789/0 conn(0x7fc5b40cc4a0 legacy=0x7fc5b4165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fc5b3400640 1 -- 172.21.15.146:0/2714094561 >> v1:172.21.15.146:6789/0 conn(0x7fc5b40cb2f0 legacy=0x7fc5b415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fc5b3400640 1 -- 172.21.15.146:0/2714094561 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b415b790 con 0x7fc5b40cc0c0 2024-09-16T01:53:37.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7f402b400640 1 -- 172.21.15.146:0/911452622 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4024031600 con 0x7f402c0c53d0 2024-09-16T01:53:37.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7f402b400640 1 -- 172.21.15.146:0/911452622 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40240319d0 con 0x7f402c0c53d0 2024-09-16T01:53:37.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa536a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa538158ea0 0x7fa5380c10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60100/0 (socket says 172.21.15.146:60100) 2024-09-16T01:53:37.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa536000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa5380cc8b0 0x7fa53815ebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44668/0 (socket says 172.21.15.146:44668) 2024-09-16T01:53:37.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa536000640 1 -- 172.21.15.146:0/1512182867 learned_addr learned my addr 172.21.15.146:0/1512182867 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:37.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.008+0000 7f4032200640 1 -- 172.21.15.146:0/911452622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 msgr2=0x7f402c166190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.008+0000 7f4032200640 1 --2- 172.21.15.146:0/911452622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c166190 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f402c0d0da0 tx=0x7f402402a6c0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.007+0000 7fa537400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa538073e90 0x7fa5380c15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.009+0000 7f4032200640 1 -- 172.21.15.146:0/911452622 shutdown_connections 2024-09-16T01:53:37.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.008+0000 7fa536a00640 1 -- 172.21.15.146:0/1512182867 >> v2:172.21.15.146:3300/0 conn(0x7fa538073e90 msgr2=0x7fa5380c15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.008+0000 7fa536a00640 1 --2- 172.21.15.146:0/1512182867 >> v2:172.21.15.146:3300/0 conn(0x7fa538073e90 0x7fa5380c15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.008+0000 7fa536a00640 1 -- 172.21.15.146:0/1512182867 >> v1:172.21.15.161:6789/0 conn(0x7fa5380cc8b0 legacy=0x7fa53815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.008+0000 7fa536a00640 1 -- 172.21.15.146:0/1512182867 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5380c73d0 con 0x7fa538158ea0 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.009+0000 7f4032200640 1 --2- 172.21.15.146:0/911452622 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c166190 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.009+0000 7f4032200640 1 -- 172.21.15.146:0/911452622 >> 172.21.15.146:0/911452622 conn(0x7f402c06e040 msgr2=0x7f402c06e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.009+0000 7f4032200640 1 -- 172.21.15.146:0/911452622 shutdown_connections 2024-09-16T01:53:37.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.009+0000 7f4032200640 1 -- 172.21.15.146:0/911452622 wait complete. 2024-09-16T01:53:37.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.012+0000 7f4032200640 1 Processor -- start 2024-09-16T01:53:37.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.012+0000 7f4032200640 1 -- start start 2024-09-16T01:53:37.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.012+0000 7fa536a00640 1 --2- 172.21.15.146:0/1512182867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538158ea0 0x7fa5380c10b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa5380cd030 tx=0x7fa53002a570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7307a9bd2c1485de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.012+0000 7fc5b3400640 1 --2- 172.21.15.146:0/2714094561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc0c0 0x7fc5b415aab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc5a00097d0 tx=0x7fc5a002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b196ac396fb1f879 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.012+0000 7fc5b2000640 1 -- 172.21.15.146:0/2714094561 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0037070 con 0x7fc5b40cc0c0 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.013+0000 7fc5b2000640 1 -- 172.21.15.146:0/2714094561 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a002ac30 con 0x7fc5b40cc0c0 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.013+0000 7fc5b2000640 1 -- 172.21.15.146:0/2714094561 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0037070 con 0x7fc5b40cc0c0 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.013+0000 7fc5ba400640 1 -- 172.21.15.146:0/2714094561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc0c0 msgr2=0x7fc5b415aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.013+0000 7fc5ba400640 1 --2- 172.21.15.146:0/2714094561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc0c0 0x7fc5b415aab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc5a00097d0 tx=0x7fc5a002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fc5ba400640 1 -- 172.21.15.146:0/2714094561 shutdown_connections 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fa535600640 1 -- 172.21.15.146:0/1512182867 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530007740 con 0x7fa538158ea0 2024-09-16T01:53:37.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fa535600640 1 -- 172.21.15.146:0/1512182867 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5300078e0 con 0x7fa538158ea0 2024-09-16T01:53:37.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fa535600640 1 -- 172.21.15.146:0/1512182867 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530007cb0 con 0x7fa538158ea0 2024-09-16T01:53:37.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fc5ba400640 1 --2- 172.21.15.146:0/2714094561 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc0c0 0x7fc5b415aab0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fc5ba400640 1 -- 172.21.15.146:0/2714094561 >> 172.21.15.146:0/2714094561 conn(0x7fc5b406d560 msgr2=0x7fc5b406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7f4032200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 0x7f402c1fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7fc5ba400640 1 -- 172.21.15.146:0/2714094561 shutdown_connections 2024-09-16T01:53:37.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.014+0000 7f4032200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7f4032200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 0x7f402c072470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7fa537e00640 1 -- 172.21.15.146:0/1512182867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538158ea0 msgr2=0x7fa5380c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7f402be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7f402be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60110/0 (socket says 172.21.15.146:60110) 2024-09-16T01:53:37.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7fa537e00640 1 --2- 172.21.15.146:0/1512182867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538158ea0 0x7fa5380c10b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa5380cd030 tx=0x7fa53002a570 comp rx=0 tx=0).stop 2024-09-16T01:53:37.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7fa537e00640 1 -- 172.21.15.146:0/1512182867 shutdown_connections 2024-09-16T01:53:37.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7fa537e00640 1 --2- 172.21.15.146:0/1512182867 >> v2:172.21.15.146:3300/0 conn(0x7fa538073e90 0x7fa5380c15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7fa537e00640 1 --2- 172.21.15.146:0/1512182867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538158ea0 0x7fa5380c10b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7fa537e00640 1 -- 172.21.15.146:0/1512182867 >> 172.21.15.146:0/1512182867 conn(0x7fa538067950 msgr2=0x7fa538067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7f4031800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 0x7f402c072470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.015+0000 7f4031800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 0x7f402c072470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47162/0 (socket says 172.21.15.146:47162) 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f4030e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 0x7f402c1fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f402be00640 1 -- 172.21.15.146:0/764960541 learned_addr learned my addr 172.21.15.146:0/764960541 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f4030e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 0x7f402c1fe0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39424/0 (socket says 172.21.15.146:39424) 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f402c154ed0 con 0x7f402c154af0 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f402c0d10b0 con 0x7f402c0c53d0 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f402c15e500 con 0x7f402c0b3030 2024-09-16T01:53:37.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7fc5ba400640 1 -- 172.21.15.146:0/2714094561 wait complete. 2024-09-16T01:53:37.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7fa537e00640 1 -- 172.21.15.146:0/1512182867 shutdown_connections 2024-09-16T01:53:37.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.017+0000 7f4031800640 1 -- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 msgr2=0x7f402c1fe0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f82f8a00640 1 Processor -- start 2024-09-16T01:53:37.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7f82f8a00640 1 -- start start 2024-09-16T01:53:37.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.016+0000 7fa537e00640 1 -- 172.21.15.146:0/1512182867 wait complete. 2024-09-16T01:53:37.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.017+0000 7f82f8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f82ec0d4e70 0x7f82ec0468c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec045710 0x7f82ec0066c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f8a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82ec006c00 0x7f82ec04bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f8a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f82ec04b1b0 con 0x7f82ec0d4e70 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f8a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f82ec04b450 con 0x7f82ec006c00 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f8a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f82ec0dab80 con 0x7f82ec045710 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec045710 0x7f82ec0066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f2800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82ec006c00 0x7f82ec04bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec045710 0x7f82ec0066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39430/0 (socket says 172.21.15.146:39430) 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7f82f1400640 1 -- 172.21.15.146:0/3905353403 learned_addr learned my addr 172.21.15.146:0/3905353403 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7fcba9400640 1 Processor -- start 2024-09-16T01:53:37.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.018+0000 7fcba9400640 1 -- start start 2024-09-16T01:53:37.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7f82f2800640 1 -- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec045710 msgr2=0x7f82ec0066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7f82f2800640 1 --2- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec045710 0x7f82ec0066c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7f82f2800640 1 -- 172.21.15.146:0/3905353403 >> v2:172.21.15.146:3300/0 conn(0x7f82ec0d4e70 msgr2=0x7f82ec0468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7f82f2800640 1 --2- 172.21.15.146:0/3905353403 >> v2:172.21.15.146:3300/0 conn(0x7f82ec0d4e70 0x7f82ec0468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7fc5ba400640 1 Processor -- start 2024-09-16T01:53:37.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7f82f2800640 1 -- 172.21.15.146:0/3905353403 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ec0dae60 con 0x7f82ec006c00 2024-09-16T01:53:37.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.019+0000 7fc5ba400640 1 -- start start 2024-09-16T01:53:37.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7f4031800640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 0x7f402c1fe0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fa537e00640 1 Processor -- start 2024-09-16T01:53:37.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7f4031800640 1 -- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 msgr2=0x7f402c155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7f4031800640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7f4031800640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402c157c30 con 0x7f402c154af0 2024-09-16T01:53:37.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7f402be00640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:37.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7f4030e00640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 0x7f402c1fe0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:37.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fcba9400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcba4158ea0 0x7fcba40c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fcba9400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba40cc8b0 0x7fcba40c15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fcba9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba406c890 con 0x7fcba4073e90 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fcba9400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcba406cb70 con 0x7fcba4158ea0 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fcba9400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcba40c70f0 con 0x7fcba40cc8b0 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fcba3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcba4158ea0 0x7fcba40c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7fcba3e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcba4158ea0 0x7fcba40c10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60126/0 (socket says 172.21.15.146:60126) 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7fcba3e00640 1 -- 172.21.15.146:0/3866632751 learned_addr learned my addr 172.21.15.146:0/3866632751 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7fcba3400640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba40cc8b0 0x7fcba40c15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.020+0000 7fa537e00640 1 -- start start 2024-09-16T01:53:37.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7f82f2800640 1 --2- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec006c00 0x7f82ec04bb40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82f40518b0 tx=0x7f82f40639f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9a9818168875b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7fcba3e00640 1 -- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba40cc8b0 msgr2=0x7fcba40c15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7fc5ba400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b40cb2f0 0x7fc5b4150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7f82f0a00640 1 -- 172.21.15.146:0/3905353403 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40b5070 con 0x7f82ec006c00 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7f82f0a00640 1 -- 172.21.15.146:0/3905353403 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f40a8bd0 con 0x7f82ec006c00 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7f82f0a00640 1 -- 172.21.15.146:0/3905353403 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40b0440 con 0x7f82ec006c00 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.021+0000 7fc5ba400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b40cc0c0 0x7fc5b414af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5ba400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc4a0 0x7fc5b414b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5ba400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b415e540 con 0x7fc5b40cc0c0 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b40cb2f0 0x7fc5b4150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5ba400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b415ba70 con 0x7fc5b40cb2f0 2024-09-16T01:53:37.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b40cb2f0 0x7fc5b4150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60132/0 (socket says 172.21.15.146:60132) 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5ba400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b40cc8b0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3400640 1 -- 172.21.15.146:0/4038606877 learned_addr learned my addr 172.21.15.146:0/4038606877 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3e00640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc4a0 0x7fc5b414b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fcba3e00640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba40cc8b0 0x7fcba40c15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fcba3e00640 1 -- 172.21.15.146:0/3866632751 >> v1:172.21.15.146:6789/0 conn(0x7fcba4073e90 legacy=0x7fcba415ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fcba3e00640 1 -- 172.21.15.146:0/3866632751 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba40c73d0 con 0x7fcba4158ea0 2024-09-16T01:53:37.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b2a00640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b40cc0c0 0x7fc5b414af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7f4031800640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 0x7f402c072470 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4024030000 tx=0x7f4024004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3e00640 1 -- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b40cb2f0 msgr2=0x7fc5b4150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3e00640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b40cb2f0 0x7fc5b4150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3e00640 1 -- 172.21.15.146:0/4038606877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b40cc0c0 msgr2=0x7fc5b414af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fc5b3e00640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b40cc0c0 0x7fc5b414af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7fcba3400640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba40cc8b0 0x7fcba40c15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:37.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7fc5b3e00640 1 -- 172.21.15.146:0/4038606877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b415d800 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7fc5b3e00640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc4a0 0x7fc5b414b440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac010fb0 tx=0x7fc5ac0154b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.022+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024036050 con 0x7f402c154af0 2024-09-16T01:53:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f402c1581b0 con 0x7f402c154af0 2024-09-16T01:53:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4024037040 con 0x7f402c154af0 2024-09-16T01:53:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024036050 con 0x7f402c154af0 2024-09-16T01:53:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402c1fd5c0 con 0x7f402c154af0 2024-09-16T01:53:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7fa537e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 0x7fa5381f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7f82f8a00640 1 -- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec006c00 msgr2=0x7f82ec04bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.023+0000 7f82f8a00640 1 --2- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec006c00 0x7f82ec04bb40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f82f40518b0 tx=0x7f82f40639f0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fa537e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5380cc8b0 0x7fa5381ea260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fcba3e00640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba4158ea0 0x7fcba40c10b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009870 tx=0x7fcb9402a1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e1f75ff4e7a3ae8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 -- 172.21.15.146:0/3905353403 shutdown_connections 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 --2- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec006c00 0x7f82ec04bb40 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 --2- 172.21.15.146:0/3905353403 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec045710 0x7f82ec0066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 --2- 172.21.15.146:0/3905353403 >> v2:172.21.15.146:3300/0 conn(0x7f82ec0d4e70 0x7f82ec0468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 -- 172.21.15.146:0/3905353403 >> 172.21.15.146:0/3905353403 conn(0x7f82ec00c230 msgr2=0x7f82ec00c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fcba2a00640 1 -- 172.21.15.146:0/3866632751 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94037070 con 0x7fcba4158ea0 2024-09-16T01:53:37.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fcba2a00640 1 -- 172.21.15.146:0/3866632751 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb94032440 con 0x7fcba4158ea0 2024-09-16T01:53:37.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fcba2a00640 1 -- 172.21.15.146:0/3866632751 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94032810 con 0x7fcba4158ea0 2024-09-16T01:53:37.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 -- 172.21.15.146:0/3905353403 shutdown_connections 2024-09-16T01:53:37.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f402c0cd9d0 con 0x7f402c154af0 2024-09-16T01:53:37.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fa536a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 0x7fa5381f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7fa536a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 0x7fa5381f4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60142/0 (socket says 172.21.15.146:60142) 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7fa536a00640 1 -- 172.21.15.146:0/734157281 learned_addr learned my addr 172.21.15.146:0/734157281 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7fa536000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5380cc8b0 0x7fa5381ea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa538158ea0 0x7fa5381e6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa53815f480 con 0x7fa538158ea0 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa53815f300 con 0x7fa538073e90 2024-09-16T01:53:37.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.025+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa53815f600 con 0x7fa5380cc8b0 2024-09-16T01:53:37.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.024+0000 7f82f8a00640 1 -- 172.21.15.146:0/3905353403 wait complete. 2024-09-16T01:53:37.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac020070 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac015dc0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0102e0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b415dae0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b4157630 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa537400640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa538158ea0 0x7fa5381e6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa536a00640 1 -- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5380cc8b0 msgr2=0x7fa5381ea260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa536a00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5380cc8b0 0x7fa5381ea260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa536a00640 1 -- 172.21.15.146:0/734157281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa538158ea0 msgr2=0x7fa5381e6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa536a00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa538158ea0 0x7fa5381e6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa536a00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5380c6200 con 0x7fa538073e90 2024-09-16T01:53:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fa537400640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa538158ea0 0x7fa5381e6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fcba9400640 1 -- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba4158ea0 msgr2=0x7fcba40c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.026+0000 7fcba9400640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba4158ea0 0x7fcba40c10b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009870 tx=0x7fcb9402a1d0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.027+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4024003990 con 0x7f402c154af0 2024-09-16T01:53:37.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7f82f8a00640 1 Processor -- start 2024-09-16T01:53:37.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7fa536a00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 0x7fa5381f4c40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa530009e80 tx=0x7fa530004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7fcba9400640 1 -- 172.21.15.146:0/3866632751 shutdown_connections 2024-09-16T01:53:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7fcba9400640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba40cc8b0 0x7fcba40c15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7fcba9400640 1 --2- 172.21.15.146:0/3866632751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba4158ea0 0x7fcba40c10b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7f82f8a00640 1 -- start start 2024-09-16T01:53:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.028+0000 7fcba9400640 1 -- 172.21.15.146:0/3866632751 >> 172.21.15.146:0/3866632751 conn(0x7fcba4067950 msgr2=0x7fcba4067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.029+0000 7fcba9400640 1 -- 172.21.15.146:0/3866632751 shutdown_connections 2024-09-16T01:53:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.029+0000 7f4028c00640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f400c06c4b0 0x7f400c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.030+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f40240abdd0 con 0x7f402c154af0 2024-09-16T01:53:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.029+0000 7fcba9400640 1 -- 172.21.15.146:0/3866632751 wait complete. 2024-09-16T01:53:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.030+0000 7f59e3200640 1 Processor -- start 2024-09-16T01:53:37.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.030+0000 7f59e3200640 1 -- start start 2024-09-16T01:53:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.030+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53003f070 con 0x7fa538073e90 2024-09-16T01:53:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.029+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc58c005290 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.031+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa530031bc0 con 0x7fa538073e90 2024-09-16T01:53:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.031+0000 7f4030e00640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f400c06c4b0 0x7f400c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.031+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530043410 con 0x7fa538073e90 2024-09-16T01:53:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.031+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5381e4df0 con 0x7fa538073e90 2024-09-16T01:53:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.031+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5381e53b0 con 0x7fa538073e90 2024-09-16T01:53:37.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f59e3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f59e40c5eb0 0x7f59e40cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f59e3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e415af00 con 0x7f59e415aa70 2024-09-16T01:53:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f59e3200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e415b1e0 con 0x7f59e40c6290 2024-09-16T01:53:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f59e3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f59e406dbd0 con 0x7f59e40c5eb0 2024-09-16T01:53:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7fcba9400640 1 Processor -- start 2024-09-16T01:53:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f59e2800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f59e40c6290 0x7f59e406c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44682/0 (socket says 172.21.15.146:44682) 2024-09-16T01:53:37.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f59e2800640 1 -- 172.21.15.146:0/266870452 learned_addr learned my addr 172.21.15.146:0/266870452 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:37.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.031+0000 7f82f8a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec006c00 0x7f82ec1720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f59e1e00640 1 --2- 172.21.15.146:0/266870452 >> v2:172.21.15.146:3300/0 conn(0x7f59e40c5eb0 0x7f59e40cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f82f8a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec045710 0x7f82ec04f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f82f8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 0x7f82ec0d5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.032+0000 7f24a4c00640 1 Processor -- start 2024-09-16T01:53:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f59e1e00640 1 -- 172.21.15.146:0/266870452 >> v1:172.21.15.161:6789/0 conn(0x7f59e40c6290 legacy=0x7f59e406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f59e1e00640 1 -- 172.21.15.146:0/266870452 >> v1:172.21.15.146:6789/0 conn(0x7f59e415aa70 legacy=0x7f59e4154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f59e1e00640 1 -- 172.21.15.146:0/266870452 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e415b4c0 con 0x7f59e40c5eb0 2024-09-16T01:53:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc5ac023070 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa510005290 con 0x7fa538073e90 2024-09-16T01:53:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7fcba9400640 1 -- start start 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 0x7fcba40c3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba9400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba40cc8b0 0x7fcba41f4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba9400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba4158ea0 0x7fcba41573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcba415f480 con 0x7fcba4073e90 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba9400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcba415f300 con 0x7fcba40cc8b0 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba9400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcba415f600 con 0x7fcba4158ea0 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7fcba3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 0x7fcba40c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fcba3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 0x7fcba40c3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47214/0 (socket says 172.21.15.146:47214) 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fcba3e00640 1 -- 172.21.15.146:0/108980891 learned_addr learned my addr 172.21.15.146:0/108980891 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7f24a4c00640 1 -- start start 2024-09-16T01:53:37.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fc5ab400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c380 0x7fc59406e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fcba8a00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba4158ea0 0x7fcba41573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fc5ac093480 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fc5b3400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c380 0x7fc59406e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7fcba3400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba40cc8b0 0x7fcba41f4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f82f8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec0db940 con 0x7f82ec0d4e70 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f82f8a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec0db420 con 0x7f82ec045710 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.033+0000 7f82f8a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f82ec04ae70 con 0x7f82ec006c00 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7f82f2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 0x7f82ec0d5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7f82f2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 0x7f82ec0d5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47208/0 (socket says 172.21.15.146:47208) 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.034+0000 7f82f2800640 1 -- 172.21.15.146:0/2726443716 learned_addr learned my addr 172.21.15.146:0/2726443716 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7f82f2800640 1 -- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec006c00 msgr2=0x7f82ec1720a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:37.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.035+0000 7f59e1e00640 1 --2- 172.21.15.146:0/266870452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c5eb0 0x7f59e40cac30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f59d40097a0 tx=0x7f59d402a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98540d590b56dd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f82f1e00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec006c00 0x7f82ec1720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f82f2800640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec006c00 0x7f82ec1720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f82f2800640 1 -- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec045710 msgr2=0x7f82ec04f5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f82f1400640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec045710 0x7f82ec04f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7fc5b3400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c380 0x7fc59406e840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0000c00 tx=0x7fc5a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f4030e00640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f400c06c4b0 0x7f400c06e970 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4018007fd0 tx=0x7f4018009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa5300435b0 con 0x7fa538073e90 2024-09-16T01:53:37.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f6797000640 1 Processor -- start 2024-09-16T01:53:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7fcba8a00640 1 -- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba40cc8b0 msgr2=0x7fcba41f4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f6797000640 1 -- start start 2024-09-16T01:53:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7fcba8a00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba40cc8b0 0x7fcba41f4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7fcba8a00640 1 -- 172.21.15.146:0/108980891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 msgr2=0x7fcba40c3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f82f2800640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec045710 0x7f82ec04f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7f82f2800640 1 -- 172.21.15.146:0/2726443716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ec0d7590 con 0x7f82ec0d4e70 2024-09-16T01:53:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.036+0000 7fcba8a00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 0x7fcba40c3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fcba8a00640 1 -- 172.21.15.146:0/108980891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba415b690 con 0x7fcba4158ea0 2024-09-16T01:53:37.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f82f1e00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec006c00 0x7f82ec1720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:37.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fcba3e00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 0x7fcba40c3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:37.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fa52ea00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa51806c380 0x7fa51806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fa5300ac700 con 0x7fa538073e90 2024-09-16T01:53:37.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fcba8a00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba4158ea0 0x7fcba41573f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00c6c0 tx=0x7fcb9c00cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc5ac0614e0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f82f2800640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 0x7f82ec0d5740 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f82f402dd50 tx=0x7f82f4074750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7fa536000640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa51806c380 0x7fa51806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f24a4c00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f24a0158f40 0x7f24a015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f24a4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 0x7f24a00c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f24a4c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24a0162c00 0x7f24a00c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f24a4c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f24a0162770 con 0x7f24a0162c00 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f24a4c00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f24a015f430 con 0x7f24a0158f40 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.037+0000 7f24a4c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f24a015f710 con 0x7f24a0159320 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 0x7f24a00c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c003b70 con 0x7fcba4158ea0 2024-09-16T01:53:37.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb9c003d10 con 0x7fcba4158ea0 2024-09-16T01:53:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c0052f0 con 0x7fcba4158ea0 2024-09-16T01:53:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 0x7f24a00c8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39482/0 (socket says 172.21.15.146:39482) 2024-09-16T01:53:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 -- 172.21.15.146:0/4220697672 learned_addr learned my addr 172.21.15.146:0/4220697672 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249de00640 1 --2- 172.21.15.146:0/4220697672 >> v2:172.21.15.161:3300/0 conn(0x7f24a0158f40 0x7f24a015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba41f4400 con 0x7fcba4158ea0 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 -- 172.21.15.146:0/4220697672 >> v2:172.21.15.161:3300/0 conn(0x7f24a0158f40 msgr2=0x7f24a015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 --2- 172.21.15.146:0/4220697672 >> v2:172.21.15.161:3300/0 conn(0x7f24a0158f40 0x7f24a015ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 -- 172.21.15.146:0/4220697672 >> v2:172.21.15.146:3300/0 conn(0x7f24a0162c00 msgr2=0x7f24a00c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 --2- 172.21.15.146:0/4220697672 >> v2:172.21.15.146:3300/0 conn(0x7f24a0162c00 0x7f24a00c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7f249d400640 1 -- 172.21.15.146:0/4220697672 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a015f9f0 con 0x7f24a0159320 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba415bc30 con 0x7fcba4158ea0 2024-09-16T01:53:37.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.039+0000 7f249d400640 1 --2- 172.21.15.146:0/4220697672 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 0x7f24a00c8b10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cd270 tx=0x7f248c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f726a45237458700 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.039+0000 7f6797000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f67900cc0c0 0x7f679015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.039+0000 7f6797000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f67900cb2f0 0x7f67900cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f6797000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679015b1d0 con 0x7f679015e9d0 2024-09-16T01:53:37.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f6797000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f67900cbd00 con 0x7f67900cb2f0 2024-09-16T01:53:37.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f6797000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f679015ba70 con 0x7f67900cc0c0 2024-09-16T01:53:37.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.038+0000 7fa536000640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa51806c380 0x7fa51806e840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa5380cc3d0 tx=0x7fa528007e20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f67900cc0c0 0x7f679015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f67900cc0c0 0x7f679015aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60172/0 (socket says 172.21.15.146:60172) 2024-09-16T01:53:37.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 -- 172.21.15.146:0/2398316261 learned_addr learned my addr 172.21.15.146:0/2398316261 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f249ca00640 1 -- 172.21.15.146:0/4220697672 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c037070 con 0x7f24a0159320 2024-09-16T01:53:37.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f249ca00640 1 -- 172.21.15.146:0/4220697672 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c032440 con 0x7f24a0159320 2024-09-16T01:53:37.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f249ca00640 1 -- 172.21.15.146:0/4220697672 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c032810 con 0x7f24a0159320 2024-09-16T01:53:37.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f59e0a00640 1 -- 172.21.15.146:0/266870452 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4037070 con 0x7f59e40c5eb0 2024-09-16T01:53:37.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f59e0a00640 1 -- 172.21.15.146:0/266870452 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d4032640 con 0x7f59e40c5eb0 2024-09-16T01:53:37.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f59e0a00640 1 -- 172.21.15.146:0/266870452 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4037070 con 0x7f59e40c5eb0 2024-09-16T01:53:37.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 -- 172.21.15.146:0/2398316261 >> v2:172.21.15.146:3300/0 conn(0x7f67900cb2f0 msgr2=0x7f67900cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:37.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 --2- 172.21.15.146:0/2398316261 >> v2:172.21.15.146:3300/0 conn(0x7f67900cb2f0 0x7f67900cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 -- 172.21.15.146:0/2398316261 >> v1:172.21.15.161:6789/0 conn(0x7f679015e9d0 legacy=0x7f67900cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f678fe00640 1 -- 172.21.15.146:0/2398316261 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679015b4b0 con 0x7f67900cc0c0 2024-09-16T01:53:37.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.040+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f40240af030 con 0x7f402c154af0 2024-09-16T01:53:37.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40b5040 con 0x7f82ec0d4e70 2024-09-16T01:53:37.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f40c0070 con 0x7f82ec0d4e70 2024-09-16T01:53:37.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40bb840 con 0x7f82ec0d4e70 2024-09-16T01:53:37.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba415f9c0 con 0x7fcba4158ea0 2024-09-16T01:53:37.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f59e3200640 1 -- 172.21.15.146:0/266870452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c5eb0 msgr2=0x7f59e40cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f59e3200640 1 --2- 172.21.15.146:0/266870452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c5eb0 0x7f59e40cac30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f59d40097a0 tx=0x7f59d402a2e0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f24a4c00640 1 -- 172.21.15.146:0/4220697672 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 msgr2=0x7f24a00c8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f24a4c00640 1 --2- 172.21.15.146:0/4220697672 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 0x7f24a00c8b10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cd270 tx=0x7f248c02a270 comp rx=0 tx=0).stop 2024-09-16T01:53:37.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f24a4c00640 1 -- 172.21.15.146:0/4220697672 shutdown_connections 2024-09-16T01:53:37.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f24a4c00640 1 --2- 172.21.15.146:0/4220697672 >> v2:172.21.15.146:3300/0 conn(0x7f24a0162c00 0x7f24a00c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ec1742d0 con 0x7f82ec0d4e70 2024-09-16T01:53:37.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.041+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ec174890 con 0x7f82ec0d4e70 2024-09-16T01:53:37.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f24a4c00640 1 --2- 172.21.15.146:0/4220697672 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0159320 0x7f24a00c8b10 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cd270 tx=0x7f248c02a270 comp rx=0 tx=0).stop 2024-09-16T01:53:37.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f24a4c00640 1 --2- 172.21.15.146:0/4220697672 >> v2:172.21.15.161:3300/0 conn(0x7f24a0158f40 0x7f24a015ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7f24a4c00640 1 -- 172.21.15.146:0/4220697672 >> 172.21.15.146:0/4220697672 conn(0x7f24a006f3c0 msgr2=0x7f24a006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcb9c005490 con 0x7fcba4158ea0 2024-09-16T01:53:37.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.042+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa53007ae40 con 0x7fa538073e90 2024-09-16T01:53:37.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82cc005290 con 0x7f82ec0d4e70 2024-09-16T01:53:37.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f59e3200640 1 -- 172.21.15.146:0/266870452 shutdown_connections 2024-09-16T01:53:37.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f59e3200640 1 --2- 172.21.15.146:0/266870452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c5eb0 0x7f59e40cac30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f59e3200640 1 -- 172.21.15.146:0/266870452 >> 172.21.15.146:0/266870452 conn(0x7f59e4067b30 msgr2=0x7f59e4067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f59e3200640 1 -- 172.21.15.146:0/266870452 shutdown_connections 2024-09-16T01:53:37.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f59e3200640 1 -- 172.21.15.146:0/266870452 wait complete. 2024-09-16T01:53:37.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f678fe00640 1 --2- 172.21.15.146:0/2398316261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67900cc0c0 0x7f679015aab0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f67900ce150 tx=0x7f678002a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9735743b0ff030b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.044+0000 7f678f400640 1 -- 172.21.15.146:0/2398316261 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780037070 con 0x7f67900cc0c0 2024-09-16T01:53:37.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.044+0000 7f678f400640 1 -- 172.21.15.146:0/2398316261 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6780032440 con 0x7f67900cc0c0 2024-09-16T01:53:37.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.044+0000 7f678f400640 1 -- 172.21.15.146:0/2398316261 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780032830 con 0x7f67900cc0c0 2024-09-16T01:53:37.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.043+0000 7f24a4c00640 1 -- 172.21.15.146:0/4220697672 shutdown_connections 2024-09-16T01:53:37.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.044+0000 7f24a4c00640 1 -- 172.21.15.146:0/4220697672 wait complete. 2024-09-16T01:53:37.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.044+0000 7fcb9be00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb8406c3b0 0x7fcb8406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.044+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fcb9c08b240 con 0x7fcba4158ea0 2024-09-16T01:53:37.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 -- 172.21.15.146:0/2398316261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67900cc0c0 msgr2=0x7f679015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 --2- 172.21.15.146:0/2398316261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67900cc0c0 0x7f679015aab0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f67900ce150 tx=0x7f678002a290 comp rx=0 tx=0).stop 2024-09-16T01:53:37.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7fcba3e00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb8406c3b0 0x7fcb8406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 -- 172.21.15.146:0/2398316261 shutdown_connections 2024-09-16T01:53:37.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 --2- 172.21.15.146:0/2398316261 >> v2:172.21.15.146:3300/0 conn(0x7f67900cb2f0 0x7f67900cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 --2- 172.21.15.146:0/2398316261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67900cc0c0 0x7f679015aab0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 -- 172.21.15.146:0/2398316261 >> 172.21.15.146:0/2398316261 conn(0x7f679006d560 msgr2=0x7f679006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f59e3200640 1 Processor -- start 2024-09-16T01:53:37.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.046+0000 7f59e3200640 1 -- start start 2024-09-16T01:53:37.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.046+0000 7f24a4c00640 1 Processor -- start 2024-09-16T01:53:37.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.046+0000 7f24a4c00640 1 -- start start 2024-09-16T01:53:37.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.047+0000 7f59e3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e40c5eb0 0x7f59e414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c6290 0x7f59e414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 0x7f59e40c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f24a4c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0158f40 0x7f24a015a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f24a4c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 0x7f24a01512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f24a4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e1a0 0x7f24a0151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f24a4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0072070 con 0x7f24a015e1a0 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f24a4c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0162400 con 0x7f24a0162c00 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f24a4c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f24a0071930 con 0x7f24a0158f40 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f249d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 0x7f24a01512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f249d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 0x7f24a01512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60182/0 (socket says 172.21.15.146:60182) 2024-09-16T01:53:37.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f249d400640 1 -- 172.21.15.146:0/536229926 learned_addr learned my addr 172.21.15.146:0/536229926 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 0x7f59e40c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e2800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 0x7f59e40c81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39498/0 (socket says 172.21.15.146:39498) 2024-09-16T01:53:37.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e2800640 1 -- 172.21.15.146:0/2913240897 learned_addr learned my addr 172.21.15.146:0/2913240897 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e1e00640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e40c5eb0 0x7f59e414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4155020 con 0x7f59e40c6290 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59e406c370 con 0x7f59e40c5eb0 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f59e41551a0 con 0x7f59e415bc60 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f249d400640 1 -- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0158f40 msgr2=0x7f24a015a7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.048+0000 7f249d400640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0158f40 0x7f24a015a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f249d400640 1 -- 172.21.15.146:0/536229926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e1a0 msgr2=0x7f24a0151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f249d400640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e1a0 0x7f24a0151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7fcba3e00640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb8406c3b0 0x7fcb8406e870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcb9402a210 tx=0x7fcb9402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f249d400640 1 -- 172.21.15.146:0/536229926 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a015ce20 con 0x7f24a0162c00 2024-09-16T01:53:37.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f249d400640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 0x7f24a01512c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f248c02a7e0 tx=0x7f248c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e2800640 1 -- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e40c5eb0 msgr2=0x7f59e414d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e2800640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e40c5eb0 0x7f59e414d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e2800640 1 -- 172.21.15.146:0/2913240897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c6290 msgr2=0x7f59e414d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e2800640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c6290 0x7f59e414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e2800640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e40c9b20 con 0x7f59e415bc60 2024-09-16T01:53:37.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c037040 con 0x7f24a0162c00 2024-09-16T01:53:37.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c042070 con 0x7f24a0162c00 2024-09-16T01:53:37.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c030900 con 0x7f24a0162c00 2024-09-16T01:53:37.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e2800640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 0x7f59e40c81c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00cad0 tx=0x7f59dc00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.049+0000 7f59e1e00640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e40c5eb0 0x7f59e414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:37.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a015d100 con 0x7f24a0162c00 2024-09-16T01:53:37.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a015d6c0 con 0x7f24a0162c00 2024-09-16T01:53:37.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc017070 con 0x7f59e415bc60 2024-09-16T01:53:37.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59dc00ac30 con 0x7f59e415bc60 2024-09-16T01:53:37.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e414e9c0 con 0x7f59e415bc60 2024-09-16T01:53:37.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.045+0000 7f6797000640 1 -- 172.21.15.146:0/2398316261 shutdown_connections 2024-09-16T01:53:37.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.046+0000 7f6797000640 1 -- 172.21.15.146:0/2398316261 wait complete. 2024-09-16T01:53:37.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.050+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc012400 con 0x7f59e415bc60 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.051+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e414ef80 con 0x7f59e415bc60 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.053+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f82f40a8a90 con 0x7f82ec0d4e70 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.054+0000 7f82ea000640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f82d406c2b0 0x7f82d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.055+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f82f4128d20 con 0x7f82ec0d4e70 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.055+0000 7f82f1e00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f82d406c2b0 0x7f82d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.055+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f82f40c9020 con 0x7f82ec0d4e70 2024-09-16T01:53:37.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.054+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcb9c059210 con 0x7fcba4158ea0 2024-09-16T01:53:37.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.052+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a0076410 con 0x7f24a0162c00 2024-09-16T01:53:37.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.054+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f248c030aa0 con 0x7f24a0162c00 2024-09-16T01:53:37.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.053+0000 7f6797000640 1 Processor -- start 2024-09-16T01:53:37.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.053+0000 7f6797000640 1 -- start start 2024-09-16T01:53:37.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.055+0000 7f6797000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 0x7f679015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.052+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e40c56f0 con 0x7f59e415bc60 2024-09-16T01:53:37.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.054+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f59dc01a070 con 0x7f59e415bc60 2024-09-16T01:53:37.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.055+0000 7f82f1e00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f82d406c2b0 0x7f82d406e770 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f82e4006fd0 tx=0x7f82e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f6797000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67900cc0c0 0x7f6790158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f6797000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f679015e9d0 0x7f6790158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f6797000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f679015e1d0 con 0x7f67900cc0c0 2024-09-16T01:53:37.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f6797000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f679015e540 con 0x7f679015e9d0 2024-09-16T01:53:37.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f6797000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f67900c5560 con 0x7f67900cb2f0 2024-09-16T01:53:37.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f59da000640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59c406c2e0 0x7f59c406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f2496000640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f248006c2b0 0x7f248006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f59dc08b090 con 0x7f59e415bc60 2024-09-16T01:53:37.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f59e1e00640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59c406c2e0 0x7f59c406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.056+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f248c0ac000 con 0x7f24a0162c00 2024-09-16T01:53:37.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.057+0000 7f249de00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f248006c2b0 0x7f248006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.057+0000 7f678fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 0x7f679015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.057+0000 7f249de00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f248006c2b0 0x7f248006e770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f24900099d0 tx=0x7f2490009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.057+0000 7f678fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 0x7f679015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39500/0 (socket says 172.21.15.146:39500) 2024-09-16T01:53:37.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.057+0000 7f678fe00640 1 -- 172.21.15.146:0/1684513174 learned_addr learned my addr 172.21.15.146:0/1684513174 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:37.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f24a0076410 con 0x7f24a0162c00 2024-09-16T01:53:37.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f59e1e00640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59c406c2e0 0x7f59c406e7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f59d4034000 tx=0x7f59d402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f6787400640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67900cc0c0 0x7f6790158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f6795000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f679015e9d0 0x7f6790158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f678fe00640 1 -- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f679015e9d0 msgr2=0x7f6790158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f678fe00640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f679015e9d0 0x7f6790158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.058+0000 7f678fe00640 1 -- 172.21.15.146:0/1684513174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67900cc0c0 msgr2=0x7f6790158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.059+0000 7f678fe00640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67900cc0c0 0x7f6790158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.059+0000 7f678fe00640 1 -- 172.21.15.146:0/1684513174 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6790159b60 con 0x7f67900cb2f0 2024-09-16T01:53:37.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.059+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f59dc059100 con 0x7f59e415bc60 2024-09-16T01:53:37.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.059+0000 7f6795000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f679015e9d0 0x7f6790158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.060+0000 7f6787400640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67900cc0c0 0x7f6790158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.060+0000 7f678fe00640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 0x7f679015cf90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f678002a2d0 tx=0x7f67800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.060+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780041020 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.060+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6780030800 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.060+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780037040 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.061+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679015a320 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.061+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67901f17b0 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.063+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6790076410 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.064+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6780044020 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.066+0000 7f678cc00640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f676c06c2b0 0x7f676c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.066+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f67800abdf0 con 0x7f67900cb2f0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.067+0000 7f6787400640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f676c06c2b0 0x7f676c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:37.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.067+0000 7f6787400640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f676c06c2b0 0x7f676c06e770 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6790151ec0 tx=0x7f6778008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:37.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.069+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6790076410 con 0x7f67900cb2f0 2024-09-16T01:53:37.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.448+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f402c0ca5f0 con 0x7f402c154af0 2024-09-16T01:53:37.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.449+0000 7f4028c00640 1 -- 172.21.15.146:0/764960541 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f402402ae20 con 0x7f402c154af0 2024-09-16T01:53:37.450 INFO:teuthology.orchestra.run.smithi146.stdout:64424509501 2024-09-16T01:53:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f400c06c4b0 msgr2=0x7f400c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f400c06c4b0 0x7f400c06e970 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4018007fd0 tx=0x7f4018009040 comp rx=0 tx=0).stop 2024-09-16T01:53:37.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 msgr2=0x7f402c072470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 0x7f402c072470 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4024030000 tx=0x7f4024004a40 comp rx=0 tx=0).stop 2024-09-16T01:53:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc58c005850 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 shutdown_connections 2024-09-16T01:53:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f400c06c4b0 0x7f400c06e970 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f402c154af0 0x7f402c072470 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f402c0c53d0 0x7f402c155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 --2- 172.21.15.146:0/764960541 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f402c0b3030 0x7f402c1fe0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.453+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 >> 172.21.15.146:0/764960541 conn(0x7f402c06e040 msgr2=0x7f402c0cb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.454+0000 7fc5ab400640 1 -- 172.21.15.146:0/4038606877 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc5ac0657e0 con 0x7fc5b40cc4a0 2024-09-16T01:53:37.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.454+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 shutdown_connections 2024-09-16T01:53:37.458 INFO:teuthology.orchestra.run.smithi146.stdout:64424509501 2024-09-16T01:53:37.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.454+0000 7f4032200640 1 -- 172.21.15.146:0/764960541 wait complete. 2024-09-16T01:53:37.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c380 msgr2=0x7fc59406e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c380 0x7fc59406e840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0000c00 tx=0x7fc5a002e000 comp rx=0 tx=0).stop 2024-09-16T01:53:37.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc4a0 msgr2=0x7fc5b414b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc4a0 0x7fc5b414b440 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac010fb0 tx=0x7fc5ac0154b0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 shutdown_connections 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc59406c380 0x7fc59406e840 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc5b40cc4a0 0x7fc5b414b440 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b40cc0c0 0x7fc5b414af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 --2- 172.21.15.146:0/4038606877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5b40cb2f0 0x7fc5b4150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.458+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 >> 172.21.15.146:0/4038606877 conn(0x7fc5b406d560 msgr2=0x7fc5b40ce570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.459+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 shutdown_connections 2024-09-16T01:53:37.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.459+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa510005580 con 0x7fa538073e90 2024-09-16T01:53:37.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.459+0000 7fc5ba400640 1 -- 172.21.15.146:0/4038606877 wait complete. 2024-09-16T01:53:37.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.459+0000 7fa52ea00640 1 -- 172.21.15.146:0/734157281 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa53007a740 con 0x7fa538073e90 2024-09-16T01:53:37.464 INFO:teuthology.orchestra.run.smithi146.stdout:55834574909 2024-09-16T01:53:37.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.463+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa51806c380 msgr2=0x7fa51806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.463+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa51806c380 0x7fa51806e840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa5380cc3d0 tx=0x7fa528007e20 comp rx=0 tx=0).stop 2024-09-16T01:53:37.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.463+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 msgr2=0x7fa5381f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.463+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 0x7fa5381f4c40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa530009e80 tx=0x7fa530004a40 comp rx=0 tx=0).stop 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.464+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 shutdown_connections 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.464+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa51806c380 0x7fa51806e840 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.464+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa538158ea0 0x7fa5381e6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.464+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa5380cc8b0 0x7fa5381ea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.464+0000 7fa537e00640 1 --2- 172.21.15.146:0/734157281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa538073e90 0x7fa5381f4c40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.464+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 >> 172.21.15.146:0/734157281 conn(0x7fa538067950 msgr2=0x7fa5380c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.465+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 shutdown_connections 2024-09-16T01:53:37.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.465+0000 7fa537e00640 1 -- 172.21.15.146:0/734157281 wait complete. 2024-09-16T01:53:37.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.471+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fcba40c7e90 con 0x7fcba4158ea0 2024-09-16T01:53:37.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.472+0000 7fcb9be00640 1 -- 172.21.15.146:0/108980891 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcb9c05d510 con 0x7fcba4158ea0 2024-09-16T01:53:37.475 INFO:teuthology.orchestra.run.smithi146.stdout:64424509501 2024-09-16T01:53:37.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.475+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f24a01630b0 con 0x7f24a0162c00 2024-09-16T01:53:37.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7f2496000640 1 -- 172.21.15.146:0/536229926 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f248c065020 con 0x7f24a0162c00 2024-09-16T01:53:37.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f82cc005850 con 0x7f82ec0d4e70 2024-09-16T01:53:37.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.477+0000 7f82ea000640 1 -- 172.21.15.146:0/2726443716 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82f40f74f0 con 0x7f82ec0d4e70 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb8406c3b0 msgr2=0x7fcb8406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb8406c3b0 0x7fcb8406e870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcb9402a210 tx=0x7fcb9402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba4158ea0 msgr2=0x7fcba41573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba4158ea0 0x7fcba41573f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00c6c0 tx=0x7fcb9c00cb80 comp rx=0 tx=0).stop 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 shutdown_connections 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fcb8406c3b0 0x7fcb8406e870 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcba4158ea0 0x7fcba41573f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba40cc8b0 0x7fcba41f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 --2- 172.21.15.146:0/108980891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba4073e90 0x7fcba40c3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.476+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 >> 172.21.15.146:0/108980891 conn(0x7fcba4067950 msgr2=0x7fcba40c0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.477+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 shutdown_connections 2024-09-16T01:53:37.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.477+0000 7fcba9400640 1 -- 172.21.15.146:0/108980891 wait complete. 2024-09-16T01:53:37.480 INFO:teuthology.orchestra.run.smithi146.stdout:60129542204 2024-09-16T01:53:37.480 INFO:teuthology.orchestra.run.smithi146.stdout:64424509500 2024-09-16T01:53:37.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.480+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f248006c2b0 msgr2=0x7f248006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.480+0000 7f24a4c00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f248006c2b0 0x7f248006e770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f24900099d0 tx=0x7f2490009190 comp rx=0 tx=0).stop 2024-09-16T01:53:37.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f82d406c2b0 msgr2=0x7f82d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f82d406c2b0 0x7f82d406e770 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f82e4006fd0 tx=0x7f82e4008040 comp rx=0 tx=0).stop 2024-09-16T01:53:37.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 msgr2=0x7f82ec0d5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 0x7f82ec0d5740 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f82f402dd50 tx=0x7f82f4074750 comp rx=0 tx=0).stop 2024-09-16T01:53:37.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509503 got 64424509501 for osd.0 2024-09-16T01:53:37.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 shutdown_connections 2024-09-16T01:53:37.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f82d406c2b0 0x7f82d406e770 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82ec0d4e70 0x7f82ec0d5740 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ec045710 0x7f82ec04f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 --2- 172.21.15.146:0/2726443716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f82ec006c00 0x7f82ec1720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 >> 172.21.15.146:0/2726443716 conn(0x7f82ec00c230 msgr2=0x7f82ec04d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.480+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 msgr2=0x7f24a01512c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.480+0000 7f24a4c00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 0x7f24a01512c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f248c02a7e0 tx=0x7f248c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 shutdown_connections 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f24a4c00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f248006c2b0 0x7f248006e770 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f24a4c00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24a015e1a0 0x7f24a0151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f24a4c00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24a0162c00 0x7f24a01512c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f24a4c00640 1 --2- 172.21.15.146:0/536229926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24a0158f40 0x7f24a015a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 >> 172.21.15.146:0/536229926 conn(0x7f24a006f3c0 msgr2=0x7f24a0153800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.481+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 shutdown_connections 2024-09-16T01:53:37.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.482+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 shutdown_connections 2024-09-16T01:53:37.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.482+0000 7f82f8a00640 1 -- 172.21.15.146:0/2726443716 wait complete. 2024-09-16T01:53:37.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.482+0000 7f24a4c00640 1 -- 172.21.15.146:0/536229926 wait complete. 2024-09-16T01:53:37.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.482+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f67900b3ec0 con 0x7f67900cb2f0 2024-09-16T01:53:37.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.483+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f59e415a0e0 con 0x7f59e415bc60 2024-09-16T01:53:37.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.483+0000 7f678cc00640 1 -- 172.21.15.146:0/1684513174 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f67900b3ec0 con 0x7f67900cb2f0 2024-09-16T01:53:37.487 INFO:teuthology.orchestra.run.smithi146.stdout:60129542204 2024-09-16T01:53:37.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.484+0000 7f59da000640 1 -- 172.21.15.146:0/2913240897 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f59dc05d400 con 0x7f59e415bc60 2024-09-16T01:53:37.487 INFO:teuthology.orchestra.run.smithi146.stdout:60129542204 2024-09-16T01:53:37.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509502 got 64424509501 for osd.2 2024-09-16T01:53:37.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f676c06c2b0 msgr2=0x7f676c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f676c06c2b0 0x7f676c06e770 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6790151ec0 tx=0x7f6778008040 comp rx=0 tx=0).stop 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 msgr2=0x7f679015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 0x7f679015cf90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f678002a2d0 tx=0x7f67800046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 shutdown_connections 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f676c06c2b0 0x7f676c06e770 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f679015e9d0 0x7f6790158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67900cc0c0 0x7f6790158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 --2- 172.21.15.146:0/1684513174 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67900cb2f0 0x7f679015cf90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 >> 172.21.15.146:0/1684513174 conn(0x7f679006d560 msgr2=0x7f67900ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 shutdown_connections 2024-09-16T01:53:37.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.488+0000 7f6797000640 1 -- 172.21.15.146:0/1684513174 wait complete. 2024-09-16T01:53:37.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59c406c2e0 msgr2=0x7f59c406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59c406c2e0 0x7f59c406e7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f59d4034000 tx=0x7f59d402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:37.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 msgr2=0x7f59e40c81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:37.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 0x7f59e40c81c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00cad0 tx=0x7f59dc00cf90 comp rx=0 tx=0).stop 2024-09-16T01:53:37.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 shutdown_connections 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f59c406c2e0 0x7f59c406e7a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f59e415bc60 0x7f59e40c81c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59e40c6290 0x7f59e414d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 --2- 172.21.15.146:0/2913240897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59e40c5eb0 0x7f59e414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 >> 172.21.15.146:0/2913240897 conn(0x7f59e4067b30 msgr2=0x7f59e40b4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 shutdown_connections 2024-09-16T01:53:37.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:37.489+0000 7f59e3200640 1 -- 172.21.15.146:0/2913240897 wait complete. 2024-09-16T01:53:37.493 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574910 got 55834574909 for osd.4 2024-09-16T01:53:37.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509503 got 64424509501 for osd.3 2024-09-16T01:53:37.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542206 got 60129542204 for osd.5 2024-09-16T01:53:37.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542206 got 60129542204 for osd.7 2024-09-16T01:53:37.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542206 got 60129542204 for osd.6 2024-09-16T01:53:37.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509502 got 64424509500 for osd.1 2024-09-16T01:53:38.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T01:53:38.490 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T01:53:38.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T01:53:38.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T01:53:38.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T01:53:38.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T01:53:38.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T01:53:38.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T01:53:38.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.735+0000 7f8f69200640 1 Processor -- start 2024-09-16T01:53:38.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.736+0000 7f8f69200640 1 -- start start 2024-09-16T01:53:38.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.738+0000 7f8f69200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f5c00ed90 0x7f8f5c049930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.738+0000 7f8f69200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c049e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.738+0000 7f8f69200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f5c0e6030 con 0x7f8f5c0dcce0 2024-09-16T01:53:38.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.739+0000 7f8f63e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c049e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.739+0000 7f8f63e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c049e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39502/0 (socket says 172.21.15.146:39502) 2024-09-16T01:53:38.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.739+0000 7f8f63e00640 1 -- 172.21.15.146:0/2248726082 learned_addr learned my addr 172.21.15.146:0/2248726082 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.739+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f8f5c048f40 con 0x7f8f5c00ed90 2024-09-16T01:53:38.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.739+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c049750 con 0x7f8f5c04fe40 2024-09-16T01:53:38.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.740+0000 7f8f62a00640 1 --2- 172.21.15.146:0/2248726082 >> v2:172.21.15.146:3300/0 conn(0x7f8f5c00ed90 0x7f8f5c049930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.740+0000 7f8f63e00640 1 -- 172.21.15.146:0/2248726082 >> v2:172.21.15.146:3300/0 conn(0x7f8f5c00ed90 msgr2=0x7f8f5c049930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.740+0000 7f8f63e00640 1 --2- 172.21.15.146:0/2248726082 >> v2:172.21.15.146:3300/0 conn(0x7f8f5c00ed90 0x7f8f5c049930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.740+0000 7f8f63e00640 1 -- 172.21.15.146:0/2248726082 >> v1:172.21.15.146:6789/0 conn(0x7f8f5c0dcce0 legacy=0x7f8f5c006380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.740+0000 7f8f63e00640 1 -- 172.21.15.146:0/2248726082 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c0504a0 con 0x7f8f5c04fe40 2024-09-16T01:53:38.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.742+0000 7f8f63e00640 1 --2- 172.21.15.146:0/2248726082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c049e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009870 tx=0x7f8f5002a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5d2279d7908c9e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.742+0000 7f8f62000640 1 -- 172.21.15.146:0/2248726082 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f50037070 con 0x7f8f5c04fe40 2024-09-16T01:53:38.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.743+0000 7f8f62000640 1 -- 172.21.15.146:0/2248726082 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f50032440 con 0x7f8f5c04fe40 2024-09-16T01:53:38.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.744+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 msgr2=0x7f8f5c049e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.744+0000 7f8f69200640 1 --2- 172.21.15.146:0/2248726082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c049e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009870 tx=0x7f8f5002a1d0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.745+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 shutdown_connections 2024-09-16T01:53:38.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.745+0000 7f8f69200640 1 --2- 172.21.15.146:0/2248726082 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c049e70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.745+0000 7f8f69200640 1 --2- 172.21.15.146:0/2248726082 >> v2:172.21.15.146:3300/0 conn(0x7f8f5c00ed90 0x7f8f5c049930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.745+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 >> 172.21.15.146:0/2248726082 conn(0x7f8f5c054470 msgr2=0x7f8f5c054880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.745+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 shutdown_connections 2024-09-16T01:53:38.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.746+0000 7f8f69200640 1 -- 172.21.15.146:0/2248726082 wait complete. 2024-09-16T01:53:38.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.749+0000 7f8f69200640 1 Processor -- start 2024-09-16T01:53:38.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.749+0000 7f8f69200640 1 -- start start 2024-09-16T01:53:38.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.751+0000 7f8f69200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 0x7f8f5c16c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.752+0000 7f8f63400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 0x7f8f5c16c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.752+0000 7f8f63400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 0x7f8f5c16c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47252/0 (socket says 172.21.15.146:47252) 2024-09-16T01:53:38.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.753+0000 7f8f63400640 1 -- 172.21.15.146:0/968877506 learned_addr learned my addr 172.21.15.146:0/968877506 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.753+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c0d7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.753+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f5c0dcce0 0x7f8f5c0d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.753+0000 7f8f62a00640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c0d7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.754+0000 7f8f63e00640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f5c0dcce0 0x7f8f5c0d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.754+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c049190 con 0x7f8f5c00ed90 2024-09-16T01:53:38.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.754+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c005ef0 con 0x7f8f5c0dcce0 2024-09-16T01:53:38.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.754+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c049470 con 0x7f8f5c04fe40 2024-09-16T01:53:38.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.755+0000 7f8f63400640 1 -- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 msgr2=0x7f8f5c0d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.755+0000 7f8f63400640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c0d7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.755+0000 7f8f63400640 1 -- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f5c0dcce0 msgr2=0x7f8f5c0d81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.755+0000 7f8f63400640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f5c0dcce0 0x7f8f5c0d81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.756+0000 7f8f63400640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c0d74a0 con 0x7f8f5c00ed90 2024-09-16T01:53:38.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.757+0000 7f8f63400640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 0x7f8f5c16c0a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8f64051500 tx=0x7f8f64069890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.757+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f64091070 con 0x7f8f5c00ed90 2024-09-16T01:53:38.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.758+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f64071840 con 0x7f8f5c00ed90 2024-09-16T01:53:38.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.758+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c04df00 con 0x7f8f5c00ed90 2024-09-16T01:53:38.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.758+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c0d7740 con 0x7f8f5c00ed90 2024-09-16T01:53:38.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.759+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f64071cd0 con 0x7f8f5c00ed90 2024-09-16T01:53:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.764+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f40005290 con 0x7f8f5c00ed90 2024-09-16T01:53:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.764+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8f64094070 con 0x7f8f5c00ed90 2024-09-16T01:53:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.766+0000 7f8f57400640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8f4406c2e0 0x7f8f4406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.766+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f8f64107790 con 0x7f8f5c00ed90 2024-09-16T01:53:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.769+0000 7f8f62a00640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8f4406c2e0 0x7f8f4406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.774+0000 7f8f62a00640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8f4406c2e0 0x7f8f4406e7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8f58006fd0 tx=0x7f8f58008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.777+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8f5c0e6030 con 0x7f8f5c00ed90 2024-09-16T01:53:38.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.801+0000 7fdff1c00640 1 Processor -- start 2024-09-16T01:53:38.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.801+0000 7fdff1c00640 1 -- start start 2024-09-16T01:53:38.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.803+0000 7fdff1c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c75a0 0x7fdfec15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.803+0000 7fdff1c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdfec0cc040 0x7fdfec0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.803+0000 7fdff1c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec06d140 con 0x7fdfec0c6b70 2024-09-16T01:53:38.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.803+0000 7fdff1c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fdfec15b880 con 0x7fdfec0cc040 2024-09-16T01:53:38.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.803+0000 7fdff1c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdfec0cab20 con 0x7fdfec0c75a0 2024-09-16T01:53:38.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.804+0000 7fdfebe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdfec0c6b70 0x7fdfec0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47194/0 (socket says 172.21.15.146:47194) 2024-09-16T01:53:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.804+0000 7fdfebe00640 1 -- 172.21.15.146:0/536226055 learned_addr learned my addr 172.21.15.146:0/536226055 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.804+0000 7fdfeb400640 1 --2- 172.21.15.146:0/536226055 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c75a0 0x7fdfec15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.805+0000 7fdff1200640 1 --2- 172.21.15.146:0/536226055 >> v2:172.21.15.146:3300/0 conn(0x7fdfec0cc040 0x7fdfec0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.805+0000 7fdfeb400640 1 -- 172.21.15.146:0/536226055 >> v2:172.21.15.146:3300/0 conn(0x7fdfec0cc040 msgr2=0x7fdfec0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.805+0000 7fdfeb400640 1 --2- 172.21.15.146:0/536226055 >> v2:172.21.15.146:3300/0 conn(0x7fdfec0cc040 0x7fdfec0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.805+0000 7fdfeb400640 1 -- 172.21.15.146:0/536226055 >> v1:172.21.15.146:6789/0 conn(0x7fdfec0c6b70 legacy=0x7fdfec0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.806+0000 7fdff1200640 1 --2- 172.21.15.146:0/536226055 >> v2:172.21.15.146:3300/0 conn(0x7fdfec0cc040 0x7fdfec0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:38.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.806+0000 7fdfeb400640 1 -- 172.21.15.146:0/536226055 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfec06d420 con 0x7fdfec0c75a0 2024-09-16T01:53:38.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.808+0000 7fdfeb400640 1 --2- 172.21.15.146:0/536226055 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c75a0 0x7fdfec15b340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009d90 tx=0x7fdfe402a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc485efdf5b48dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.808+0000 7fdfeaa00640 1 -- 172.21.15.146:0/536226055 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4007740 con 0x7fdfec0c75a0 2024-09-16T01:53:38.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.808+0000 7fdfeaa00640 1 -- 172.21.15.146:0/536226055 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe40078e0 con 0x7fdfec0c75a0 2024-09-16T01:53:38.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.808+0000 7fdfeaa00640 1 -- 172.21.15.146:0/536226055 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4007cb0 con 0x7fdfec0c75a0 2024-09-16T01:53:38.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7f4976200640 1 Processor -- start 2024-09-16T01:53:38.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7f4976200640 1 -- start start 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7fdff1c00640 1 -- 172.21.15.146:0/536226055 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c75a0 msgr2=0x7fdfec15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7fdff1c00640 1 --2- 172.21.15.146:0/536226055 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c75a0 0x7fdfec15b340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009d90 tx=0x7fdfe402a4d0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7fdff1c00640 1 -- 172.21.15.146:0/536226055 shutdown_connections 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7fdff1c00640 1 --2- 172.21.15.146:0/536226055 >> v2:172.21.15.146:3300/0 conn(0x7fdfec0cc040 0x7fdfec0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7fdff1c00640 1 --2- 172.21.15.146:0/536226055 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c75a0 0x7fdfec15b340 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009d90 tx=0x7fdfe402a4d0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.809+0000 7fdff1c00640 1 -- 172.21.15.146:0/536226055 >> 172.21.15.146:0/536226055 conn(0x7fdfec0ccc20 msgr2=0x7fdfec0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.810+0000 7fdff1c00640 1 -- 172.21.15.146:0/536226055 shutdown_connections 2024-09-16T01:53:38.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.810+0000 7fdff1c00640 1 -- 172.21.15.146:0/536226055 wait complete. 2024-09-16T01:53:38.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.813+0000 7fdff1c00640 1 Processor -- start 2024-09-16T01:53:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.811+0000 7f4976200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49700cc0c0 0x7f497015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.811+0000 7f4976200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49700cb2f0 0x7f49700cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.811+0000 7f4976200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f497015b1d0 con 0x7f497015e9d0 2024-09-16T01:53:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.811+0000 7f4976200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f49700cbd00 con 0x7f49700cb2f0 2024-09-16T01:53:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.811+0000 7f4976200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f497015ba70 con 0x7f49700cc0c0 2024-09-16T01:53:38.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.812+0000 7f496f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49700cc0c0 0x7f497015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.812+0000 7f496f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49700cc0c0 0x7f497015aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60206/0 (socket says 172.21.15.146:60206) 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.812+0000 7f496f400640 1 -- 172.21.15.146:0/3904907270 learned_addr learned my addr 172.21.15.146:0/3904907270 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.812+0000 7f496ea00640 1 --2- 172.21.15.146:0/3904907270 >> v2:172.21.15.146:3300/0 conn(0x7f49700cb2f0 0x7f49700cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.813+0000 7f496f400640 1 -- 172.21.15.146:0/3904907270 >> v2:172.21.15.146:3300/0 conn(0x7f49700cb2f0 msgr2=0x7f49700cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.813+0000 7f496f400640 1 --2- 172.21.15.146:0/3904907270 >> v2:172.21.15.146:3300/0 conn(0x7f49700cb2f0 0x7f49700cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.813+0000 7f496f400640 1 -- 172.21.15.146:0/3904907270 >> v1:172.21.15.161:6789/0 conn(0x7f497015e9d0 legacy=0x7f49700cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.813+0000 7f496f400640 1 -- 172.21.15.146:0/3904907270 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497015b4b0 con 0x7f49700cc0c0 2024-09-16T01:53:38.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.814+0000 7fdff1c00640 1 -- start start 2024-09-16T01:53:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.815+0000 7f496ea00640 1 --2- 172.21.15.146:0/3904907270 >> v2:172.21.15.146:3300/0 conn(0x7f49700cb2f0 0x7f49700cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.815+0000 7fdff1c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c6b70 0x7fdfec159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.815+0000 7fdff1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfec0cc040 0x7fdfec0c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.815+0000 7fdff1c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 0x7fdfec1fd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.815+0000 7f496f400640 1 --2- 172.21.15.146:0/3904907270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 0x7f497015aab0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f49700ce150 tx=0x7f496002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fbd950fb014a8d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdff1c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdfec0cb050 con 0x7fdfec0cc040 2024-09-16T01:53:38.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7f496e000640 1 -- 172.21.15.146:0/3904907270 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960037070 con 0x7f49700cc0c0 2024-09-16T01:53:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdff1c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdfec0cad70 con 0x7fdfec159da0 2024-09-16T01:53:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7f496e000640 1 -- 172.21.15.146:0/3904907270 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4960032640 con 0x7f49700cc0c0 2024-09-16T01:53:38.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7f496e000640 1 -- 172.21.15.146:0/3904907270 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960032a50 con 0x7f49700cc0c0 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdff1c00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdfec0cbc80 con 0x7fdfec0c6b70 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdff1200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 0x7fdfec1fd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdff1200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 0x7fdfec1fd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60212/0 (socket says 172.21.15.146:60212) 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdff1200640 1 -- 172.21.15.146:0/3272796595 learned_addr learned my addr 172.21.15.146:0/3272796595 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.816+0000 7fdfeb400640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfec0cc040 0x7fdfec0c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.817+0000 7fdff1200640 1 -- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c6b70 msgr2=0x7fdfec159860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:38.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.817+0000 7fdfebe00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c6b70 0x7fdfec159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.817+0000 7f4976200640 1 -- 172.21.15.146:0/3904907270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 msgr2=0x7f497015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.817+0000 7fa2e5400640 1 Processor -- start 2024-09-16T01:53:38.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.817+0000 7fdff1200640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c6b70 0x7fdfec159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fdff1200640 1 -- 172.21.15.146:0/3272796595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfec0cc040 msgr2=0x7fdfec0c19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fdff1200640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfec0cc040 0x7fdfec0c19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fdff1200640 1 -- 172.21.15.146:0/3272796595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfec1feb90 con 0x7fdfec159da0 2024-09-16T01:53:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.817+0000 7f4976200640 1 --2- 172.21.15.146:0/3904907270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 0x7f497015aab0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f49700ce150 tx=0x7f496002a2c0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fa2e5400640 1 -- start start 2024-09-16T01:53:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fdfeb400640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfec0cc040 0x7fdfec0c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:38.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fdfebe00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c6b70 0x7fdfec159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f4976200640 1 -- 172.21.15.146:0/3904907270 shutdown_connections 2024-09-16T01:53:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f4976200640 1 --2- 172.21.15.146:0/3904907270 >> v2:172.21.15.146:3300/0 conn(0x7f49700cb2f0 0x7f49700cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f4976200640 1 --2- 172.21.15.146:0/3904907270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 0x7f497015aab0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f4976200640 1 -- 172.21.15.146:0/3904907270 >> 172.21.15.146:0/3904907270 conn(0x7f497006d560 msgr2=0x7f497006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f4976200640 1 -- 172.21.15.146:0/3904907270 shutdown_connections 2024-09-16T01:53:38.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7fdff1200640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 0x7fdfec1fd720 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00c850 tx=0x7fdfdc00cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f2128a00640 1 Processor -- start 2024-09-16T01:53:38.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.818+0000 7f4976200640 1 -- 172.21.15.146:0/3904907270 wait complete. 2024-09-16T01:53:38.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.819+0000 7f2128a00640 1 -- start start 2024-09-16T01:53:38.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.819+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc015070 con 0x7fdfec159da0 2024-09-16T01:53:38.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.819+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfdc00a8c0 con 0x7fdfec159da0 2024-09-16T01:53:38.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.819+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc00ac90 con 0x7fdfec159da0 2024-09-16T01:53:38.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.819+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfec0c5ee0 con 0x7fdfec159da0 2024-09-16T01:53:38.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.819+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfec1fe3c0 con 0x7fdfec159da0 2024-09-16T01:53:38.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.820+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfec06cd20 con 0x7fdfec159da0 2024-09-16T01:53:38.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7fa2e5400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa2e00cb2f0 0x7fa2e015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.820+0000 7efcef400640 1 Processor -- start 2024-09-16T01:53:38.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7efcef400640 1 -- start start 2024-09-16T01:53:38.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7f4976200640 1 Processor -- start 2024-09-16T01:53:38.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7fa2e5400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e015b1d0 con 0x7fa2e00cc0c0 2024-09-16T01:53:38.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7f2128a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f212415e040 0x7f21240c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7f2128a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 0x7f21240cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7f2128a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f212406c950 con 0x7f2124154d00 2024-09-16T01:53:38.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7f2128a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f21240707e0 con 0x7f212415e040 2024-09-16T01:53:38.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7f2128a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2124070a60 con 0x7f2124071010 2024-09-16T01:53:38.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.821+0000 7fa2de800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa2e00cc0c0 0x7fa2e015e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47202/0 (socket says 172.21.15.146:47202) 2024-09-16T01:53:38.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7fa2e5400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e015b4b0 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7fa2e5400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e00cbd00 con 0x7fa2e00cb2f0 2024-09-16T01:53:38.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7fa2df200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa2e00cc4a0 0x7fa2e0165430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:44714/0 (socket says 172.21.15.146:44714) 2024-09-16T01:53:38.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7fa2df200640 1 -- 172.21.15.146:0/1607065145 learned_addr learned my addr 172.21.15.146:0/1607065145 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:38.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7fa2dde00640 1 --2- 172.21.15.146:0/1607065145 >> v2:172.21.15.146:3300/0 conn(0x7fa2e00cb2f0 0x7fa2e015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7f2122800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 0x7f21240cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7f2122800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 0x7f21240cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39528/0 (socket says 172.21.15.146:39528) 2024-09-16T01:53:38.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7f2122800640 1 -- 172.21.15.146:0/1230236278 learned_addr learned my addr 172.21.15.146:0/1230236278 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.822+0000 7f4976200640 1 -- start start 2024-09-16T01:53:38.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fa2dd400640 1 -- 172.21.15.146:0/1607065145 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136401034 0 0) 0x7fa2e015b4b0 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fa2dd400640 1 -- 172.21.15.146:0/1607065145 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2c4003520 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7f2121400640 1 --2- 172.21.15.146:0/1230236278 >> v2:172.21.15.146:3300/0 conn(0x7f212415e040 0x7f21240c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fa2dde00640 1 -- 172.21.15.146:0/1607065145 >> v1:172.21.15.161:6789/0 conn(0x7fa2e00cc4a0 legacy=0x7fa2e0165430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fa2dde00640 1 -- 172.21.15.146:0/1607065145 >> v1:172.21.15.146:6789/0 conn(0x7fa2e00cc0c0 legacy=0x7fa2e015e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fa2dde00640 1 -- 172.21.15.146:0/1607065145 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e015b790 con 0x7fa2e00cb2f0 2024-09-16T01:53:38.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fc3a3200640 1 Processor -- start 2024-09-16T01:53:38.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7fc3a3200640 1 -- start start 2024-09-16T01:53:38.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.823+0000 7f2121400640 1 -- 172.21.15.146:0/1230236278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 msgr2=0x7f21240cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7efcef400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efce8158f40 0x7efce815ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7efcef400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efce815f430 con 0x7efce80c8b10 2024-09-16T01:53:38.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7f2121400640 1 --2- 172.21.15.146:0/1230236278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 0x7f21240cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7f2121400640 1 -- 172.21.15.146:0/1230236278 >> v1:172.21.15.146:6789/0 conn(0x7f2124154d00 legacy=0x7f21240c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7f2121400640 1 -- 172.21.15.146:0/1230236278 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212406c390 con 0x7f212415e040 2024-09-16T01:53:38.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7efcef400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efce815f710 con 0x7efce8159320 2024-09-16T01:53:38.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7efcef400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7efce8162770 con 0x7efce8158f40 2024-09-16T01:53:38.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7f2122800640 1 --2- 172.21.15.146:0/1230236278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 0x7f21240cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:38.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.824+0000 7efced400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efce80c8b10 0x7efce80c7430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47230/0 (socket says 172.21.15.146:47230) 2024-09-16T01:53:38.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7efced400640 1 -- 172.21.15.146:0/2970081708 learned_addr learned my addr 172.21.15.146:0/2970081708 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:38.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7efceca00640 1 --2- 172.21.15.146:0/2970081708 >> v2:172.21.15.146:3300/0 conn(0x7efce8158f40 0x7efce815ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7f4976200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 0x7f49701503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7efce7400640 1 -- 172.21.15.146:0/2970081708 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1342057359 0 0) 0x7efce815f430 con 0x7efce80c8b10 2024-09-16T01:53:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7fa2dde00640 1 --2- 172.21.15.146:0/1607065145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e015aab0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0097a0 tx=0x7fa2cc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e576f7cab533fd8b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7fc3a3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 0x7fc39c0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fc3a3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c15af00 con 0x7fc39c0c5eb0 2024-09-16T01:53:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fc3a3200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc39c15b1e0 con 0x7fc39c0c6290 2024-09-16T01:53:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fc3a3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c06dbd0 con 0x7fc39c15aa70 2024-09-16T01:53:38.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7f2121400640 1 --2- 172.21.15.146:0/1230236278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212415e040 0x7f21240c7bc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f21140097a0 tx=0x7f211402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e3668fb9396cf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fc3a1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 0x7fc39c0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdfdc018070 con 0x7fdfec159da0 2024-09-16T01:53:38.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.825+0000 7efce7400640 1 -- 172.21.15.146:0/2970081708 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcd0003520 con 0x7efce80c8b10 2024-09-16T01:53:38.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7efceca00640 1 -- 172.21.15.146:0/2970081708 >> v1:172.21.15.161:6789/0 conn(0x7efce8159320 legacy=0x7efce8162c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7efce7400640 1 -- 172.21.15.146:0/2970081708 <== mon.0 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1336846515 0 0) 0x7efcd0003520 con 0x7efce80c8b10 2024-09-16T01:53:38.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7efceca00640 1 -- 172.21.15.146:0/2970081708 >> v1:172.21.15.146:6789/0 conn(0x7efce80c8b10 legacy=0x7efce80c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7efceca00640 1 -- 172.21.15.146:0/2970081708 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce815f9f0 con 0x7efce8158f40 2024-09-16T01:53:38.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7f4976200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 0x7f49700b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fc3a1400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 0x7fc39c0cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39536/0 (socket says 172.21.15.146:39536) 2024-09-16T01:53:38.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7fc3a1400640 1 -- 172.21.15.146:0/4240268253 learned_addr learned my addr 172.21.15.146:0/4240268253 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f4976200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f497015e9d0 0x7f4970151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.826+0000 7f2120a00640 1 -- 172.21.15.146:0/1230236278 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2114037070 con 0x7f212415e040 2024-09-16T01:53:38.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f2120a00640 1 -- 172.21.15.146:0/1230236278 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2114032440 con 0x7f212415e040 2024-09-16T01:53:38.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f2120a00640 1 -- 172.21.15.146:0/1230236278 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2114032810 con 0x7f212415e040 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 0x7f49701503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f4976200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f497015e1d0 con 0x7f49700cb2f0 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f4976200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f497015e540 con 0x7f49700cc0c0 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f4976200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f49700c5560 con 0x7f497015e9d0 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 0x7f49701503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47312/0 (socket says 172.21.15.146:47312) 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 -- 172.21.15.146:0/1845171462 learned_addr learned my addr 172.21.15.146:0/1845171462 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7fc3a1400640 1 -- 172.21.15.146:0/4240268253 >> v1:172.21.15.161:6789/0 conn(0x7fc39c0c6290 legacy=0x7fc39c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7fa2dd400640 1 -- 172.21.15.146:0/1607065145 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc037070 con 0x7fa2e00cb2f0 2024-09-16T01:53:38.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7fa2dd400640 1 -- 172.21.15.146:0/1607065145 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2cc032440 con 0x7fa2e00cb2f0 2024-09-16T01:53:38.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7fa2dd400640 1 -- 172.21.15.146:0/1607065145 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc032850 con 0x7fa2e00cb2f0 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 -- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f497015e9d0 msgr2=0x7f4970151690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f497015e9d0 0x7f4970151690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 -- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 msgr2=0x7f49700b6050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 0x7f49700b6050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.827+0000 7f496f400640 1 -- 172.21.15.146:0/1845171462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49701f90a0 con 0x7f49700cb2f0 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f2128a00640 1 -- 172.21.15.146:0/1230236278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212415e040 msgr2=0x7f21240c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f2128a00640 1 --2- 172.21.15.146:0/1230236278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212415e040 0x7f21240c7bc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f21140097a0 tx=0x7f211402a180 comp rx=0 tx=0).stop 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f2128a00640 1 -- 172.21.15.146:0/1230236278 shutdown_connections 2024-09-16T01:53:38.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f2128a00640 1 --2- 172.21.15.146:0/1230236278 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2124071010 0x7f21240cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f2128a00640 1 --2- 172.21.15.146:0/1230236278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212415e040 0x7f21240c7bc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f2128a00640 1 -- 172.21.15.146:0/1230236278 >> 172.21.15.146:0/1230236278 conn(0x7f2124067950 msgr2=0x7f2124067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7efceca00640 1 --2- 172.21.15.146:0/2970081708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce8158f40 0x7efce815ee10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0097a0 tx=0x7efcdc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a40bde037e32e2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f51fb200640 1 Processor -- start 2024-09-16T01:53:38.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7f496f400640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 0x7f49701503c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4960000c00 tx=0x7f49600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7efce7400640 1 -- 172.21.15.146:0/2970081708 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc037070 con 0x7efce8158f40 2024-09-16T01:53:38.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7efce7400640 1 -- 172.21.15.146:0/2970081708 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcdc02ace0 con 0x7efce8158f40 2024-09-16T01:53:38.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fa2e5400640 1 -- 172.21.15.146:0/1607065145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cb2f0 msgr2=0x7fa2e015aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fa2e5400640 1 --2- 172.21.15.146:0/1607065145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e015aab0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0097a0 tx=0x7fa2cc02a250 comp rx=0 tx=0).stop 2024-09-16T01:53:38.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7efce7400640 1 -- 172.21.15.146:0/2970081708 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc037070 con 0x7efce8158f40 2024-09-16T01:53:38.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960037040 con 0x7f49700cb2f0 2024-09-16T01:53:38.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4960042070 con 0x7f49700cb2f0 2024-09-16T01:53:38.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960030990 con 0x7f49700cb2f0 2024-09-16T01:53:38.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7fc3a1400640 1 -- 172.21.15.146:0/4240268253 >> v1:172.21.15.146:6789/0 conn(0x7fc39c0c5eb0 legacy=0x7fc39c154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.828+0000 7fc3a1400640 1 -- 172.21.15.146:0/4240268253 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c15b4c0 con 0x7fc39c15aa70 2024-09-16T01:53:38.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fa2e5400640 1 -- 172.21.15.146:0/1607065145 shutdown_connections 2024-09-16T01:53:38.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fa2e5400640 1 --2- 172.21.15.146:0/1607065145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e015aab0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49700ca150 con 0x7f49700cb2f0 2024-09-16T01:53:38.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49700ca6f0 con 0x7f49700cb2f0 2024-09-16T01:53:38.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fa2e5400640 1 -- 172.21.15.146:0/1607065145 >> 172.21.15.146:0/1607065145 conn(0x7fa2e006d560 msgr2=0x7fa2e006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f2128a00640 1 -- 172.21.15.146:0/1230236278 shutdown_connections 2024-09-16T01:53:38.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f51fb200640 1 -- start start 2024-09-16T01:53:38.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fdfe3e00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfcc06c380 0x7fdfcc06e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fdfdc08b3e0 con 0x7fdfec159da0 2024-09-16T01:53:38.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7fc3a1400640 1 --2- 172.21.15.146:0/4240268253 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 0x7fc39c0cac30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0098a0 tx=0x7fc38c02a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37b7ad3b18c00dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7fa2e5400640 1 -- 172.21.15.146:0/1607065145 shutdown_connections 2024-09-16T01:53:38.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.829+0000 7f2128a00640 1 -- 172.21.15.146:0/1230236278 wait complete. 2024-09-16T01:53:38.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7efcef400640 1 -- 172.21.15.146:0/2970081708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce8158f40 msgr2=0x7efce815ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7efcef400640 1 --2- 172.21.15.146:0/2970081708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce8158f40 0x7efce815ee10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0097a0 tx=0x7efcdc02a250 comp rx=0 tx=0).stop 2024-09-16T01:53:38.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7fa2e5400640 1 -- 172.21.15.146:0/1607065145 wait complete. 2024-09-16T01:53:38.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7efcef400640 1 -- 172.21.15.146:0/2970081708 shutdown_connections 2024-09-16T01:53:38.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7efcef400640 1 --2- 172.21.15.146:0/2970081708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce8158f40 0x7efce815ee10 secure :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0097a0 tx=0x7efcdc02a250 comp rx=0 tx=0).stop 2024-09-16T01:53:38.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7efcef400640 1 -- 172.21.15.146:0/2970081708 >> 172.21.15.146:0/2970081708 conn(0x7efce806f3c0 msgr2=0x7efce806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7efcef400640 1 -- 172.21.15.146:0/2970081708 shutdown_connections 2024-09-16T01:53:38.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7fc3a0a00640 1 -- 172.21.15.146:0/4240268253 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c037070 con 0x7fc39c15aa70 2024-09-16T01:53:38.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7fc3a0a00640 1 -- 172.21.15.146:0/4240268253 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc38c032640 con 0x7fc39c15aa70 2024-09-16T01:53:38.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.830+0000 7fc3a0a00640 1 -- 172.21.15.146:0/4240268253 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c032a10 con 0x7fc39c15aa70 2024-09-16T01:53:38.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7f51fb200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 0x7f51fc15bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7f51fb200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51fc15c050 0x7f51fc0c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f51fb200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51fc158630 con 0x7f51fc06c4f0 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f51fb200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f51fc158f20 con 0x7f51fc15c050 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f51fb200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f51fc073f00 con 0x7f51fc14f1d0 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7fdfebe00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfcc06c380 0x7fdfcc06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdfdc0594d0 con 0x7fdfec159da0 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7fc3a3200640 1 -- 172.21.15.146:0/4240268253 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 msgr2=0x7fc39c0cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7fc3a3200640 1 --2- 172.21.15.146:0/4240268253 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 0x7fc39c0cac30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0098a0 tx=0x7fc38c02a330 comp rx=0 tx=0).stop 2024-09-16T01:53:38.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f2128a00640 1 Processor -- start 2024-09-16T01:53:38.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f51f9e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 0x7f51fc15bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f51f9e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 0x7f51fc15bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39548/0 (socket says 172.21.15.146:39548) 2024-09-16T01:53:38.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7f51f9e00640 1 -- 172.21.15.146:0/980503745 learned_addr learned my addr 172.21.15.146:0/980503745 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7efcef400640 1 -- 172.21.15.146:0/2970081708 wait complete. 2024-09-16T01:53:38.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f2128a00640 1 -- start start 2024-09-16T01:53:38.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7fc3a3200640 1 -- 172.21.15.146:0/4240268253 shutdown_connections 2024-09-16T01:53:38.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7fc3a3200640 1 --2- 172.21.15.146:0/4240268253 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c15aa70 0x7fc39c0cac30 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc38c0098a0 tx=0x7fc38c02a330 comp rx=0 tx=0).stop 2024-09-16T01:53:38.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.832+0000 7fc3a3200640 1 -- 172.21.15.146:0/4240268253 >> 172.21.15.146:0/4240268253 conn(0x7fc39c067b30 msgr2=0x7fc39c067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f9400640 1 --2- 172.21.15.146:0/980503745 >> v2:172.21.15.146:3300/0 conn(0x7f51fc15c050 0x7f51fc0c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f8a00640 1 -- 172.21.15.146:0/980503745 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639862917 0 0) 0x7f51fc158630 con 0x7f51fc06c4f0 2024-09-16T01:53:38.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7fa2e5400640 1 Processor -- start 2024-09-16T01:53:38.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.831+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49701586f0 con 0x7f49700cb2f0 2024-09-16T01:53:38.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7fdfebe00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfcc06c380 0x7fdfcc06e840 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8000f60 tx=0x7fdfd8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7fa2e5400640 1 -- start start 2024-09-16T01:53:38.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f8a00640 1 -- 172.21.15.146:0/980503745 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51e0003520 con 0x7f51fc06c4f0 2024-09-16T01:53:38.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f9e00640 1 -- 172.21.15.146:0/980503745 >> v2:172.21.15.146:3300/0 conn(0x7f51fc15c050 msgr2=0x7f51fc0c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:38.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f9e00640 1 --2- 172.21.15.146:0/980503745 >> v2:172.21.15.146:3300/0 conn(0x7f51fc15c050 0x7f51fc0c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f9e00640 1 -- 172.21.15.146:0/980503745 >> v1:172.21.15.161:6789/0 conn(0x7f51fc06c4f0 legacy=0x7f51fc0cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.833+0000 7f51f9e00640 1 -- 172.21.15.146:0/980503745 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc1593b0 con 0x7f51fc14f1d0 2024-09-16T01:53:38.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.834+0000 7fc3a3200640 1 -- 172.21.15.146:0/4240268253 shutdown_connections 2024-09-16T01:53:38.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.834+0000 7fc3a3200640 1 -- 172.21.15.146:0/4240268253 wait complete. 2024-09-16T01:53:38.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.834+0000 7efcef400640 1 Processor -- start 2024-09-16T01:53:38.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7f2128a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2124071010 0x7f21241e4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7efcef400640 1 -- start start 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7f2128a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 0x7f21241e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7f2128a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 0x7f21240c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7fa2e5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e0151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7fa2e5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cc0c0 0x7fa2e0150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7fa2e5400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa2e00cc4a0 0x7fa2e00c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7fa2e5400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e015e540 con 0x7fa2e00cc0c0 2024-09-16T01:53:38.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7fa2e5400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e015ba70 con 0x7fa2e00cb2f0 2024-09-16T01:53:38.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7fa2e5400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e00cc8b0 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4960030b30 con 0x7f49700cb2f0 2024-09-16T01:53:38.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7fa2de800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e0151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7fa2de800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e0151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60222/0 (socket says 172.21.15.146:60222) 2024-09-16T01:53:38.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7fa2de800640 1 -- 172.21.15.146:0/395772286 learned_addr learned my addr 172.21.15.146:0/395772286 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.835+0000 7f51f9e00640 1 --2- 172.21.15.146:0/980503745 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 0x7f51fc15bb10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009870 tx=0x7f51ec02e620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40c7f75d0d9b485 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f51f8a00640 1 -- 172.21.15.146:0/980503745 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec03a070 con 0x7f51fc14f1d0 2024-09-16T01:53:38.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f51f8a00640 1 -- 172.21.15.146:0/980503745 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec033510 con 0x7f51fc14f1d0 2024-09-16T01:53:38.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f51f8a00640 1 -- 172.21.15.146:0/980503745 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0338e0 con 0x7f51fc14f1d0 2024-09-16T01:53:38.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f2122800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 0x7f21240c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7fa2df200640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa2e00cc4a0 0x7fa2e00c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f2122800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 0x7f21240c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39570/0 (socket says 172.21.15.146:39570) 2024-09-16T01:53:38.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.836+0000 7f2121400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 0x7f21241e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fa2df200640 1 -- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2e00cb2f0 msgr2=0x7fa2e0151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fa2df200640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e0151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fa2df200640 1 -- 172.21.15.146:0/395772286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cc0c0 msgr2=0x7fa2e0150720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:38.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f2121400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 0x7f21241e5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60234/0 (socket says 172.21.15.146:60234) 2024-09-16T01:53:38.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efcef400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efce80c8b10 0x7efce81f8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efcef400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efce8159320 0x7efce81618d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efcef400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 0x7efce81531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f2122800640 1 -- 172.21.15.146:0/3084177054 learned_addr learned my addr 172.21.15.146:0/3084177054 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f51fb200640 1 -- 172.21.15.146:0/980503745 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 msgr2=0x7f51fc15bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f51fb200640 1 --2- 172.21.15.146:0/980503745 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 0x7f51fc15bb10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009870 tx=0x7f51ec02e620 comp rx=0 tx=0).stop 2024-09-16T01:53:38.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efcef400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efce8163310 con 0x7efce815daa0 2024-09-16T01:53:38.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f21240c75a0 con 0x7f2124071010 2024-09-16T01:53:38.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fa2dde00640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cc0c0 0x7fa2e0150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efcef400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efce8162400 con 0x7efce8159320 2024-09-16T01:53:38.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efcef400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efce8072070 con 0x7efce80c8b10 2024-09-16T01:53:38.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fc3a3200640 1 Processor -- start 2024-09-16T01:53:38.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efced400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 0x7efce81531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efced400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 0x7efce81531e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47338/0 (socket says 172.21.15.146:47338) 2024-09-16T01:53:38.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7efced400640 1 -- 172.21.15.146:0/2926722518 learned_addr learned my addr 172.21.15.146:0/2926722518 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fa2df200640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cc0c0 0x7fa2e0150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7fa2df200640 1 -- 172.21.15.146:0/395772286 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e00c7e40 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2124070d40 con 0x7f2124154d00 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f2121e00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2124071010 0x7f21241e4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f51fb200640 1 -- 172.21.15.146:0/980503745 shutdown_connections 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f51fb200640 1 --2- 172.21.15.146:0/980503745 >> v2:172.21.15.146:3300/0 conn(0x7f51fc15c050 0x7f51fc0c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f51fb200640 1 --2- 172.21.15.146:0/980503745 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc14f1d0 0x7f51fc15bb10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f51fb200640 1 -- 172.21.15.146:0/980503745 >> 172.21.15.146:0/980503745 conn(0x7f51fc067950 msgr2=0x7f51fc067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:38.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7fa2dde00640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cc0c0 0x7fa2e0150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:38.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7f51fb200640 1 -- 172.21.15.146:0/980503745 shutdown_connections 2024-09-16T01:53:38.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7fa2df200640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa2e00cc4a0 0x7fa2e00c8080 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800cae0 tx=0x7fa2d800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8017070 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7efceca00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efce80c8b10 0x7efce81f8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d800ac40 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.839+0000 7efceca00640 1 -- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efce8159320 msgr2=0x7efce81618d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:53:38.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f21241550e0 con 0x7f212415e040 2024-09-16T01:53:38.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7f51fb200640 1 -- 172.21.15.146:0/980503745 wait complete. 2024-09-16T01:53:38.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e00c7b60 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.839+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e01f1380 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.838+0000 7fc3a3200640 1 -- start start 2024-09-16T01:53:38.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.839+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8012400 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f2122800640 1 -- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 msgr2=0x7f21241e5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f2122800640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 0x7f21241e5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f2122800640 1 -- 172.21.15.146:0/3084177054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2124071010 msgr2=0x7f21241e4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f2122800640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2124071010 0x7f21241e4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f51fb200640 1 Processor -- start 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f51fb200640 1 -- start start 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7efceca00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efce8159320 0x7efce81618d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7efceca00640 1 -- 172.21.15.146:0/2926722518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 msgr2=0x7efce81531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7efceca00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 0x7efce81531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7efceca00640 1 -- 172.21.15.146:0/2926722518 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce81f9310 con 0x7efce80c8b10 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7efced400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 0x7efce81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:38.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7efceca00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efce80c8b10 0x7efce81f8dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efcdc031000 tx=0x7efcdc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7f2122800640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21241f8d90 con 0x7f212415e040 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7f2121400640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 0x7f21241e5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7f2121e00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2124071010 0x7f21241e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.840+0000 7fc3a3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc39c0c5eb0 0x7fc39c1f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7fc3a3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc39c0c6290 0x7fc39c1e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7fc3a3200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c153e90 0x7fc39c1e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7fc3a3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c155020 con 0x7fc39c0c6290 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7fc3a3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c06c370 con 0x7fc39c0c5eb0 2024-09-16T01:53:38.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.841+0000 7fc3a3200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c1551a0 con 0x7fc39c153e90 2024-09-16T01:53:38.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f4967400640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f495006c2e0 0x7f495006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.837+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f49600abca0 con 0x7f49700cb2f0 2024-09-16T01:53:38.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7fc3a1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc39c0c5eb0 0x7fc39c1f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc037070 con 0x7efce80c8b10 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7f51fb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51fc06c4f0 0x7f51fc1ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7fc3a1e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc39c0c5eb0 0x7fc39c1f4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60246/0 (socket says 172.21.15.146:60246) 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7fc3a1e00640 1 -- 172.21.15.146:0/2555026919 learned_addr learned my addr 172.21.15.146:0/2555026919 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7f2122800640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 0x7f21240c6010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f210c00e930 tx=0x7f210c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c017070 con 0x7f212415e040 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210c00ca00 con 0x7f212415e040 2024-09-16T01:53:38.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c00cdd0 con 0x7f212415e040 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fb200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51fc14f1d0 0x7f51fc0c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fb200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 0x7f51fc0c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc0741c0 con 0x7f51fc06c4f0 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fb200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc158bb0 con 0x7f51fc14f1d0 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fb200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f51fc152b30 con 0x7f51fc15c050 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fa800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 0x7f51fc0c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fa800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 0x7f51fc0c4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:39608/0 (socket says 172.21.15.146:39608) 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51fa800640 1 -- 172.21.15.146:0/1380726891 learned_addr learned my addr 172.21.15.146:0/1380726891 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51f9400640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51fc14f1d0 0x7f51fc0c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7f51f9e00640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51fc06c4f0 0x7f51fc1ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7f51fa800640 1 -- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51fc14f1d0 msgr2=0x7f51fc0c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7f51fa800640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51fc14f1d0 0x7f51fc0c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7f51fa800640 1 -- 172.21.15.146:0/1380726891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51fc06c4f0 msgr2=0x7f51fc1ec840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7fc3a1400640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc39c0c6290 0x7fc39c1e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7fc3a2800640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c153e90 0x7fc39c1e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7fc3a2800640 1 -- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc39c0c5eb0 msgr2=0x7fc39c1f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7fc3a2800640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc39c0c5eb0 0x7fc39c1f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.843+0000 7fc3a2800640 1 -- 172.21.15.146:0/2555026919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc39c0c6290 msgr2=0x7fc39c1e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b8005290 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7f51fa800640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51fc06c4f0 0x7f51fc1ec840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7f51fa800640 1 -- 172.21.15.146:0/1380726891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc0c3ff0 con 0x7f51fc15c050 2024-09-16T01:53:38.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7f51f9e00640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51fc06c4f0 0x7f51fc1ec840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2124152b50 con 0x7f212415e040 2024-09-16T01:53:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21241f9070 con 0x7f212415e040 2024-09-16T01:53:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7f496ea00640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f495006c2e0 0x7f495006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f49701586f0 con 0x7f49700cb2f0 2024-09-16T01:53:38.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa2d801a070 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcdc03c620 con 0x7efce80c8b10 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.842+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce8159b30 con 0x7efce80c8b10 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce00019f0 con 0x7efce80c8b10 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.844+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efce815a0f0 con 0x7efce80c8b10 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.846+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efcdc048060 con 0x7efce80c8b10 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.846+0000 7f51fa800640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 0x7f51fc0c4a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f51f400c700 tx=0x7f51f400cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7fc3a2800640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc39c0c6290 0x7fc39c1e4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:38.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7fc3a2800640 1 -- 172.21.15.146:0/2555026919 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c1e6560 con 0x7fc39c153e90 2024-09-16T01:53:38.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7fc3a1400640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc39c0c6290 0x7fc39c1e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:53:38.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.845+0000 7fc3a2800640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c153e90 0x7fc39c1e53c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc39800c730 tx=0x7fc39800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.846+0000 7f496ea00640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f495006c2e0 0x7f495006e7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f49700cb1a0 tx=0x7f495c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.846+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce815a330 con 0x7efce80c8b10 2024-09-16T01:53:38.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.846+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2124153070 con 0x7f212415e040 2024-09-16T01:53:38.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.847+0000 7fa2d6a00640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa2c006c2b0 0x7fa2c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.847+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fa2d808b070 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.847+0000 7efce4c00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efcc806c330 0x7efcc806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.847+0000 7fa2de800640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa2c006c2b0 0x7fa2c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.847+0000 7efce7e00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efcc806c330 0x7efcc806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.848+0000 7fa2de800640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa2c006c2b0 0x7fa2c006e770 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0004de0 tx=0x7fa2d0007f10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.848+0000 7efce7e00640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efcc806c330 0x7efcc806e7f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efce81f2ad0 tx=0x7efcd40079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4003da0 con 0x7f51fc15c050 2024-09-16T01:53:38.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa2d80590e0 con 0x7fa2e00cc4a0 2024-09-16T01:53:38.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398016020 con 0x7fc39c153e90 2024-09-16T01:53:38.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7efcdc0a8780 con 0x7efce80c8b10 2024-09-16T01:53:38.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c1f4b20 con 0x7fc39c153e90 2024-09-16T01:53:38.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39c151f90 con 0x7fc39c153e90 2024-09-16T01:53:38.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.850+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51fc0c4f60 con 0x7f51fc15c050 2024-09-16T01:53:38.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.851+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f51fc1f0070 con 0x7f51fc15c050 2024-09-16T01:53:38.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.852+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39c0c56f0 con 0x7fc39c153e90 2024-09-16T01:53:38.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.852+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51fc0c6320 con 0x7f51fc15c050 2024-09-16T01:53:38.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.852+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f210c004020 con 0x7f212415e040 2024-09-16T01:53:38.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7f211a000640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f210006c300 0x7f210006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f210c08b740 con 0x7f212415e040 2024-09-16T01:53:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7f2121e00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f210006c300 0x7f210006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc398003970 con 0x7fc39c153e90 2024-09-16T01:53:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398003d20 con 0x7fc39c153e90 2024-09-16T01:53:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f400ce10 con 0x7f51fc15c050 2024-09-16T01:53:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.854+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4005270 con 0x7f51fc15c050 2024-09-16T01:53:38.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.855+0000 7f2121e00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f210006c300 0x7f210006e7c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f211c0012e0 tx=0x7f211c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.857+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efce815a330 con 0x7efce80c8b10 2024-09-16T01:53:38.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.859+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2124153070 con 0x7f212415e040 2024-09-16T01:53:38.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.862+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f51f4005410 con 0x7f51fc15c050 2024-09-16T01:53:38.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.862+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc3980050d0 con 0x7fc39c153e90 2024-09-16T01:53:38.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.864+0000 7f51f2000640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f51dc06c2b0 0x7f51dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.864+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f51f408be90 con 0x7f51fc15c050 2024-09-16T01:53:38.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.864+0000 7fc396000640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc38006c2e0 0x7fc38006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:38.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.864+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7fc39808c060 con 0x7fc39c153e90 2024-09-16T01:53:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.865+0000 7f51f9e00640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f51dc06c2b0 0x7f51dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.865+0000 7fc3a1e00640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc38006c2e0 0x7fc38006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.865+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc398020020 con 0x7fc39c153e90 2024-09-16T01:53:38.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.866+0000 7f51f9e00640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f51dc06c2b0 0x7f51dc06e770 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f51ec037000 tx=0x7f51ec008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.866+0000 7fc3a1e00640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc38006c2e0 0x7fc38006e7a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc390009990 tx=0x7fc390009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:38.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:38.869+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f51fc0c4f60 con 0x7f51fc15c050 2024-09-16T01:53:39.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.176+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8f40005580 con 0x7f8f5c00ed90 2024-09-16T01:53:39.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.177+0000 7f8f57400640 1 -- 172.21.15.146:0/968877506 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f640d5e00 con 0x7f8f5c00ed90 2024-09-16T01:53:39.179 INFO:teuthology.orchestra.run.smithi146.stdout:64424509503 2024-09-16T01:53:39.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.181+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8f4406c2e0 msgr2=0x7f8f4406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.181+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8f4406c2e0 0x7f8f4406e7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8f58006fd0 tx=0x7f8f58008040 comp rx=0 tx=0).stop 2024-09-16T01:53:39.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.182+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 msgr2=0x7f8f5c16c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.182+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 0x7f8f5c16c0a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8f64051500 tx=0x7f8f64069890 comp rx=0 tx=0).stop 2024-09-16T01:53:39.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.182+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 shutdown_connections 2024-09-16T01:53:39.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.182+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f8f4406c2e0 0x7f8f4406e7a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.183+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8f5c0dcce0 0x7f8f5c0d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.183+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8f5c04fe40 0x7f8f5c0d7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.183+0000 7f8f69200640 1 --2- 172.21.15.146:0/968877506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f5c00ed90 0x7f8f5c16c0a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.183+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 >> 172.21.15.146:0/968877506 conn(0x7f8f5c054470 msgr2=0x7f8f5c049900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.183+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 shutdown_connections 2024-09-16T01:53:39.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.183+0000 7f8f69200640 1 -- 172.21.15.146:0/968877506 wait complete. 2024-09-16T01:53:39.211 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509503 got 64424509503 for osd.0 2024-09-16T01:53:39.211 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.236+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f21240c7bc0 con 0x7f212415e040 2024-09-16T01:53:39.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.236+0000 7f211a000640 1 -- 172.21.15.146:0/3084177054 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f210c0597a0 con 0x7f212415e040 2024-09-16T01:53:39.237 INFO:teuthology.orchestra.run.smithi146.stdout:60129542206 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f210006c300 msgr2=0x7f210006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f210006c300 0x7f210006e7c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f211c0012e0 tx=0x7f211c009040 comp rx=0 tx=0).stop 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 msgr2=0x7f21240c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 0x7f21240c6010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f210c00e930 tx=0x7f210c00edf0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 shutdown_connections 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f210006c300 0x7f210006e7c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f212415e040 0x7f21240c6010 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2124154d00 0x7f21241e5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 --2- 172.21.15.146:0/3084177054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2124071010 0x7f21241e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 >> 172.21.15.146:0/3084177054 conn(0x7f2124067950 msgr2=0x7f212406cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 shutdown_connections 2024-09-16T01:53:39.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.239+0000 7f2128a00640 1 -- 172.21.15.146:0/3084177054 wait complete. 2024-09-16T01:53:39.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.242+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f49700cbea0 con 0x7f49700cb2f0 2024-09-16T01:53:39.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.243+0000 7f4967400640 1 -- 172.21.15.146:0/1845171462 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4960030e20 con 0x7f49700cb2f0 2024-09-16T01:53:39.248 INFO:teuthology.orchestra.run.smithi146.stdout:55834574910 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f495006c2e0 msgr2=0x7f495006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f495006c2e0 0x7f495006e7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f49700cb1a0 tx=0x7f495c009190 comp rx=0 tx=0).stop 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 msgr2=0x7f49701503c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 0x7f49701503c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4960000c00 tx=0x7f49600046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 shutdown_connections 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f495006c2e0 0x7f495006e7a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f497015e9d0 0x7f4970151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49700cc0c0 0x7f49700b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 --2- 172.21.15.146:0/1845171462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49700cb2f0 0x7f49701503c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 >> 172.21.15.146:0/1845171462 conn(0x7f497006d560 msgr2=0x7f49700d0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.248+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 shutdown_connections 2024-09-16T01:53:39.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.249+0000 7f4976200640 1 -- 172.21.15.146:0/1845171462 wait complete. 2024-09-16T01:53:39.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.249+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdfec073120 con 0x7fdfec159da0 2024-09-16T01:53:39.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.250+0000 7fdfe3e00640 1 -- 172.21.15.146:0/3272796595 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdfdc05d7d0 con 0x7fdfec159da0 2024-09-16T01:53:39.251 INFO:teuthology.orchestra.run.smithi146.stdout:64424509503 2024-09-16T01:53:39.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.253+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfcc06c380 msgr2=0x7fdfcc06e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.253+0000 7fdff1c00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfcc06c380 0x7fdfcc06e840 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8000f60 tx=0x7fdfd8009040 comp rx=0 tx=0).stop 2024-09-16T01:53:39.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.253+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 msgr2=0x7fdfec1fd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.253+0000 7fdff1c00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 0x7fdfec1fd720 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00c850 tx=0x7fdfdc00cd10 comp rx=0 tx=0).stop 2024-09-16T01:53:39.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 shutdown_connections 2024-09-16T01:53:39.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fdfcc06c380 0x7fdfcc06e840 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdfec159da0 0x7fdfec1fd720 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfec0cc040 0x7fdfec0c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 --2- 172.21.15.146:0/3272796595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdfec0c6b70 0x7fdfec159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 >> 172.21.15.146:0/3272796595 conn(0x7fdfec0ccc20 msgr2=0x7fdfec0cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 shutdown_connections 2024-09-16T01:53:39.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.254+0000 7fdff1c00640 1 -- 172.21.15.146:0/3272796595 wait complete. 2024-09-16T01:53:39.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.260+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc39c0cdb50 con 0x7fc39c153e90 2024-09-16T01:53:39.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.260+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa2b8005580 con 0x7fa2e00cc4a0 2024-09-16T01:53:39.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.260+0000 7fc396000640 1 -- 172.21.15.146:0/2555026919 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc39805a720 con 0x7fc39c153e90 2024-09-16T01:53:39.262 INFO:teuthology.orchestra.run.smithi146.stdout:64424509502 2024-09-16T01:53:39.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.261+0000 7fa2d6a00640 1 -- 172.21.15.146:0/395772286 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa2d805d3e0 con 0x7fa2e00cc4a0 2024-09-16T01:53:39.263 INFO:teuthology.orchestra.run.smithi146.stdout:64424509503 2024-09-16T01:53:39.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542206 got 60129542206 for osd.7 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc38006c2e0 msgr2=0x7fc38006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc38006c2e0 0x7fc38006e7a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc390009990 tx=0x7fc390009110 comp rx=0 tx=0).stop 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c153e90 msgr2=0x7fc39c1e53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c153e90 0x7fc39c1e53c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc39800c730 tx=0x7fc39800cbf0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 shutdown_connections 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc38006c2e0 0x7fc38006e7a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc39c153e90 0x7fc39c1e53c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc39c0c6290 0x7fc39c1e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 --2- 172.21.15.146:0/2555026919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc39c0c5eb0 0x7fc39c1f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.264+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 >> 172.21.15.146:0/2555026919 conn(0x7fc39c067b30 msgr2=0x7fc39c1e0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.267 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa2c006c2b0 msgr2=0x7fa2c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa2c006c2b0 0x7fa2c006e770 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0004de0 tx=0x7fa2d0007f10 comp rx=0 tx=0).stop 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa2e00cc4a0 msgr2=0x7fa2e00c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa2e00cc4a0 0x7fa2e00c8080 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800cae0 tx=0x7fa2d800a060 comp rx=0 tx=0).stop 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 shutdown_connections 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fa2c006c2b0 0x7fa2c006e770 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa2e00cc4a0 0x7fa2e00c8080 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2e00cc0c0 0x7fa2e0150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 --2- 172.21.15.146:0/395772286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2e00cb2f0 0x7fa2e0151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 >> 172.21.15.146:0/395772286 conn(0x7fa2e006d560 msgr2=0x7fa2e00c6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 shutdown_connections 2024-09-16T01:53:39.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fc3a3200640 1 -- 172.21.15.146:0/2555026919 wait complete. 2024-09-16T01:53:39.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 shutdown_connections 2024-09-16T01:53:39.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.265+0000 7fa2e5400640 1 -- 172.21.15.146:0/395772286 wait complete. 2024-09-16T01:53:39.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574910 got 55834574910 for osd.4 2024-09-16T01:53:39.273 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.274+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f51fc14e4a0 con 0x7f51fc15c050 2024-09-16T01:53:39.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.275+0000 7f51f2000640 1 -- 172.21.15.146:0/1380726891 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f51fc0c4f60 con 0x7f51fc15c050 2024-09-16T01:53:39.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.275+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7efce8071930 con 0x7efce80c8b10 2024-09-16T01:53:39.276 INFO:teuthology.orchestra.run.smithi146.stdout:60129542206 2024-09-16T01:53:39.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.276+0000 7efce4c00640 1 -- 172.21.15.146:0/2926722518 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efcdc078c30 con 0x7efce80c8b10 2024-09-16T01:53:39.277 INFO:teuthology.orchestra.run.smithi146.stdout:60129542206 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.277+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f51dc06c2b0 msgr2=0x7f51dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.277+0000 7f51fb200640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f51dc06c2b0 0x7f51dc06e770 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f51ec037000 tx=0x7f51ec008030 comp rx=0 tx=0).stop 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.277+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 msgr2=0x7f51fc0c4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.277+0000 7f51fb200640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 0x7f51fc0c4a20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f51f400c700 tx=0x7f51f400cbc0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 shutdown_connections 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f51dc06c2b0 0x7f51dc06e770 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f51fc15c050 0x7f51fc0c4a20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51fc14f1d0 0x7f51fc0c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 --2- 172.21.15.146:0/1380726891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51fc06c4f0 0x7f51fc1ec840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 >> 172.21.15.146:0/1380726891 conn(0x7f51fc067950 msgr2=0x7f51fc0c67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 shutdown_connections 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.278+0000 7f51fb200640 1 -- 172.21.15.146:0/1380726891 wait complete. 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.279+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efcc806c330 msgr2=0x7efcc806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.279+0000 7efcef400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efcc806c330 0x7efcc806e7f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efce81f2ad0 tx=0x7efcd40079d0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.279+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efce80c8b10 msgr2=0x7efce81f8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.279+0000 7efcef400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efce80c8b10 0x7efce81f8dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efcdc031000 tx=0x7efcdc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.281+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 shutdown_connections 2024-09-16T01:53:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.281+0000 7efcef400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efcc806c330 0x7efcc806e7f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.281+0000 7efcef400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efce815daa0 0x7efce81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.281+0000 7efcef400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efce8159320 0x7efce81618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.281+0000 7efcef400640 1 --2- 172.21.15.146:0/2926722518 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efce80c8b10 0x7efce81f8dd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.281+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 >> 172.21.15.146:0/2926722518 conn(0x7efce806f3c0 msgr2=0x7efce81579a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509502 got 64424509503 for osd.2 2024-09-16T01:53:39.284 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.283+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 shutdown_connections 2024-09-16T01:53:39.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.283+0000 7efcef400640 1 -- 172.21.15.146:0/2926722518 wait complete. 2024-09-16T01:53:39.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509502 got 64424509502 for osd.1 2024-09-16T01:53:39.293 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509503 got 64424509503 for osd.3 2024-09-16T01:53:39.294 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.298 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542206 got 60129542206 for osd.6 2024-09-16T01:53:39.298 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542206 got 60129542206 for osd.5 2024-09-16T01:53:39.307 DEBUG:teuthology.parallel:result is None 2024-09-16T01:53:39.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T01:53:39.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:39.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.500+0000 7f8634e00640 1 Processor -- start 2024-09-16T01:53:39.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.500+0000 7f8634e00640 1 -- start start 2024-09-16T01:53:39.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f8634e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8630152850 0x7f8630158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f8634e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8630072f50 con 0x7f8630158bb0 2024-09-16T01:53:39.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f8634e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f863015cb50 con 0x7f86300c6bb0 2024-09-16T01:53:39.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f8634e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8630073210 con 0x7f8630152850 2024-09-16T01:53:39.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f862d400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8630158bb0 0x7f86300c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47240/0 (socket says 172.21.15.146:47240) 2024-09-16T01:53:39.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f862d400640 1 -- 172.21.15.146:0/4040354074 learned_addr learned my addr 172.21.15.146:0/4040354074 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:39.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.501+0000 7f862de00640 1 --2- 172.21.15.146:0/4040354074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8630152850 0x7f8630158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.502+0000 7f862de00640 1 -- 172.21.15.146:0/4040354074 >> v1:172.21.15.161:6789/0 conn(0x7f86300c6bb0 legacy=0x7f8630162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.502+0000 7f862de00640 1 -- 172.21.15.146:0/4040354074 >> v1:172.21.15.146:6789/0 conn(0x7f8630158bb0 legacy=0x7f86300c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.502+0000 7f862de00640 1 -- 172.21.15.146:0/4040354074 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863015bd90 con 0x7f8630152850 2024-09-16T01:53:39.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.502+0000 7f862ca00640 1 -- 172.21.15.146:0/4040354074 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2954844263 0 0) 0x7f8630072f50 con 0x7f8630158bb0 2024-09-16T01:53:39.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.503+0000 7f862de00640 1 --2- 172.21.15.146:0/4040354074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8630152850 0x7f8630158670 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8620009dc0 tx=0x7f862002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877a6776cc55e243 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.503+0000 7f862ca00640 1 -- 172.21.15.146:0/4040354074 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620007740 con 0x7f8630152850 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.503+0000 7f862ca00640 1 -- 172.21.15.146:0/4040354074 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86200078e0 con 0x7f8630152850 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.503+0000 7f862ca00640 1 -- 172.21.15.146:0/4040354074 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620007cb0 con 0x7f8630152850 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 -- 172.21.15.146:0/4040354074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8630152850 msgr2=0x7f8630158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 --2- 172.21.15.146:0/4040354074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8630152850 0x7f8630158670 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8620009dc0 tx=0x7f862002a5f0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 -- 172.21.15.146:0/4040354074 shutdown_connections 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 --2- 172.21.15.146:0/4040354074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8630152850 0x7f8630158670 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 -- 172.21.15.146:0/4040354074 >> 172.21.15.146:0/4040354074 conn(0x7f863006f3c0 msgr2=0x7f863006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 -- 172.21.15.146:0/4040354074 shutdown_connections 2024-09-16T01:53:39.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.504+0000 7f8634e00640 1 -- 172.21.15.146:0/4040354074 wait complete. 2024-09-16T01:53:39.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.505+0000 7f8634e00640 1 Processor -- start 2024-09-16T01:53:39.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.505+0000 7f8634e00640 1 -- start start 2024-09-16T01:53:39.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f8634e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86300c6bb0 0x7f863014b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f8634e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 0x7f86301f1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f8634e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8630158bb0 0x7f86301f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f8634e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f863015c630 con 0x7f8630152850 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f8634e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f863015c350 con 0x7f8630158bb0 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f8634e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8630070200 con 0x7f86300c6bb0 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f862d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 0x7f86301f1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.506+0000 7f862d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 0x7f86301f1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47360/0 (socket says 172.21.15.146:47360) 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 -- 172.21.15.146:0/3291459551 learned_addr learned my addr 172.21.15.146:0/3291459551 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862de00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86300c6bb0 0x7f863014b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862e800640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8630158bb0 0x7f86301f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 -- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86300c6bb0 msgr2=0x7f863014b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86300c6bb0 0x7f863014b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 -- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8630158bb0 msgr2=0x7f86301f1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8630158bb0 0x7f86301f1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 -- 172.21.15.146:0/3291459551 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86301f20d0 con 0x7f8630152850 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f862d400640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 0x7f86301f1300 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f861c00c480 tx=0x7f861c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c018070 con 0x7f8630152850 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861c0074e0 con 0x7f8630152850 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8630158940 con 0x7f8630152850 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.507+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c004a20 con 0x7f8630152850 2024-09-16T01:53:39.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.508+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8630154a90 con 0x7f8630152850 2024-09-16T01:53:39.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.509+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86300718a0 con 0x7f8630152850 2024-09-16T01:53:39.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.510+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f861c01b070 con 0x7f8630152850 2024-09-16T01:53:39.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.511+0000 7f8626000640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f861006c2b0 0x7f861006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.511+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f861c08a710 con 0x7f8630152850 2024-09-16T01:53:39.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.511+0000 7f862de00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f861006c2b0 0x7f861006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.512+0000 7f862de00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f861006c2b0 0x7f861006e770 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8620034000 tx=0x7f862002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:39.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.515+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f861c058820 con 0x7f8630152850 2024-09-16T01:53:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.688+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f86300726a0 con 0x7f861006c2b0 2024-09-16T01:53:39.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.695+0000 7f8626000640 1 -- 172.21.15.146:0/3291459551 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253932 (secure 0 0 0) 0x7f86300726a0 con 0x7f861006c2b0 2024-09-16T01:53:39.699 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:39.701 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:39.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f861006c2b0 msgr2=0x7f861006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f861006c2b0 0x7f861006e770 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8620034000 tx=0x7f862002f000 comp rx=0 tx=0).stop 2024-09-16T01:53:39.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 msgr2=0x7f86301f1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 0x7f86301f1300 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f861c00c480 tx=0x7f861c00c940 comp rx=0 tx=0).stop 2024-09-16T01:53:39.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 shutdown_connections 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f861006c2b0 0x7f861006e770 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8630158bb0 0x7f86301f1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8630152850 0x7f86301f1300 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 --2- 172.21.15.146:0/3291459551 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f86300c6bb0 0x7f863014b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 >> 172.21.15.146:0/3291459551 conn(0x7f863006f3c0 msgr2=0x7f8630150e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 shutdown_connections 2024-09-16T01:53:39.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.699+0000 7f8634e00640 1 -- 172.21.15.146:0/3291459551 wait complete. 2024-09-16T01:53:39.716 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-16T01:53:38.573627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":988881,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9794,"num_write_kb":25168838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4698,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11254,"ondisk_log_size":11254,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":281136,"kb_used_data":14956,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693584,"statfs":{"total":773094113280,"available":772806230016,"internally_reserved":0,"allocated":15314944,"data_stored":8903554,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-114,"num_bytes_recovered":-808,"num_keys_recovered":-40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.059399"},"pg_stats":[{"pgid":"3.3f","version":"127'159","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553902+0000","last_change":"2024-09-16T01:53:22.525372+0000","last_active":"2024-09-16T01:53:36.553902+0000","last_peered":"2024-09-16T01:53:36.553902+0000","last_clean":"2024-09-16T01:53:36.553902+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:36.553902+0000","last_undegraded":"2024-09-16T01:53:36.553902+0000","last_fullsized":"2024-09-16T01:53:36.553902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:00.392212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551885+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:36.551885+0000","last_peered":"2024-09-16T01:53:36.551885+0000","last_clean":"2024-09-16T01:53:36.551885+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:36.551885+0000","last_undegraded":"2024-09-16T01:53:36.551885+0000","last_fullsized":"2024-09-16T01:53:36.551885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":799,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551822+0000","last_change":"2024-09-16T01:53:23.518810+0000","last_active":"2024-09-16T01:53:36.551822+0000","last_peered":"2024-09-16T01:53:36.551822+0000","last_clean":"2024-09-16T01:53:36.551822+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:53:36.551822+0000","last_undegraded":"2024-09-16T01:53:36.551822+0000","last_fullsized":"2024-09-16T01:53:36.551822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"127'185","reported_seq":583,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556840+0000","last_change":"2024-09-16T01:53:32.511261+0000","last_active":"2024-09-16T01:53:36.556840+0000","last_peered":"2024-09-16T01:53:36.556840+0000","last_clean":"2024-09-16T01:53:36.556840+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:36.556840+0000","last_undegraded":"2024-09-16T01:53:36.556840+0000","last_fullsized":"2024-09-16T01:53:36.556840+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:07.896976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"127'170","reported_seq":522,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557933+0000","last_change":"2024-09-16T01:53:22.521371+0000","last_active":"2024-09-16T01:53:36.557933+0000","last_peered":"2024-09-16T01:53:36.557933+0000","last_clean":"2024-09-16T01:53:36.557933+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:36.557933+0000","last_undegraded":"2024-09-16T01:53:36.557933+0000","last_fullsized":"2024-09-16T01:53:36.557933+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:39.028997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"127'156","reported_seq":576,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555886+0000","last_change":"2024-09-16T01:53:23.521468+0000","last_active":"2024-09-16T01:53:36.555886+0000","last_peered":"2024-09-16T01:53:36.555886+0000","last_clean":"2024-09-16T01:53:36.555886+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:36.555886+0000","last_undegraded":"2024-09-16T01:53:36.555886+0000","last_fullsized":"2024-09-16T01:53:36.555886+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:34:42.522900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"127'183","reported_seq":633,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552387+0000","last_change":"2024-09-16T01:53:22.523352+0000","last_active":"2024-09-16T01:53:36.552387+0000","last_peered":"2024-09-16T01:53:36.552387+0000","last_clean":"2024-09-16T01:53:36.552387+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:36.552387+0000","last_undegraded":"2024-09-16T01:53:36.552387+0000","last_fullsized":"2024-09-16T01:53:36.552387+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.078885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":722,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551418+0000","last_change":"2024-09-16T01:53:22.523335+0000","last_active":"2024-09-16T01:53:36.551418+0000","last_peered":"2024-09-16T01:53:36.551418+0000","last_clean":"2024-09-16T01:53:36.551418+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:53:36.551418+0000","last_undegraded":"2024-09-16T01:53:36.551418+0000","last_fullsized":"2024-09-16T01:53:36.551418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"127'167","reported_seq":440,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555973+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:36.555973+0000","last_peered":"2024-09-16T01:53:36.555973+0000","last_clean":"2024-09-16T01:53:36.555973+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:36.555973+0000","last_undegraded":"2024-09-16T01:53:36.555973+0000","last_fullsized":"2024-09-16T01:53:36.555973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"127'175","reported_seq":555,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553462+0000","last_change":"2024-09-16T01:53:22.526050+0000","last_active":"2024-09-16T01:53:36.553462+0000","last_peered":"2024-09-16T01:53:36.553462+0000","last_clean":"2024-09-16T01:53:36.553462+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:36.553462+0000","last_undegraded":"2024-09-16T01:53:36.553462+0000","last_fullsized":"2024-09-16T01:53:36.553462+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:00.473148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"127'208","reported_seq":611,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551226+0000","last_change":"2024-09-16T01:53:23.519544+0000","last_active":"2024-09-16T01:53:36.551226+0000","last_peered":"2024-09-16T01:53:36.551226+0000","last_clean":"2024-09-16T01:53:36.551226+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:36.551226+0000","last_undegraded":"2024-09-16T01:53:36.551226+0000","last_fullsized":"2024-09-16T01:53:36.551226+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:26.797344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"127'159","reported_seq":355,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552856+0000","last_change":"2024-09-16T01:53:22.519665+0000","last_active":"2024-09-16T01:53:36.552856+0000","last_peered":"2024-09-16T01:53:36.552856+0000","last_clean":"2024-09-16T01:53:36.552856+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:36.552856+0000","last_undegraded":"2024-09-16T01:53:36.552856+0000","last_fullsized":"2024-09-16T01:53:36.552856+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:32.846352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545527+0000","last_change":"2024-09-16T01:53:34.533587+0000","last_active":"2024-09-16T01:53:36.545527+0000","last_peered":"2024-09-16T01:53:36.545527+0000","last_clean":"2024-09-16T01:53:36.545527+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:36.545527+0000","last_undegraded":"2024-09-16T01:53:36.545527+0000","last_fullsized":"2024-09-16T01:53:36.545527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:15.453058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":487,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545288+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:36.545288+0000","last_peered":"2024-09-16T01:53:36.545288+0000","last_clean":"2024-09-16T01:53:36.545288+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:36.545288+0000","last_undegraded":"2024-09-16T01:53:36.545288+0000","last_fullsized":"2024-09-16T01:53:36.545288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":650,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551144+0000","last_change":"2024-09-16T01:53:22.527702+0000","last_active":"2024-09-16T01:53:36.551144+0000","last_peered":"2024-09-16T01:53:36.551144+0000","last_clean":"2024-09-16T01:53:36.551144+0000","last_became_active":"2024-09-16T01:53:22.526994+0000","last_became_peered":"2024-09-16T01:53:22.526994+0000","last_unstale":"2024-09-16T01:53:36.551144+0000","last_undegraded":"2024-09-16T01:53:36.551144+0000","last_fullsized":"2024-09-16T01:53:36.551144+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"127'153","reported_seq":423,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169770+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:36.169770+0000","last_peered":"2024-09-16T01:53:36.169770+0000","last_clean":"2024-09-16T01:53:36.169770+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:36.169770+0000","last_undegraded":"2024-09-16T01:53:36.169770+0000","last_fullsized":"2024-09-16T01:53:36.169770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"127'168","reported_seq":636,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170335+0000","last_change":"2024-09-16T01:53:23.526130+0000","last_active":"2024-09-16T01:53:36.170335+0000","last_peered":"2024-09-16T01:53:36.170335+0000","last_clean":"2024-09-16T01:53:36.170335+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:53:36.170335+0000","last_undegraded":"2024-09-16T01:53:36.170335+0000","last_fullsized":"2024-09-16T01:53:36.170335+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:02.127294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"127'169","reported_seq":575,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563883+0000","last_change":"2024-09-16T01:53:36.563883+0000","last_active":"2024-09-16T01:53:36.563883+0000","last_peered":"2024-09-16T01:53:36.563883+0000","last_clean":"2024-09-16T01:53:36.563883+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:36.563883+0000","last_undegraded":"2024-09-16T01:53:36.563883+0000","last_fullsized":"2024-09-16T01:53:36.563883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554778+0000","last_change":"2024-09-16T01:53:36.554778+0000","last_active":"2024-09-16T01:53:36.554778+0000","last_peered":"2024-09-16T01:53:36.554778+0000","last_clean":"2024-09-16T01:53:36.554778+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:36.554778+0000","last_undegraded":"2024-09-16T01:53:36.554778+0000","last_fullsized":"2024-09-16T01:53:36.554778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"127'171","reported_seq":588,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552764+0000","last_change":"2024-09-16T01:53:23.524759+0000","last_active":"2024-09-16T01:53:36.552764+0000","last_peered":"2024-09-16T01:53:36.552764+0000","last_clean":"2024-09-16T01:53:36.552764+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:36.552764+0000","last_undegraded":"2024-09-16T01:53:36.552764+0000","last_fullsized":"2024-09-16T01:53:36.552764+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"127'143","reported_seq":455,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545707+0000","last_change":"2024-09-16T01:53:23.520715+0000","last_active":"2024-09-16T01:53:36.545707+0000","last_peered":"2024-09-16T01:53:36.545707+0000","last_clean":"2024-09-16T01:53:36.545707+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:36.545707+0000","last_undegraded":"2024-09-16T01:53:36.545707+0000","last_fullsized":"2024-09-16T01:53:36.545707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:45.628132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"127'189","reported_seq":821,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556048+0000","last_change":"2024-09-16T01:53:22.524407+0000","last_active":"2024-09-16T01:53:36.556048+0000","last_peered":"2024-09-16T01:53:36.556048+0000","last_clean":"2024-09-16T01:53:36.556048+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:36.556048+0000","last_undegraded":"2024-09-16T01:53:36.556048+0000","last_fullsized":"2024-09-16T01:53:36.556048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551482+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:36.551482+0000","last_peered":"2024-09-16T01:53:36.551482+0000","last_clean":"2024-09-16T01:53:36.551482+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:36.551482+0000","last_undegraded":"2024-09-16T01:53:36.551482+0000","last_fullsized":"2024-09-16T01:53:36.551482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"127'201","reported_seq":472,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169762+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:36.169762+0000","last_peered":"2024-09-16T01:53:36.169762+0000","last_clean":"2024-09-16T01:53:36.169762+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:36.169762+0000","last_undegraded":"2024-09-16T01:53:36.169762+0000","last_fullsized":"2024-09-16T01:53:36.169762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":431,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552017+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:36.552017+0000","last_peered":"2024-09-16T01:53:36.552017+0000","last_clean":"2024-09-16T01:53:36.552017+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:36.552017+0000","last_undegraded":"2024-09-16T01:53:36.552017+0000","last_fullsized":"2024-09-16T01:53:36.552017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"127'202","reported_seq":476,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559877+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:36.559877+0000","last_peered":"2024-09-16T01:53:36.559877+0000","last_clean":"2024-09-16T01:53:36.559877+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:36.559877+0000","last_undegraded":"2024-09-16T01:53:36.559877+0000","last_fullsized":"2024-09-16T01:53:36.559877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":664,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545722+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:36.545722+0000","last_peered":"2024-09-16T01:53:36.545722+0000","last_clean":"2024-09-16T01:53:36.545722+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:36.545722+0000","last_undegraded":"2024-09-16T01:53:36.545722+0000","last_fullsized":"2024-09-16T01:53:36.545722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545688+0000","last_change":"2024-09-16T01:53:34.533524+0000","last_active":"2024-09-16T01:53:36.545688+0000","last_peered":"2024-09-16T01:53:36.545688+0000","last_clean":"2024-09-16T01:53:36.545688+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:36.545688+0000","last_undegraded":"2024-09-16T01:53:36.545688+0000","last_fullsized":"2024-09-16T01:53:36.545688+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:57.738639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":683,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551567+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:36.551567+0000","last_peered":"2024-09-16T01:53:36.551567+0000","last_clean":"2024-09-16T01:53:36.551567+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:36.551567+0000","last_undegraded":"2024-09-16T01:53:36.551567+0000","last_fullsized":"2024-09-16T01:53:36.551567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"127'145","reported_seq":416,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169917+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:36.169917+0000","last_peered":"2024-09-16T01:53:36.169917+0000","last_clean":"2024-09-16T01:53:36.169917+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:36.169917+0000","last_undegraded":"2024-09-16T01:53:36.169917+0000","last_fullsized":"2024-09-16T01:53:36.169917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"127'146","reported_seq":558,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563260+0000","last_change":"2024-09-16T01:53:36.563260+0000","last_active":"2024-09-16T01:53:36.563260+0000","last_peered":"2024-09-16T01:53:36.563260+0000","last_clean":"2024-09-16T01:53:36.563260+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:36.563260+0000","last_undegraded":"2024-09-16T01:53:36.563260+0000","last_fullsized":"2024-09-16T01:53:36.563260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:31.539361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551939+0000","last_change":"2024-09-16T01:53:36.551939+0000","last_active":"2024-09-16T01:53:36.551939+0000","last_peered":"2024-09-16T01:53:36.551939+0000","last_clean":"2024-09-16T01:53:36.551939+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:36.551939+0000","last_undegraded":"2024-09-16T01:53:36.551939+0000","last_fullsized":"2024-09-16T01:53:36.551939+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556226+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:36.556226+0000","last_peered":"2024-09-16T01:53:36.556226+0000","last_clean":"2024-09-16T01:53:36.556226+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:36.556226+0000","last_undegraded":"2024-09-16T01:53:36.556226+0000","last_fullsized":"2024-09-16T01:53:36.556226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551629+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:36.551629+0000","last_peered":"2024-09-16T01:53:36.551629+0000","last_clean":"2024-09-16T01:53:36.551629+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:36.551629+0000","last_undegraded":"2024-09-16T01:53:36.551629+0000","last_fullsized":"2024-09-16T01:53:36.551629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"127'154","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545597+0000","last_change":"2024-09-16T01:53:22.520820+0000","last_active":"2024-09-16T01:53:36.545597+0000","last_peered":"2024-09-16T01:53:36.545597+0000","last_clean":"2024-09-16T01:53:36.545597+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:36.545597+0000","last_undegraded":"2024-09-16T01:53:36.545597+0000","last_fullsized":"2024-09-16T01:53:36.545597+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:51.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":305,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170091+0000","last_change":"2024-09-16T01:53:19.532463+0000","last_active":"2024-09-16T01:53:36.170091+0000","last_peered":"2024-09-16T01:53:36.170091+0000","last_clean":"2024-09-16T01:53:36.170091+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:36.170091+0000","last_undegraded":"2024-09-16T01:53:36.170091+0000","last_fullsized":"2024-09-16T01:53:36.170091+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:52.176146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":275,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545279+0000","last_change":"2024-09-16T01:53:34.532869+0000","last_active":"2024-09-16T01:53:36.545279+0000","last_peered":"2024-09-16T01:53:36.545279+0000","last_clean":"2024-09-16T01:53:36.545279+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:36.545279+0000","last_undegraded":"2024-09-16T01:53:36.545279+0000","last_fullsized":"2024-09-16T01:53:36.545279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:13.148781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":454,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551995+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:36.551995+0000","last_peered":"2024-09-16T01:53:36.551995+0000","last_clean":"2024-09-16T01:53:36.551995+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:36.551995+0000","last_undegraded":"2024-09-16T01:53:36.551995+0000","last_fullsized":"2024-09-16T01:53:36.551995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":275,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169864+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:36.169864+0000","last_peered":"2024-09-16T01:53:36.169864+0000","last_clean":"2024-09-16T01:53:36.169864+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:36.169864+0000","last_undegraded":"2024-09-16T01:53:36.169864+0000","last_fullsized":"2024-09-16T01:53:36.169864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"127'177","reported_seq":418,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545675+0000","last_change":"2024-09-16T01:53:32.513845+0000","last_active":"2024-09-16T01:53:36.545675+0000","last_peered":"2024-09-16T01:53:36.545675+0000","last_clean":"2024-09-16T01:53:36.545675+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:36.545675+0000","last_undegraded":"2024-09-16T01:53:36.545675+0000","last_fullsized":"2024-09-16T01:53:36.545675+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:02.738813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":318,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556079+0000","last_change":"2024-09-16T01:53:20.522781+0000","last_active":"2024-09-16T01:53:36.556079+0000","last_peered":"2024-09-16T01:53:36.556079+0000","last_clean":"2024-09-16T01:53:36.556079+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:36.556079+0000","last_undegraded":"2024-09-16T01:53:36.556079+0000","last_fullsized":"2024-09-16T01:53:36.556079+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:25.742037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551737+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:36.551737+0000","last_peered":"2024-09-16T01:53:36.551737+0000","last_clean":"2024-09-16T01:53:36.551737+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:36.551737+0000","last_undegraded":"2024-09-16T01:53:36.551737+0000","last_fullsized":"2024-09-16T01:53:36.551737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553376+0000","last_change":"2024-09-16T01:53:19.579965+0000","last_active":"2024-09-16T01:53:36.553376+0000","last_peered":"2024-09-16T01:53:36.553376+0000","last_clean":"2024-09-16T01:53:36.553376+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:36.553376+0000","last_undegraded":"2024-09-16T01:53:36.553376+0000","last_fullsized":"2024-09-16T01:53:36.553376+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:29.737898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":423,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552118+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:36.552118+0000","last_peered":"2024-09-16T01:53:36.552118+0000","last_clean":"2024-09-16T01:53:36.552118+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:36.552118+0000","last_undegraded":"2024-09-16T01:53:36.552118+0000","last_fullsized":"2024-09-16T01:53:36.552118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"133'11","reported_seq":160,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557008+0000","last_change":"2024-09-16T01:53:36.557008+0000","last_active":"2024-09-16T01:53:36.557008+0000","last_peered":"2024-09-16T01:53:36.557008+0000","last_clean":"2024-09-16T01:53:36.557008+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:36.557008+0000","last_undegraded":"2024-09-16T01:53:36.557008+0000","last_fullsized":"2024-09-16T01:53:36.557008+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"127'157","reported_seq":427,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169964+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:36.169964+0000","last_peered":"2024-09-16T01:53:36.169964+0000","last_clean":"2024-09-16T01:53:36.169964+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:36.169964+0000","last_undegraded":"2024-09-16T01:53:36.169964+0000","last_fullsized":"2024-09-16T01:53:36.169964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"134'175","reported_seq":492,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556933+0000","last_change":"2024-09-16T01:53:23.518694+0000","last_active":"2024-09-16T01:53:36.556933+0000","last_peered":"2024-09-16T01:53:36.556933+0000","last_clean":"2024-09-16T01:53:36.556933+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:36.556933+0000","last_undegraded":"2024-09-16T01:53:36.556933+0000","last_fullsized":"2024-09-16T01:53:36.556933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:11.376091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391327,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"127'170","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551424+0000","last_change":"2024-09-16T01:53:22.524097+0000","last_active":"2024-09-16T01:53:36.551424+0000","last_peered":"2024-09-16T01:53:36.551424+0000","last_clean":"2024-09-16T01:53:36.551424+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:36.551424+0000","last_undegraded":"2024-09-16T01:53:36.551424+0000","last_fullsized":"2024-09-16T01:53:36.551424+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:14.041864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559516+0000","last_change":"2024-09-16T01:53:36.559516+0000","last_active":"2024-09-16T01:53:36.559516+0000","last_peered":"2024-09-16T01:53:36.559516+0000","last_clean":"2024-09-16T01:53:36.559516+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:36.559516+0000","last_undegraded":"2024-09-16T01:53:36.559516+0000","last_fullsized":"2024-09-16T01:53:36.559516+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":410,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546404+0000","last_change":"2024-09-16T01:53:34.529911+0000","last_active":"2024-09-16T01:53:36.546404+0000","last_peered":"2024-09-16T01:53:36.546404+0000","last_clean":"2024-09-16T01:53:36.546404+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:36.546404+0000","last_undegraded":"2024-09-16T01:53:36.546404+0000","last_fullsized":"2024-09-16T01:53:36.546404+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":355,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170203+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:36.170203+0000","last_peered":"2024-09-16T01:53:36.170203+0000","last_clean":"2024-09-16T01:53:36.170203+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:36.170203+0000","last_undegraded":"2024-09-16T01:53:36.170203+0000","last_fullsized":"2024-09-16T01:53:36.170203+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"127'186","reported_seq":700,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170014+0000","last_change":"2024-09-16T01:53:23.521939+0000","last_active":"2024-09-16T01:53:36.170014+0000","last_peered":"2024-09-16T01:53:36.170014+0000","last_clean":"2024-09-16T01:53:36.170014+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:53:36.170014+0000","last_undegraded":"2024-09-16T01:53:36.170014+0000","last_fullsized":"2024-09-16T01:53:36.170014+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:55.352896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552282+0000","last_change":"2024-09-16T01:53:19.550015+0000","last_active":"2024-09-16T01:53:36.552282+0000","last_peered":"2024-09-16T01:53:36.552282+0000","last_clean":"2024-09-16T01:53:36.552282+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:36.552282+0000","last_undegraded":"2024-09-16T01:53:36.552282+0000","last_fullsized":"2024-09-16T01:53:36.552282+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"127'191","reported_seq":464,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555603+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:36.555603+0000","last_peered":"2024-09-16T01:53:36.555603+0000","last_clean":"2024-09-16T01:53:36.555603+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:36.555603+0000","last_undegraded":"2024-09-16T01:53:36.555603+0000","last_fullsized":"2024-09-16T01:53:36.555603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"133'18","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545354+0000","last_change":"2024-09-16T01:53:20.541056+0000","last_active":"2024-09-16T01:53:36.545354+0000","last_peered":"2024-09-16T01:53:36.545354+0000","last_clean":"2024-09-16T01:53:36.545354+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:36.545354+0000","last_undegraded":"2024-09-16T01:53:36.545354+0000","last_fullsized":"2024-09-16T01:53:36.545354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:21.564935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"127'209","reported_seq":479,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169815+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:36.169815+0000","last_peered":"2024-09-16T01:53:36.169815+0000","last_clean":"2024-09-16T01:53:36.169815+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:36.169815+0000","last_undegraded":"2024-09-16T01:53:36.169815+0000","last_fullsized":"2024-09-16T01:53:36.169815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553629+0000","last_change":"2024-09-16T01:53:19.563306+0000","last_active":"2024-09-16T01:53:36.553629+0000","last_peered":"2024-09-16T01:53:36.553629+0000","last_clean":"2024-09-16T01:53:36.553629+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:36.553629+0000","last_undegraded":"2024-09-16T01:53:36.553629+0000","last_fullsized":"2024-09-16T01:53:36.553629+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:04.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"127'170","reported_seq":648,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.560008+0000","last_change":"2024-09-16T01:53:22.527907+0000","last_active":"2024-09-16T01:53:36.560008+0000","last_peered":"2024-09-16T01:53:36.560008+0000","last_clean":"2024-09-16T01:53:36.560008+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:36.560008+0000","last_undegraded":"2024-09-16T01:53:36.560008+0000","last_fullsized":"2024-09-16T01:53:36.560008+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:51.042095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546066+0000","last_change":"2024-09-16T01:53:20.547257+0000","last_active":"2024-09-16T01:53:36.546066+0000","last_peered":"2024-09-16T01:53:36.546066+0000","last_clean":"2024-09-16T01:53:36.546066+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:36.546066+0000","last_undegraded":"2024-09-16T01:53:36.546066+0000","last_fullsized":"2024-09-16T01:53:36.546066+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:04.416789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"127'153","reported_seq":523,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553716+0000","last_change":"2024-09-16T01:53:23.527787+0000","last_active":"2024-09-16T01:53:36.553716+0000","last_peered":"2024-09-16T01:53:36.553716+0000","last_clean":"2024-09-16T01:53:36.553716+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:53:36.553716+0000","last_undegraded":"2024-09-16T01:53:36.553716+0000","last_fullsized":"2024-09-16T01:53:36.553716+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551843+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:36.551843+0000","last_peered":"2024-09-16T01:53:36.551843+0000","last_clean":"2024-09-16T01:53:36.551843+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:36.551843+0000","last_undegraded":"2024-09-16T01:53:36.551843+0000","last_fullsized":"2024-09-16T01:53:36.551843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"127'159","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552996+0000","last_change":"2024-09-16T01:53:22.522539+0000","last_active":"2024-09-16T01:53:36.552996+0000","last_peered":"2024-09-16T01:53:36.552996+0000","last_clean":"2024-09-16T01:53:36.552996+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:36.552996+0000","last_undegraded":"2024-09-16T01:53:36.552996+0000","last_fullsized":"2024-09-16T01:53:36.552996+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:01.840411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":263,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545924+0000","last_change":"2024-09-16T01:53:34.533293+0000","last_active":"2024-09-16T01:53:36.545924+0000","last_peered":"2024-09-16T01:53:36.545924+0000","last_clean":"2024-09-16T01:53:36.545924+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:36.545924+0000","last_undegraded":"2024-09-16T01:53:36.545924+0000","last_fullsized":"2024-09-16T01:53:36.545924+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:39.949872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"127'168","reported_seq":439,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170093+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:36.170093+0000","last_peered":"2024-09-16T01:53:36.170093+0000","last_clean":"2024-09-16T01:53:36.170093+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:36.170093+0000","last_undegraded":"2024-09-16T01:53:36.170093+0000","last_fullsized":"2024-09-16T01:53:36.170093+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":432,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555710+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:36.555710+0000","last_peered":"2024-09-16T01:53:36.555710+0000","last_clean":"2024-09-16T01:53:36.555710+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:36.555710+0000","last_undegraded":"2024-09-16T01:53:36.555710+0000","last_fullsized":"2024-09-16T01:53:36.555710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":279,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170218+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:36.170218+0000","last_peered":"2024-09-16T01:53:36.170218+0000","last_clean":"2024-09-16T01:53:36.170218+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:36.170218+0000","last_undegraded":"2024-09-16T01:53:36.170218+0000","last_fullsized":"2024-09-16T01:53:36.170218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"127'175","reported_seq":592,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545790+0000","last_change":"2024-09-16T01:53:22.524529+0000","last_active":"2024-09-16T01:53:36.545790+0000","last_peered":"2024-09-16T01:53:36.545790+0000","last_clean":"2024-09-16T01:53:36.545790+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:36.545790+0000","last_undegraded":"2024-09-16T01:53:36.545790+0000","last_fullsized":"2024-09-16T01:53:36.545790+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:39.441997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":277,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169885+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:36.169885+0000","last_peered":"2024-09-16T01:53:36.169885+0000","last_clean":"2024-09-16T01:53:36.169885+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:36.169885+0000","last_undegraded":"2024-09-16T01:53:36.169885+0000","last_fullsized":"2024-09-16T01:53:36.169885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":403,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545398+0000","last_change":"2024-09-16T01:53:34.532478+0000","last_active":"2024-09-16T01:53:36.545398+0000","last_peered":"2024-09-16T01:53:36.545398+0000","last_clean":"2024-09-16T01:53:36.545398+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:36.545398+0000","last_undegraded":"2024-09-16T01:53:36.545398+0000","last_fullsized":"2024-09-16T01:53:36.545398+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"133'12","reported_seq":311,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554302+0000","last_change":"2024-09-16T01:53:36.554302+0000","last_active":"2024-09-16T01:53:36.554302+0000","last_peered":"2024-09-16T01:53:36.554302+0000","last_clean":"2024-09-16T01:53:36.554302+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:36.554302+0000","last_undegraded":"2024-09-16T01:53:36.554302+0000","last_fullsized":"2024-09-16T01:53:36.554302+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551550+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:36.551550+0000","last_peered":"2024-09-16T01:53:36.551550+0000","last_clean":"2024-09-16T01:53:36.551550+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:36.551550+0000","last_undegraded":"2024-09-16T01:53:36.551550+0000","last_fullsized":"2024-09-16T01:53:36.551550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":278,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170313+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:36.170313+0000","last_peered":"2024-09-16T01:53:36.170313+0000","last_clean":"2024-09-16T01:53:36.170313+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:36.170313+0000","last_undegraded":"2024-09-16T01:53:36.170313+0000","last_fullsized":"2024-09-16T01:53:36.170313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"127'170","reported_seq":543,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.558045+0000","last_change":"2024-09-16T01:53:23.527347+0000","last_active":"2024-09-16T01:53:36.558045+0000","last_peered":"2024-09-16T01:53:36.558045+0000","last_clean":"2024-09-16T01:53:36.558045+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:53:36.558045+0000","last_undegraded":"2024-09-16T01:53:36.558045+0000","last_fullsized":"2024-09-16T01:53:36.558045+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":276,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551717+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:36.551717+0000","last_peered":"2024-09-16T01:53:36.551717+0000","last_clean":"2024-09-16T01:53:36.551717+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:36.551717+0000","last_undegraded":"2024-09-16T01:53:36.551717+0000","last_fullsized":"2024-09-16T01:53:36.551717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"127'155","reported_seq":429,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557405+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:36.557405+0000","last_peered":"2024-09-16T01:53:36.557405+0000","last_clean":"2024-09-16T01:53:36.557405+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:36.557405+0000","last_undegraded":"2024-09-16T01:53:36.557405+0000","last_fullsized":"2024-09-16T01:53:36.557405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":119,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557300+0000","last_change":"2024-09-16T01:53:20.534486+0000","last_active":"2024-09-16T01:53:36.557300+0000","last_peered":"2024-09-16T01:53:36.557300+0000","last_clean":"2024-09-16T01:53:36.557300+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:36.557300+0000","last_undegraded":"2024-09-16T01:53:36.557300+0000","last_fullsized":"2024-09-16T01:53:36.557300+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:26.318068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":396,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545805+0000","last_change":"2024-09-16T01:53:34.539129+0000","last_active":"2024-09-16T01:53:36.545805+0000","last_peered":"2024-09-16T01:53:36.545805+0000","last_clean":"2024-09-16T01:53:36.545805+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:36.545805+0000","last_undegraded":"2024-09-16T01:53:36.545805+0000","last_fullsized":"2024-09-16T01:53:36.545805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"133'25","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546105+0000","last_change":"2024-09-16T01:53:20.586608+0000","last_active":"2024-09-16T01:53:36.546105+0000","last_peered":"2024-09-16T01:53:36.546105+0000","last_clean":"2024-09-16T01:53:36.546105+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:36.546105+0000","last_undegraded":"2024-09-16T01:53:36.546105+0000","last_fullsized":"2024-09-16T01:53:36.546105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:55.614045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":466,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546041+0000","last_change":"2024-09-16T01:53:23.514351+0000","last_active":"2024-09-16T01:53:36.546041+0000","last_peered":"2024-09-16T01:53:36.546041+0000","last_clean":"2024-09-16T01:53:36.546041+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:36.546041+0000","last_undegraded":"2024-09-16T01:53:36.546041+0000","last_fullsized":"2024-09-16T01:53:36.546041+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:14.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"130'8","reported_seq":311,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170374+0000","last_change":"2024-09-16T01:53:19.543649+0000","last_active":"2024-09-16T01:53:36.170374+0000","last_peered":"2024-09-16T01:53:36.170374+0000","last_clean":"2024-09-16T01:53:36.170374+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:53:36.170374+0000","last_undegraded":"2024-09-16T01:53:36.170374+0000","last_fullsized":"2024-09-16T01:53:36.170374+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:12.126459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"127'176","reported_seq":541,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546195+0000","last_change":"2024-09-16T01:53:23.518054+0000","last_active":"2024-09-16T01:53:36.546195+0000","last_peered":"2024-09-16T01:53:36.546195+0000","last_clean":"2024-09-16T01:53:36.546195+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:36.546195+0000","last_undegraded":"2024-09-16T01:53:36.546195+0000","last_fullsized":"2024-09-16T01:53:36.546195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:50.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551978+0000","last_change":"2024-09-16T01:53:19.575999+0000","last_active":"2024-09-16T01:53:36.551978+0000","last_peered":"2024-09-16T01:53:36.551978+0000","last_clean":"2024-09-16T01:53:36.551978+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:36.551978+0000","last_undegraded":"2024-09-16T01:53:36.551978+0000","last_fullsized":"2024-09-16T01:53:36.551978+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:59.687176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"127'160","reported_seq":583,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169874+0000","last_change":"2024-09-16T01:53:23.514324+0000","last_active":"2024-09-16T01:53:36.169874+0000","last_peered":"2024-09-16T01:53:36.169874+0000","last_clean":"2024-09-16T01:53:36.169874+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:53:36.169874+0000","last_undegraded":"2024-09-16T01:53:36.169874+0000","last_fullsized":"2024-09-16T01:53:36.169874+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:27.575957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546415+0000","last_change":"2024-09-16T01:53:20.576951+0000","last_active":"2024-09-16T01:53:36.546415+0000","last_peered":"2024-09-16T01:53:36.546415+0000","last_clean":"2024-09-16T01:53:36.546415+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:36.546415+0000","last_undegraded":"2024-09-16T01:53:36.546415+0000","last_fullsized":"2024-09-16T01:53:36.546415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:20.021875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"127'164","reported_seq":494,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545261+0000","last_change":"2024-09-16T01:53:23.517332+0000","last_active":"2024-09-16T01:53:36.545261+0000","last_peered":"2024-09-16T01:53:36.545261+0000","last_clean":"2024-09-16T01:53:36.545261+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:36.545261+0000","last_undegraded":"2024-09-16T01:53:36.545261+0000","last_fullsized":"2024-09-16T01:53:36.545261+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:15.929317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":248,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545542+0000","last_change":"2024-09-16T01:53:34.538675+0000","last_active":"2024-09-16T01:53:36.545542+0000","last_peered":"2024-09-16T01:53:36.545542+0000","last_clean":"2024-09-16T01:53:36.545542+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:36.545542+0000","last_undegraded":"2024-09-16T01:53:36.545542+0000","last_fullsized":"2024-09-16T01:53:36.545542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":624,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554614+0000","last_change":"2024-09-16T01:53:36.554614+0000","last_active":"2024-09-16T01:53:36.554614+0000","last_peered":"2024-09-16T01:53:36.554614+0000","last_clean":"2024-09-16T01:53:36.554614+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:36.554614+0000","last_undegraded":"2024-09-16T01:53:36.554614+0000","last_fullsized":"2024-09-16T01:53:36.554614+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:23.435521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":278,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170120+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:36.170120+0000","last_peered":"2024-09-16T01:53:36.170120+0000","last_clean":"2024-09-16T01:53:36.170120+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:36.170120+0000","last_undegraded":"2024-09-16T01:53:36.170120+0000","last_fullsized":"2024-09-16T01:53:36.170120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"127'149","reported_seq":420,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169976+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:36.169976+0000","last_peered":"2024-09-16T01:53:36.169976+0000","last_clean":"2024-09-16T01:53:36.169976+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:36.169976+0000","last_undegraded":"2024-09-16T01:53:36.169976+0000","last_fullsized":"2024-09-16T01:53:36.169976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"127'190","reported_seq":700,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170007+0000","last_change":"2024-09-16T01:53:22.527742+0000","last_active":"2024-09-16T01:53:36.170007+0000","last_peered":"2024-09-16T01:53:36.170007+0000","last_clean":"2024-09-16T01:53:36.170007+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:53:36.170007+0000","last_undegraded":"2024-09-16T01:53:36.170007+0000","last_fullsized":"2024-09-16T01:53:36.170007+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553134+0000","last_change":"2024-09-16T01:53:19.535031+0000","last_active":"2024-09-16T01:53:36.553134+0000","last_peered":"2024-09-16T01:53:36.553134+0000","last_clean":"2024-09-16T01:53:36.553134+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:36.553134+0000","last_undegraded":"2024-09-16T01:53:36.553134+0000","last_fullsized":"2024-09-16T01:53:36.553134+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:36.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"127'153","reported_seq":618,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556668+0000","last_change":"2024-09-16T01:53:22.521982+0000","last_active":"2024-09-16T01:53:36.556668+0000","last_peered":"2024-09-16T01:53:36.556668+0000","last_clean":"2024-09-16T01:53:36.556668+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:36.556668+0000","last_undegraded":"2024-09-16T01:53:36.556668+0000","last_fullsized":"2024-09-16T01:53:36.556668+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:35.196217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551329+0000","last_change":"2024-09-16T01:53:19.559597+0000","last_active":"2024-09-16T01:53:36.551329+0000","last_peered":"2024-09-16T01:53:36.551329+0000","last_clean":"2024-09-16T01:53:36.551329+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:36.551329+0000","last_undegraded":"2024-09-16T01:53:36.551329+0000","last_fullsized":"2024-09-16T01:53:36.551329+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552068+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:36.552068+0000","last_peered":"2024-09-16T01:53:36.552068+0000","last_clean":"2024-09-16T01:53:36.552068+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:36.552068+0000","last_undegraded":"2024-09-16T01:53:36.552068+0000","last_fullsized":"2024-09-16T01:53:36.552068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552067+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:36.552067+0000","last_peered":"2024-09-16T01:53:36.552067+0000","last_clean":"2024-09-16T01:53:36.552067+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:36.552067+0000","last_undegraded":"2024-09-16T01:53:36.552067+0000","last_fullsized":"2024-09-16T01:53:36.552067+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":449,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551955+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:36.551955+0000","last_peered":"2024-09-16T01:53:36.551955+0000","last_clean":"2024-09-16T01:53:36.551955+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:36.551955+0000","last_undegraded":"2024-09-16T01:53:36.551955+0000","last_fullsized":"2024-09-16T01:53:36.551955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":56,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545971+0000","last_change":"2024-09-16T01:53:20.557667+0000","last_active":"2024-09-16T01:53:36.545971+0000","last_peered":"2024-09-16T01:53:36.545971+0000","last_clean":"2024-09-16T01:53:36.545971+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:36.545971+0000","last_undegraded":"2024-09-16T01:53:36.545971+0000","last_fullsized":"2024-09-16T01:53:36.545971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:05:38.159129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4442,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10755,"ondisk_log_size":10755,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398513,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":419,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797026,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509503,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37152,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334688,"statfs":{"total":96636764160,"available":96598720512,"internally_reserved":0,"allocated":1843200,"data_stored":868593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2224128,"data_stored":1257231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509502,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1457506,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509503,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1843200,"data_stored":868525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542206,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1847296,"data_stored":867283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542206,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1826816,"data_stored":868104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542206,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1900544,"data_stored":1457423,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41404,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330436,"statfs":{"total":96636764160,"available":96594366464,"internally_reserved":0,"allocated":1937408,"data_stored":1258889,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393165,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:39.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:39.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.916+0000 7efd0be00640 1 Processor -- start 2024-09-16T01:53:39.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.916+0000 7efd0be00640 1 -- start start 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.917+0000 7efd0be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd04151a90 0x7efd040c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd0be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd04151e70 con 0x7efd040c6410 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd0be00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd040c8130 con 0x7efd040c5280 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd0be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7efd040c8410 con 0x7efd04151a90 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd09400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd04151a90 0x7efd040c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd08a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efd040c6410 0x7efd04158040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47248/0 (socket says 172.21.15.146:47248) 2024-09-16T01:53:39.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd08a00640 1 -- 172.21.15.146:0/1484889946 learned_addr learned my addr 172.21.15.146:0/1484889946 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:39.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd09400640 1 -- 172.21.15.146:0/1484889946 >> v1:172.21.15.161:6789/0 conn(0x7efd040c5280 legacy=0x7efd04161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd09400640 1 -- 172.21.15.146:0/1484889946 >> v1:172.21.15.146:6789/0 conn(0x7efd040c6410 legacy=0x7efd04158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd09400640 1 -- 172.21.15.146:0/1484889946 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd040c86f0 con 0x7efd04151a90 2024-09-16T01:53:39.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.918+0000 7efd03e00640 1 -- 172.21.15.146:0/1484889946 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394669733 0 0) 0x7efd04151e70 con 0x7efd040c6410 2024-09-16T01:53:39.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.919+0000 7efd09400640 1 --2- 172.21.15.146:0/1484889946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd04151a90 0x7efd040c36b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097a0 tx=0x7efcf802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a201fbb758641b44 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:39.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd03e00640 1 -- 172.21.15.146:0/1484889946 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8037070 con 0x7efd04151a90 2024-09-16T01:53:39.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd03e00640 1 -- 172.21.15.146:0/1484889946 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf8032440 con 0x7efd04151a90 2024-09-16T01:53:39.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd03e00640 1 -- 172.21.15.146:0/1484889946 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8032810 con 0x7efd04151a90 2024-09-16T01:53:39.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 -- 172.21.15.146:0/1484889946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd04151a90 msgr2=0x7efd040c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 --2- 172.21.15.146:0/1484889946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd04151a90 0x7efd040c36b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097a0 tx=0x7efcf802a230 comp rx=0 tx=0).stop 2024-09-16T01:53:39.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 -- 172.21.15.146:0/1484889946 shutdown_connections 2024-09-16T01:53:39.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 --2- 172.21.15.146:0/1484889946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd04151a90 0x7efd040c36b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 -- 172.21.15.146:0/1484889946 >> 172.21.15.146:0/1484889946 conn(0x7efd0406f3c0 msgr2=0x7efd0406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:39.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 -- 172.21.15.146:0/1484889946 shutdown_connections 2024-09-16T01:53:39.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.920+0000 7efd0be00640 1 -- 172.21.15.146:0/1484889946 wait complete. 2024-09-16T01:53:39.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.922+0000 7efd0be00640 1 Processor -- start 2024-09-16T01:53:39.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.922+0000 7efd0be00640 1 -- start start 2024-09-16T01:53:39.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd0be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 0x7efd041f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd0be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd040c6410 0x7efd041e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd0be00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd04151a90 0x7efd041f1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd0be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efd04157b20 con 0x7efd040c5280 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd0be00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd04157840 con 0x7efd040c6410 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd0be00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efd040c3bf0 con 0x7efd04151a90 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd09400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 0x7efd041f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd08a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd040c6410 0x7efd041e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd09e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd04151a90 0x7efd041f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd08a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd040c6410 0x7efd041e5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:60278/0 (socket says 172.21.15.146:60278) 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd09400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 0x7efd041f49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47370/0 (socket says 172.21.15.146:47370) 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd08a00640 1 -- 172.21.15.146:0/2777703269 learned_addr learned my addr 172.21.15.146:0/2777703269 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd09400640 1 -- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd04151a90 msgr2=0x7efd041f1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd09400640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd04151a90 0x7efd041f1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.923+0000 7efd09400640 1 -- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd040c6410 msgr2=0x7efd041e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd09400640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd040c6410 0x7efd041e5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:39.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd09400640 1 -- 172.21.15.146:0/2777703269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0406e290 con 0x7efd040c5280 2024-09-16T01:53:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd09400640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 0x7efd041f49d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efcf8031000 tx=0x7efcf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8037050 con 0x7efd040c5280 2024-09-16T01:53:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf8038040 con 0x7efd040c5280 2024-09-16T01:53:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd041535e0 con 0x7efd040c5280 2024-09-16T01:53:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8004470 con 0x7efd040c5280 2024-09-16T01:53:39.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.924+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd04153b60 con 0x7efd040c5280 2024-09-16T01:53:39.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.925+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd040c79b0 con 0x7efd040c5280 2024-09-16T01:53:39.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.926+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efcf8030900 con 0x7efd040c5280 2024-09-16T01:53:39.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.927+0000 7efd01600640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efce406c2b0 0x7efce406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:39.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.927+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7efcf80ac100 con 0x7efd040c5280 2024-09-16T01:53:39.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.927+0000 7efd08a00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efce406c2b0 0x7efce406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:39.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.928+0000 7efd08a00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efce406c2b0 0x7efce406e770 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efcf0007fd0 tx=0x7efcf0007f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:39.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:39.931+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efcf807a200 con 0x7efd040c5280 2024-09-16T01:53:40.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.113+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd04073370 con 0x7efce406c2b0 2024-09-16T01:53:40.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.120+0000 7efd01600640 1 -- 172.21.15.146:0/2777703269 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253932 (secure 0 0 0) 0x7efd04073370 con 0x7efce406c2b0 2024-09-16T01:53:40.123 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:40.125 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:40.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efce406c2b0 msgr2=0x7efce406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efce406c2b0 0x7efce406e770 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efcf0007fd0 tx=0x7efcf0007f60 comp rx=0 tx=0).stop 2024-09-16T01:53:40.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 msgr2=0x7efd041f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 0x7efd041f49d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efcf8031000 tx=0x7efcf80046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 shutdown_connections 2024-09-16T01:53:40.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7efce406c2b0 0x7efce406e770 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efd04151a90 0x7efd041f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd040c6410 0x7efd041e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 --2- 172.21.15.146:0/2777703269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd040c5280 0x7efd041f49d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 >> 172.21.15.146:0/2777703269 conn(0x7efd0406f3c0 msgr2=0x7efd040c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:40.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.123+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 shutdown_connections 2024-09-16T01:53:40.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.124+0000 7efd0be00640 1 -- 172.21.15.146:0/2777703269 wait complete. 2024-09-16T01:53:40.151 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-16T01:53:38.573627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":988881,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9794,"num_write_kb":25168838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4698,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11254,"ondisk_log_size":11254,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":281136,"kb_used_data":14956,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693584,"statfs":{"total":773094113280,"available":772806230016,"internally_reserved":0,"allocated":15314944,"data_stored":8903554,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-114,"num_bytes_recovered":-808,"num_keys_recovered":-40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.059399"},"pg_stats":[{"pgid":"3.3f","version":"127'159","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553902+0000","last_change":"2024-09-16T01:53:22.525372+0000","last_active":"2024-09-16T01:53:36.553902+0000","last_peered":"2024-09-16T01:53:36.553902+0000","last_clean":"2024-09-16T01:53:36.553902+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:36.553902+0000","last_undegraded":"2024-09-16T01:53:36.553902+0000","last_fullsized":"2024-09-16T01:53:36.553902+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:00.392212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551885+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:36.551885+0000","last_peered":"2024-09-16T01:53:36.551885+0000","last_clean":"2024-09-16T01:53:36.551885+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:36.551885+0000","last_undegraded":"2024-09-16T01:53:36.551885+0000","last_fullsized":"2024-09-16T01:53:36.551885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":799,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551822+0000","last_change":"2024-09-16T01:53:23.518810+0000","last_active":"2024-09-16T01:53:36.551822+0000","last_peered":"2024-09-16T01:53:36.551822+0000","last_clean":"2024-09-16T01:53:36.551822+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:53:36.551822+0000","last_undegraded":"2024-09-16T01:53:36.551822+0000","last_fullsized":"2024-09-16T01:53:36.551822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"127'185","reported_seq":583,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556840+0000","last_change":"2024-09-16T01:53:32.511261+0000","last_active":"2024-09-16T01:53:36.556840+0000","last_peered":"2024-09-16T01:53:36.556840+0000","last_clean":"2024-09-16T01:53:36.556840+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:36.556840+0000","last_undegraded":"2024-09-16T01:53:36.556840+0000","last_fullsized":"2024-09-16T01:53:36.556840+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:07.896976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"127'170","reported_seq":522,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557933+0000","last_change":"2024-09-16T01:53:22.521371+0000","last_active":"2024-09-16T01:53:36.557933+0000","last_peered":"2024-09-16T01:53:36.557933+0000","last_clean":"2024-09-16T01:53:36.557933+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:36.557933+0000","last_undegraded":"2024-09-16T01:53:36.557933+0000","last_fullsized":"2024-09-16T01:53:36.557933+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:39.028997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"127'156","reported_seq":576,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555886+0000","last_change":"2024-09-16T01:53:23.521468+0000","last_active":"2024-09-16T01:53:36.555886+0000","last_peered":"2024-09-16T01:53:36.555886+0000","last_clean":"2024-09-16T01:53:36.555886+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:36.555886+0000","last_undegraded":"2024-09-16T01:53:36.555886+0000","last_fullsized":"2024-09-16T01:53:36.555886+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:34:42.522900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"127'183","reported_seq":633,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552387+0000","last_change":"2024-09-16T01:53:22.523352+0000","last_active":"2024-09-16T01:53:36.552387+0000","last_peered":"2024-09-16T01:53:36.552387+0000","last_clean":"2024-09-16T01:53:36.552387+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:36.552387+0000","last_undegraded":"2024-09-16T01:53:36.552387+0000","last_fullsized":"2024-09-16T01:53:36.552387+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.078885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":722,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551418+0000","last_change":"2024-09-16T01:53:22.523335+0000","last_active":"2024-09-16T01:53:36.551418+0000","last_peered":"2024-09-16T01:53:36.551418+0000","last_clean":"2024-09-16T01:53:36.551418+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:53:36.551418+0000","last_undegraded":"2024-09-16T01:53:36.551418+0000","last_fullsized":"2024-09-16T01:53:36.551418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"127'167","reported_seq":440,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555973+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:36.555973+0000","last_peered":"2024-09-16T01:53:36.555973+0000","last_clean":"2024-09-16T01:53:36.555973+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:36.555973+0000","last_undegraded":"2024-09-16T01:53:36.555973+0000","last_fullsized":"2024-09-16T01:53:36.555973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"127'175","reported_seq":555,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553462+0000","last_change":"2024-09-16T01:53:22.526050+0000","last_active":"2024-09-16T01:53:36.553462+0000","last_peered":"2024-09-16T01:53:36.553462+0000","last_clean":"2024-09-16T01:53:36.553462+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:36.553462+0000","last_undegraded":"2024-09-16T01:53:36.553462+0000","last_fullsized":"2024-09-16T01:53:36.553462+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:00.473148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"127'208","reported_seq":611,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551226+0000","last_change":"2024-09-16T01:53:23.519544+0000","last_active":"2024-09-16T01:53:36.551226+0000","last_peered":"2024-09-16T01:53:36.551226+0000","last_clean":"2024-09-16T01:53:36.551226+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:36.551226+0000","last_undegraded":"2024-09-16T01:53:36.551226+0000","last_fullsized":"2024-09-16T01:53:36.551226+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:26.797344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"127'159","reported_seq":355,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552856+0000","last_change":"2024-09-16T01:53:22.519665+0000","last_active":"2024-09-16T01:53:36.552856+0000","last_peered":"2024-09-16T01:53:36.552856+0000","last_clean":"2024-09-16T01:53:36.552856+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:36.552856+0000","last_undegraded":"2024-09-16T01:53:36.552856+0000","last_fullsized":"2024-09-16T01:53:36.552856+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:32.846352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545527+0000","last_change":"2024-09-16T01:53:34.533587+0000","last_active":"2024-09-16T01:53:36.545527+0000","last_peered":"2024-09-16T01:53:36.545527+0000","last_clean":"2024-09-16T01:53:36.545527+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:36.545527+0000","last_undegraded":"2024-09-16T01:53:36.545527+0000","last_fullsized":"2024-09-16T01:53:36.545527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:15.453058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":487,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545288+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:36.545288+0000","last_peered":"2024-09-16T01:53:36.545288+0000","last_clean":"2024-09-16T01:53:36.545288+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:36.545288+0000","last_undegraded":"2024-09-16T01:53:36.545288+0000","last_fullsized":"2024-09-16T01:53:36.545288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":650,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551144+0000","last_change":"2024-09-16T01:53:22.527702+0000","last_active":"2024-09-16T01:53:36.551144+0000","last_peered":"2024-09-16T01:53:36.551144+0000","last_clean":"2024-09-16T01:53:36.551144+0000","last_became_active":"2024-09-16T01:53:22.526994+0000","last_became_peered":"2024-09-16T01:53:22.526994+0000","last_unstale":"2024-09-16T01:53:36.551144+0000","last_undegraded":"2024-09-16T01:53:36.551144+0000","last_fullsized":"2024-09-16T01:53:36.551144+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"127'153","reported_seq":423,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169770+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:36.169770+0000","last_peered":"2024-09-16T01:53:36.169770+0000","last_clean":"2024-09-16T01:53:36.169770+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:36.169770+0000","last_undegraded":"2024-09-16T01:53:36.169770+0000","last_fullsized":"2024-09-16T01:53:36.169770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"127'168","reported_seq":636,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170335+0000","last_change":"2024-09-16T01:53:23.526130+0000","last_active":"2024-09-16T01:53:36.170335+0000","last_peered":"2024-09-16T01:53:36.170335+0000","last_clean":"2024-09-16T01:53:36.170335+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:53:36.170335+0000","last_undegraded":"2024-09-16T01:53:36.170335+0000","last_fullsized":"2024-09-16T01:53:36.170335+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:02.127294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"127'169","reported_seq":575,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563883+0000","last_change":"2024-09-16T01:53:36.563883+0000","last_active":"2024-09-16T01:53:36.563883+0000","last_peered":"2024-09-16T01:53:36.563883+0000","last_clean":"2024-09-16T01:53:36.563883+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:36.563883+0000","last_undegraded":"2024-09-16T01:53:36.563883+0000","last_fullsized":"2024-09-16T01:53:36.563883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554778+0000","last_change":"2024-09-16T01:53:36.554778+0000","last_active":"2024-09-16T01:53:36.554778+0000","last_peered":"2024-09-16T01:53:36.554778+0000","last_clean":"2024-09-16T01:53:36.554778+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:36.554778+0000","last_undegraded":"2024-09-16T01:53:36.554778+0000","last_fullsized":"2024-09-16T01:53:36.554778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"127'171","reported_seq":588,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552764+0000","last_change":"2024-09-16T01:53:23.524759+0000","last_active":"2024-09-16T01:53:36.552764+0000","last_peered":"2024-09-16T01:53:36.552764+0000","last_clean":"2024-09-16T01:53:36.552764+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:36.552764+0000","last_undegraded":"2024-09-16T01:53:36.552764+0000","last_fullsized":"2024-09-16T01:53:36.552764+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"127'143","reported_seq":455,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545707+0000","last_change":"2024-09-16T01:53:23.520715+0000","last_active":"2024-09-16T01:53:36.545707+0000","last_peered":"2024-09-16T01:53:36.545707+0000","last_clean":"2024-09-16T01:53:36.545707+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:36.545707+0000","last_undegraded":"2024-09-16T01:53:36.545707+0000","last_fullsized":"2024-09-16T01:53:36.545707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:45.628132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"127'189","reported_seq":821,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556048+0000","last_change":"2024-09-16T01:53:22.524407+0000","last_active":"2024-09-16T01:53:36.556048+0000","last_peered":"2024-09-16T01:53:36.556048+0000","last_clean":"2024-09-16T01:53:36.556048+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:36.556048+0000","last_undegraded":"2024-09-16T01:53:36.556048+0000","last_fullsized":"2024-09-16T01:53:36.556048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551482+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:36.551482+0000","last_peered":"2024-09-16T01:53:36.551482+0000","last_clean":"2024-09-16T01:53:36.551482+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:36.551482+0000","last_undegraded":"2024-09-16T01:53:36.551482+0000","last_fullsized":"2024-09-16T01:53:36.551482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"127'201","reported_seq":472,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169762+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:36.169762+0000","last_peered":"2024-09-16T01:53:36.169762+0000","last_clean":"2024-09-16T01:53:36.169762+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:36.169762+0000","last_undegraded":"2024-09-16T01:53:36.169762+0000","last_fullsized":"2024-09-16T01:53:36.169762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":431,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552017+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:36.552017+0000","last_peered":"2024-09-16T01:53:36.552017+0000","last_clean":"2024-09-16T01:53:36.552017+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:36.552017+0000","last_undegraded":"2024-09-16T01:53:36.552017+0000","last_fullsized":"2024-09-16T01:53:36.552017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"127'202","reported_seq":476,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559877+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:36.559877+0000","last_peered":"2024-09-16T01:53:36.559877+0000","last_clean":"2024-09-16T01:53:36.559877+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:36.559877+0000","last_undegraded":"2024-09-16T01:53:36.559877+0000","last_fullsized":"2024-09-16T01:53:36.559877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":664,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545722+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:36.545722+0000","last_peered":"2024-09-16T01:53:36.545722+0000","last_clean":"2024-09-16T01:53:36.545722+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:36.545722+0000","last_undegraded":"2024-09-16T01:53:36.545722+0000","last_fullsized":"2024-09-16T01:53:36.545722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545688+0000","last_change":"2024-09-16T01:53:34.533524+0000","last_active":"2024-09-16T01:53:36.545688+0000","last_peered":"2024-09-16T01:53:36.545688+0000","last_clean":"2024-09-16T01:53:36.545688+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:36.545688+0000","last_undegraded":"2024-09-16T01:53:36.545688+0000","last_fullsized":"2024-09-16T01:53:36.545688+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:57.738639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":683,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551567+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:36.551567+0000","last_peered":"2024-09-16T01:53:36.551567+0000","last_clean":"2024-09-16T01:53:36.551567+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:36.551567+0000","last_undegraded":"2024-09-16T01:53:36.551567+0000","last_fullsized":"2024-09-16T01:53:36.551567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"127'145","reported_seq":416,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169917+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:36.169917+0000","last_peered":"2024-09-16T01:53:36.169917+0000","last_clean":"2024-09-16T01:53:36.169917+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:36.169917+0000","last_undegraded":"2024-09-16T01:53:36.169917+0000","last_fullsized":"2024-09-16T01:53:36.169917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"127'146","reported_seq":558,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563260+0000","last_change":"2024-09-16T01:53:36.563260+0000","last_active":"2024-09-16T01:53:36.563260+0000","last_peered":"2024-09-16T01:53:36.563260+0000","last_clean":"2024-09-16T01:53:36.563260+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:36.563260+0000","last_undegraded":"2024-09-16T01:53:36.563260+0000","last_fullsized":"2024-09-16T01:53:36.563260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:31.539361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551939+0000","last_change":"2024-09-16T01:53:36.551939+0000","last_active":"2024-09-16T01:53:36.551939+0000","last_peered":"2024-09-16T01:53:36.551939+0000","last_clean":"2024-09-16T01:53:36.551939+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:36.551939+0000","last_undegraded":"2024-09-16T01:53:36.551939+0000","last_fullsized":"2024-09-16T01:53:36.551939+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556226+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:36.556226+0000","last_peered":"2024-09-16T01:53:36.556226+0000","last_clean":"2024-09-16T01:53:36.556226+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:36.556226+0000","last_undegraded":"2024-09-16T01:53:36.556226+0000","last_fullsized":"2024-09-16T01:53:36.556226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551629+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:36.551629+0000","last_peered":"2024-09-16T01:53:36.551629+0000","last_clean":"2024-09-16T01:53:36.551629+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:36.551629+0000","last_undegraded":"2024-09-16T01:53:36.551629+0000","last_fullsized":"2024-09-16T01:53:36.551629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"127'154","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545597+0000","last_change":"2024-09-16T01:53:22.520820+0000","last_active":"2024-09-16T01:53:36.545597+0000","last_peered":"2024-09-16T01:53:36.545597+0000","last_clean":"2024-09-16T01:53:36.545597+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:36.545597+0000","last_undegraded":"2024-09-16T01:53:36.545597+0000","last_fullsized":"2024-09-16T01:53:36.545597+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:51.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":305,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170091+0000","last_change":"2024-09-16T01:53:19.532463+0000","last_active":"2024-09-16T01:53:36.170091+0000","last_peered":"2024-09-16T01:53:36.170091+0000","last_clean":"2024-09-16T01:53:36.170091+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:36.170091+0000","last_undegraded":"2024-09-16T01:53:36.170091+0000","last_fullsized":"2024-09-16T01:53:36.170091+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:52.176146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":275,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545279+0000","last_change":"2024-09-16T01:53:34.532869+0000","last_active":"2024-09-16T01:53:36.545279+0000","last_peered":"2024-09-16T01:53:36.545279+0000","last_clean":"2024-09-16T01:53:36.545279+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:36.545279+0000","last_undegraded":"2024-09-16T01:53:36.545279+0000","last_fullsized":"2024-09-16T01:53:36.545279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:13.148781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":454,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551995+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:36.551995+0000","last_peered":"2024-09-16T01:53:36.551995+0000","last_clean":"2024-09-16T01:53:36.551995+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:36.551995+0000","last_undegraded":"2024-09-16T01:53:36.551995+0000","last_fullsized":"2024-09-16T01:53:36.551995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":275,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169864+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:36.169864+0000","last_peered":"2024-09-16T01:53:36.169864+0000","last_clean":"2024-09-16T01:53:36.169864+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:36.169864+0000","last_undegraded":"2024-09-16T01:53:36.169864+0000","last_fullsized":"2024-09-16T01:53:36.169864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"127'177","reported_seq":418,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545675+0000","last_change":"2024-09-16T01:53:32.513845+0000","last_active":"2024-09-16T01:53:36.545675+0000","last_peered":"2024-09-16T01:53:36.545675+0000","last_clean":"2024-09-16T01:53:36.545675+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:36.545675+0000","last_undegraded":"2024-09-16T01:53:36.545675+0000","last_fullsized":"2024-09-16T01:53:36.545675+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:02.738813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":318,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556079+0000","last_change":"2024-09-16T01:53:20.522781+0000","last_active":"2024-09-16T01:53:36.556079+0000","last_peered":"2024-09-16T01:53:36.556079+0000","last_clean":"2024-09-16T01:53:36.556079+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:36.556079+0000","last_undegraded":"2024-09-16T01:53:36.556079+0000","last_fullsized":"2024-09-16T01:53:36.556079+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:25.742037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551737+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:36.551737+0000","last_peered":"2024-09-16T01:53:36.551737+0000","last_clean":"2024-09-16T01:53:36.551737+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:36.551737+0000","last_undegraded":"2024-09-16T01:53:36.551737+0000","last_fullsized":"2024-09-16T01:53:36.551737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553376+0000","last_change":"2024-09-16T01:53:19.579965+0000","last_active":"2024-09-16T01:53:36.553376+0000","last_peered":"2024-09-16T01:53:36.553376+0000","last_clean":"2024-09-16T01:53:36.553376+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:36.553376+0000","last_undegraded":"2024-09-16T01:53:36.553376+0000","last_fullsized":"2024-09-16T01:53:36.553376+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:29.737898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":423,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552118+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:36.552118+0000","last_peered":"2024-09-16T01:53:36.552118+0000","last_clean":"2024-09-16T01:53:36.552118+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:36.552118+0000","last_undegraded":"2024-09-16T01:53:36.552118+0000","last_fullsized":"2024-09-16T01:53:36.552118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"133'11","reported_seq":160,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557008+0000","last_change":"2024-09-16T01:53:36.557008+0000","last_active":"2024-09-16T01:53:36.557008+0000","last_peered":"2024-09-16T01:53:36.557008+0000","last_clean":"2024-09-16T01:53:36.557008+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:36.557008+0000","last_undegraded":"2024-09-16T01:53:36.557008+0000","last_fullsized":"2024-09-16T01:53:36.557008+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"127'157","reported_seq":427,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169964+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:36.169964+0000","last_peered":"2024-09-16T01:53:36.169964+0000","last_clean":"2024-09-16T01:53:36.169964+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:36.169964+0000","last_undegraded":"2024-09-16T01:53:36.169964+0000","last_fullsized":"2024-09-16T01:53:36.169964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"134'175","reported_seq":492,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556933+0000","last_change":"2024-09-16T01:53:23.518694+0000","last_active":"2024-09-16T01:53:36.556933+0000","last_peered":"2024-09-16T01:53:36.556933+0000","last_clean":"2024-09-16T01:53:36.556933+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:36.556933+0000","last_undegraded":"2024-09-16T01:53:36.556933+0000","last_fullsized":"2024-09-16T01:53:36.556933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:11.376091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391327,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"127'170","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551424+0000","last_change":"2024-09-16T01:53:22.524097+0000","last_active":"2024-09-16T01:53:36.551424+0000","last_peered":"2024-09-16T01:53:36.551424+0000","last_clean":"2024-09-16T01:53:36.551424+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:36.551424+0000","last_undegraded":"2024-09-16T01:53:36.551424+0000","last_fullsized":"2024-09-16T01:53:36.551424+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:14.041864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559516+0000","last_change":"2024-09-16T01:53:36.559516+0000","last_active":"2024-09-16T01:53:36.559516+0000","last_peered":"2024-09-16T01:53:36.559516+0000","last_clean":"2024-09-16T01:53:36.559516+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:36.559516+0000","last_undegraded":"2024-09-16T01:53:36.559516+0000","last_fullsized":"2024-09-16T01:53:36.559516+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":410,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546404+0000","last_change":"2024-09-16T01:53:34.529911+0000","last_active":"2024-09-16T01:53:36.546404+0000","last_peered":"2024-09-16T01:53:36.546404+0000","last_clean":"2024-09-16T01:53:36.546404+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:36.546404+0000","last_undegraded":"2024-09-16T01:53:36.546404+0000","last_fullsized":"2024-09-16T01:53:36.546404+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":355,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170203+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:36.170203+0000","last_peered":"2024-09-16T01:53:36.170203+0000","last_clean":"2024-09-16T01:53:36.170203+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:36.170203+0000","last_undegraded":"2024-09-16T01:53:36.170203+0000","last_fullsized":"2024-09-16T01:53:36.170203+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"127'186","reported_seq":700,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170014+0000","last_change":"2024-09-16T01:53:23.521939+0000","last_active":"2024-09-16T01:53:36.170014+0000","last_peered":"2024-09-16T01:53:36.170014+0000","last_clean":"2024-09-16T01:53:36.170014+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:53:36.170014+0000","last_undegraded":"2024-09-16T01:53:36.170014+0000","last_fullsized":"2024-09-16T01:53:36.170014+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:55.352896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552282+0000","last_change":"2024-09-16T01:53:19.550015+0000","last_active":"2024-09-16T01:53:36.552282+0000","last_peered":"2024-09-16T01:53:36.552282+0000","last_clean":"2024-09-16T01:53:36.552282+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:36.552282+0000","last_undegraded":"2024-09-16T01:53:36.552282+0000","last_fullsized":"2024-09-16T01:53:36.552282+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":847,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"127'191","reported_seq":464,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555603+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:36.555603+0000","last_peered":"2024-09-16T01:53:36.555603+0000","last_clean":"2024-09-16T01:53:36.555603+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:36.555603+0000","last_undegraded":"2024-09-16T01:53:36.555603+0000","last_fullsized":"2024-09-16T01:53:36.555603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"133'18","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545354+0000","last_change":"2024-09-16T01:53:20.541056+0000","last_active":"2024-09-16T01:53:36.545354+0000","last_peered":"2024-09-16T01:53:36.545354+0000","last_clean":"2024-09-16T01:53:36.545354+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:36.545354+0000","last_undegraded":"2024-09-16T01:53:36.545354+0000","last_fullsized":"2024-09-16T01:53:36.545354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:21.564935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"127'209","reported_seq":479,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169815+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:36.169815+0000","last_peered":"2024-09-16T01:53:36.169815+0000","last_clean":"2024-09-16T01:53:36.169815+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:36.169815+0000","last_undegraded":"2024-09-16T01:53:36.169815+0000","last_fullsized":"2024-09-16T01:53:36.169815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553629+0000","last_change":"2024-09-16T01:53:19.563306+0000","last_active":"2024-09-16T01:53:36.553629+0000","last_peered":"2024-09-16T01:53:36.553629+0000","last_clean":"2024-09-16T01:53:36.553629+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:36.553629+0000","last_undegraded":"2024-09-16T01:53:36.553629+0000","last_fullsized":"2024-09-16T01:53:36.553629+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:04.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"127'170","reported_seq":648,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.560008+0000","last_change":"2024-09-16T01:53:22.527907+0000","last_active":"2024-09-16T01:53:36.560008+0000","last_peered":"2024-09-16T01:53:36.560008+0000","last_clean":"2024-09-16T01:53:36.560008+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:36.560008+0000","last_undegraded":"2024-09-16T01:53:36.560008+0000","last_fullsized":"2024-09-16T01:53:36.560008+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:51.042095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546066+0000","last_change":"2024-09-16T01:53:20.547257+0000","last_active":"2024-09-16T01:53:36.546066+0000","last_peered":"2024-09-16T01:53:36.546066+0000","last_clean":"2024-09-16T01:53:36.546066+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:36.546066+0000","last_undegraded":"2024-09-16T01:53:36.546066+0000","last_fullsized":"2024-09-16T01:53:36.546066+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:04.416789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"127'153","reported_seq":523,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553716+0000","last_change":"2024-09-16T01:53:23.527787+0000","last_active":"2024-09-16T01:53:36.553716+0000","last_peered":"2024-09-16T01:53:36.553716+0000","last_clean":"2024-09-16T01:53:36.553716+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:53:36.553716+0000","last_undegraded":"2024-09-16T01:53:36.553716+0000","last_fullsized":"2024-09-16T01:53:36.553716+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551843+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:36.551843+0000","last_peered":"2024-09-16T01:53:36.551843+0000","last_clean":"2024-09-16T01:53:36.551843+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:36.551843+0000","last_undegraded":"2024-09-16T01:53:36.551843+0000","last_fullsized":"2024-09-16T01:53:36.551843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"127'159","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552996+0000","last_change":"2024-09-16T01:53:22.522539+0000","last_active":"2024-09-16T01:53:36.552996+0000","last_peered":"2024-09-16T01:53:36.552996+0000","last_clean":"2024-09-16T01:53:36.552996+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:36.552996+0000","last_undegraded":"2024-09-16T01:53:36.552996+0000","last_fullsized":"2024-09-16T01:53:36.552996+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:01.840411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":263,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545924+0000","last_change":"2024-09-16T01:53:34.533293+0000","last_active":"2024-09-16T01:53:36.545924+0000","last_peered":"2024-09-16T01:53:36.545924+0000","last_clean":"2024-09-16T01:53:36.545924+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:36.545924+0000","last_undegraded":"2024-09-16T01:53:36.545924+0000","last_fullsized":"2024-09-16T01:53:36.545924+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:39.949872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"127'168","reported_seq":439,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170093+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:36.170093+0000","last_peered":"2024-09-16T01:53:36.170093+0000","last_clean":"2024-09-16T01:53:36.170093+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:36.170093+0000","last_undegraded":"2024-09-16T01:53:36.170093+0000","last_fullsized":"2024-09-16T01:53:36.170093+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":432,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555710+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:36.555710+0000","last_peered":"2024-09-16T01:53:36.555710+0000","last_clean":"2024-09-16T01:53:36.555710+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:36.555710+0000","last_undegraded":"2024-09-16T01:53:36.555710+0000","last_fullsized":"2024-09-16T01:53:36.555710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":279,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170218+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:36.170218+0000","last_peered":"2024-09-16T01:53:36.170218+0000","last_clean":"2024-09-16T01:53:36.170218+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:36.170218+0000","last_undegraded":"2024-09-16T01:53:36.170218+0000","last_fullsized":"2024-09-16T01:53:36.170218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"127'175","reported_seq":592,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545790+0000","last_change":"2024-09-16T01:53:22.524529+0000","last_active":"2024-09-16T01:53:36.545790+0000","last_peered":"2024-09-16T01:53:36.545790+0000","last_clean":"2024-09-16T01:53:36.545790+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:36.545790+0000","last_undegraded":"2024-09-16T01:53:36.545790+0000","last_fullsized":"2024-09-16T01:53:36.545790+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:39.441997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":277,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169885+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:36.169885+0000","last_peered":"2024-09-16T01:53:36.169885+0000","last_clean":"2024-09-16T01:53:36.169885+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:36.169885+0000","last_undegraded":"2024-09-16T01:53:36.169885+0000","last_fullsized":"2024-09-16T01:53:36.169885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":403,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545398+0000","last_change":"2024-09-16T01:53:34.532478+0000","last_active":"2024-09-16T01:53:36.545398+0000","last_peered":"2024-09-16T01:53:36.545398+0000","last_clean":"2024-09-16T01:53:36.545398+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:36.545398+0000","last_undegraded":"2024-09-16T01:53:36.545398+0000","last_fullsized":"2024-09-16T01:53:36.545398+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"133'12","reported_seq":311,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554302+0000","last_change":"2024-09-16T01:53:36.554302+0000","last_active":"2024-09-16T01:53:36.554302+0000","last_peered":"2024-09-16T01:53:36.554302+0000","last_clean":"2024-09-16T01:53:36.554302+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:36.554302+0000","last_undegraded":"2024-09-16T01:53:36.554302+0000","last_fullsized":"2024-09-16T01:53:36.554302+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551550+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:36.551550+0000","last_peered":"2024-09-16T01:53:36.551550+0000","last_clean":"2024-09-16T01:53:36.551550+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:36.551550+0000","last_undegraded":"2024-09-16T01:53:36.551550+0000","last_fullsized":"2024-09-16T01:53:36.551550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":278,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170313+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:36.170313+0000","last_peered":"2024-09-16T01:53:36.170313+0000","last_clean":"2024-09-16T01:53:36.170313+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:36.170313+0000","last_undegraded":"2024-09-16T01:53:36.170313+0000","last_fullsized":"2024-09-16T01:53:36.170313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"127'170","reported_seq":543,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.558045+0000","last_change":"2024-09-16T01:53:23.527347+0000","last_active":"2024-09-16T01:53:36.558045+0000","last_peered":"2024-09-16T01:53:36.558045+0000","last_clean":"2024-09-16T01:53:36.558045+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:53:36.558045+0000","last_undegraded":"2024-09-16T01:53:36.558045+0000","last_fullsized":"2024-09-16T01:53:36.558045+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":276,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551717+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:36.551717+0000","last_peered":"2024-09-16T01:53:36.551717+0000","last_clean":"2024-09-16T01:53:36.551717+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:36.551717+0000","last_undegraded":"2024-09-16T01:53:36.551717+0000","last_fullsized":"2024-09-16T01:53:36.551717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"127'155","reported_seq":429,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557405+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:36.557405+0000","last_peered":"2024-09-16T01:53:36.557405+0000","last_clean":"2024-09-16T01:53:36.557405+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:36.557405+0000","last_undegraded":"2024-09-16T01:53:36.557405+0000","last_fullsized":"2024-09-16T01:53:36.557405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":119,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557300+0000","last_change":"2024-09-16T01:53:20.534486+0000","last_active":"2024-09-16T01:53:36.557300+0000","last_peered":"2024-09-16T01:53:36.557300+0000","last_clean":"2024-09-16T01:53:36.557300+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:36.557300+0000","last_undegraded":"2024-09-16T01:53:36.557300+0000","last_fullsized":"2024-09-16T01:53:36.557300+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:26.318068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":396,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545805+0000","last_change":"2024-09-16T01:53:34.539129+0000","last_active":"2024-09-16T01:53:36.545805+0000","last_peered":"2024-09-16T01:53:36.545805+0000","last_clean":"2024-09-16T01:53:36.545805+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:36.545805+0000","last_undegraded":"2024-09-16T01:53:36.545805+0000","last_fullsized":"2024-09-16T01:53:36.545805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"133'25","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546105+0000","last_change":"2024-09-16T01:53:20.586608+0000","last_active":"2024-09-16T01:53:36.546105+0000","last_peered":"2024-09-16T01:53:36.546105+0000","last_clean":"2024-09-16T01:53:36.546105+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:36.546105+0000","last_undegraded":"2024-09-16T01:53:36.546105+0000","last_fullsized":"2024-09-16T01:53:36.546105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:55.614045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":466,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546041+0000","last_change":"2024-09-16T01:53:23.514351+0000","last_active":"2024-09-16T01:53:36.546041+0000","last_peered":"2024-09-16T01:53:36.546041+0000","last_clean":"2024-09-16T01:53:36.546041+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:36.546041+0000","last_undegraded":"2024-09-16T01:53:36.546041+0000","last_fullsized":"2024-09-16T01:53:36.546041+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:14.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"130'8","reported_seq":311,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170374+0000","last_change":"2024-09-16T01:53:19.543649+0000","last_active":"2024-09-16T01:53:36.170374+0000","last_peered":"2024-09-16T01:53:36.170374+0000","last_clean":"2024-09-16T01:53:36.170374+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:53:36.170374+0000","last_undegraded":"2024-09-16T01:53:36.170374+0000","last_fullsized":"2024-09-16T01:53:36.170374+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:12.126459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"127'176","reported_seq":541,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546195+0000","last_change":"2024-09-16T01:53:23.518054+0000","last_active":"2024-09-16T01:53:36.546195+0000","last_peered":"2024-09-16T01:53:36.546195+0000","last_clean":"2024-09-16T01:53:36.546195+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:36.546195+0000","last_undegraded":"2024-09-16T01:53:36.546195+0000","last_fullsized":"2024-09-16T01:53:36.546195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:50.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551978+0000","last_change":"2024-09-16T01:53:19.575999+0000","last_active":"2024-09-16T01:53:36.551978+0000","last_peered":"2024-09-16T01:53:36.551978+0000","last_clean":"2024-09-16T01:53:36.551978+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:36.551978+0000","last_undegraded":"2024-09-16T01:53:36.551978+0000","last_fullsized":"2024-09-16T01:53:36.551978+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:59.687176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"127'160","reported_seq":583,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169874+0000","last_change":"2024-09-16T01:53:23.514324+0000","last_active":"2024-09-16T01:53:36.169874+0000","last_peered":"2024-09-16T01:53:36.169874+0000","last_clean":"2024-09-16T01:53:36.169874+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:53:36.169874+0000","last_undegraded":"2024-09-16T01:53:36.169874+0000","last_fullsized":"2024-09-16T01:53:36.169874+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:27.575957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546415+0000","last_change":"2024-09-16T01:53:20.576951+0000","last_active":"2024-09-16T01:53:36.546415+0000","last_peered":"2024-09-16T01:53:36.546415+0000","last_clean":"2024-09-16T01:53:36.546415+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:36.546415+0000","last_undegraded":"2024-09-16T01:53:36.546415+0000","last_fullsized":"2024-09-16T01:53:36.546415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:20.021875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"127'164","reported_seq":494,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545261+0000","last_change":"2024-09-16T01:53:23.517332+0000","last_active":"2024-09-16T01:53:36.545261+0000","last_peered":"2024-09-16T01:53:36.545261+0000","last_clean":"2024-09-16T01:53:36.545261+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:36.545261+0000","last_undegraded":"2024-09-16T01:53:36.545261+0000","last_fullsized":"2024-09-16T01:53:36.545261+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:15.929317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":248,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545542+0000","last_change":"2024-09-16T01:53:34.538675+0000","last_active":"2024-09-16T01:53:36.545542+0000","last_peered":"2024-09-16T01:53:36.545542+0000","last_clean":"2024-09-16T01:53:36.545542+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:36.545542+0000","last_undegraded":"2024-09-16T01:53:36.545542+0000","last_fullsized":"2024-09-16T01:53:36.545542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":624,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554614+0000","last_change":"2024-09-16T01:53:36.554614+0000","last_active":"2024-09-16T01:53:36.554614+0000","last_peered":"2024-09-16T01:53:36.554614+0000","last_clean":"2024-09-16T01:53:36.554614+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:36.554614+0000","last_undegraded":"2024-09-16T01:53:36.554614+0000","last_fullsized":"2024-09-16T01:53:36.554614+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:23.435521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":278,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170120+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:36.170120+0000","last_peered":"2024-09-16T01:53:36.170120+0000","last_clean":"2024-09-16T01:53:36.170120+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:36.170120+0000","last_undegraded":"2024-09-16T01:53:36.170120+0000","last_fullsized":"2024-09-16T01:53:36.170120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"127'149","reported_seq":420,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.169976+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:36.169976+0000","last_peered":"2024-09-16T01:53:36.169976+0000","last_clean":"2024-09-16T01:53:36.169976+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:36.169976+0000","last_undegraded":"2024-09-16T01:53:36.169976+0000","last_fullsized":"2024-09-16T01:53:36.169976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"127'190","reported_seq":700,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.170007+0000","last_change":"2024-09-16T01:53:22.527742+0000","last_active":"2024-09-16T01:53:36.170007+0000","last_peered":"2024-09-16T01:53:36.170007+0000","last_clean":"2024-09-16T01:53:36.170007+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:53:36.170007+0000","last_undegraded":"2024-09-16T01:53:36.170007+0000","last_fullsized":"2024-09-16T01:53:36.170007+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553134+0000","last_change":"2024-09-16T01:53:19.535031+0000","last_active":"2024-09-16T01:53:36.553134+0000","last_peered":"2024-09-16T01:53:36.553134+0000","last_clean":"2024-09-16T01:53:36.553134+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:36.553134+0000","last_undegraded":"2024-09-16T01:53:36.553134+0000","last_fullsized":"2024-09-16T01:53:36.553134+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:36.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"127'153","reported_seq":618,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556668+0000","last_change":"2024-09-16T01:53:22.521982+0000","last_active":"2024-09-16T01:53:36.556668+0000","last_peered":"2024-09-16T01:53:36.556668+0000","last_clean":"2024-09-16T01:53:36.556668+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:36.556668+0000","last_undegraded":"2024-09-16T01:53:36.556668+0000","last_fullsized":"2024-09-16T01:53:36.556668+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:35.196217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551329+0000","last_change":"2024-09-16T01:53:19.559597+0000","last_active":"2024-09-16T01:53:36.551329+0000","last_peered":"2024-09-16T01:53:36.551329+0000","last_clean":"2024-09-16T01:53:36.551329+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:36.551329+0000","last_undegraded":"2024-09-16T01:53:36.551329+0000","last_fullsized":"2024-09-16T01:53:36.551329+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552068+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:36.552068+0000","last_peered":"2024-09-16T01:53:36.552068+0000","last_clean":"2024-09-16T01:53:36.552068+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:36.552068+0000","last_undegraded":"2024-09-16T01:53:36.552068+0000","last_fullsized":"2024-09-16T01:53:36.552068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552067+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:36.552067+0000","last_peered":"2024-09-16T01:53:36.552067+0000","last_clean":"2024-09-16T01:53:36.552067+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:36.552067+0000","last_undegraded":"2024-09-16T01:53:36.552067+0000","last_fullsized":"2024-09-16T01:53:36.552067+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":449,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551955+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:36.551955+0000","last_peered":"2024-09-16T01:53:36.551955+0000","last_clean":"2024-09-16T01:53:36.551955+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:36.551955+0000","last_undegraded":"2024-09-16T01:53:36.551955+0000","last_fullsized":"2024-09-16T01:53:36.551955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":56,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545971+0000","last_change":"2024-09-16T01:53:20.557667+0000","last_active":"2024-09-16T01:53:36.545971+0000","last_peered":"2024-09-16T01:53:36.545971+0000","last_clean":"2024-09-16T01:53:36.545971+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:36.545971+0000","last_undegraded":"2024-09-16T01:53:36.545971+0000","last_fullsized":"2024-09-16T01:53:36.545971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:05:38.159129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4442,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10755,"ondisk_log_size":10755,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398513,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":419,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797026,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":382,"ondisk_log_size":382,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509503,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37152,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334688,"statfs":{"total":96636764160,"available":96598720512,"internally_reserved":0,"allocated":1843200,"data_stored":868593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2224128,"data_stored":1257231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509502,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1892352,"data_stored":1457506,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509503,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32992,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338848,"statfs":{"total":96636764160,"available":96602980352,"internally_reserved":0,"allocated":1843200,"data_stored":868525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542206,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1847296,"data_stored":867283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542206,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1826816,"data_stored":868104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542206,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1900544,"data_stored":1457423,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41404,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330436,"statfs":{"total":96636764160,"available":96594366464,"internally_reserved":0,"allocated":1937408,"data_stored":1258889,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393165,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:40.156 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T01:53:40.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T01:53:40.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.352+0000 7f0e09400640 1 Processor -- start 2024-09-16T01:53:40.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.352+0000 7f0e09400640 1 -- start start 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e09400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e09400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 0x7f0e040c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e09400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0e0406fee0 0x7f0e0415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e09400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0e040c8530 con 0x7f0e040c6cc0 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e09400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0e04158890 con 0x7f0e0406fee0 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e09400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0e04158b00 con 0x7f0e04152760 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e01e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 0x7f0e040c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e03200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0e0406fee0 0x7f0e0415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e01e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 0x7f0e040c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47372/0 (socket says 172.21.15.146:47372) 2024-09-16T01:53:40.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e01e00640 1 -- 172.21.15.146:0/77285573 learned_addr learned my addr 172.21.15.146:0/77285573 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.353+0000 7f0e02800640 1 --2- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.354+0000 7f0e03200640 1 -- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 msgr2=0x7f0e040c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.354+0000 7f0e03200640 1 --2- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.354+0000 7f0e03200640 1 -- 172.21.15.146:0/77285573 >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 msgr2=0x7f0e040c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.354+0000 7f0e03200640 1 --2- 172.21.15.146:0/77285573 >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 0x7f0e040c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.354+0000 7f0e03200640 1 -- 172.21.15.146:0/77285573 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e04158de0 con 0x7f0e0406fee0 2024-09-16T01:53:40.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.354+0000 7f0e01e00640 1 --2- 172.21.15.146:0/77285573 >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 0x7f0e040c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.355+0000 7f0e03200640 1 --2- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0415bd70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0e040c6950 tx=0x7f0dfc02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da90cae0c5af38b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.355+0000 7f0e01400640 1 -- 172.21.15.146:0/77285573 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc007740 con 0x7f0e0406fee0 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.355+0000 7f0e01400640 1 -- 172.21.15.146:0/77285573 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dfc0078e0 con 0x7f0e0406fee0 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.355+0000 7f0e01400640 1 -- 172.21.15.146:0/77285573 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc007cb0 con 0x7f0e0406fee0 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 -- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 msgr2=0x7f0e0415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 --2- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0415bd70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0e040c6950 tx=0x7f0dfc02a660 comp rx=0 tx=0).stop 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 -- 172.21.15.146:0/77285573 shutdown_connections 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 --2- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0415bd70 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 --2- 172.21.15.146:0/77285573 >> v2:172.21.15.146:3300/0 conn(0x7f0e040c6cc0 0x7f0e040c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 --2- 172.21.15.146:0/77285573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 -- 172.21.15.146:0/77285573 >> 172.21.15.146:0/77285573 conn(0x7f0e0406e670 msgr2=0x7f0e0406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:40.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 -- 172.21.15.146:0/77285573 shutdown_connections 2024-09-16T01:53:40.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.356+0000 7f0e09400640 1 -- 172.21.15.146:0/77285573 wait complete. 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.357+0000 7f0e09400640 1 Processor -- start 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.357+0000 7f0e09400640 1 -- start start 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.358+0000 7f0e09400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.358+0000 7f0e09400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 0x7f0e040cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.358+0000 7f0e09400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.358+0000 7f0e09400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0e04152b40 con 0x7f0e040c6cc0 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.358+0000 7f0e09400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e041593a0 con 0x7f0e0406fee0 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.358+0000 7f0e09400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0e04152cc0 con 0x7f0e04152760 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 0x7f0e040cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 0x7f0e040cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47378/0 (socket says 172.21.15.146:47378) 2024-09-16T01:53:40.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 -- 172.21.15.146:0/3814966309 learned_addr learned my addr 172.21.15.146:0/3814966309 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e02800640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e03200640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 -- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 msgr2=0x7f0e040ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 -- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 msgr2=0x7f0e0414f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0414f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 -- 172.21.15.146:0/3814966309 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0414cba0 con 0x7f0e040c6cc0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e03200640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0e01e00640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 0x7f0e040cab10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0df000e980 tx=0x7f0df000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df000ca00 con 0x7f0e040c6cc0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df000cba0 con 0x7f0e040c6cc0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.359+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0004030 con 0x7f0e040c6cc0 2024-09-16T01:53:40.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.360+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e040cb050 con 0x7f0e040c6cc0 2024-09-16T01:53:40.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.360+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0414ce40 con 0x7f0e040c6cc0 2024-09-16T01:53:40.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.361+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e040c7230 con 0x7f0e040c6cc0 2024-09-16T01:53:40.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.361+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0df00041d0 con 0x7f0e040c6cc0 2024-09-16T01:53:40.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.363+0000 7f0dfaa00640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0de406c2b0 0x7f0de406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:40.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.363+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f0df008b560 con 0x7f0e040c6cc0 2024-09-16T01:53:40.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.363+0000 7f0e02800640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0de406c2b0 0x7f0de406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:40.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.364+0000 7f0e02800640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0de406c2b0 0x7f0de406e770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009aa0 tx=0x7f0df4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:40.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.367+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0df0017020 con 0x7f0e040c6cc0 2024-09-16T01:53:40.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.543+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) -- 0x7f0e04072140 con 0x7f0e040c6cc0 2024-09-16T01:53:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.546+0000 7f0dfaa00640 1 -- 172.21.15.146:0/3814966309 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v11) ==== 118+0+0 (secure 0 0 0) 0x7f0df00594b0 con 0x7f0e040c6cc0 2024-09-16T01:53:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.548+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0de406c2b0 msgr2=0x7f0de406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.548+0000 7f0e09400640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0de406c2b0 0x7f0de406e770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009aa0 tx=0x7f0df4009190 comp rx=0 tx=0).stop 2024-09-16T01:53:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.548+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 msgr2=0x7f0e040cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.548+0000 7f0e09400640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 0x7f0e040cab10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0df000e980 tx=0x7f0df000ee40 comp rx=0 tx=0).stop 2024-09-16T01:53:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.548+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 shutdown_connections 2024-09-16T01:53:40.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f0de406c2b0 0x7f0de406e770 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0e04152760 0x7f0e040ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e040c6cc0 0x7f0e040cab10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 --2- 172.21.15.146:0/3814966309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e0406fee0 0x7f0e0414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 >> 172.21.15.146:0/3814966309 conn(0x7f0e0406e670 msgr2=0x7f0e040c59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 shutdown_connections 2024-09-16T01:53:40.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:40.549+0000 7f0e09400640 1 -- 172.21.15.146:0/3814966309 wait complete. 2024-09-16T01:53:40.565 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-16T01:53:40.712 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-16T01:53:40.712 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:53:40.712 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:53:40.749 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-16T01:53:40.749 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:53:40.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:53:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.017+0000 7f18cf200640 1 Processor -- start 2024-09-16T01:53:41.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.017+0000 7f18cf200640 1 -- start start 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cf200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cf200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18d00cc040 0x7f18d00cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cf200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d006d140 con 0x7f18d00c6b70 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cf200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f18d015b880 con 0x7f18d00cc040 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cf200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f18d00cab20 con 0x7f18d00c75a0 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cde00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f18d00c6b70 0x7f18d00cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47264/0 (socket says 172.21.15.146:47264) 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cde00640 1 -- 172.21.15.146:0/1669032473 learned_addr learned my addr 172.21.15.146:0/1669032473 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:41.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18cd400640 1 --2- 172.21.15.146:0/1669032473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.018+0000 7f18ce800640 1 --2- 172.21.15.146:0/1669032473 >> v2:172.21.15.161:3300/0 conn(0x7f18d00cc040 0x7f18d00cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.019+0000 7f18cca00640 1 -- 172.21.15.146:0/1669032473 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1325125923 0 0) 0x7f18d006d140 con 0x7f18d00c6b70 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.019+0000 7f18cca00640 1 -- 172.21.15.146:0/1669032473 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18b4003520 con 0x7f18d00c6b70 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.019+0000 7f18cd400640 1 -- 172.21.15.146:0/1669032473 >> v2:172.21.15.161:3300/0 conn(0x7f18d00cc040 msgr2=0x7f18d00cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.019+0000 7f18cd400640 1 --2- 172.21.15.146:0/1669032473 >> v2:172.21.15.161:3300/0 conn(0x7f18d00cc040 0x7f18d00cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.019+0000 7f18cd400640 1 -- 172.21.15.146:0/1669032473 >> v1:172.21.15.146:6789/0 conn(0x7f18d00c6b70 legacy=0x7f18d00cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:41.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.019+0000 7f18cd400640 1 -- 172.21.15.146:0/1669032473 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d006d420 con 0x7f18d00c75a0 2024-09-16T01:53:41.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.020+0000 7f18cd400640 1 --2- 172.21.15.146:0/1669032473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015b340 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0097a0 tx=0x7f18bc02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d48b95f5de19a59e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:41.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.020+0000 7f18cca00640 1 -- 172.21.15.146:0/1669032473 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc037070 con 0x7f18d00c75a0 2024-09-16T01:53:41.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.020+0000 7f18cca00640 1 -- 172.21.15.146:0/1669032473 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc02ae50 con 0x7f18d00c75a0 2024-09-16T01:53:41.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.020+0000 7f18cca00640 1 -- 172.21.15.146:0/1669032473 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc032640 con 0x7f18d00c75a0 2024-09-16T01:53:41.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 -- 172.21.15.146:0/1669032473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 msgr2=0x7f18d015b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 --2- 172.21.15.146:0/1669032473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015b340 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0097a0 tx=0x7f18bc02a100 comp rx=0 tx=0).stop 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 -- 172.21.15.146:0/1669032473 shutdown_connections 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 --2- 172.21.15.146:0/1669032473 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015b340 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 --2- 172.21.15.146:0/1669032473 >> v2:172.21.15.161:3300/0 conn(0x7f18d00cc040 0x7f18d00cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 -- 172.21.15.146:0/1669032473 >> 172.21.15.146:0/1669032473 conn(0x7f18d00ccc20 msgr2=0x7f18d00cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 -- 172.21.15.146:0/1669032473 shutdown_connections 2024-09-16T01:53:41.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.021+0000 7f18cf200640 1 -- 172.21.15.146:0/1669032473 wait complete. 2024-09-16T01:53:41.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.022+0000 7f18cf200640 1 Processor -- start 2024-09-16T01:53:41.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 -- start start 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18d00c6b70 0x7f18d01f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 0x7f18d01514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f18d00cb050 con 0x7f18d00cc040 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f18d00cad70 con 0x7f18d00c6b70 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.023+0000 7f18cf200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f18d00cbc80 con 0x7f18d00c75a0 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 0x7f18d01514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:41.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 0x7f18d01514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47388/0 (socket says 172.21.15.146:47388) 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 -- 172.21.15.146:0/2536303307 learned_addr learned my addr 172.21.15.146:0/2536303307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18cde00640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18d00c6b70 0x7f18d01f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18cd400640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 -- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 msgr2=0x7f18d015ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 -- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18d00c6b70 msgr2=0x7f18d01f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18d00c6b70 0x7f18d01f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 -- 172.21.15.146:0/2536303307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d00c2890 con 0x7f18d00cc040 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.024+0000 7f18ce800640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 0x7f18d01514c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f18c800efc0 tx=0x7f18c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.025+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8018020 con 0x7f18d00cc040 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.025+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c8014480 con 0x7f18d00cc040 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.025+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c8014870 con 0x7f18d00cc040 2024-09-16T01:53:41.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.025+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d00c2b30 con 0x7f18d00cc040 2024-09-16T01:53:41.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.025+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d015bea0 con 0x7f18d00cc040 2024-09-16T01:53:41.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.026+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d006cd20 con 0x7f18d00cc040 2024-09-16T01:53:41.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.027+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f18c801b020 con 0x7f18d00cc040 2024-09-16T01:53:41.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.028+0000 7f18c6000640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f18b006c2b0 0x7f18b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:41.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.028+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5857+0+0 (secure 0 0 0) 0x7f18c808c1f0 con 0x7f18d00cc040 2024-09-16T01:53:41.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.028+0000 7f18cde00640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f18b006c2b0 0x7f18b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:41.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.029+0000 7f18cde00640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f18b006c2b0 0x7f18b006e770 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f18c0005880 tx=0x7f18c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:41.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.033+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f18c8022020 con 0x7f18d00cc040 2024-09-16T01:53:41.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.217+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f18d00ca280 con 0x7f18d00cc040 2024-09-16T01:53:41.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.217+0000 7f18c6000640 1 -- 172.21.15.146:0/2536303307 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v152) ==== 80+0+0 (secure 0 0 0) 0x7f18c8010040 con 0x7f18d00cc040 2024-09-16T01:53:41.218 INFO:teuthology.orchestra.run.smithi146.stderr:listed 0 entries 2024-09-16T01:53:41.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f18b006c2b0 msgr2=0x7f18b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f18b006c2b0 0x7f18b006e770 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f18c0005880 tx=0x7f18c0009040 comp rx=0 tx=0).stop 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 msgr2=0x7f18d01514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 0x7f18d01514c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f18c800efc0 tx=0x7f18c800c490 comp rx=0 tx=0).stop 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 shutdown_connections 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f18b006c2b0 0x7f18b006e770 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d00cc040 0x7f18d01514c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d00c75a0 0x7f18d015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 --2- 172.21.15.146:0/2536303307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18d00c6b70 0x7f18d01f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 >> 172.21.15.146:0/2536303307 conn(0x7f18d00ccc20 msgr2=0x7f18d00cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 shutdown_connections 2024-09-16T01:53:41.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:41.219+0000 7f18cf200640 1 -- 172.21.15.146:0/2536303307 wait complete. 2024-09-16T01:53:41.237 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:53:41.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:53:41.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:53:41.324 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:53:41.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:53:41.325 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:53:41.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:53:41.394 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-09-16T01:53:41.429 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:53:41.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:53:41.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:53:41.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:53:41.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-09-16T01:53:41.852 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T01:53:41.862 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:53:41.862 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:42.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.018+0000 7f47da400640 1 Processor -- start 2024-09-16T01:53:42.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.018+0000 7f47da400640 1 -- start start 2024-09-16T01:53:42.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47da400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f47d4152850 0x7f47d4158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47da400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47d40c6bb0 0x7f47d40c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47da400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d4072f50 con 0x7f47d4158bb0 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47da400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f47d4073210 con 0x7f47d4152850 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47da400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f47d415cb50 con 0x7f47d40c6bb0 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47d3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f47d4152850 0x7f47d4158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47caa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f47d4158bb0 0x7f47d40c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47272/0 (socket says 172.21.15.146:47272) 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47caa00640 1 -- 172.21.15.146:0/3252601810 learned_addr learned my addr 172.21.15.146:0/3252601810 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47d3e00640 1 --2- 172.21.15.146:0/3252601810 >> v2:172.21.15.161:3300/0 conn(0x7f47d40c6bb0 0x7f47d40c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47d3400640 1 -- 172.21.15.146:0/3252601810 >> v2:172.21.15.161:3300/0 conn(0x7f47d40c6bb0 msgr2=0x7f47d40c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47d3400640 1 --2- 172.21.15.146:0/3252601810 >> v2:172.21.15.161:3300/0 conn(0x7f47d40c6bb0 0x7f47d40c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47d3400640 1 -- 172.21.15.146:0/3252601810 >> v1:172.21.15.146:6789/0 conn(0x7f47d4158bb0 legacy=0x7f47d40c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.020+0000 7f47d3400640 1 -- 172.21.15.146:0/3252601810 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d415bd90 con 0x7f47d4152850 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.021+0000 7f47d3400640 1 --2- 172.21.15.146:0/3252601810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4152850 0x7f47d4158670 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f47d406f350 tx=0x7f47c402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ed8b3652e92f66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47d2a00640 1 -- 172.21.15.146:0/3252601810 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4037070 con 0x7f47d4152850 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47d2a00640 1 -- 172.21.15.146:0/3252601810 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c4032440 con 0x7f47d4152850 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47d2a00640 1 -- 172.21.15.146:0/3252601810 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4032810 con 0x7f47d4152850 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 -- 172.21.15.146:0/3252601810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4152850 msgr2=0x7f47d4158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 --2- 172.21.15.146:0/3252601810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4152850 0x7f47d4158670 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f47d406f350 tx=0x7f47c402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 -- 172.21.15.146:0/3252601810 shutdown_connections 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 --2- 172.21.15.146:0/3252601810 >> v2:172.21.15.161:3300/0 conn(0x7f47d40c6bb0 0x7f47d40c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 --2- 172.21.15.146:0/3252601810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4152850 0x7f47d4158670 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 -- 172.21.15.146:0/3252601810 >> 172.21.15.146:0/3252601810 conn(0x7f47d406f3c0 msgr2=0x7f47d406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:42.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 -- 172.21.15.146:0/3252601810 shutdown_connections 2024-09-16T01:53:42.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.022+0000 7f47da400640 1 -- 172.21.15.146:0/3252601810 wait complete. 2024-09-16T01:53:42.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.024+0000 7f47da400640 1 Processor -- start 2024-09-16T01:53:42.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.024+0000 7f47da400640 1 -- start start 2024-09-16T01:53:42.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47da400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f47d40c6bb0 0x7f47d414cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47da400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47d4152850 0x7f47d414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47da400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 0x7f47d414bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47da400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f47d415c630 con 0x7f47d4158bb0 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47da400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47d415c350 con 0x7f47d4152850 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47da400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f47d4070200 con 0x7f47d40c6bb0 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 0x7f47d414bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 0x7f47d414bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47402/0 (socket says 172.21.15.146:47402) 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47d3e00640 1 -- 172.21.15.146:0/1554805581 learned_addr learned my addr 172.21.15.146:0/1554805581 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47caa00640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47d4152850 0x7f47d414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.025+0000 7f47d3400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f47d40c6bb0 0x7f47d414cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47d3e00640 1 -- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f47d40c6bb0 msgr2=0x7f47d414cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47d3e00640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f47d40c6bb0 0x7f47d414cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47d3e00640 1 -- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47d4152850 msgr2=0x7f47d414b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47d3e00640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47d4152850 0x7f47d414b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47d3e00640 1 -- 172.21.15.146:0/1554805581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d41f18a0 con 0x7f47d4158bb0 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47d3e00640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 0x7f47d414bd00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00efc0 tx=0x7f47cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc018020 con 0x7f47d4158bb0 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47cc014480 con 0x7f47d4158bb0 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d41f1b80 con 0x7f47d4158bb0 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc014870 con 0x7f47d4158bb0 2024-09-16T01:53:42.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.026+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d41f2140 con 0x7f47d4158bb0 2024-09-16T01:53:42.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.027+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d40c7620 con 0x7f47d4158bb0 2024-09-16T01:53:42.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.028+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f47cc01b020 con 0x7f47d4158bb0 2024-09-16T01:53:42.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.029+0000 7f47cbe00640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47b006c2b0 0x7f47b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.029+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5886+0+0 (secure 0 0 0) 0x7f47cc08c210 con 0x7f47d4158bb0 2024-09-16T01:53:42.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.030+0000 7f47d3400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47b006c2b0 0x7f47b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.030+0000 7f47d3400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47b006c2b0 0x7f47b006e770 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f47c4034000 tx=0x7f47c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.033+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f47d40c7620 con 0x7f47d4158bb0 2024-09-16T01:53:42.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.206+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f47d40718c0 con 0x7f47b006c2b0 2024-09-16T01:53:42.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.212+0000 7f47cbe00640 1 -- 172.21.15.146:0/1554805581 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253935 (secure 0 0 0) 0x7f47d40718c0 con 0x7f47b006c2b0 2024-09-16T01:53:42.216 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:42.218 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47b006c2b0 msgr2=0x7f47b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47b006c2b0 0x7f47b006e770 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f47c4034000 tx=0x7f47c402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 msgr2=0x7f47d414bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 0x7f47d414bd00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00efc0 tx=0x7f47cc00c490 comp rx=0 tx=0).stop 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 shutdown_connections 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f47b006c2b0 0x7f47b006e770 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d4158bb0 0x7f47d414bd00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47d4152850 0x7f47d414b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 --2- 172.21.15.146:0/1554805581 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f47d40c6bb0 0x7f47d414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 >> 172.21.15.146:0/1554805581 conn(0x7f47d406f3c0 msgr2=0x7f47d4150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:42.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 shutdown_connections 2024-09-16T01:53:42.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.217+0000 7f47da400640 1 -- 172.21.15.146:0/1554805581 wait complete. 2024-09-16T01:53:42.235 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-16T01:53:41.645072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":989022,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9795,"num_write_kb":25168839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4698,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11471,"ondisk_log_size":11471,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":281212,"kb_used_data":15040,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693508,"statfs":{"total":773094113280,"available":772806152192,"internally_reserved":0,"allocated":15400960,"data_stored":8946150,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-163,"num_bytes_recovered":-808,"num_keys_recovered":-40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.072109"},"pg_stats":[{"pgid":"3.3f","version":"152'167","reported_seq":492,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361768+0000","last_change":"2024-09-16T01:53:22.525372+0000","last_active":"2024-09-16T01:53:41.361768+0000","last_peered":"2024-09-16T01:53:41.361768+0000","last_clean":"2024-09-16T01:53:41.361768+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:41.361768+0000","last_undegraded":"2024-09-16T01:53:41.361768+0000","last_fullsized":"2024-09-16T01:53:41.361768+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:00.392212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551885+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:36.551885+0000","last_peered":"2024-09-16T01:53:36.551885+0000","last_clean":"2024-09-16T01:53:36.551885+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:36.551885+0000","last_undegraded":"2024-09-16T01:53:36.551885+0000","last_fullsized":"2024-09-16T01:53:36.551885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":799,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551822+0000","last_change":"2024-09-16T01:53:23.518810+0000","last_active":"2024-09-16T01:53:36.551822+0000","last_peered":"2024-09-16T01:53:36.551822+0000","last_clean":"2024-09-16T01:53:36.551822+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:53:36.551822+0000","last_undegraded":"2024-09-16T01:53:36.551822+0000","last_fullsized":"2024-09-16T01:53:36.551822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"127'185","reported_seq":583,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556840+0000","last_change":"2024-09-16T01:53:32.511261+0000","last_active":"2024-09-16T01:53:36.556840+0000","last_peered":"2024-09-16T01:53:36.556840+0000","last_clean":"2024-09-16T01:53:36.556840+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:36.556840+0000","last_undegraded":"2024-09-16T01:53:36.556840+0000","last_fullsized":"2024-09-16T01:53:36.556840+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:07.896976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"152'174","reported_seq":526,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.342604+0000","last_change":"2024-09-16T01:53:22.521371+0000","last_active":"2024-09-16T01:53:41.342604+0000","last_peered":"2024-09-16T01:53:41.342604+0000","last_clean":"2024-09-16T01:53:41.342604+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:41.342604+0000","last_undegraded":"2024-09-16T01:53:41.342604+0000","last_fullsized":"2024-09-16T01:53:41.342604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:39.028997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"127'156","reported_seq":576,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555886+0000","last_change":"2024-09-16T01:53:23.521468+0000","last_active":"2024-09-16T01:53:36.555886+0000","last_peered":"2024-09-16T01:53:36.555886+0000","last_clean":"2024-09-16T01:53:36.555886+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:36.555886+0000","last_undegraded":"2024-09-16T01:53:36.555886+0000","last_fullsized":"2024-09-16T01:53:36.555886+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:34:42.522900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"152'188","reported_seq":638,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.345763+0000","last_change":"2024-09-16T01:53:22.523352+0000","last_active":"2024-09-16T01:53:41.345763+0000","last_peered":"2024-09-16T01:53:41.345763+0000","last_clean":"2024-09-16T01:53:41.345763+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:41.345763+0000","last_undegraded":"2024-09-16T01:53:41.345763+0000","last_fullsized":"2024-09-16T01:53:41.345763+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.078885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":722,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551418+0000","last_change":"2024-09-16T01:53:22.523335+0000","last_active":"2024-09-16T01:53:36.551418+0000","last_peered":"2024-09-16T01:53:36.551418+0000","last_clean":"2024-09-16T01:53:36.551418+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:53:36.551418+0000","last_undegraded":"2024-09-16T01:53:36.551418+0000","last_fullsized":"2024-09-16T01:53:36.551418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"152'178","reported_seq":451,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.357816+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:41.357816+0000","last_peered":"2024-09-16T01:53:41.357816+0000","last_clean":"2024-09-16T01:53:41.357816+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:41.357816+0000","last_undegraded":"2024-09-16T01:53:41.357816+0000","last_fullsized":"2024-09-16T01:53:41.357816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"152'190","reported_seq":570,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.356987+0000","last_change":"2024-09-16T01:53:22.526050+0000","last_active":"2024-09-16T01:53:41.356987+0000","last_peered":"2024-09-16T01:53:41.356987+0000","last_clean":"2024-09-16T01:53:41.356987+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:41.356987+0000","last_undegraded":"2024-09-16T01:53:41.356987+0000","last_fullsized":"2024-09-16T01:53:41.356987+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:00.473148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"152'214","reported_seq":617,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.357952+0000","last_change":"2024-09-16T01:53:23.519544+0000","last_active":"2024-09-16T01:53:41.357952+0000","last_peered":"2024-09-16T01:53:41.357952+0000","last_clean":"2024-09-16T01:53:41.357952+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:41.357952+0000","last_undegraded":"2024-09-16T01:53:41.357952+0000","last_fullsized":"2024-09-16T01:53:41.357952+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:26.797344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"152'166","reported_seq":362,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343823+0000","last_change":"2024-09-16T01:53:22.519665+0000","last_active":"2024-09-16T01:53:41.343823+0000","last_peered":"2024-09-16T01:53:41.343823+0000","last_clean":"2024-09-16T01:53:41.343823+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:41.343823+0000","last_undegraded":"2024-09-16T01:53:41.343823+0000","last_fullsized":"2024-09-16T01:53:41.343823+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:32.846352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545527+0000","last_change":"2024-09-16T01:53:34.533587+0000","last_active":"2024-09-16T01:53:36.545527+0000","last_peered":"2024-09-16T01:53:36.545527+0000","last_clean":"2024-09-16T01:53:36.545527+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:36.545527+0000","last_undegraded":"2024-09-16T01:53:36.545527+0000","last_fullsized":"2024-09-16T01:53:36.545527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:15.453058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":488,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545392+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:36.545392+0000","last_peered":"2024-09-16T01:53:36.545392+0000","last_clean":"2024-09-16T01:53:36.545392+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:36.545392+0000","last_undegraded":"2024-09-16T01:53:36.545392+0000","last_fullsized":"2024-09-16T01:53:36.545392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551212+0000","last_change":"2024-09-16T01:53:22.527702+0000","last_active":"2024-09-16T01:53:36.551212+0000","last_peered":"2024-09-16T01:53:36.551212+0000","last_clean":"2024-09-16T01:53:36.551212+0000","last_became_active":"2024-09-16T01:53:22.526994+0000","last_became_peered":"2024-09-16T01:53:22.526994+0000","last_unstale":"2024-09-16T01:53:36.551212+0000","last_undegraded":"2024-09-16T01:53:36.551212+0000","last_fullsized":"2024-09-16T01:53:36.551212+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"152'162","reported_seq":435,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.359260+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:41.359260+0000","last_peered":"2024-09-16T01:53:41.359260+0000","last_clean":"2024-09-16T01:53:41.359260+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:41.359260+0000","last_undegraded":"2024-09-16T01:53:41.359260+0000","last_fullsized":"2024-09-16T01:53:41.359260+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"152'176","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361166+0000","last_change":"2024-09-16T01:53:23.526130+0000","last_active":"2024-09-16T01:53:41.361166+0000","last_peered":"2024-09-16T01:53:41.361166+0000","last_clean":"2024-09-16T01:53:41.361166+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:53:41.361166+0000","last_undegraded":"2024-09-16T01:53:41.361166+0000","last_fullsized":"2024-09-16T01:53:41.361166+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:02.127294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"127'169","reported_seq":575,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563883+0000","last_change":"2024-09-16T01:53:36.563883+0000","last_active":"2024-09-16T01:53:36.563883+0000","last_peered":"2024-09-16T01:53:36.563883+0000","last_clean":"2024-09-16T01:53:36.563883+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:36.563883+0000","last_undegraded":"2024-09-16T01:53:36.563883+0000","last_fullsized":"2024-09-16T01:53:36.563883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554778+0000","last_change":"2024-09-16T01:53:36.554778+0000","last_active":"2024-09-16T01:53:36.554778+0000","last_peered":"2024-09-16T01:53:36.554778+0000","last_clean":"2024-09-16T01:53:36.554778+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:36.554778+0000","last_undegraded":"2024-09-16T01:53:36.554778+0000","last_fullsized":"2024-09-16T01:53:36.554778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"152'179","reported_seq":596,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343701+0000","last_change":"2024-09-16T01:53:23.524759+0000","last_active":"2024-09-16T01:53:41.343701+0000","last_peered":"2024-09-16T01:53:41.343701+0000","last_clean":"2024-09-16T01:53:41.343701+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:41.343701+0000","last_undegraded":"2024-09-16T01:53:41.343701+0000","last_fullsized":"2024-09-16T01:53:41.343701+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"127'143","reported_seq":455,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545707+0000","last_change":"2024-09-16T01:53:23.520715+0000","last_active":"2024-09-16T01:53:36.545707+0000","last_peered":"2024-09-16T01:53:36.545707+0000","last_clean":"2024-09-16T01:53:36.545707+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:36.545707+0000","last_undegraded":"2024-09-16T01:53:36.545707+0000","last_fullsized":"2024-09-16T01:53:36.545707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:45.628132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"152'199","reported_seq":831,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.353111+0000","last_change":"2024-09-16T01:53:22.524407+0000","last_active":"2024-09-16T01:53:41.353111+0000","last_peered":"2024-09-16T01:53:41.353111+0000","last_clean":"2024-09-16T01:53:41.353111+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:41.353111+0000","last_undegraded":"2024-09-16T01:53:41.353111+0000","last_fullsized":"2024-09-16T01:53:41.353111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551482+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:36.551482+0000","last_peered":"2024-09-16T01:53:36.551482+0000","last_clean":"2024-09-16T01:53:36.551482+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:36.551482+0000","last_undegraded":"2024-09-16T01:53:36.551482+0000","last_fullsized":"2024-09-16T01:53:36.551482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"152'207","reported_seq":480,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.359084+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:41.359084+0000","last_peered":"2024-09-16T01:53:41.359084+0000","last_clean":"2024-09-16T01:53:41.359084+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:41.359084+0000","last_undegraded":"2024-09-16T01:53:41.359084+0000","last_fullsized":"2024-09-16T01:53:41.359084+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":431,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552017+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:36.552017+0000","last_peered":"2024-09-16T01:53:36.552017+0000","last_clean":"2024-09-16T01:53:36.552017+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:36.552017+0000","last_undegraded":"2024-09-16T01:53:36.552017+0000","last_fullsized":"2024-09-16T01:53:36.552017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"127'202","reported_seq":476,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559877+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:36.559877+0000","last_peered":"2024-09-16T01:53:36.559877+0000","last_clean":"2024-09-16T01:53:36.559877+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:36.559877+0000","last_undegraded":"2024-09-16T01:53:36.559877+0000","last_fullsized":"2024-09-16T01:53:36.559877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":664,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545722+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:36.545722+0000","last_peered":"2024-09-16T01:53:36.545722+0000","last_clean":"2024-09-16T01:53:36.545722+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:36.545722+0000","last_undegraded":"2024-09-16T01:53:36.545722+0000","last_fullsized":"2024-09-16T01:53:36.545722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545688+0000","last_change":"2024-09-16T01:53:34.533524+0000","last_active":"2024-09-16T01:53:36.545688+0000","last_peered":"2024-09-16T01:53:36.545688+0000","last_clean":"2024-09-16T01:53:36.545688+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:36.545688+0000","last_undegraded":"2024-09-16T01:53:36.545688+0000","last_fullsized":"2024-09-16T01:53:36.545688+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:57.738639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":683,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551567+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:36.551567+0000","last_peered":"2024-09-16T01:53:36.551567+0000","last_clean":"2024-09-16T01:53:36.551567+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:36.551567+0000","last_undegraded":"2024-09-16T01:53:36.551567+0000","last_fullsized":"2024-09-16T01:53:36.551567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"152'153","reported_seq":426,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.355574+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:41.355574+0000","last_peered":"2024-09-16T01:53:41.355574+0000","last_clean":"2024-09-16T01:53:41.355574+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:41.355574+0000","last_undegraded":"2024-09-16T01:53:41.355574+0000","last_fullsized":"2024-09-16T01:53:41.355574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"127'146","reported_seq":558,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563260+0000","last_change":"2024-09-16T01:53:36.563260+0000","last_active":"2024-09-16T01:53:36.563260+0000","last_peered":"2024-09-16T01:53:36.563260+0000","last_clean":"2024-09-16T01:53:36.563260+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:36.563260+0000","last_undegraded":"2024-09-16T01:53:36.563260+0000","last_fullsized":"2024-09-16T01:53:36.563260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:31.539361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551939+0000","last_change":"2024-09-16T01:53:36.551939+0000","last_active":"2024-09-16T01:53:36.551939+0000","last_peered":"2024-09-16T01:53:36.551939+0000","last_clean":"2024-09-16T01:53:36.551939+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:36.551939+0000","last_undegraded":"2024-09-16T01:53:36.551939+0000","last_fullsized":"2024-09-16T01:53:36.551939+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"152'149","reported_seq":422,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.338192+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:41.338192+0000","last_peered":"2024-09-16T01:53:41.338192+0000","last_clean":"2024-09-16T01:53:41.338192+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:41.338192+0000","last_undegraded":"2024-09-16T01:53:41.338192+0000","last_fullsized":"2024-09-16T01:53:41.338192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551629+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:36.551629+0000","last_peered":"2024-09-16T01:53:36.551629+0000","last_clean":"2024-09-16T01:53:36.551629+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:36.551629+0000","last_undegraded":"2024-09-16T01:53:36.551629+0000","last_fullsized":"2024-09-16T01:53:36.551629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"127'154","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545597+0000","last_change":"2024-09-16T01:53:22.520820+0000","last_active":"2024-09-16T01:53:36.545597+0000","last_peered":"2024-09-16T01:53:36.545597+0000","last_clean":"2024-09-16T01:53:36.545597+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:36.545597+0000","last_undegraded":"2024-09-16T01:53:36.545597+0000","last_fullsized":"2024-09-16T01:53:36.545597+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:51.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842491+0000","last_change":"2024-09-16T01:53:19.532463+0000","last_active":"2024-09-16T01:53:36.842491+0000","last_peered":"2024-09-16T01:53:36.842491+0000","last_clean":"2024-09-16T01:53:36.842491+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:36.842491+0000","last_undegraded":"2024-09-16T01:53:36.842491+0000","last_fullsized":"2024-09-16T01:53:36.842491+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:52.176146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":275,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545279+0000","last_change":"2024-09-16T01:53:34.532869+0000","last_active":"2024-09-16T01:53:36.545279+0000","last_peered":"2024-09-16T01:53:36.545279+0000","last_clean":"2024-09-16T01:53:36.545279+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:36.545279+0000","last_undegraded":"2024-09-16T01:53:36.545279+0000","last_fullsized":"2024-09-16T01:53:36.545279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:13.148781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":454,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551995+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:36.551995+0000","last_peered":"2024-09-16T01:53:36.551995+0000","last_clean":"2024-09-16T01:53:36.551995+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:36.551995+0000","last_undegraded":"2024-09-16T01:53:36.551995+0000","last_fullsized":"2024-09-16T01:53:36.551995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":277,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.840791+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:36.840791+0000","last_peered":"2024-09-16T01:53:36.840791+0000","last_clean":"2024-09-16T01:53:36.840791+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:36.840791+0000","last_undegraded":"2024-09-16T01:53:36.840791+0000","last_fullsized":"2024-09-16T01:53:36.840791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"127'177","reported_seq":418,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545675+0000","last_change":"2024-09-16T01:53:32.513845+0000","last_active":"2024-09-16T01:53:36.545675+0000","last_peered":"2024-09-16T01:53:36.545675+0000","last_clean":"2024-09-16T01:53:36.545675+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:36.545675+0000","last_undegraded":"2024-09-16T01:53:36.545675+0000","last_fullsized":"2024-09-16T01:53:36.545675+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:02.738813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":318,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556079+0000","last_change":"2024-09-16T01:53:20.522781+0000","last_active":"2024-09-16T01:53:36.556079+0000","last_peered":"2024-09-16T01:53:36.556079+0000","last_clean":"2024-09-16T01:53:36.556079+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:36.556079+0000","last_undegraded":"2024-09-16T01:53:36.556079+0000","last_fullsized":"2024-09-16T01:53:36.556079+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:25.742037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551737+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:36.551737+0000","last_peered":"2024-09-16T01:53:36.551737+0000","last_clean":"2024-09-16T01:53:36.551737+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:36.551737+0000","last_undegraded":"2024-09-16T01:53:36.551737+0000","last_fullsized":"2024-09-16T01:53:36.551737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553376+0000","last_change":"2024-09-16T01:53:19.579965+0000","last_active":"2024-09-16T01:53:36.553376+0000","last_peered":"2024-09-16T01:53:36.553376+0000","last_clean":"2024-09-16T01:53:36.553376+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:36.553376+0000","last_undegraded":"2024-09-16T01:53:36.553376+0000","last_fullsized":"2024-09-16T01:53:36.553376+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:29.737898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":423,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552118+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:36.552118+0000","last_peered":"2024-09-16T01:53:36.552118+0000","last_clean":"2024-09-16T01:53:36.552118+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:36.552118+0000","last_undegraded":"2024-09-16T01:53:36.552118+0000","last_fullsized":"2024-09-16T01:53:36.552118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"133'11","reported_seq":160,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557008+0000","last_change":"2024-09-16T01:53:36.557008+0000","last_active":"2024-09-16T01:53:36.557008+0000","last_peered":"2024-09-16T01:53:36.557008+0000","last_clean":"2024-09-16T01:53:36.557008+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:36.557008+0000","last_undegraded":"2024-09-16T01:53:36.557008+0000","last_fullsized":"2024-09-16T01:53:36.557008+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'168","reported_seq":441,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362450+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:41.362450+0000","last_peered":"2024-09-16T01:53:41.362450+0000","last_clean":"2024-09-16T01:53:41.362450+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:41.362450+0000","last_undegraded":"2024-09-16T01:53:41.362450+0000","last_fullsized":"2024-09-16T01:53:41.362450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"134'175","reported_seq":492,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556933+0000","last_change":"2024-09-16T01:53:23.518694+0000","last_active":"2024-09-16T01:53:36.556933+0000","last_peered":"2024-09-16T01:53:36.556933+0000","last_clean":"2024-09-16T01:53:36.556933+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:36.556933+0000","last_undegraded":"2024-09-16T01:53:36.556933+0000","last_fullsized":"2024-09-16T01:53:36.556933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:11.376091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391327,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"152'176","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343763+0000","last_change":"2024-09-16T01:53:22.524097+0000","last_active":"2024-09-16T01:53:41.343763+0000","last_peered":"2024-09-16T01:53:41.343763+0000","last_clean":"2024-09-16T01:53:41.343763+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:41.343763+0000","last_undegraded":"2024-09-16T01:53:41.343763+0000","last_fullsized":"2024-09-16T01:53:41.343763+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:14.041864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559516+0000","last_change":"2024-09-16T01:53:36.559516+0000","last_active":"2024-09-16T01:53:36.559516+0000","last_peered":"2024-09-16T01:53:36.559516+0000","last_clean":"2024-09-16T01:53:36.559516+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:36.559516+0000","last_undegraded":"2024-09-16T01:53:36.559516+0000","last_fullsized":"2024-09-16T01:53:36.559516+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":411,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546458+0000","last_change":"2024-09-16T01:53:34.529911+0000","last_active":"2024-09-16T01:53:36.546458+0000","last_peered":"2024-09-16T01:53:36.546458+0000","last_clean":"2024-09-16T01:53:36.546458+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:36.546458+0000","last_undegraded":"2024-09-16T01:53:36.546458+0000","last_fullsized":"2024-09-16T01:53:36.546458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":358,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842660+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:36.842660+0000","last_peered":"2024-09-16T01:53:36.842660+0000","last_clean":"2024-09-16T01:53:36.842660+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:36.842660+0000","last_undegraded":"2024-09-16T01:53:36.842660+0000","last_fullsized":"2024-09-16T01:53:36.842660+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"152'201","reported_seq":718,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362908+0000","last_change":"2024-09-16T01:53:23.521939+0000","last_active":"2024-09-16T01:53:41.362908+0000","last_peered":"2024-09-16T01:53:41.362908+0000","last_clean":"2024-09-16T01:53:41.362908+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:53:41.362908+0000","last_undegraded":"2024-09-16T01:53:41.362908+0000","last_fullsized":"2024-09-16T01:53:41.362908+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:55.352896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"152'7","reported_seq":53,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552327+0000","last_change":"2024-09-16T01:53:19.550015+0000","last_active":"2024-09-16T01:53:36.552327+0000","last_peered":"2024-09-16T01:53:36.552327+0000","last_clean":"2024-09-16T01:53:36.552327+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:36.552327+0000","last_undegraded":"2024-09-16T01:53:36.552327+0000","last_fullsized":"2024-09-16T01:53:36.552327+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"152'197","reported_seq":471,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.337310+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:41.337310+0000","last_peered":"2024-09-16T01:53:41.337310+0000","last_clean":"2024-09-16T01:53:41.337310+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:41.337310+0000","last_undegraded":"2024-09-16T01:53:41.337310+0000","last_fullsized":"2024-09-16T01:53:41.337310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"133'18","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545354+0000","last_change":"2024-09-16T01:53:20.541056+0000","last_active":"2024-09-16T01:53:36.545354+0000","last_peered":"2024-09-16T01:53:36.545354+0000","last_clean":"2024-09-16T01:53:36.545354+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:36.545354+0000","last_undegraded":"2024-09-16T01:53:36.545354+0000","last_fullsized":"2024-09-16T01:53:36.545354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:21.564935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"152'215","reported_seq":488,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.356427+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:41.356427+0000","last_peered":"2024-09-16T01:53:41.356427+0000","last_clean":"2024-09-16T01:53:41.356427+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:41.356427+0000","last_undegraded":"2024-09-16T01:53:41.356427+0000","last_fullsized":"2024-09-16T01:53:41.356427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553629+0000","last_change":"2024-09-16T01:53:19.563306+0000","last_active":"2024-09-16T01:53:36.553629+0000","last_peered":"2024-09-16T01:53:36.553629+0000","last_clean":"2024-09-16T01:53:36.553629+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:36.553629+0000","last_undegraded":"2024-09-16T01:53:36.553629+0000","last_fullsized":"2024-09-16T01:53:36.553629+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:04.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"127'170","reported_seq":648,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.560008+0000","last_change":"2024-09-16T01:53:22.527907+0000","last_active":"2024-09-16T01:53:36.560008+0000","last_peered":"2024-09-16T01:53:36.560008+0000","last_clean":"2024-09-16T01:53:36.560008+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:36.560008+0000","last_undegraded":"2024-09-16T01:53:36.560008+0000","last_fullsized":"2024-09-16T01:53:36.560008+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:51.042095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546066+0000","last_change":"2024-09-16T01:53:20.547257+0000","last_active":"2024-09-16T01:53:36.546066+0000","last_peered":"2024-09-16T01:53:36.546066+0000","last_clean":"2024-09-16T01:53:36.546066+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:36.546066+0000","last_undegraded":"2024-09-16T01:53:36.546066+0000","last_fullsized":"2024-09-16T01:53:36.546066+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:04.416789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"152'162","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.350317+0000","last_change":"2024-09-16T01:53:23.527787+0000","last_active":"2024-09-16T01:53:41.350317+0000","last_peered":"2024-09-16T01:53:41.350317+0000","last_clean":"2024-09-16T01:53:41.350317+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:53:41.350317+0000","last_undegraded":"2024-09-16T01:53:41.350317+0000","last_fullsized":"2024-09-16T01:53:41.350317+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551843+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:36.551843+0000","last_peered":"2024-09-16T01:53:36.551843+0000","last_clean":"2024-09-16T01:53:36.551843+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:36.551843+0000","last_undegraded":"2024-09-16T01:53:36.551843+0000","last_fullsized":"2024-09-16T01:53:36.551843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"152'165","reported_seq":768,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.334297+0000","last_change":"2024-09-16T01:53:22.522539+0000","last_active":"2024-09-16T01:53:41.334297+0000","last_peered":"2024-09-16T01:53:41.334297+0000","last_clean":"2024-09-16T01:53:41.334297+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:41.334297+0000","last_undegraded":"2024-09-16T01:53:41.334297+0000","last_fullsized":"2024-09-16T01:53:41.334297+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:01.840411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":263,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545924+0000","last_change":"2024-09-16T01:53:34.533293+0000","last_active":"2024-09-16T01:53:36.545924+0000","last_peered":"2024-09-16T01:53:36.545924+0000","last_clean":"2024-09-16T01:53:36.545924+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:36.545924+0000","last_undegraded":"2024-09-16T01:53:36.545924+0000","last_fullsized":"2024-09-16T01:53:36.545924+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:39.949872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"152'175","reported_seq":448,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.358312+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:41.358312+0000","last_peered":"2024-09-16T01:53:41.358312+0000","last_clean":"2024-09-16T01:53:41.358312+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:41.358312+0000","last_undegraded":"2024-09-16T01:53:41.358312+0000","last_fullsized":"2024-09-16T01:53:41.358312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":432,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555710+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:36.555710+0000","last_peered":"2024-09-16T01:53:36.555710+0000","last_clean":"2024-09-16T01:53:36.555710+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:36.555710+0000","last_undegraded":"2024-09-16T01:53:36.555710+0000","last_fullsized":"2024-09-16T01:53:36.555710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.841615+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:36.841615+0000","last_peered":"2024-09-16T01:53:36.841615+0000","last_clean":"2024-09-16T01:53:36.841615+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:36.841615+0000","last_undegraded":"2024-09-16T01:53:36.841615+0000","last_fullsized":"2024-09-16T01:53:36.841615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"127'175","reported_seq":592,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545790+0000","last_change":"2024-09-16T01:53:22.524529+0000","last_active":"2024-09-16T01:53:36.545790+0000","last_peered":"2024-09-16T01:53:36.545790+0000","last_clean":"2024-09-16T01:53:36.545790+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:36.545790+0000","last_undegraded":"2024-09-16T01:53:36.545790+0000","last_fullsized":"2024-09-16T01:53:36.545790+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:39.441997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.840890+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:36.840890+0000","last_peered":"2024-09-16T01:53:36.840890+0000","last_clean":"2024-09-16T01:53:36.840890+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:36.840890+0000","last_undegraded":"2024-09-16T01:53:36.840890+0000","last_fullsized":"2024-09-16T01:53:36.840890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":403,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545398+0000","last_change":"2024-09-16T01:53:34.532478+0000","last_active":"2024-09-16T01:53:36.545398+0000","last_peered":"2024-09-16T01:53:36.545398+0000","last_clean":"2024-09-16T01:53:36.545398+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:36.545398+0000","last_undegraded":"2024-09-16T01:53:36.545398+0000","last_fullsized":"2024-09-16T01:53:36.545398+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"133'12","reported_seq":311,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554302+0000","last_change":"2024-09-16T01:53:36.554302+0000","last_active":"2024-09-16T01:53:36.554302+0000","last_peered":"2024-09-16T01:53:36.554302+0000","last_clean":"2024-09-16T01:53:36.554302+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:36.554302+0000","last_undegraded":"2024-09-16T01:53:36.554302+0000","last_fullsized":"2024-09-16T01:53:36.554302+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551550+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:36.551550+0000","last_peered":"2024-09-16T01:53:36.551550+0000","last_clean":"2024-09-16T01:53:36.551550+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:36.551550+0000","last_undegraded":"2024-09-16T01:53:36.551550+0000","last_fullsized":"2024-09-16T01:53:36.551550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842714+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:36.842714+0000","last_peered":"2024-09-16T01:53:36.842714+0000","last_clean":"2024-09-16T01:53:36.842714+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:36.842714+0000","last_undegraded":"2024-09-16T01:53:36.842714+0000","last_fullsized":"2024-09-16T01:53:36.842714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"152'177","reported_seq":550,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.351445+0000","last_change":"2024-09-16T01:53:23.527347+0000","last_active":"2024-09-16T01:53:41.351445+0000","last_peered":"2024-09-16T01:53:41.351445+0000","last_clean":"2024-09-16T01:53:41.351445+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:53:41.351445+0000","last_undegraded":"2024-09-16T01:53:41.351445+0000","last_fullsized":"2024-09-16T01:53:41.351445+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":276,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551717+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:36.551717+0000","last_peered":"2024-09-16T01:53:36.551717+0000","last_clean":"2024-09-16T01:53:36.551717+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:36.551717+0000","last_undegraded":"2024-09-16T01:53:36.551717+0000","last_fullsized":"2024-09-16T01:53:36.551717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"127'155","reported_seq":429,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557405+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:36.557405+0000","last_peered":"2024-09-16T01:53:36.557405+0000","last_clean":"2024-09-16T01:53:36.557405+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:36.557405+0000","last_undegraded":"2024-09-16T01:53:36.557405+0000","last_fullsized":"2024-09-16T01:53:36.557405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":119,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557300+0000","last_change":"2024-09-16T01:53:20.534486+0000","last_active":"2024-09-16T01:53:36.557300+0000","last_peered":"2024-09-16T01:53:36.557300+0000","last_clean":"2024-09-16T01:53:36.557300+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:36.557300+0000","last_undegraded":"2024-09-16T01:53:36.557300+0000","last_fullsized":"2024-09-16T01:53:36.557300+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:26.318068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":396,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545805+0000","last_change":"2024-09-16T01:53:34.539129+0000","last_active":"2024-09-16T01:53:36.545805+0000","last_peered":"2024-09-16T01:53:36.545805+0000","last_clean":"2024-09-16T01:53:36.545805+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:36.545805+0000","last_undegraded":"2024-09-16T01:53:36.545805+0000","last_fullsized":"2024-09-16T01:53:36.545805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"133'25","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546105+0000","last_change":"2024-09-16T01:53:20.586608+0000","last_active":"2024-09-16T01:53:36.546105+0000","last_peered":"2024-09-16T01:53:36.546105+0000","last_clean":"2024-09-16T01:53:36.546105+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:36.546105+0000","last_undegraded":"2024-09-16T01:53:36.546105+0000","last_fullsized":"2024-09-16T01:53:36.546105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:55.614045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":466,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546041+0000","last_change":"2024-09-16T01:53:23.514351+0000","last_active":"2024-09-16T01:53:36.546041+0000","last_peered":"2024-09-16T01:53:36.546041+0000","last_clean":"2024-09-16T01:53:36.546041+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:36.546041+0000","last_undegraded":"2024-09-16T01:53:36.546041+0000","last_fullsized":"2024-09-16T01:53:36.546041+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:14.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"130'8","reported_seq":313,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842568+0000","last_change":"2024-09-16T01:53:19.543649+0000","last_active":"2024-09-16T01:53:36.842568+0000","last_peered":"2024-09-16T01:53:36.842568+0000","last_clean":"2024-09-16T01:53:36.842568+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:53:36.842568+0000","last_undegraded":"2024-09-16T01:53:36.842568+0000","last_fullsized":"2024-09-16T01:53:36.842568+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:12.126459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"127'176","reported_seq":541,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546195+0000","last_change":"2024-09-16T01:53:23.518054+0000","last_active":"2024-09-16T01:53:36.546195+0000","last_peered":"2024-09-16T01:53:36.546195+0000","last_clean":"2024-09-16T01:53:36.546195+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:36.546195+0000","last_undegraded":"2024-09-16T01:53:36.546195+0000","last_fullsized":"2024-09-16T01:53:36.546195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:50.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551978+0000","last_change":"2024-09-16T01:53:19.575999+0000","last_active":"2024-09-16T01:53:36.551978+0000","last_peered":"2024-09-16T01:53:36.551978+0000","last_clean":"2024-09-16T01:53:36.551978+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:36.551978+0000","last_undegraded":"2024-09-16T01:53:36.551978+0000","last_fullsized":"2024-09-16T01:53:36.551978+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:59.687176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"152'162","reported_seq":587,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.319456+0000","last_change":"2024-09-16T01:53:23.514324+0000","last_active":"2024-09-16T01:53:41.319456+0000","last_peered":"2024-09-16T01:53:41.319456+0000","last_clean":"2024-09-16T01:53:41.319456+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:53:41.319456+0000","last_undegraded":"2024-09-16T01:53:41.319456+0000","last_fullsized":"2024-09-16T01:53:41.319456+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:27.575957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546415+0000","last_change":"2024-09-16T01:53:20.576951+0000","last_active":"2024-09-16T01:53:36.546415+0000","last_peered":"2024-09-16T01:53:36.546415+0000","last_clean":"2024-09-16T01:53:36.546415+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:36.546415+0000","last_undegraded":"2024-09-16T01:53:36.546415+0000","last_fullsized":"2024-09-16T01:53:36.546415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:20.021875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"127'164","reported_seq":494,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545261+0000","last_change":"2024-09-16T01:53:23.517332+0000","last_active":"2024-09-16T01:53:36.545261+0000","last_peered":"2024-09-16T01:53:36.545261+0000","last_clean":"2024-09-16T01:53:36.545261+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:36.545261+0000","last_undegraded":"2024-09-16T01:53:36.545261+0000","last_fullsized":"2024-09-16T01:53:36.545261+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:15.929317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":248,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545542+0000","last_change":"2024-09-16T01:53:34.538675+0000","last_active":"2024-09-16T01:53:36.545542+0000","last_peered":"2024-09-16T01:53:36.545542+0000","last_clean":"2024-09-16T01:53:36.545542+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:36.545542+0000","last_undegraded":"2024-09-16T01:53:36.545542+0000","last_fullsized":"2024-09-16T01:53:36.545542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":624,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554614+0000","last_change":"2024-09-16T01:53:36.554614+0000","last_active":"2024-09-16T01:53:36.554614+0000","last_peered":"2024-09-16T01:53:36.554614+0000","last_clean":"2024-09-16T01:53:36.554614+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:36.554614+0000","last_undegraded":"2024-09-16T01:53:36.554614+0000","last_fullsized":"2024-09-16T01:53:36.554614+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:23.435521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.841495+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:36.841495+0000","last_peered":"2024-09-16T01:53:36.841495+0000","last_clean":"2024-09-16T01:53:36.841495+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:36.841495+0000","last_undegraded":"2024-09-16T01:53:36.841495+0000","last_fullsized":"2024-09-16T01:53:36.841495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"152'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343195+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:41.343195+0000","last_peered":"2024-09-16T01:53:41.343195+0000","last_clean":"2024-09-16T01:53:41.343195+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:41.343195+0000","last_undegraded":"2024-09-16T01:53:41.343195+0000","last_fullsized":"2024-09-16T01:53:41.343195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"152'205","reported_seq":717,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361493+0000","last_change":"2024-09-16T01:53:22.527742+0000","last_active":"2024-09-16T01:53:41.361493+0000","last_peered":"2024-09-16T01:53:41.361493+0000","last_clean":"2024-09-16T01:53:41.361493+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:53:41.361493+0000","last_undegraded":"2024-09-16T01:53:41.361493+0000","last_fullsized":"2024-09-16T01:53:41.361493+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553134+0000","last_change":"2024-09-16T01:53:19.535031+0000","last_active":"2024-09-16T01:53:36.553134+0000","last_peered":"2024-09-16T01:53:36.553134+0000","last_clean":"2024-09-16T01:53:36.553134+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:36.553134+0000","last_undegraded":"2024-09-16T01:53:36.553134+0000","last_fullsized":"2024-09-16T01:53:36.553134+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:36.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"152'162","reported_seq":627,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.353319+0000","last_change":"2024-09-16T01:53:22.521982+0000","last_active":"2024-09-16T01:53:41.353319+0000","last_peered":"2024-09-16T01:53:41.353319+0000","last_clean":"2024-09-16T01:53:41.353319+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:41.353319+0000","last_undegraded":"2024-09-16T01:53:41.353319+0000","last_fullsized":"2024-09-16T01:53:41.353319+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:35.196217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551329+0000","last_change":"2024-09-16T01:53:19.559597+0000","last_active":"2024-09-16T01:53:36.551329+0000","last_peered":"2024-09-16T01:53:36.551329+0000","last_clean":"2024-09-16T01:53:36.551329+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:36.551329+0000","last_undegraded":"2024-09-16T01:53:36.551329+0000","last_fullsized":"2024-09-16T01:53:36.551329+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552068+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:36.552068+0000","last_peered":"2024-09-16T01:53:36.552068+0000","last_clean":"2024-09-16T01:53:36.552068+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:36.552068+0000","last_undegraded":"2024-09-16T01:53:36.552068+0000","last_fullsized":"2024-09-16T01:53:36.552068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552067+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:36.552067+0000","last_peered":"2024-09-16T01:53:36.552067+0000","last_clean":"2024-09-16T01:53:36.552067+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:36.552067+0000","last_undegraded":"2024-09-16T01:53:36.552067+0000","last_fullsized":"2024-09-16T01:53:36.552067+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":449,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551955+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:36.551955+0000","last_peered":"2024-09-16T01:53:36.551955+0000","last_clean":"2024-09-16T01:53:36.551955+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:36.551955+0000","last_undegraded":"2024-09-16T01:53:36.551955+0000","last_fullsized":"2024-09-16T01:53:36.551955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":56,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545971+0000","last_change":"2024-09-16T01:53:20.557667+0000","last_active":"2024-09-16T01:53:36.545971+0000","last_peered":"2024-09-16T01:53:36.545971+0000","last_clean":"2024-09-16T01:53:36.545971+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:36.545971+0000","last_undegraded":"2024-09-16T01:53:36.545971+0000","last_fullsized":"2024-09-16T01:53:36.545971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:05:38.159129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4442,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10971,"ondisk_log_size":10971,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398654,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":420,"num_write_kb":599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797308,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":383,"ondisk_log_size":383,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509504,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1855488,"data_stored":874658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37524,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334316,"statfs":{"total":96636764160,"available":96598339584,"internally_reserved":0,"allocated":2224128,"data_stored":1257231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1904640,"data_stored":1463571,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509504,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1855488,"data_stored":874731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542207,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":873348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542207,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1839104,"data_stored":874169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1912832,"data_stored":1463488,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41420,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330420,"statfs":{"total":96636764160,"available":96594350080,"internally_reserved":0,"allocated":1949696,"data_stored":1264954,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393165,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:42.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.428+0000 7f1b5ca00640 1 Processor -- start 2024-09-16T01:53:42.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.428+0000 7f1b5ca00640 1 -- start start 2024-09-16T01:53:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.429+0000 7f1b5ca00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1b58151a90 0x7f1b580c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.429+0000 7f1b5ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b580c5280 0x7f1b580c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.429+0000 7f1b5ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b58151e70 con 0x7f1b580c6410 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.429+0000 7f1b5ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f1b580c8130 con 0x7f1b580c5280 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.429+0000 7f1b5ca00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1b580c8410 con 0x7f1b58151a90 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b56800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b580c5280 0x7f1b580c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b55400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1b580c6410 0x7f1b58158040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47286/0 (socket says 172.21.15.146:47286) 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b55400640 1 -- 172.21.15.146:0/3099351904 learned_addr learned my addr 172.21.15.146:0/3099351904 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b55e00640 1 --2- 172.21.15.146:0/3099351904 >> v2:172.21.15.161:3300/0 conn(0x7f1b58151a90 0x7f1b580c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b56800640 1 -- 172.21.15.146:0/3099351904 >> v2:172.21.15.161:3300/0 conn(0x7f1b58151a90 msgr2=0x7f1b580c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b56800640 1 --2- 172.21.15.146:0/3099351904 >> v2:172.21.15.161:3300/0 conn(0x7f1b58151a90 0x7f1b580c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b56800640 1 -- 172.21.15.146:0/3099351904 >> v1:172.21.15.146:6789/0 conn(0x7f1b580c6410 legacy=0x7f1b58158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.430+0000 7f1b56800640 1 -- 172.21.15.146:0/3099351904 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b580c86f0 con 0x7f1b580c5280 2024-09-16T01:53:42.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.431+0000 7f1b56800640 1 --2- 172.21.15.146:0/3099351904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b580c5280 0x7f1b580c56f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1b58069950 tx=0x7f1b5002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926d1a43a7fc83c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.431+0000 7f1b54a00640 1 -- 172.21.15.146:0/3099351904 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50007740 con 0x7f1b580c5280 2024-09-16T01:53:42.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.431+0000 7f1b54a00640 1 -- 172.21.15.146:0/3099351904 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b500078e0 con 0x7f1b580c5280 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.431+0000 7f1b54a00640 1 -- 172.21.15.146:0/3099351904 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50007cb0 con 0x7f1b580c5280 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 -- 172.21.15.146:0/3099351904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b580c5280 msgr2=0x7f1b580c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/3099351904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b580c5280 0x7f1b580c56f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1b58069950 tx=0x7f1b5002a6a0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 -- 172.21.15.146:0/3099351904 shutdown_connections 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/3099351904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b580c5280 0x7f1b580c56f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/3099351904 >> v2:172.21.15.161:3300/0 conn(0x7f1b58151a90 0x7f1b580c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 -- 172.21.15.146:0/3099351904 >> 172.21.15.146:0/3099351904 conn(0x7f1b5806f3c0 msgr2=0x7f1b5806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 -- 172.21.15.146:0/3099351904 shutdown_connections 2024-09-16T01:53:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.432+0000 7f1b5ca00640 1 -- 172.21.15.146:0/3099351904 wait complete. 2024-09-16T01:53:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.433+0000 7f1b5ca00640 1 Processor -- start 2024-09-16T01:53:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 -- start start 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1b580c5280 0x7f1b580cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b580c6410 0x7f1b58153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 0x7f1b58147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1b58157b20 con 0x7f1b58151a90 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b58157840 con 0x7f1b580c6410 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.434+0000 7f1b5ca00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1b580c3bf0 con 0x7f1b580c5280 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b56800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 0x7f1b58147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1b580c5280 0x7f1b580cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b56800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 0x7f1b58147c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47414/0 (socket says 172.21.15.146:47414) 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b56800640 1 -- 172.21.15.146:0/1516233297 learned_addr learned my addr 172.21.15.146:0/1516233297 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55400640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b580c6410 0x7f1b58153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 -- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b580c6410 msgr2=0x7f1b58153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b580c6410 0x7f1b58153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 -- 172.21.15.146:0/1516233297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 msgr2=0x7f1b58147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 0x7f1b58147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 -- 172.21.15.146:0/1516233297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b580ca700 con 0x7f1b580c5280 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b56800640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 0x7f1b58147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.435+0000 7f1b55e00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1b580c5280 0x7f1b580cc950 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e8a0 tx=0x7f1b4800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.436+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48017070 con 0x7f1b580c5280 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.436+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4800c940 con 0x7f1b580c5280 2024-09-16T01:53:42.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.436+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b580cad00 con 0x7f1b580c5280 2024-09-16T01:53:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.436+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48009e60 con 0x7f1b580c5280 2024-09-16T01:53:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.436+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b580c9fb0 con 0x7f1b580c5280 2024-09-16T01:53:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.437+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b580c71f0 con 0x7f1b580c5280 2024-09-16T01:53:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.438+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1b4801a070 con 0x7f1b580c5280 2024-09-16T01:53:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.439+0000 7f1b4e000640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1b3806c2b0 0x7f1b3806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.439+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(153..153 src has 1..153) ==== 5886+0+0 (secure 0 0 0) 0x7f1b4808b590 con 0x7f1b580c5280 2024-09-16T01:53:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.439+0000 7f1b55400640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1b3806c2b0 0x7f1b3806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.439+0000 7f1b55400640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1b3806c2b0 0x7f1b3806e770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1b581547d0 tx=0x7f1b44007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.443+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1b480595f0 con 0x7f1b580c5280 2024-09-16T01:53:42.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.616+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b58144150 con 0x7f1b3806c2b0 2024-09-16T01:53:42.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.623+0000 7f1b4e000640 1 -- 172.21.15.146:0/1516233297 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253947 (secure 0 0 0) 0x7f1b58144150 con 0x7f1b3806c2b0 2024-09-16T01:53:42.627 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:42.628 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1b3806c2b0 msgr2=0x7f1b3806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1b3806c2b0 0x7f1b3806e770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1b581547d0 tx=0x7f1b44007e80 comp rx=0 tx=0).stop 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1b580c5280 msgr2=0x7f1b580cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1b580c5280 0x7f1b580cc950 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e8a0 tx=0x7f1b4800ed60 comp rx=0 tx=0).stop 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 shutdown_connections 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f1b3806c2b0 0x7f1b3806e770 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b58151a90 0x7f1b58147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b580c6410 0x7f1b58153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 --2- 172.21.15.146:0/1516233297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1b580c5280 0x7f1b580cc950 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.627+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 >> 172.21.15.146:0/1516233297 conn(0x7f1b5806f3c0 msgr2=0x7f1b580c7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:42.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.628+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 shutdown_connections 2024-09-16T01:53:42.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.628+0000 7f1b5ca00640 1 -- 172.21.15.146:0/1516233297 wait complete. 2024-09-16T01:53:42.644 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-16T01:53:42.574909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":989271,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9798,"num_write_kb":25168842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4698,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11513,"ondisk_log_size":11513,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":281216,"kb_used_data":15052,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693504,"statfs":{"total":773094113280,"available":772806148096,"internally_reserved":0,"allocated":15413248,"data_stored":8952464,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-163,"num_bytes_recovered":-808,"num_keys_recovered":-40,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.042652"},"pg_stats":[{"pgid":"3.3f","version":"152'167","reported_seq":492,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361768+0000","last_change":"2024-09-16T01:53:22.525372+0000","last_active":"2024-09-16T01:53:41.361768+0000","last_peered":"2024-09-16T01:53:41.361768+0000","last_clean":"2024-09-16T01:53:41.361768+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:41.361768+0000","last_undegraded":"2024-09-16T01:53:41.361768+0000","last_fullsized":"2024-09-16T01:53:41.361768+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:00.392212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551885+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:36.551885+0000","last_peered":"2024-09-16T01:53:36.551885+0000","last_clean":"2024-09-16T01:53:36.551885+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:36.551885+0000","last_undegraded":"2024-09-16T01:53:36.551885+0000","last_fullsized":"2024-09-16T01:53:36.551885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":799,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551822+0000","last_change":"2024-09-16T01:53:23.518810+0000","last_active":"2024-09-16T01:53:36.551822+0000","last_peered":"2024-09-16T01:53:36.551822+0000","last_clean":"2024-09-16T01:53:36.551822+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:53:36.551822+0000","last_undegraded":"2024-09-16T01:53:36.551822+0000","last_fullsized":"2024-09-16T01:53:36.551822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"152'193","reported_seq":591,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.334545+0000","last_change":"2024-09-16T01:53:32.511261+0000","last_active":"2024-09-16T01:53:41.334545+0000","last_peered":"2024-09-16T01:53:41.334545+0000","last_clean":"2024-09-16T01:53:41.334545+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:41.334545+0000","last_undegraded":"2024-09-16T01:53:41.334545+0000","last_fullsized":"2024-09-16T01:53:41.334545+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:07.896976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"152'174","reported_seq":526,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.342604+0000","last_change":"2024-09-16T01:53:22.521371+0000","last_active":"2024-09-16T01:53:41.342604+0000","last_peered":"2024-09-16T01:53:41.342604+0000","last_clean":"2024-09-16T01:53:41.342604+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:41.342604+0000","last_undegraded":"2024-09-16T01:53:41.342604+0000","last_fullsized":"2024-09-16T01:53:41.342604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:39.028997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"152'162","reported_seq":582,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.341075+0000","last_change":"2024-09-16T01:53:23.521468+0000","last_active":"2024-09-16T01:53:41.341075+0000","last_peered":"2024-09-16T01:53:41.341075+0000","last_clean":"2024-09-16T01:53:41.341075+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:41.341075+0000","last_undegraded":"2024-09-16T01:53:41.341075+0000","last_fullsized":"2024-09-16T01:53:41.341075+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:34:42.522900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"152'188","reported_seq":638,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.345763+0000","last_change":"2024-09-16T01:53:22.523352+0000","last_active":"2024-09-16T01:53:41.345763+0000","last_peered":"2024-09-16T01:53:41.345763+0000","last_clean":"2024-09-16T01:53:41.345763+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:41.345763+0000","last_undegraded":"2024-09-16T01:53:41.345763+0000","last_fullsized":"2024-09-16T01:53:41.345763+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.078885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":722,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551418+0000","last_change":"2024-09-16T01:53:22.523335+0000","last_active":"2024-09-16T01:53:36.551418+0000","last_peered":"2024-09-16T01:53:36.551418+0000","last_clean":"2024-09-16T01:53:36.551418+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:53:36.551418+0000","last_undegraded":"2024-09-16T01:53:36.551418+0000","last_fullsized":"2024-09-16T01:53:36.551418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"152'178","reported_seq":451,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.357816+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:41.357816+0000","last_peered":"2024-09-16T01:53:41.357816+0000","last_clean":"2024-09-16T01:53:41.357816+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:41.357816+0000","last_undegraded":"2024-09-16T01:53:41.357816+0000","last_fullsized":"2024-09-16T01:53:41.357816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"152'190","reported_seq":570,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.356987+0000","last_change":"2024-09-16T01:53:22.526050+0000","last_active":"2024-09-16T01:53:41.356987+0000","last_peered":"2024-09-16T01:53:41.356987+0000","last_clean":"2024-09-16T01:53:41.356987+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:41.356987+0000","last_undegraded":"2024-09-16T01:53:41.356987+0000","last_fullsized":"2024-09-16T01:53:41.356987+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:00.473148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"152'214","reported_seq":617,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.357952+0000","last_change":"2024-09-16T01:53:23.519544+0000","last_active":"2024-09-16T01:53:41.357952+0000","last_peered":"2024-09-16T01:53:41.357952+0000","last_clean":"2024-09-16T01:53:41.357952+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:41.357952+0000","last_undegraded":"2024-09-16T01:53:41.357952+0000","last_fullsized":"2024-09-16T01:53:41.357952+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:26.797344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"152'166","reported_seq":362,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343823+0000","last_change":"2024-09-16T01:53:22.519665+0000","last_active":"2024-09-16T01:53:41.343823+0000","last_peered":"2024-09-16T01:53:41.343823+0000","last_clean":"2024-09-16T01:53:41.343823+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:41.343823+0000","last_undegraded":"2024-09-16T01:53:41.343823+0000","last_fullsized":"2024-09-16T01:53:41.343823+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:32.846352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545527+0000","last_change":"2024-09-16T01:53:34.533587+0000","last_active":"2024-09-16T01:53:36.545527+0000","last_peered":"2024-09-16T01:53:36.545527+0000","last_clean":"2024-09-16T01:53:36.545527+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:36.545527+0000","last_undegraded":"2024-09-16T01:53:36.545527+0000","last_fullsized":"2024-09-16T01:53:36.545527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:15.453058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":488,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545392+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:36.545392+0000","last_peered":"2024-09-16T01:53:36.545392+0000","last_clean":"2024-09-16T01:53:36.545392+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:36.545392+0000","last_undegraded":"2024-09-16T01:53:36.545392+0000","last_fullsized":"2024-09-16T01:53:36.545392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551212+0000","last_change":"2024-09-16T01:53:22.527702+0000","last_active":"2024-09-16T01:53:36.551212+0000","last_peered":"2024-09-16T01:53:36.551212+0000","last_clean":"2024-09-16T01:53:36.551212+0000","last_became_active":"2024-09-16T01:53:22.526994+0000","last_became_peered":"2024-09-16T01:53:22.526994+0000","last_unstale":"2024-09-16T01:53:36.551212+0000","last_undegraded":"2024-09-16T01:53:36.551212+0000","last_fullsized":"2024-09-16T01:53:36.551212+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"152'162","reported_seq":435,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.359260+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:41.359260+0000","last_peered":"2024-09-16T01:53:41.359260+0000","last_clean":"2024-09-16T01:53:41.359260+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:41.359260+0000","last_undegraded":"2024-09-16T01:53:41.359260+0000","last_fullsized":"2024-09-16T01:53:41.359260+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"152'176","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361166+0000","last_change":"2024-09-16T01:53:23.526130+0000","last_active":"2024-09-16T01:53:41.361166+0000","last_peered":"2024-09-16T01:53:41.361166+0000","last_clean":"2024-09-16T01:53:41.361166+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:53:41.361166+0000","last_undegraded":"2024-09-16T01:53:41.361166+0000","last_fullsized":"2024-09-16T01:53:41.361166+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:02.127294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"127'169","reported_seq":575,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563883+0000","last_change":"2024-09-16T01:53:36.563883+0000","last_active":"2024-09-16T01:53:36.563883+0000","last_peered":"2024-09-16T01:53:36.563883+0000","last_clean":"2024-09-16T01:53:36.563883+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:36.563883+0000","last_undegraded":"2024-09-16T01:53:36.563883+0000","last_fullsized":"2024-09-16T01:53:36.563883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554778+0000","last_change":"2024-09-16T01:53:36.554778+0000","last_active":"2024-09-16T01:53:36.554778+0000","last_peered":"2024-09-16T01:53:36.554778+0000","last_clean":"2024-09-16T01:53:36.554778+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:36.554778+0000","last_undegraded":"2024-09-16T01:53:36.554778+0000","last_fullsized":"2024-09-16T01:53:36.554778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"152'179","reported_seq":596,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343701+0000","last_change":"2024-09-16T01:53:23.524759+0000","last_active":"2024-09-16T01:53:41.343701+0000","last_peered":"2024-09-16T01:53:41.343701+0000","last_clean":"2024-09-16T01:53:41.343701+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:41.343701+0000","last_undegraded":"2024-09-16T01:53:41.343701+0000","last_fullsized":"2024-09-16T01:53:41.343701+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"127'143","reported_seq":455,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545707+0000","last_change":"2024-09-16T01:53:23.520715+0000","last_active":"2024-09-16T01:53:36.545707+0000","last_peered":"2024-09-16T01:53:36.545707+0000","last_clean":"2024-09-16T01:53:36.545707+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:36.545707+0000","last_undegraded":"2024-09-16T01:53:36.545707+0000","last_fullsized":"2024-09-16T01:53:36.545707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:45.628132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"152'199","reported_seq":831,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.353111+0000","last_change":"2024-09-16T01:53:22.524407+0000","last_active":"2024-09-16T01:53:41.353111+0000","last_peered":"2024-09-16T01:53:41.353111+0000","last_clean":"2024-09-16T01:53:41.353111+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:41.353111+0000","last_undegraded":"2024-09-16T01:53:41.353111+0000","last_fullsized":"2024-09-16T01:53:41.353111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551482+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:36.551482+0000","last_peered":"2024-09-16T01:53:36.551482+0000","last_clean":"2024-09-16T01:53:36.551482+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:36.551482+0000","last_undegraded":"2024-09-16T01:53:36.551482+0000","last_fullsized":"2024-09-16T01:53:36.551482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"152'207","reported_seq":480,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.359084+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:41.359084+0000","last_peered":"2024-09-16T01:53:41.359084+0000","last_clean":"2024-09-16T01:53:41.359084+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:41.359084+0000","last_undegraded":"2024-09-16T01:53:41.359084+0000","last_fullsized":"2024-09-16T01:53:41.359084+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":431,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552017+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:36.552017+0000","last_peered":"2024-09-16T01:53:36.552017+0000","last_clean":"2024-09-16T01:53:36.552017+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:36.552017+0000","last_undegraded":"2024-09-16T01:53:36.552017+0000","last_fullsized":"2024-09-16T01:53:36.552017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"152'217","reported_seq":491,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362690+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:41.362690+0000","last_peered":"2024-09-16T01:53:41.362690+0000","last_clean":"2024-09-16T01:53:41.362690+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:41.362690+0000","last_undegraded":"2024-09-16T01:53:41.362690+0000","last_fullsized":"2024-09-16T01:53:41.362690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":664,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545722+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:36.545722+0000","last_peered":"2024-09-16T01:53:36.545722+0000","last_clean":"2024-09-16T01:53:36.545722+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:36.545722+0000","last_undegraded":"2024-09-16T01:53:36.545722+0000","last_fullsized":"2024-09-16T01:53:36.545722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545688+0000","last_change":"2024-09-16T01:53:34.533524+0000","last_active":"2024-09-16T01:53:36.545688+0000","last_peered":"2024-09-16T01:53:36.545688+0000","last_clean":"2024-09-16T01:53:36.545688+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:36.545688+0000","last_undegraded":"2024-09-16T01:53:36.545688+0000","last_fullsized":"2024-09-16T01:53:36.545688+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:57.738639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":683,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551567+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:36.551567+0000","last_peered":"2024-09-16T01:53:36.551567+0000","last_clean":"2024-09-16T01:53:36.551567+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:36.551567+0000","last_undegraded":"2024-09-16T01:53:36.551567+0000","last_fullsized":"2024-09-16T01:53:36.551567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"152'153","reported_seq":426,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.355574+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:41.355574+0000","last_peered":"2024-09-16T01:53:41.355574+0000","last_clean":"2024-09-16T01:53:41.355574+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:41.355574+0000","last_undegraded":"2024-09-16T01:53:41.355574+0000","last_fullsized":"2024-09-16T01:53:41.355574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"127'146","reported_seq":558,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563260+0000","last_change":"2024-09-16T01:53:36.563260+0000","last_active":"2024-09-16T01:53:36.563260+0000","last_peered":"2024-09-16T01:53:36.563260+0000","last_clean":"2024-09-16T01:53:36.563260+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:36.563260+0000","last_undegraded":"2024-09-16T01:53:36.563260+0000","last_fullsized":"2024-09-16T01:53:36.563260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:31.539361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551939+0000","last_change":"2024-09-16T01:53:36.551939+0000","last_active":"2024-09-16T01:53:36.551939+0000","last_peered":"2024-09-16T01:53:36.551939+0000","last_clean":"2024-09-16T01:53:36.551939+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:36.551939+0000","last_undegraded":"2024-09-16T01:53:36.551939+0000","last_fullsized":"2024-09-16T01:53:36.551939+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"152'149","reported_seq":422,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.338192+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:41.338192+0000","last_peered":"2024-09-16T01:53:41.338192+0000","last_clean":"2024-09-16T01:53:41.338192+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:41.338192+0000","last_undegraded":"2024-09-16T01:53:41.338192+0000","last_fullsized":"2024-09-16T01:53:41.338192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551629+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:36.551629+0000","last_peered":"2024-09-16T01:53:36.551629+0000","last_clean":"2024-09-16T01:53:36.551629+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:36.551629+0000","last_undegraded":"2024-09-16T01:53:36.551629+0000","last_fullsized":"2024-09-16T01:53:36.551629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"127'154","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545597+0000","last_change":"2024-09-16T01:53:22.520820+0000","last_active":"2024-09-16T01:53:36.545597+0000","last_peered":"2024-09-16T01:53:36.545597+0000","last_clean":"2024-09-16T01:53:36.545597+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:36.545597+0000","last_undegraded":"2024-09-16T01:53:36.545597+0000","last_fullsized":"2024-09-16T01:53:36.545597+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:51.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842491+0000","last_change":"2024-09-16T01:53:19.532463+0000","last_active":"2024-09-16T01:53:36.842491+0000","last_peered":"2024-09-16T01:53:36.842491+0000","last_clean":"2024-09-16T01:53:36.842491+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:36.842491+0000","last_undegraded":"2024-09-16T01:53:36.842491+0000","last_fullsized":"2024-09-16T01:53:36.842491+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:52.176146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":275,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545279+0000","last_change":"2024-09-16T01:53:34.532869+0000","last_active":"2024-09-16T01:53:36.545279+0000","last_peered":"2024-09-16T01:53:36.545279+0000","last_clean":"2024-09-16T01:53:36.545279+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:36.545279+0000","last_undegraded":"2024-09-16T01:53:36.545279+0000","last_fullsized":"2024-09-16T01:53:36.545279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:13.148781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":454,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551995+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:36.551995+0000","last_peered":"2024-09-16T01:53:36.551995+0000","last_clean":"2024-09-16T01:53:36.551995+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:36.551995+0000","last_undegraded":"2024-09-16T01:53:36.551995+0000","last_fullsized":"2024-09-16T01:53:36.551995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":277,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.840791+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:36.840791+0000","last_peered":"2024-09-16T01:53:36.840791+0000","last_clean":"2024-09-16T01:53:36.840791+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:36.840791+0000","last_undegraded":"2024-09-16T01:53:36.840791+0000","last_fullsized":"2024-09-16T01:53:36.840791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"127'177","reported_seq":418,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545675+0000","last_change":"2024-09-16T01:53:32.513845+0000","last_active":"2024-09-16T01:53:36.545675+0000","last_peered":"2024-09-16T01:53:36.545675+0000","last_clean":"2024-09-16T01:53:36.545675+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:36.545675+0000","last_undegraded":"2024-09-16T01:53:36.545675+0000","last_fullsized":"2024-09-16T01:53:36.545675+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:02.738813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":318,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556079+0000","last_change":"2024-09-16T01:53:20.522781+0000","last_active":"2024-09-16T01:53:36.556079+0000","last_peered":"2024-09-16T01:53:36.556079+0000","last_clean":"2024-09-16T01:53:36.556079+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:36.556079+0000","last_undegraded":"2024-09-16T01:53:36.556079+0000","last_fullsized":"2024-09-16T01:53:36.556079+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:25.742037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551737+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:36.551737+0000","last_peered":"2024-09-16T01:53:36.551737+0000","last_clean":"2024-09-16T01:53:36.551737+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:36.551737+0000","last_undegraded":"2024-09-16T01:53:36.551737+0000","last_fullsized":"2024-09-16T01:53:36.551737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553376+0000","last_change":"2024-09-16T01:53:19.579965+0000","last_active":"2024-09-16T01:53:36.553376+0000","last_peered":"2024-09-16T01:53:36.553376+0000","last_clean":"2024-09-16T01:53:36.553376+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:36.553376+0000","last_undegraded":"2024-09-16T01:53:36.553376+0000","last_fullsized":"2024-09-16T01:53:36.553376+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:29.737898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":423,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552118+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:36.552118+0000","last_peered":"2024-09-16T01:53:36.552118+0000","last_clean":"2024-09-16T01:53:36.552118+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:36.552118+0000","last_undegraded":"2024-09-16T01:53:36.552118+0000","last_fullsized":"2024-09-16T01:53:36.552118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"133'11","reported_seq":160,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557008+0000","last_change":"2024-09-16T01:53:36.557008+0000","last_active":"2024-09-16T01:53:36.557008+0000","last_peered":"2024-09-16T01:53:36.557008+0000","last_clean":"2024-09-16T01:53:36.557008+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:36.557008+0000","last_undegraded":"2024-09-16T01:53:36.557008+0000","last_fullsized":"2024-09-16T01:53:36.557008+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'168","reported_seq":441,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362450+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:41.362450+0000","last_peered":"2024-09-16T01:53:41.362450+0000","last_clean":"2024-09-16T01:53:41.362450+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:41.362450+0000","last_undegraded":"2024-09-16T01:53:41.362450+0000","last_fullsized":"2024-09-16T01:53:41.362450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"152'178","reported_seq":496,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.364517+0000","last_change":"2024-09-16T01:53:23.518694+0000","last_active":"2024-09-16T01:53:41.364517+0000","last_peered":"2024-09-16T01:53:41.364517+0000","last_clean":"2024-09-16T01:53:41.364517+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:41.364517+0000","last_undegraded":"2024-09-16T01:53:41.364517+0000","last_fullsized":"2024-09-16T01:53:41.364517+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:11.376091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391576,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"152'176","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343763+0000","last_change":"2024-09-16T01:53:22.524097+0000","last_active":"2024-09-16T01:53:41.343763+0000","last_peered":"2024-09-16T01:53:41.343763+0000","last_clean":"2024-09-16T01:53:41.343763+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:41.343763+0000","last_undegraded":"2024-09-16T01:53:41.343763+0000","last_fullsized":"2024-09-16T01:53:41.343763+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:14.041864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559516+0000","last_change":"2024-09-16T01:53:36.559516+0000","last_active":"2024-09-16T01:53:36.559516+0000","last_peered":"2024-09-16T01:53:36.559516+0000","last_clean":"2024-09-16T01:53:36.559516+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:36.559516+0000","last_undegraded":"2024-09-16T01:53:36.559516+0000","last_fullsized":"2024-09-16T01:53:36.559516+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":411,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546458+0000","last_change":"2024-09-16T01:53:34.529911+0000","last_active":"2024-09-16T01:53:36.546458+0000","last_peered":"2024-09-16T01:53:36.546458+0000","last_clean":"2024-09-16T01:53:36.546458+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:36.546458+0000","last_undegraded":"2024-09-16T01:53:36.546458+0000","last_fullsized":"2024-09-16T01:53:36.546458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":358,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842660+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:36.842660+0000","last_peered":"2024-09-16T01:53:36.842660+0000","last_clean":"2024-09-16T01:53:36.842660+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:36.842660+0000","last_undegraded":"2024-09-16T01:53:36.842660+0000","last_fullsized":"2024-09-16T01:53:36.842660+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"152'201","reported_seq":718,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362908+0000","last_change":"2024-09-16T01:53:23.521939+0000","last_active":"2024-09-16T01:53:41.362908+0000","last_peered":"2024-09-16T01:53:41.362908+0000","last_clean":"2024-09-16T01:53:41.362908+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:53:41.362908+0000","last_undegraded":"2024-09-16T01:53:41.362908+0000","last_fullsized":"2024-09-16T01:53:41.362908+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:55.352896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"152'7","reported_seq":53,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552327+0000","last_change":"2024-09-16T01:53:19.550015+0000","last_active":"2024-09-16T01:53:36.552327+0000","last_peered":"2024-09-16T01:53:36.552327+0000","last_clean":"2024-09-16T01:53:36.552327+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:36.552327+0000","last_undegraded":"2024-09-16T01:53:36.552327+0000","last_fullsized":"2024-09-16T01:53:36.552327+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"152'197","reported_seq":471,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.337310+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:41.337310+0000","last_peered":"2024-09-16T01:53:41.337310+0000","last_clean":"2024-09-16T01:53:41.337310+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:41.337310+0000","last_undegraded":"2024-09-16T01:53:41.337310+0000","last_fullsized":"2024-09-16T01:53:41.337310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"133'18","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545354+0000","last_change":"2024-09-16T01:53:20.541056+0000","last_active":"2024-09-16T01:53:36.545354+0000","last_peered":"2024-09-16T01:53:36.545354+0000","last_clean":"2024-09-16T01:53:36.545354+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:36.545354+0000","last_undegraded":"2024-09-16T01:53:36.545354+0000","last_fullsized":"2024-09-16T01:53:36.545354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:21.564935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"152'215","reported_seq":488,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.356427+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:41.356427+0000","last_peered":"2024-09-16T01:53:41.356427+0000","last_clean":"2024-09-16T01:53:41.356427+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:41.356427+0000","last_undegraded":"2024-09-16T01:53:41.356427+0000","last_fullsized":"2024-09-16T01:53:41.356427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553629+0000","last_change":"2024-09-16T01:53:19.563306+0000","last_active":"2024-09-16T01:53:36.553629+0000","last_peered":"2024-09-16T01:53:36.553629+0000","last_clean":"2024-09-16T01:53:36.553629+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:36.553629+0000","last_undegraded":"2024-09-16T01:53:36.553629+0000","last_fullsized":"2024-09-16T01:53:36.553629+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:04.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"152'175","reported_seq":653,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361366+0000","last_change":"2024-09-16T01:53:22.527907+0000","last_active":"2024-09-16T01:53:41.361366+0000","last_peered":"2024-09-16T01:53:41.361366+0000","last_clean":"2024-09-16T01:53:41.361366+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:41.361366+0000","last_undegraded":"2024-09-16T01:53:41.361366+0000","last_fullsized":"2024-09-16T01:53:41.361366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:51.042095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546066+0000","last_change":"2024-09-16T01:53:20.547257+0000","last_active":"2024-09-16T01:53:36.546066+0000","last_peered":"2024-09-16T01:53:36.546066+0000","last_clean":"2024-09-16T01:53:36.546066+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:36.546066+0000","last_undegraded":"2024-09-16T01:53:36.546066+0000","last_fullsized":"2024-09-16T01:53:36.546066+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:04.416789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"152'162","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.350317+0000","last_change":"2024-09-16T01:53:23.527787+0000","last_active":"2024-09-16T01:53:41.350317+0000","last_peered":"2024-09-16T01:53:41.350317+0000","last_clean":"2024-09-16T01:53:41.350317+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:53:41.350317+0000","last_undegraded":"2024-09-16T01:53:41.350317+0000","last_fullsized":"2024-09-16T01:53:41.350317+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551843+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:36.551843+0000","last_peered":"2024-09-16T01:53:36.551843+0000","last_clean":"2024-09-16T01:53:36.551843+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:36.551843+0000","last_undegraded":"2024-09-16T01:53:36.551843+0000","last_fullsized":"2024-09-16T01:53:36.551843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"152'165","reported_seq":768,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.334297+0000","last_change":"2024-09-16T01:53:22.522539+0000","last_active":"2024-09-16T01:53:41.334297+0000","last_peered":"2024-09-16T01:53:41.334297+0000","last_clean":"2024-09-16T01:53:41.334297+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:41.334297+0000","last_undegraded":"2024-09-16T01:53:41.334297+0000","last_fullsized":"2024-09-16T01:53:41.334297+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:01.840411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":263,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545924+0000","last_change":"2024-09-16T01:53:34.533293+0000","last_active":"2024-09-16T01:53:36.545924+0000","last_peered":"2024-09-16T01:53:36.545924+0000","last_clean":"2024-09-16T01:53:36.545924+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:36.545924+0000","last_undegraded":"2024-09-16T01:53:36.545924+0000","last_fullsized":"2024-09-16T01:53:36.545924+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:39.949872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"152'175","reported_seq":448,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.358312+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:41.358312+0000","last_peered":"2024-09-16T01:53:41.358312+0000","last_clean":"2024-09-16T01:53:41.358312+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:41.358312+0000","last_undegraded":"2024-09-16T01:53:41.358312+0000","last_fullsized":"2024-09-16T01:53:41.358312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":432,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555710+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:36.555710+0000","last_peered":"2024-09-16T01:53:36.555710+0000","last_clean":"2024-09-16T01:53:36.555710+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:36.555710+0000","last_undegraded":"2024-09-16T01:53:36.555710+0000","last_fullsized":"2024-09-16T01:53:36.555710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.841615+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:36.841615+0000","last_peered":"2024-09-16T01:53:36.841615+0000","last_clean":"2024-09-16T01:53:36.841615+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:36.841615+0000","last_undegraded":"2024-09-16T01:53:36.841615+0000","last_fullsized":"2024-09-16T01:53:36.841615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"127'175","reported_seq":592,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545790+0000","last_change":"2024-09-16T01:53:22.524529+0000","last_active":"2024-09-16T01:53:36.545790+0000","last_peered":"2024-09-16T01:53:36.545790+0000","last_clean":"2024-09-16T01:53:36.545790+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:36.545790+0000","last_undegraded":"2024-09-16T01:53:36.545790+0000","last_fullsized":"2024-09-16T01:53:36.545790+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:39.441997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.840890+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:36.840890+0000","last_peered":"2024-09-16T01:53:36.840890+0000","last_clean":"2024-09-16T01:53:36.840890+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:36.840890+0000","last_undegraded":"2024-09-16T01:53:36.840890+0000","last_fullsized":"2024-09-16T01:53:36.840890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":403,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545398+0000","last_change":"2024-09-16T01:53:34.532478+0000","last_active":"2024-09-16T01:53:36.545398+0000","last_peered":"2024-09-16T01:53:36.545398+0000","last_clean":"2024-09-16T01:53:36.545398+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:36.545398+0000","last_undegraded":"2024-09-16T01:53:36.545398+0000","last_fullsized":"2024-09-16T01:53:36.545398+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"133'12","reported_seq":311,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554302+0000","last_change":"2024-09-16T01:53:36.554302+0000","last_active":"2024-09-16T01:53:36.554302+0000","last_peered":"2024-09-16T01:53:36.554302+0000","last_clean":"2024-09-16T01:53:36.554302+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:36.554302+0000","last_undegraded":"2024-09-16T01:53:36.554302+0000","last_fullsized":"2024-09-16T01:53:36.554302+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551550+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:36.551550+0000","last_peered":"2024-09-16T01:53:36.551550+0000","last_clean":"2024-09-16T01:53:36.551550+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:36.551550+0000","last_undegraded":"2024-09-16T01:53:36.551550+0000","last_fullsized":"2024-09-16T01:53:36.551550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842714+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:36.842714+0000","last_peered":"2024-09-16T01:53:36.842714+0000","last_clean":"2024-09-16T01:53:36.842714+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:36.842714+0000","last_undegraded":"2024-09-16T01:53:36.842714+0000","last_fullsized":"2024-09-16T01:53:36.842714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"152'177","reported_seq":550,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.351445+0000","last_change":"2024-09-16T01:53:23.527347+0000","last_active":"2024-09-16T01:53:41.351445+0000","last_peered":"2024-09-16T01:53:41.351445+0000","last_clean":"2024-09-16T01:53:41.351445+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:53:41.351445+0000","last_undegraded":"2024-09-16T01:53:41.351445+0000","last_fullsized":"2024-09-16T01:53:41.351445+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":276,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551717+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:36.551717+0000","last_peered":"2024-09-16T01:53:36.551717+0000","last_clean":"2024-09-16T01:53:36.551717+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:36.551717+0000","last_undegraded":"2024-09-16T01:53:36.551717+0000","last_fullsized":"2024-09-16T01:53:36.551717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"152'160","reported_seq":434,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.332794+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:41.332794+0000","last_peered":"2024-09-16T01:53:41.332794+0000","last_clean":"2024-09-16T01:53:41.332794+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:41.332794+0000","last_undegraded":"2024-09-16T01:53:41.332794+0000","last_fullsized":"2024-09-16T01:53:41.332794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":119,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557300+0000","last_change":"2024-09-16T01:53:20.534486+0000","last_active":"2024-09-16T01:53:36.557300+0000","last_peered":"2024-09-16T01:53:36.557300+0000","last_clean":"2024-09-16T01:53:36.557300+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:36.557300+0000","last_undegraded":"2024-09-16T01:53:36.557300+0000","last_fullsized":"2024-09-16T01:53:36.557300+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:26.318068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":396,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545805+0000","last_change":"2024-09-16T01:53:34.539129+0000","last_active":"2024-09-16T01:53:36.545805+0000","last_peered":"2024-09-16T01:53:36.545805+0000","last_clean":"2024-09-16T01:53:36.545805+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:36.545805+0000","last_undegraded":"2024-09-16T01:53:36.545805+0000","last_fullsized":"2024-09-16T01:53:36.545805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"133'25","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546105+0000","last_change":"2024-09-16T01:53:20.586608+0000","last_active":"2024-09-16T01:53:36.546105+0000","last_peered":"2024-09-16T01:53:36.546105+0000","last_clean":"2024-09-16T01:53:36.546105+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:36.546105+0000","last_undegraded":"2024-09-16T01:53:36.546105+0000","last_fullsized":"2024-09-16T01:53:36.546105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:55.614045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":466,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546041+0000","last_change":"2024-09-16T01:53:23.514351+0000","last_active":"2024-09-16T01:53:36.546041+0000","last_peered":"2024-09-16T01:53:36.546041+0000","last_clean":"2024-09-16T01:53:36.546041+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:36.546041+0000","last_undegraded":"2024-09-16T01:53:36.546041+0000","last_fullsized":"2024-09-16T01:53:36.546041+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:14.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"130'8","reported_seq":313,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842568+0000","last_change":"2024-09-16T01:53:19.543649+0000","last_active":"2024-09-16T01:53:36.842568+0000","last_peered":"2024-09-16T01:53:36.842568+0000","last_clean":"2024-09-16T01:53:36.842568+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:53:36.842568+0000","last_undegraded":"2024-09-16T01:53:36.842568+0000","last_fullsized":"2024-09-16T01:53:36.842568+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:12.126459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"127'176","reported_seq":541,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546195+0000","last_change":"2024-09-16T01:53:23.518054+0000","last_active":"2024-09-16T01:53:36.546195+0000","last_peered":"2024-09-16T01:53:36.546195+0000","last_clean":"2024-09-16T01:53:36.546195+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:36.546195+0000","last_undegraded":"2024-09-16T01:53:36.546195+0000","last_fullsized":"2024-09-16T01:53:36.546195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:50.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551978+0000","last_change":"2024-09-16T01:53:19.575999+0000","last_active":"2024-09-16T01:53:36.551978+0000","last_peered":"2024-09-16T01:53:36.551978+0000","last_clean":"2024-09-16T01:53:36.551978+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:36.551978+0000","last_undegraded":"2024-09-16T01:53:36.551978+0000","last_fullsized":"2024-09-16T01:53:36.551978+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:59.687176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"152'162","reported_seq":587,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.319456+0000","last_change":"2024-09-16T01:53:23.514324+0000","last_active":"2024-09-16T01:53:41.319456+0000","last_peered":"2024-09-16T01:53:41.319456+0000","last_clean":"2024-09-16T01:53:41.319456+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:53:41.319456+0000","last_undegraded":"2024-09-16T01:53:41.319456+0000","last_fullsized":"2024-09-16T01:53:41.319456+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:27.575957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546415+0000","last_change":"2024-09-16T01:53:20.576951+0000","last_active":"2024-09-16T01:53:36.546415+0000","last_peered":"2024-09-16T01:53:36.546415+0000","last_clean":"2024-09-16T01:53:36.546415+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:36.546415+0000","last_undegraded":"2024-09-16T01:53:36.546415+0000","last_fullsized":"2024-09-16T01:53:36.546415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:20.021875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"127'164","reported_seq":494,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545261+0000","last_change":"2024-09-16T01:53:23.517332+0000","last_active":"2024-09-16T01:53:36.545261+0000","last_peered":"2024-09-16T01:53:36.545261+0000","last_clean":"2024-09-16T01:53:36.545261+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:36.545261+0000","last_undegraded":"2024-09-16T01:53:36.545261+0000","last_fullsized":"2024-09-16T01:53:36.545261+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:15.929317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":248,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545542+0000","last_change":"2024-09-16T01:53:34.538675+0000","last_active":"2024-09-16T01:53:36.545542+0000","last_peered":"2024-09-16T01:53:36.545542+0000","last_clean":"2024-09-16T01:53:36.545542+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:36.545542+0000","last_undegraded":"2024-09-16T01:53:36.545542+0000","last_fullsized":"2024-09-16T01:53:36.545542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":624,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554614+0000","last_change":"2024-09-16T01:53:36.554614+0000","last_active":"2024-09-16T01:53:36.554614+0000","last_peered":"2024-09-16T01:53:36.554614+0000","last_clean":"2024-09-16T01:53:36.554614+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:36.554614+0000","last_undegraded":"2024-09-16T01:53:36.554614+0000","last_fullsized":"2024-09-16T01:53:36.554614+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:23.435521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.841495+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:36.841495+0000","last_peered":"2024-09-16T01:53:36.841495+0000","last_clean":"2024-09-16T01:53:36.841495+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:36.841495+0000","last_undegraded":"2024-09-16T01:53:36.841495+0000","last_fullsized":"2024-09-16T01:53:36.841495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"152'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343195+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:41.343195+0000","last_peered":"2024-09-16T01:53:41.343195+0000","last_clean":"2024-09-16T01:53:41.343195+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:41.343195+0000","last_undegraded":"2024-09-16T01:53:41.343195+0000","last_fullsized":"2024-09-16T01:53:41.343195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"152'205","reported_seq":717,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361493+0000","last_change":"2024-09-16T01:53:22.527742+0000","last_active":"2024-09-16T01:53:41.361493+0000","last_peered":"2024-09-16T01:53:41.361493+0000","last_clean":"2024-09-16T01:53:41.361493+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:53:41.361493+0000","last_undegraded":"2024-09-16T01:53:41.361493+0000","last_fullsized":"2024-09-16T01:53:41.361493+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553134+0000","last_change":"2024-09-16T01:53:19.535031+0000","last_active":"2024-09-16T01:53:36.553134+0000","last_peered":"2024-09-16T01:53:36.553134+0000","last_clean":"2024-09-16T01:53:36.553134+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:36.553134+0000","last_undegraded":"2024-09-16T01:53:36.553134+0000","last_fullsized":"2024-09-16T01:53:36.553134+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:36.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"152'162","reported_seq":627,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.353319+0000","last_change":"2024-09-16T01:53:22.521982+0000","last_active":"2024-09-16T01:53:41.353319+0000","last_peered":"2024-09-16T01:53:41.353319+0000","last_clean":"2024-09-16T01:53:41.353319+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:41.353319+0000","last_undegraded":"2024-09-16T01:53:41.353319+0000","last_fullsized":"2024-09-16T01:53:41.353319+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:35.196217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551329+0000","last_change":"2024-09-16T01:53:19.559597+0000","last_active":"2024-09-16T01:53:36.551329+0000","last_peered":"2024-09-16T01:53:36.551329+0000","last_clean":"2024-09-16T01:53:36.551329+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:36.551329+0000","last_undegraded":"2024-09-16T01:53:36.551329+0000","last_fullsized":"2024-09-16T01:53:36.551329+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552068+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:36.552068+0000","last_peered":"2024-09-16T01:53:36.552068+0000","last_clean":"2024-09-16T01:53:36.552068+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:36.552068+0000","last_undegraded":"2024-09-16T01:53:36.552068+0000","last_fullsized":"2024-09-16T01:53:36.552068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552067+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:36.552067+0000","last_peered":"2024-09-16T01:53:36.552067+0000","last_clean":"2024-09-16T01:53:36.552067+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:36.552067+0000","last_undegraded":"2024-09-16T01:53:36.552067+0000","last_fullsized":"2024-09-16T01:53:36.552067+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":449,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551955+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:36.551955+0000","last_peered":"2024-09-16T01:53:36.551955+0000","last_clean":"2024-09-16T01:53:36.551955+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:36.551955+0000","last_undegraded":"2024-09-16T01:53:36.551955+0000","last_fullsized":"2024-09-16T01:53:36.551955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":56,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545971+0000","last_change":"2024-09-16T01:53:20.557667+0000","last_active":"2024-09-16T01:53:36.545971+0000","last_peered":"2024-09-16T01:53:36.545971+0000","last_clean":"2024-09-16T01:53:36.545971+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:36.545971+0000","last_undegraded":"2024-09-16T01:53:36.545971+0000","last_fullsized":"2024-09-16T01:53:36.545971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:05:38.159129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4442,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11010,"ondisk_log_size":11010,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398903,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":423,"num_write_kb":602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797557,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509504,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1855488,"data_stored":874658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509504,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37528,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334312,"statfs":{"total":96636764160,"available":96598335488,"internally_reserved":0,"allocated":2236416,"data_stored":1263545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1904640,"data_stored":1463571,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509504,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1855488,"data_stored":874731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542207,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":873348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542207,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1839104,"data_stored":874169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1912832,"data_stored":1463488,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41420,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330420,"statfs":{"total":96636764160,"available":96594350080,"internally_reserved":0,"allocated":1949696,"data_stored":1264954,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393165,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:42.648 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:53:42.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:42.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.838+0000 7f707d600640 1 Processor -- start 2024-09-16T01:53:42.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.838+0000 7f707d600640 1 -- start start 2024-09-16T01:53:42.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f707d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780c6b70 0x7f707815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f707d600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70780cc040 0x7f70780cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f707d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f707806d140 con 0x7f70780c75a0 2024-09-16T01:53:42.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f707d600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f707815b880 con 0x7f70780cc040 2024-09-16T01:53:42.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f707d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70780cab20 con 0x7f70780c6b70 2024-09-16T01:53:42.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f7077400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f70780c75a0 0x7f70780cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47302/0 (socket says 172.21.15.146:47302) 2024-09-16T01:53:42.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f7077400640 1 -- 172.21.15.146:0/1978722832 learned_addr learned my addr 172.21.15.146:0/1978722832 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:42.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f707cc00640 1 --2- 172.21.15.146:0/1978722832 >> v2:172.21.15.161:3300/0 conn(0x7f70780cc040 0x7f70780cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.839+0000 7f7077e00640 1 --2- 172.21.15.146:0/1978722832 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780c6b70 0x7f707815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f7076a00640 1 -- 172.21.15.146:0/1978722832 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944064134 0 0) 0x7f707806d140 con 0x7f70780c75a0 2024-09-16T01:53:42.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f7076a00640 1 -- 172.21.15.146:0/1978722832 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f705c003520 con 0x7f70780c75a0 2024-09-16T01:53:42.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f7077e00640 1 -- 172.21.15.146:0/1978722832 >> v2:172.21.15.161:3300/0 conn(0x7f70780cc040 msgr2=0x7f70780cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:42.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f7077e00640 1 --2- 172.21.15.146:0/1978722832 >> v2:172.21.15.161:3300/0 conn(0x7f70780cc040 0x7f70780cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f7077e00640 1 -- 172.21.15.146:0/1978722832 >> v1:172.21.15.146:6789/0 conn(0x7f70780c75a0 legacy=0x7f70780cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f7077e00640 1 -- 172.21.15.146:0/1978722832 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707806d420 con 0x7f70780c6b70 2024-09-16T01:53:42.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.840+0000 7f707cc00640 1 --2- 172.21.15.146:0/1978722832 >> v2:172.21.15.161:3300/0 conn(0x7f70780cc040 0x7f70780cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:42.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.841+0000 7f7077e00640 1 --2- 172.21.15.146:0/1978722832 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780c6b70 0x7f707815b340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f70680097a0 tx=0x7f706802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=319b887bedcd9e07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.841+0000 7f7076a00640 1 -- 172.21.15.146:0/1978722832 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068037070 con 0x7f70780c6b70 2024-09-16T01:53:42.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.841+0000 7f7076a00640 1 -- 172.21.15.146:0/1978722832 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706802ae50 con 0x7f70780c6b70 2024-09-16T01:53:42.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.841+0000 7f7076a00640 1 -- 172.21.15.146:0/1978722832 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068032640 con 0x7f70780c6b70 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 -- 172.21.15.146:0/1978722832 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780c6b70 msgr2=0x7f707815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 --2- 172.21.15.146:0/1978722832 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780c6b70 0x7f707815b340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f70680097a0 tx=0x7f706802a100 comp rx=0 tx=0).stop 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 -- 172.21.15.146:0/1978722832 shutdown_connections 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 --2- 172.21.15.146:0/1978722832 >> v2:172.21.15.161:3300/0 conn(0x7f70780cc040 0x7f70780cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 --2- 172.21.15.146:0/1978722832 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780c6b70 0x7f707815b340 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 -- 172.21.15.146:0/1978722832 >> 172.21.15.146:0/1978722832 conn(0x7f70780ccc20 msgr2=0x7f70780cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 -- 172.21.15.146:0/1978722832 shutdown_connections 2024-09-16T01:53:42.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.842+0000 7f707d600640 1 -- 172.21.15.146:0/1978722832 wait complete. 2024-09-16T01:53:42.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.843+0000 7f707d600640 1 Processor -- start 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.843+0000 7f707d600640 1 -- start start 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.844+0000 7f707d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 0x7f70781e6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.844+0000 7f707d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70780c75a0 0x7f707815e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.844+0000 7f707d600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780cc040 0x7f707815d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.844+0000 7f707d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f70780cb050 con 0x7f70780c75a0 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.844+0000 7f707d600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70780cad70 con 0x7f70780c6b70 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.844+0000 7f707d600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70780cbc80 con 0x7f70780cc040 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f7077e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 0x7f70781e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f7077400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70780c75a0 0x7f707815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780cc040 0x7f707815d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f7077e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 0x7f70781e6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55008/0 (socket says 172.21.15.146:55008) 2024-09-16T01:53:42.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f7077e00640 1 -- 172.21.15.146:0/1755600688 learned_addr learned my addr 172.21.15.146:0/1755600688 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 -- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 msgr2=0x7f70781e6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 0x7f70781e6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 -- 172.21.15.146:0/1755600688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70780c75a0 msgr2=0x7f707815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70780c75a0 0x7f707815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 -- 172.21.15.146:0/1755600688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707815c680 con 0x7f70780cc040 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f7077e00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 0x7f70781e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f7077400640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70780c75a0 0x7f707815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f707cc00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780cc040 0x7f707815d0a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f707000ef90 tx=0x7f707000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.845+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070018020 con 0x7f70780cc040 2024-09-16T01:53:42.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.846+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7070014480 con 0x7f70780cc040 2024-09-16T01:53:42.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.846+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070014870 con 0x7f70780cc040 2024-09-16T01:53:42.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.846+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70780c5600 con 0x7f70780cc040 2024-09-16T01:53:42.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.846+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70780c9950 con 0x7f70780cc040 2024-09-16T01:53:42.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.847+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f707815dd90 con 0x7f70780cc040 2024-09-16T01:53:42.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.848+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f707001b020 con 0x7f70780cc040 2024-09-16T01:53:42.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.849+0000 7f706fe00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f705806c2b0 0x7f705806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:42.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.849+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(154..154 src has 1..154) ==== 5911+0+0 (secure 0 0 0) 0x7f707008c290 con 0x7f70780cc040 2024-09-16T01:53:42.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.849+0000 7f7077e00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f705806c2b0 0x7f705806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:42.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.849+0000 7f7077e00640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f705806c2b0 0x7f705806e770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f70680097a0 tx=0x7f706802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:42.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:42.853+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7070022020 con 0x7f70780cc040 2024-09-16T01:53:43.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.024+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f707815c430 con 0x7f705806c2b0 2024-09-16T01:53:43.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.031+0000 7f706fe00640 1 -- 172.21.15.146:0/1755600688 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253939 (secure 0 0 0) 0x7f707815c430 con 0x7f705806c2b0 2024-09-16T01:53:43.036 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:43.037 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.034+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f705806c2b0 msgr2=0x7f705806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.034+0000 7f707d600640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f705806c2b0 0x7f705806e770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f70680097a0 tx=0x7f706802e000 comp rx=0 tx=0).stop 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.034+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780cc040 msgr2=0x7f707815d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.034+0000 7f707d600640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780cc040 0x7f707815d0a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f707000ef90 tx=0x7f707000c490 comp rx=0 tx=0).stop 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 shutdown_connections 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f705806c2b0 0x7f705806e770 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70780cc040 0x7f707815d0a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70780c75a0 0x7f707815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 --2- 172.21.15.146:0/1755600688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70780c6b70 0x7f70781e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 >> 172.21.15.146:0/1755600688 conn(0x7f70780ccc20 msgr2=0x7f70780c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 shutdown_connections 2024-09-16T01:53:43.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.035+0000 7f707d600640 1 -- 172.21.15.146:0/1755600688 wait complete. 2024-09-16T01:53:43.053 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T01:53:42.649924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":989271,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9798,"num_write_kb":25168842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4698,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11513,"ondisk_log_size":11513,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":281216,"kb_used_data":15052,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754693504,"statfs":{"total":773094113280,"available":772806148096,"internally_reserved":0,"allocated":15413248,"data_stored":8952464,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.107400"},"pg_stats":[{"pgid":"3.3f","version":"152'167","reported_seq":492,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361768+0000","last_change":"2024-09-16T01:53:22.525372+0000","last_active":"2024-09-16T01:53:41.361768+0000","last_peered":"2024-09-16T01:53:41.361768+0000","last_clean":"2024-09-16T01:53:41.361768+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:41.361768+0000","last_undegraded":"2024-09-16T01:53:41.361768+0000","last_fullsized":"2024-09-16T01:53:41.361768+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:00.392212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"127'148","reported_seq":421,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551885+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:36.551885+0000","last_peered":"2024-09-16T01:53:36.551885+0000","last_clean":"2024-09-16T01:53:36.551885+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:36.551885+0000","last_undegraded":"2024-09-16T01:53:36.551885+0000","last_fullsized":"2024-09-16T01:53:36.551885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"127'171","reported_seq":799,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551822+0000","last_change":"2024-09-16T01:53:23.518810+0000","last_active":"2024-09-16T01:53:36.551822+0000","last_peered":"2024-09-16T01:53:36.551822+0000","last_clean":"2024-09-16T01:53:36.551822+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:53:36.551822+0000","last_undegraded":"2024-09-16T01:53:36.551822+0000","last_fullsized":"2024-09-16T01:53:36.551822+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"152'193","reported_seq":591,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.334545+0000","last_change":"2024-09-16T01:53:32.511261+0000","last_active":"2024-09-16T01:53:41.334545+0000","last_peered":"2024-09-16T01:53:41.334545+0000","last_clean":"2024-09-16T01:53:41.334545+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:41.334545+0000","last_undegraded":"2024-09-16T01:53:41.334545+0000","last_fullsized":"2024-09-16T01:53:41.334545+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:07.896976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"152'174","reported_seq":526,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.342604+0000","last_change":"2024-09-16T01:53:22.521371+0000","last_active":"2024-09-16T01:53:41.342604+0000","last_peered":"2024-09-16T01:53:41.342604+0000","last_clean":"2024-09-16T01:53:41.342604+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:41.342604+0000","last_undegraded":"2024-09-16T01:53:41.342604+0000","last_fullsized":"2024-09-16T01:53:41.342604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:39.028997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"152'162","reported_seq":582,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.341075+0000","last_change":"2024-09-16T01:53:23.521468+0000","last_active":"2024-09-16T01:53:41.341075+0000","last_peered":"2024-09-16T01:53:41.341075+0000","last_clean":"2024-09-16T01:53:41.341075+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:41.341075+0000","last_undegraded":"2024-09-16T01:53:41.341075+0000","last_fullsized":"2024-09-16T01:53:41.341075+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:34:42.522900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"152'188","reported_seq":638,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.345763+0000","last_change":"2024-09-16T01:53:22.523352+0000","last_active":"2024-09-16T01:53:41.345763+0000","last_peered":"2024-09-16T01:53:41.345763+0000","last_clean":"2024-09-16T01:53:41.345763+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:41.345763+0000","last_undegraded":"2024-09-16T01:53:41.345763+0000","last_fullsized":"2024-09-16T01:53:41.345763+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.078885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"127'195","reported_seq":722,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551418+0000","last_change":"2024-09-16T01:53:22.523335+0000","last_active":"2024-09-16T01:53:36.551418+0000","last_peered":"2024-09-16T01:53:36.551418+0000","last_clean":"2024-09-16T01:53:36.551418+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:53:36.551418+0000","last_undegraded":"2024-09-16T01:53:36.551418+0000","last_fullsized":"2024-09-16T01:53:36.551418+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"152'178","reported_seq":451,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.357816+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:41.357816+0000","last_peered":"2024-09-16T01:53:41.357816+0000","last_clean":"2024-09-16T01:53:41.357816+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:41.357816+0000","last_undegraded":"2024-09-16T01:53:41.357816+0000","last_fullsized":"2024-09-16T01:53:41.357816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:56.250967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"152'190","reported_seq":570,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.356987+0000","last_change":"2024-09-16T01:53:22.526050+0000","last_active":"2024-09-16T01:53:41.356987+0000","last_peered":"2024-09-16T01:53:41.356987+0000","last_clean":"2024-09-16T01:53:41.356987+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:41.356987+0000","last_undegraded":"2024-09-16T01:53:41.356987+0000","last_fullsized":"2024-09-16T01:53:41.356987+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:51:00.473148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"152'214","reported_seq":617,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.357952+0000","last_change":"2024-09-16T01:53:23.519544+0000","last_active":"2024-09-16T01:53:41.357952+0000","last_peered":"2024-09-16T01:53:41.357952+0000","last_clean":"2024-09-16T01:53:41.357952+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:41.357952+0000","last_undegraded":"2024-09-16T01:53:41.357952+0000","last_fullsized":"2024-09-16T01:53:41.357952+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:26.797344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"152'166","reported_seq":362,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343823+0000","last_change":"2024-09-16T01:53:22.519665+0000","last_active":"2024-09-16T01:53:41.343823+0000","last_peered":"2024-09-16T01:53:41.343823+0000","last_clean":"2024-09-16T01:53:41.343823+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:41.343823+0000","last_undegraded":"2024-09-16T01:53:41.343823+0000","last_fullsized":"2024-09-16T01:53:41.343823+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:32.846352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"127'167","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545527+0000","last_change":"2024-09-16T01:53:34.533587+0000","last_active":"2024-09-16T01:53:36.545527+0000","last_peered":"2024-09-16T01:53:36.545527+0000","last_clean":"2024-09-16T01:53:36.545527+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:36.545527+0000","last_undegraded":"2024-09-16T01:53:36.545527+0000","last_fullsized":"2024-09-16T01:53:36.545527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:15.453058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"127'164","reported_seq":488,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545392+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:36.545392+0000","last_peered":"2024-09-16T01:53:36.545392+0000","last_clean":"2024-09-16T01:53:36.545392+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:36.545392+0000","last_undegraded":"2024-09-16T01:53:36.545392+0000","last_fullsized":"2024-09-16T01:53:36.545392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"127'170","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551212+0000","last_change":"2024-09-16T01:53:22.527702+0000","last_active":"2024-09-16T01:53:36.551212+0000","last_peered":"2024-09-16T01:53:36.551212+0000","last_clean":"2024-09-16T01:53:36.551212+0000","last_became_active":"2024-09-16T01:53:22.526994+0000","last_became_peered":"2024-09-16T01:53:22.526994+0000","last_unstale":"2024-09-16T01:53:36.551212+0000","last_undegraded":"2024-09-16T01:53:36.551212+0000","last_fullsized":"2024-09-16T01:53:36.551212+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"152'162","reported_seq":435,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.359260+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:41.359260+0000","last_peered":"2024-09-16T01:53:41.359260+0000","last_clean":"2024-09-16T01:53:41.359260+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:41.359260+0000","last_undegraded":"2024-09-16T01:53:41.359260+0000","last_fullsized":"2024-09-16T01:53:41.359260+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"152'176","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361166+0000","last_change":"2024-09-16T01:53:23.526130+0000","last_active":"2024-09-16T01:53:41.361166+0000","last_peered":"2024-09-16T01:53:41.361166+0000","last_clean":"2024-09-16T01:53:41.361166+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:53:41.361166+0000","last_undegraded":"2024-09-16T01:53:41.361166+0000","last_fullsized":"2024-09-16T01:53:41.361166+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:02.127294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"127'169","reported_seq":575,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563883+0000","last_change":"2024-09-16T01:53:36.563883+0000","last_active":"2024-09-16T01:53:36.563883+0000","last_peered":"2024-09-16T01:53:36.563883+0000","last_clean":"2024-09-16T01:53:36.563883+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:36.563883+0000","last_undegraded":"2024-09-16T01:53:36.563883+0000","last_fullsized":"2024-09-16T01:53:36.563883+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"127'198","reported_seq":651,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554778+0000","last_change":"2024-09-16T01:53:36.554778+0000","last_active":"2024-09-16T01:53:36.554778+0000","last_peered":"2024-09-16T01:53:36.554778+0000","last_clean":"2024-09-16T01:53:36.554778+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:36.554778+0000","last_undegraded":"2024-09-16T01:53:36.554778+0000","last_fullsized":"2024-09-16T01:53:36.554778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"152'179","reported_seq":596,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343701+0000","last_change":"2024-09-16T01:53:23.524759+0000","last_active":"2024-09-16T01:53:41.343701+0000","last_peered":"2024-09-16T01:53:41.343701+0000","last_clean":"2024-09-16T01:53:41.343701+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:41.343701+0000","last_undegraded":"2024-09-16T01:53:41.343701+0000","last_fullsized":"2024-09-16T01:53:41.343701+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"127'143","reported_seq":455,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545707+0000","last_change":"2024-09-16T01:53:23.520715+0000","last_active":"2024-09-16T01:53:36.545707+0000","last_peered":"2024-09-16T01:53:36.545707+0000","last_clean":"2024-09-16T01:53:36.545707+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:36.545707+0000","last_undegraded":"2024-09-16T01:53:36.545707+0000","last_fullsized":"2024-09-16T01:53:36.545707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:45.628132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"152'199","reported_seq":831,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.353111+0000","last_change":"2024-09-16T01:53:22.524407+0000","last_active":"2024-09-16T01:53:41.353111+0000","last_peered":"2024-09-16T01:53:41.353111+0000","last_clean":"2024-09-16T01:53:41.353111+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:41.353111+0000","last_undegraded":"2024-09-16T01:53:41.353111+0000","last_fullsized":"2024-09-16T01:53:41.353111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"127'142","reported_seq":415,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551482+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:36.551482+0000","last_peered":"2024-09-16T01:53:36.551482+0000","last_clean":"2024-09-16T01:53:36.551482+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:36.551482+0000","last_undegraded":"2024-09-16T01:53:36.551482+0000","last_fullsized":"2024-09-16T01:53:36.551482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"152'207","reported_seq":480,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.359084+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:41.359084+0000","last_peered":"2024-09-16T01:53:41.359084+0000","last_clean":"2024-09-16T01:53:41.359084+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:41.359084+0000","last_undegraded":"2024-09-16T01:53:41.359084+0000","last_fullsized":"2024-09-16T01:53:41.359084+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"127'158","reported_seq":431,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552017+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:36.552017+0000","last_peered":"2024-09-16T01:53:36.552017+0000","last_clean":"2024-09-16T01:53:36.552017+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:36.552017+0000","last_undegraded":"2024-09-16T01:53:36.552017+0000","last_fullsized":"2024-09-16T01:53:36.552017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"152'217","reported_seq":491,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362690+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:41.362690+0000","last_peered":"2024-09-16T01:53:41.362690+0000","last_clean":"2024-09-16T01:53:41.362690+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:41.362690+0000","last_undegraded":"2024-09-16T01:53:41.362690+0000","last_fullsized":"2024-09-16T01:53:41.362690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:01.899687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"127'160","reported_seq":664,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545722+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:36.545722+0000","last_peered":"2024-09-16T01:53:36.545722+0000","last_clean":"2024-09-16T01:53:36.545722+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:36.545722+0000","last_undegraded":"2024-09-16T01:53:36.545722+0000","last_fullsized":"2024-09-16T01:53:36.545722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"127'148","reported_seq":484,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545688+0000","last_change":"2024-09-16T01:53:34.533524+0000","last_active":"2024-09-16T01:53:36.545688+0000","last_peered":"2024-09-16T01:53:36.545688+0000","last_clean":"2024-09-16T01:53:36.545688+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:36.545688+0000","last_undegraded":"2024-09-16T01:53:36.545688+0000","last_fullsized":"2024-09-16T01:53:36.545688+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:57.738639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"127'127","reported_seq":683,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551567+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:36.551567+0000","last_peered":"2024-09-16T01:53:36.551567+0000","last_clean":"2024-09-16T01:53:36.551567+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:36.551567+0000","last_undegraded":"2024-09-16T01:53:36.551567+0000","last_fullsized":"2024-09-16T01:53:36.551567+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"152'153","reported_seq":426,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.355574+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:41.355574+0000","last_peered":"2024-09-16T01:53:41.355574+0000","last_clean":"2024-09-16T01:53:41.355574+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:41.355574+0000","last_undegraded":"2024-09-16T01:53:41.355574+0000","last_fullsized":"2024-09-16T01:53:41.355574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"127'146","reported_seq":558,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.563260+0000","last_change":"2024-09-16T01:53:36.563260+0000","last_active":"2024-09-16T01:53:36.563260+0000","last_peered":"2024-09-16T01:53:36.563260+0000","last_clean":"2024-09-16T01:53:36.563260+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:36.563260+0000","last_undegraded":"2024-09-16T01:53:36.563260+0000","last_fullsized":"2024-09-16T01:53:36.563260+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:31.539361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"127'171","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551939+0000","last_change":"2024-09-16T01:53:36.551939+0000","last_active":"2024-09-16T01:53:36.551939+0000","last_peered":"2024-09-16T01:53:36.551939+0000","last_clean":"2024-09-16T01:53:36.551939+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:36.551939+0000","last_undegraded":"2024-09-16T01:53:36.551939+0000","last_fullsized":"2024-09-16T01:53:36.551939+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"152'149","reported_seq":422,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.338192+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:41.338192+0000","last_peered":"2024-09-16T01:53:41.338192+0000","last_clean":"2024-09-16T01:53:41.338192+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:41.338192+0000","last_undegraded":"2024-09-16T01:53:41.338192+0000","last_fullsized":"2024-09-16T01:53:41.338192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.332032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551629+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:36.551629+0000","last_peered":"2024-09-16T01:53:36.551629+0000","last_clean":"2024-09-16T01:53:36.551629+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:36.551629+0000","last_undegraded":"2024-09-16T01:53:36.551629+0000","last_fullsized":"2024-09-16T01:53:36.551629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"127'154","reported_seq":762,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545597+0000","last_change":"2024-09-16T01:53:22.520820+0000","last_active":"2024-09-16T01:53:36.545597+0000","last_peered":"2024-09-16T01:53:36.545597+0000","last_clean":"2024-09-16T01:53:36.545597+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:36.545597+0000","last_undegraded":"2024-09-16T01:53:36.545597+0000","last_fullsized":"2024-09-16T01:53:36.545597+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:51.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842491+0000","last_change":"2024-09-16T01:53:19.532463+0000","last_active":"2024-09-16T01:53:36.842491+0000","last_peered":"2024-09-16T01:53:36.842491+0000","last_clean":"2024-09-16T01:53:36.842491+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:36.842491+0000","last_undegraded":"2024-09-16T01:53:36.842491+0000","last_fullsized":"2024-09-16T01:53:36.842491+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:52.176146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":275,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545279+0000","last_change":"2024-09-16T01:53:34.532869+0000","last_active":"2024-09-16T01:53:36.545279+0000","last_peered":"2024-09-16T01:53:36.545279+0000","last_clean":"2024-09-16T01:53:36.545279+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:36.545279+0000","last_undegraded":"2024-09-16T01:53:36.545279+0000","last_fullsized":"2024-09-16T01:53:36.545279+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:13.148781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"127'182","reported_seq":454,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551995+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:36.551995+0000","last_peered":"2024-09-16T01:53:36.551995+0000","last_clean":"2024-09-16T01:53:36.551995+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:36.551995+0000","last_undegraded":"2024-09-16T01:53:36.551995+0000","last_fullsized":"2024-09-16T01:53:36.551995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":277,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.840791+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:36.840791+0000","last_peered":"2024-09-16T01:53:36.840791+0000","last_clean":"2024-09-16T01:53:36.840791+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:36.840791+0000","last_undegraded":"2024-09-16T01:53:36.840791+0000","last_fullsized":"2024-09-16T01:53:36.840791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"127'177","reported_seq":418,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545675+0000","last_change":"2024-09-16T01:53:32.513845+0000","last_active":"2024-09-16T01:53:36.545675+0000","last_peered":"2024-09-16T01:53:36.545675+0000","last_clean":"2024-09-16T01:53:36.545675+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:36.545675+0000","last_undegraded":"2024-09-16T01:53:36.545675+0000","last_fullsized":"2024-09-16T01:53:36.545675+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:02.738813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":318,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.556079+0000","last_change":"2024-09-16T01:53:20.522781+0000","last_active":"2024-09-16T01:53:36.556079+0000","last_peered":"2024-09-16T01:53:36.556079+0000","last_clean":"2024-09-16T01:53:36.556079+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:36.556079+0000","last_undegraded":"2024-09-16T01:53:36.556079+0000","last_fullsized":"2024-09-16T01:53:36.556079+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:25.742037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"127'241","reported_seq":527,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551737+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:36.551737+0000","last_peered":"2024-09-16T01:53:36.551737+0000","last_clean":"2024-09-16T01:53:36.551737+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:36.551737+0000","last_undegraded":"2024-09-16T01:53:36.551737+0000","last_fullsized":"2024-09-16T01:53:36.551737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553376+0000","last_change":"2024-09-16T01:53:19.579965+0000","last_active":"2024-09-16T01:53:36.553376+0000","last_peered":"2024-09-16T01:53:36.553376+0000","last_clean":"2024-09-16T01:53:36.553376+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:36.553376+0000","last_undegraded":"2024-09-16T01:53:36.553376+0000","last_fullsized":"2024-09-16T01:53:36.553376+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:29.737898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"127'151","reported_seq":423,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552118+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:36.552118+0000","last_peered":"2024-09-16T01:53:36.552118+0000","last_clean":"2024-09-16T01:53:36.552118+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:36.552118+0000","last_undegraded":"2024-09-16T01:53:36.552118+0000","last_fullsized":"2024-09-16T01:53:36.552118+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"133'11","reported_seq":160,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557008+0000","last_change":"2024-09-16T01:53:36.557008+0000","last_active":"2024-09-16T01:53:36.557008+0000","last_peered":"2024-09-16T01:53:36.557008+0000","last_clean":"2024-09-16T01:53:36.557008+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:36.557008+0000","last_undegraded":"2024-09-16T01:53:36.557008+0000","last_fullsized":"2024-09-16T01:53:36.557008+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'168","reported_seq":441,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362450+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:41.362450+0000","last_peered":"2024-09-16T01:53:41.362450+0000","last_clean":"2024-09-16T01:53:41.362450+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:41.362450+0000","last_undegraded":"2024-09-16T01:53:41.362450+0000","last_fullsized":"2024-09-16T01:53:41.362450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"152'178","reported_seq":496,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.364517+0000","last_change":"2024-09-16T01:53:23.518694+0000","last_active":"2024-09-16T01:53:41.364517+0000","last_peered":"2024-09-16T01:53:41.364517+0000","last_clean":"2024-09-16T01:53:41.364517+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:41.364517+0000","last_undegraded":"2024-09-16T01:53:41.364517+0000","last_fullsized":"2024-09-16T01:53:41.364517+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:11.376091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391576,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"152'176","reported_seq":533,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343763+0000","last_change":"2024-09-16T01:53:22.524097+0000","last_active":"2024-09-16T01:53:41.343763+0000","last_peered":"2024-09-16T01:53:41.343763+0000","last_clean":"2024-09-16T01:53:41.343763+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:41.343763+0000","last_undegraded":"2024-09-16T01:53:41.343763+0000","last_fullsized":"2024-09-16T01:53:41.343763+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:14.041864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":264,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.559516+0000","last_change":"2024-09-16T01:53:36.559516+0000","last_active":"2024-09-16T01:53:36.559516+0000","last_peered":"2024-09-16T01:53:36.559516+0000","last_clean":"2024-09-16T01:53:36.559516+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:36.559516+0000","last_undegraded":"2024-09-16T01:53:36.559516+0000","last_fullsized":"2024-09-16T01:53:36.559516+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"127'168","reported_seq":411,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546458+0000","last_change":"2024-09-16T01:53:34.529911+0000","last_active":"2024-09-16T01:53:36.546458+0000","last_peered":"2024-09-16T01:53:36.546458+0000","last_clean":"2024-09-16T01:53:36.546458+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:36.546458+0000","last_undegraded":"2024-09-16T01:53:36.546458+0000","last_fullsized":"2024-09-16T01:53:36.546458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":358,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842660+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:36.842660+0000","last_peered":"2024-09-16T01:53:36.842660+0000","last_clean":"2024-09-16T01:53:36.842660+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:36.842660+0000","last_undegraded":"2024-09-16T01:53:36.842660+0000","last_fullsized":"2024-09-16T01:53:36.842660+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"152'201","reported_seq":718,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.362908+0000","last_change":"2024-09-16T01:53:23.521939+0000","last_active":"2024-09-16T01:53:41.362908+0000","last_peered":"2024-09-16T01:53:41.362908+0000","last_clean":"2024-09-16T01:53:41.362908+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:53:41.362908+0000","last_undegraded":"2024-09-16T01:53:41.362908+0000","last_fullsized":"2024-09-16T01:53:41.362908+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:55.352896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"152'7","reported_seq":53,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552327+0000","last_change":"2024-09-16T01:53:19.550015+0000","last_active":"2024-09-16T01:53:36.552327+0000","last_peered":"2024-09-16T01:53:36.552327+0000","last_clean":"2024-09-16T01:53:36.552327+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:36.552327+0000","last_undegraded":"2024-09-16T01:53:36.552327+0000","last_fullsized":"2024-09-16T01:53:36.552327+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"152'197","reported_seq":471,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.337310+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:41.337310+0000","last_peered":"2024-09-16T01:53:41.337310+0000","last_clean":"2024-09-16T01:53:41.337310+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:41.337310+0000","last_undegraded":"2024-09-16T01:53:41.337310+0000","last_fullsized":"2024-09-16T01:53:41.337310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:20.997448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"133'18","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545354+0000","last_change":"2024-09-16T01:53:20.541056+0000","last_active":"2024-09-16T01:53:36.545354+0000","last_peered":"2024-09-16T01:53:36.545354+0000","last_clean":"2024-09-16T01:53:36.545354+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:36.545354+0000","last_undegraded":"2024-09-16T01:53:36.545354+0000","last_fullsized":"2024-09-16T01:53:36.545354+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:21.564935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"152'215","reported_seq":488,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.356427+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:41.356427+0000","last_peered":"2024-09-16T01:53:41.356427+0000","last_clean":"2024-09-16T01:53:41.356427+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:41.356427+0000","last_undegraded":"2024-09-16T01:53:41.356427+0000","last_fullsized":"2024-09-16T01:53:41.356427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":50,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553629+0000","last_change":"2024-09-16T01:53:19.563306+0000","last_active":"2024-09-16T01:53:36.553629+0000","last_peered":"2024-09-16T01:53:36.553629+0000","last_clean":"2024-09-16T01:53:36.553629+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:36.553629+0000","last_undegraded":"2024-09-16T01:53:36.553629+0000","last_fullsized":"2024-09-16T01:53:36.553629+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:04.089748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"152'175","reported_seq":653,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361366+0000","last_change":"2024-09-16T01:53:22.527907+0000","last_active":"2024-09-16T01:53:41.361366+0000","last_peered":"2024-09-16T01:53:41.361366+0000","last_clean":"2024-09-16T01:53:41.361366+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:41.361366+0000","last_undegraded":"2024-09-16T01:53:41.361366+0000","last_fullsized":"2024-09-16T01:53:41.361366+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:51.042095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546066+0000","last_change":"2024-09-16T01:53:20.547257+0000","last_active":"2024-09-16T01:53:36.546066+0000","last_peered":"2024-09-16T01:53:36.546066+0000","last_clean":"2024-09-16T01:53:36.546066+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:36.546066+0000","last_undegraded":"2024-09-16T01:53:36.546066+0000","last_fullsized":"2024-09-16T01:53:36.546066+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:04.416789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"152'162","reported_seq":532,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.350317+0000","last_change":"2024-09-16T01:53:23.527787+0000","last_active":"2024-09-16T01:53:41.350317+0000","last_peered":"2024-09-16T01:53:41.350317+0000","last_clean":"2024-09-16T01:53:41.350317+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:53:41.350317+0000","last_undegraded":"2024-09-16T01:53:41.350317+0000","last_fullsized":"2024-09-16T01:53:41.350317+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551843+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:36.551843+0000","last_peered":"2024-09-16T01:53:36.551843+0000","last_clean":"2024-09-16T01:53:36.551843+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:36.551843+0000","last_undegraded":"2024-09-16T01:53:36.551843+0000","last_fullsized":"2024-09-16T01:53:36.551843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"152'165","reported_seq":768,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.334297+0000","last_change":"2024-09-16T01:53:22.522539+0000","last_active":"2024-09-16T01:53:41.334297+0000","last_peered":"2024-09-16T01:53:41.334297+0000","last_clean":"2024-09-16T01:53:41.334297+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:41.334297+0000","last_undegraded":"2024-09-16T01:53:41.334297+0000","last_fullsized":"2024-09-16T01:53:41.334297+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:01.840411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":263,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545924+0000","last_change":"2024-09-16T01:53:34.533293+0000","last_active":"2024-09-16T01:53:36.545924+0000","last_peered":"2024-09-16T01:53:36.545924+0000","last_clean":"2024-09-16T01:53:36.545924+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:36.545924+0000","last_undegraded":"2024-09-16T01:53:36.545924+0000","last_fullsized":"2024-09-16T01:53:36.545924+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:39.949872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"152'175","reported_seq":448,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.358312+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:41.358312+0000","last_peered":"2024-09-16T01:53:41.358312+0000","last_clean":"2024-09-16T01:53:41.358312+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:41.358312+0000","last_undegraded":"2024-09-16T01:53:41.358312+0000","last_fullsized":"2024-09-16T01:53:41.358312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":432,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.555710+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:36.555710+0000","last_peered":"2024-09-16T01:53:36.555710+0000","last_clean":"2024-09-16T01:53:36.555710+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:36.555710+0000","last_undegraded":"2024-09-16T01:53:36.555710+0000","last_fullsized":"2024-09-16T01:53:36.555710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:42.331279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.841615+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:36.841615+0000","last_peered":"2024-09-16T01:53:36.841615+0000","last_clean":"2024-09-16T01:53:36.841615+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:36.841615+0000","last_undegraded":"2024-09-16T01:53:36.841615+0000","last_fullsized":"2024-09-16T01:53:36.841615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"127'175","reported_seq":592,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545790+0000","last_change":"2024-09-16T01:53:22.524529+0000","last_active":"2024-09-16T01:53:36.545790+0000","last_peered":"2024-09-16T01:53:36.545790+0000","last_clean":"2024-09-16T01:53:36.545790+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:36.545790+0000","last_undegraded":"2024-09-16T01:53:36.545790+0000","last_fullsized":"2024-09-16T01:53:36.545790+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:39.441997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.840890+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:36.840890+0000","last_peered":"2024-09-16T01:53:36.840890+0000","last_clean":"2024-09-16T01:53:36.840890+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:36.840890+0000","last_undegraded":"2024-09-16T01:53:36.840890+0000","last_fullsized":"2024-09-16T01:53:36.840890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"127'160","reported_seq":403,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545398+0000","last_change":"2024-09-16T01:53:34.532478+0000","last_active":"2024-09-16T01:53:36.545398+0000","last_peered":"2024-09-16T01:53:36.545398+0000","last_clean":"2024-09-16T01:53:36.545398+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:36.545398+0000","last_undegraded":"2024-09-16T01:53:36.545398+0000","last_fullsized":"2024-09-16T01:53:36.545398+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"133'12","reported_seq":311,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554302+0000","last_change":"2024-09-16T01:53:36.554302+0000","last_active":"2024-09-16T01:53:36.554302+0000","last_peered":"2024-09-16T01:53:36.554302+0000","last_clean":"2024-09-16T01:53:36.554302+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:36.554302+0000","last_undegraded":"2024-09-16T01:53:36.554302+0000","last_fullsized":"2024-09-16T01:53:36.554302+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"127'157","reported_seq":647,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551550+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:36.551550+0000","last_peered":"2024-09-16T01:53:36.551550+0000","last_clean":"2024-09-16T01:53:36.551550+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:36.551550+0000","last_undegraded":"2024-09-16T01:53:36.551550+0000","last_fullsized":"2024-09-16T01:53:36.551550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842714+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:36.842714+0000","last_peered":"2024-09-16T01:53:36.842714+0000","last_clean":"2024-09-16T01:53:36.842714+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:36.842714+0000","last_undegraded":"2024-09-16T01:53:36.842714+0000","last_fullsized":"2024-09-16T01:53:36.842714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"152'177","reported_seq":550,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.351445+0000","last_change":"2024-09-16T01:53:23.527347+0000","last_active":"2024-09-16T01:53:41.351445+0000","last_peered":"2024-09-16T01:53:41.351445+0000","last_clean":"2024-09-16T01:53:41.351445+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:53:41.351445+0000","last_undegraded":"2024-09-16T01:53:41.351445+0000","last_fullsized":"2024-09-16T01:53:41.351445+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:13.133402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":276,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551717+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:36.551717+0000","last_peered":"2024-09-16T01:53:36.551717+0000","last_clean":"2024-09-16T01:53:36.551717+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:36.551717+0000","last_undegraded":"2024-09-16T01:53:36.551717+0000","last_fullsized":"2024-09-16T01:53:36.551717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"152'160","reported_seq":434,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.332794+0000","last_change":"2024-09-16T01:48:58.824723+0000","last_active":"2024-09-16T01:53:41.332794+0000","last_peered":"2024-09-16T01:53:41.332794+0000","last_clean":"2024-09-16T01:53:41.332794+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:41.332794+0000","last_undegraded":"2024-09-16T01:53:41.332794+0000","last_fullsized":"2024-09-16T01:53:41.332794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:03.166891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":119,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.557300+0000","last_change":"2024-09-16T01:53:20.534486+0000","last_active":"2024-09-16T01:53:36.557300+0000","last_peered":"2024-09-16T01:53:36.557300+0000","last_clean":"2024-09-16T01:53:36.557300+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:36.557300+0000","last_undegraded":"2024-09-16T01:53:36.557300+0000","last_fullsized":"2024-09-16T01:53:36.557300+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:26.318068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'153","reported_seq":396,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545805+0000","last_change":"2024-09-16T01:53:34.539129+0000","last_active":"2024-09-16T01:53:36.545805+0000","last_peered":"2024-09-16T01:53:36.545805+0000","last_clean":"2024-09-16T01:53:36.545805+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:36.545805+0000","last_undegraded":"2024-09-16T01:53:36.545805+0000","last_fullsized":"2024-09-16T01:53:36.545805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"133'25","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546105+0000","last_change":"2024-09-16T01:53:20.586608+0000","last_active":"2024-09-16T01:53:36.546105+0000","last_peered":"2024-09-16T01:53:36.546105+0000","last_clean":"2024-09-16T01:53:36.546105+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:36.546105+0000","last_undegraded":"2024-09-16T01:53:36.546105+0000","last_fullsized":"2024-09-16T01:53:36.546105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:55.614045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"127'161","reported_seq":466,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546041+0000","last_change":"2024-09-16T01:53:23.514351+0000","last_active":"2024-09-16T01:53:36.546041+0000","last_peered":"2024-09-16T01:53:36.546041+0000","last_clean":"2024-09-16T01:53:36.546041+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:36.546041+0000","last_undegraded":"2024-09-16T01:53:36.546041+0000","last_fullsized":"2024-09-16T01:53:36.546041+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:14.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"130'8","reported_seq":313,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.842568+0000","last_change":"2024-09-16T01:53:19.543649+0000","last_active":"2024-09-16T01:53:36.842568+0000","last_peered":"2024-09-16T01:53:36.842568+0000","last_clean":"2024-09-16T01:53:36.842568+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:53:36.842568+0000","last_undegraded":"2024-09-16T01:53:36.842568+0000","last_fullsized":"2024-09-16T01:53:36.842568+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:12.126459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"127'176","reported_seq":541,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546195+0000","last_change":"2024-09-16T01:53:23.518054+0000","last_active":"2024-09-16T01:53:36.546195+0000","last_peered":"2024-09-16T01:53:36.546195+0000","last_clean":"2024-09-16T01:53:36.546195+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:36.546195+0000","last_undegraded":"2024-09-16T01:53:36.546195+0000","last_fullsized":"2024-09-16T01:53:36.546195+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:50.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":48,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551978+0000","last_change":"2024-09-16T01:53:19.575999+0000","last_active":"2024-09-16T01:53:36.551978+0000","last_peered":"2024-09-16T01:53:36.551978+0000","last_clean":"2024-09-16T01:53:36.551978+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:36.551978+0000","last_undegraded":"2024-09-16T01:53:36.551978+0000","last_fullsized":"2024-09-16T01:53:36.551978+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:59.687176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"152'162","reported_seq":587,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.319456+0000","last_change":"2024-09-16T01:53:23.514324+0000","last_active":"2024-09-16T01:53:41.319456+0000","last_peered":"2024-09-16T01:53:41.319456+0000","last_clean":"2024-09-16T01:53:41.319456+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:53:41.319456+0000","last_undegraded":"2024-09-16T01:53:41.319456+0000","last_fullsized":"2024-09-16T01:53:41.319456+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:27.575957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":58,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.546415+0000","last_change":"2024-09-16T01:53:20.576951+0000","last_active":"2024-09-16T01:53:36.546415+0000","last_peered":"2024-09-16T01:53:36.546415+0000","last_clean":"2024-09-16T01:53:36.546415+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:36.546415+0000","last_undegraded":"2024-09-16T01:53:36.546415+0000","last_fullsized":"2024-09-16T01:53:36.546415+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:20.021875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"127'164","reported_seq":494,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545261+0000","last_change":"2024-09-16T01:53:23.517332+0000","last_active":"2024-09-16T01:53:36.545261+0000","last_peered":"2024-09-16T01:53:36.545261+0000","last_clean":"2024-09-16T01:53:36.545261+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:36.545261+0000","last_undegraded":"2024-09-16T01:53:36.545261+0000","last_fullsized":"2024-09-16T01:53:36.545261+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:15.929317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":248,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545542+0000","last_change":"2024-09-16T01:53:34.538675+0000","last_active":"2024-09-16T01:53:36.545542+0000","last_peered":"2024-09-16T01:53:36.545542+0000","last_clean":"2024-09-16T01:53:36.545542+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:36.545542+0000","last_undegraded":"2024-09-16T01:53:36.545542+0000","last_fullsized":"2024-09-16T01:53:36.545542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"127'156","reported_seq":624,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.554614+0000","last_change":"2024-09-16T01:53:36.554614+0000","last_active":"2024-09-16T01:53:36.554614+0000","last_peered":"2024-09-16T01:53:36.554614+0000","last_clean":"2024-09-16T01:53:36.554614+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:36.554614+0000","last_undegraded":"2024-09-16T01:53:36.554614+0000","last_fullsized":"2024-09-16T01:53:36.554614+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:23.435521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.841495+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:36.841495+0000","last_peered":"2024-09-16T01:53:36.841495+0000","last_clean":"2024-09-16T01:53:36.841495+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:36.841495+0000","last_undegraded":"2024-09-16T01:53:36.841495+0000","last_fullsized":"2024-09-16T01:53:36.841495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"152'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.343195+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:41.343195+0000","last_peered":"2024-09-16T01:53:41.343195+0000","last_clean":"2024-09-16T01:53:41.343195+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:41.343195+0000","last_undegraded":"2024-09-16T01:53:41.343195+0000","last_fullsized":"2024-09-16T01:53:41.343195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"152'205","reported_seq":717,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.361493+0000","last_change":"2024-09-16T01:53:22.527742+0000","last_active":"2024-09-16T01:53:41.361493+0000","last_peered":"2024-09-16T01:53:41.361493+0000","last_clean":"2024-09-16T01:53:41.361493+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:53:41.361493+0000","last_undegraded":"2024-09-16T01:53:41.361493+0000","last_fullsized":"2024-09-16T01:53:41.361493+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'5","reported_seq":52,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.553134+0000","last_change":"2024-09-16T01:53:19.535031+0000","last_active":"2024-09-16T01:53:36.553134+0000","last_peered":"2024-09-16T01:53:36.553134+0000","last_clean":"2024-09-16T01:53:36.553134+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:36.553134+0000","last_undegraded":"2024-09-16T01:53:36.553134+0000","last_fullsized":"2024-09-16T01:53:36.553134+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:36.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"152'162","reported_seq":627,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:41.353319+0000","last_change":"2024-09-16T01:53:22.521982+0000","last_active":"2024-09-16T01:53:41.353319+0000","last_peered":"2024-09-16T01:53:41.353319+0000","last_clean":"2024-09-16T01:53:41.353319+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:41.353319+0000","last_undegraded":"2024-09-16T01:53:41.353319+0000","last_fullsized":"2024-09-16T01:53:41.353319+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:50:35.196217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":60,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551329+0000","last_change":"2024-09-16T01:53:19.559597+0000","last_active":"2024-09-16T01:53:36.551329+0000","last_peered":"2024-09-16T01:53:36.551329+0000","last_clean":"2024-09-16T01:53:36.551329+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:36.551329+0000","last_undegraded":"2024-09-16T01:53:36.551329+0000","last_fullsized":"2024-09-16T01:53:36.551329+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:16.037628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"127'154","reported_seq":427,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552068+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:36.552068+0000","last_peered":"2024-09-16T01:53:36.552068+0000","last_clean":"2024-09-16T01:53:36.552068+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:36.552068+0000","last_undegraded":"2024-09-16T01:53:36.552068+0000","last_fullsized":"2024-09-16T01:53:36.552068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":307,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.552067+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:36.552067+0000","last_peered":"2024-09-16T01:53:36.552067+0000","last_clean":"2024-09-16T01:53:36.552067+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:36.552067+0000","last_undegraded":"2024-09-16T01:53:36.552067+0000","last_fullsized":"2024-09-16T01:53:36.552067+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"127'176","reported_seq":449,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.551955+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:36.551955+0000","last_peered":"2024-09-16T01:53:36.551955+0000","last_clean":"2024-09-16T01:53:36.551955+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:36.551955+0000","last_undegraded":"2024-09-16T01:53:36.551955+0000","last_fullsized":"2024-09-16T01:53:36.551955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":56,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T01:53:36.545971+0000","last_change":"2024-09-16T01:53:20.557667+0000","last_active":"2024-09-16T01:53:36.545971+0000","last_peered":"2024-09-16T01:53:36.545971+0000","last_clean":"2024-09-16T01:53:36.545971+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:36.545971+0000","last_undegraded":"2024-09-16T01:53:36.545971+0000","last_fullsized":"2024-09-16T01:53:36.545971+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:05:38.159129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4442,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11010,"ondisk_log_size":11010,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398903,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":423,"num_write_kb":602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797557,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509504,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1855488,"data_stored":874658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509504,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37528,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334312,"statfs":{"total":96636764160,"available":96598335488,"internally_reserved":0,"allocated":2236416,"data_stored":1263545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1904640,"data_stored":1463571,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509504,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32996,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338844,"statfs":{"total":96636764160,"available":96602976256,"internally_reserved":0,"allocated":1855488,"data_stored":874731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542207,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":873348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542207,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1839104,"data_stored":874169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1912832,"data_stored":1463488,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41420,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330420,"statfs":{"total":96636764160,"available":96594350080,"internally_reserved":0,"allocated":1949696,"data_stored":1264954,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393165,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:43.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:53:43.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.259+0000 7f2300e00640 1 Processor -- start 2024-09-16T01:53:43.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.259+0000 7f2300e00640 1 -- start start 2024-09-16T01:53:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.260+0000 7f2300e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc15b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.260+0000 7f2300e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22fc0cc040 0x7f22fc0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.260+0000 7f2300e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22fc06d140 con 0x7f22fc0c6b70 2024-09-16T01:53:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.260+0000 7f2300e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f22fc15b880 con 0x7f22fc0cc040 2024-09-16T01:53:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.260+0000 7f2300e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f22fc0cab20 con 0x7f22fc0c75a0 2024-09-16T01:53:43.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:43.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22fbe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22fc0cc040 0x7f22fc0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:43.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc15b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:46194/0 (socket says 172.21.15.146:46194) 2024-09-16T01:53:43.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22fbe00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22fc0cc040 0x7f22fc0cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55012/0 (socket says 172.21.15.146:55012) 2024-09-16T01:53:43.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 -- 172.21.15.146:0/3464122296 learned_addr learned my addr 172.21.15.146:0/3464122296 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:43.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 -- 172.21.15.146:0/3464122296 >> v2:172.21.15.161:3300/0 conn(0x7f22fc0cc040 msgr2=0x7f22fc0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:43.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 --2- 172.21.15.146:0/3464122296 >> v2:172.21.15.161:3300/0 conn(0x7f22fc0cc040 0x7f22fc0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 -- 172.21.15.146:0/3464122296 >> v1:172.21.15.161:6789/0 conn(0x7f22fc0c6b70 legacy=0x7f22fc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22faa00640 1 -- 172.21.15.146:0/3464122296 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 250798201 0 0) 0x7f22fc06d140 con 0x7f22fc0c6b70 2024-09-16T01:53:43.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.261+0000 7f22f2a00640 1 -- 172.21.15.146:0/3464122296 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc06d420 con 0x7f22fc0c75a0 2024-09-16T01:53:43.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.262+0000 7f22f2a00640 1 --2- 172.21.15.146:0/3464122296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc15b340 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009870 tx=0x7f22e402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8d52e5064575c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:43.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.262+0000 7f22faa00640 1 -- 172.21.15.146:0/3464122296 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4037070 con 0x7f22fc0c75a0 2024-09-16T01:53:43.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.262+0000 7f22faa00640 1 -- 172.21.15.146:0/3464122296 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e4032440 con 0x7f22fc0c75a0 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.262+0000 7f22faa00640 1 -- 172.21.15.146:0/3464122296 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e4032810 con 0x7f22fc0c75a0 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 -- 172.21.15.146:0/3464122296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 msgr2=0x7f22fc15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 --2- 172.21.15.146:0/3464122296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc15b340 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009870 tx=0x7f22e402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 -- 172.21.15.146:0/3464122296 shutdown_connections 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 --2- 172.21.15.146:0/3464122296 >> v2:172.21.15.161:3300/0 conn(0x7f22fc0cc040 0x7f22fc0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 --2- 172.21.15.146:0/3464122296 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc15b340 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 -- 172.21.15.146:0/3464122296 >> 172.21.15.146:0/3464122296 conn(0x7f22fc0ccc20 msgr2=0x7f22fc0cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 -- 172.21.15.146:0/3464122296 shutdown_connections 2024-09-16T01:53:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.263+0000 7f2300e00640 1 -- 172.21.15.146:0/3464122296 wait complete. 2024-09-16T01:53:43.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.265+0000 7f2300e00640 1 Processor -- start 2024-09-16T01:53:43.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.265+0000 7f2300e00640 1 -- start start 2024-09-16T01:53:43.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f2300e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22fc0c6b70 0x7f22fc1f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f2300e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc1f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f2300e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 0x7f22fc151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f2300e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc0cb050 con 0x7f22fc0cc040 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f2300e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc0cad70 con 0x7f22fc0c6b70 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f2300e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f22fc0cbc80 con 0x7f22fc0c75a0 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 0x7f22fc151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 0x7f22fc151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43044/0 (socket says 172.21.15.146:43044) 2024-09-16T01:53:43.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 -- 172.21.15.146:0/125512543 learned_addr learned my addr 172.21.15.146:0/125512543 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fb400640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22fc0c6b70 0x7f22fc1f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22f2a00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc1f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 -- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 msgr2=0x7f22fc1f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc1f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 -- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22fc0c6b70 msgr2=0x7f22fc1f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22fc0c6b70 0x7f22fc1f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.266+0000 7f22fbe00640 1 -- 172.21.15.146:0/125512543 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc1f48d0 con 0x7f22fc0cc040 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.267+0000 7f22fbe00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 0x7f22fc151450 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22f400e880 tx=0x7f22f400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.267+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4017070 con 0x7f22fc0cc040 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.267+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f400c940 con 0x7f22fc0cc040 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.267+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f400cd30 con 0x7f22fc0cc040 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.267+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22fc15ba90 con 0x7f22fc0cc040 2024-09-16T01:53:43.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.267+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc15ccb0 con 0x7f22fc0cc040 2024-09-16T01:53:43.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.268+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22fc06cd20 con 0x7f22fc0cc040 2024-09-16T01:53:43.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.269+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f22f401a070 con 0x7f22fc0cc040 2024-09-16T01:53:43.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.270+0000 7f22f3e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f22d806c2e0 0x7f22d806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:43.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.270+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 5911+0+0 (secure 0 0 0) 0x7f22f408b4f0 con 0x7f22fc0cc040 2024-09-16T01:53:43.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.270+0000 7f22fb400640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f22d806c2e0 0x7f22d806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:43.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.271+0000 7f22fb400640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f22d806c2e0 0x7f22d806e7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0012e0 tx=0x7f22ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:43.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.274+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f22f40594a0 con 0x7f22fc0cc040 2024-09-16T01:53:43.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.451+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22fc0ca280 con 0x7f22fc0cc040 2024-09-16T01:53:43.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.453+0000 7f22f3e00640 1 -- 172.21.15.146:0/125512543 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) ==== 74+0+17012 (secure 0 0 0) 0x7f22f405d7a0 con 0x7f22fc0cc040 2024-09-16T01:53:43.455 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:43.455 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":154,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:53:42.644828+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:53:19.491406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":2,"to":5}]},{"pgid":"3.20","mappings":[{"from":2,"to":7}]},{"pgid":"3.2d","mappings":[{"from":3,"to":7}]},{"pgid":"3.31","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.31","osds":[5]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:53:43.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f22d806c2e0 msgr2=0x7f22d806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f22d806c2e0 0x7f22d806e7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0012e0 tx=0x7f22ec009040 comp rx=0 tx=0).stop 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 msgr2=0x7f22fc151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 0x7f22fc151450 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22f400e880 tx=0x7f22f400ed40 comp rx=0 tx=0).stop 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 shutdown_connections 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f22d806c2e0 0x7f22d806e7a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22fc0cc040 0x7f22fc151450 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22fc0c75a0 0x7f22fc1f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 --2- 172.21.15.146:0/125512543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22fc0c6b70 0x7f22fc1f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 >> 172.21.15.146:0/125512543 conn(0x7f22fc0ccc20 msgr2=0x7f22fc0c4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 shutdown_connections 2024-09-16T01:53:43.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:43.455+0000 7f2300e00640 1 -- 172.21.15.146:0/125512543 wait complete. 2024-09-16T01:53:44.475 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T01:53:44.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:44.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.636+0000 7f2d66000640 1 Processor -- start 2024-09-16T01:53:44.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.636+0000 7f2d66000640 1 -- start start 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d66000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d600c73c0 0x7f2d6006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d66000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d600c81e0 con 0x7f2d60154fa0 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d66000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d600c7890 con 0x7f2d6015e7f0 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d66000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2d600c84f0 con 0x7f2d600c73c0 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d5f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2d60154fa0 0x7f2d6015e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44244/0 (socket says 172.21.15.146:44244) 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d5f400640 1 -- 172.21.15.146:0/3443404542 learned_addr learned my addr 172.21.15.146:0/3443404542 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:44.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d56a00640 1 --2- 172.21.15.146:0/3443404542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d600c73c0 0x7f2d6006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d5ea00640 1 -- 172.21.15.146:0/3443404542 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2014165041 0 0) 0x7f2d600c7890 con 0x7f2d6015e7f0 2024-09-16T01:53:44.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d5ea00640 1 -- 172.21.15.146:0/3443404542 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d44003520 con 0x7f2d6015e7f0 2024-09-16T01:53:44.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d5ea00640 1 -- 172.21.15.146:0/3443404542 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238630096 0 0) 0x7f2d600c81e0 con 0x7f2d60154fa0 2024-09-16T01:53:44.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d56a00640 1 -- 172.21.15.146:0/3443404542 >> v1:172.21.15.161:6789/0 conn(0x7f2d6015e7f0 legacy=0x7f2d6006ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d56a00640 1 -- 172.21.15.146:0/3443404542 >> v1:172.21.15.146:6789/0 conn(0x7f2d60154fa0 legacy=0x7f2d6015e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.638+0000 7f2d56a00640 1 -- 172.21.15.146:0/3443404542 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d600c87d0 con 0x7f2d600c73c0 2024-09-16T01:53:44.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.639+0000 7f2d56a00640 1 --2- 172.21.15.146:0/3443404542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d600c73c0 0x7f2d6006e6e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2d480097d0 tx=0x7f2d4802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=217b5f374f63508a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d5ea00640 1 -- 172.21.15.146:0/3443404542 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48037070 con 0x7f2d600c73c0 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d5ea00640 1 -- 172.21.15.146:0/3443404542 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d48032440 con 0x7f2d600c73c0 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d5ea00640 1 -- 172.21.15.146:0/3443404542 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d480327f0 con 0x7f2d600c73c0 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 -- 172.21.15.146:0/3443404542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d600c73c0 msgr2=0x7f2d6006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 --2- 172.21.15.146:0/3443404542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d600c73c0 0x7f2d6006e6e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2d480097d0 tx=0x7f2d4802a290 comp rx=0 tx=0).stop 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 -- 172.21.15.146:0/3443404542 shutdown_connections 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 --2- 172.21.15.146:0/3443404542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d600c73c0 0x7f2d6006e6e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 -- 172.21.15.146:0/3443404542 >> 172.21.15.146:0/3443404542 conn(0x7f2d600cc0b0 msgr2=0x7f2d600cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 -- 172.21.15.146:0/3443404542 shutdown_connections 2024-09-16T01:53:44.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.640+0000 7f2d66000640 1 -- 172.21.15.146:0/3443404542 wait complete. 2024-09-16T01:53:44.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.642+0000 7f2d66000640 1 Processor -- start 2024-09-16T01:53:44.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.642+0000 7f2d66000640 1 -- start start 2024-09-16T01:53:44.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d66000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d600c73c0 0x7f2d6015a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d66000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d60154fa0 0x7f2d60157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d66000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d6015e7f0 0x7f2d601f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d66000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6006f330 con 0x7f2d600c73c0 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d66000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6006fb40 con 0x7f2d60154fa0 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d66000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6006f4b0 con 0x7f2d6015e7f0 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d5fe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d6015e7f0 0x7f2d601f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d60154fa0 0x7f2d60157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d5f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d600c73c0 0x7f2d6015a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d60154fa0 0x7f2d60157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55018/0 (socket says 172.21.15.146:55018) 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 -- 172.21.15.146:0/1872861072 learned_addr learned my addr 172.21.15.146:0/1872861072 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 -- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d6015e7f0 msgr2=0x7f2d601f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d6015e7f0 0x7f2d601f0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 -- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d600c73c0 msgr2=0x7f2d6015a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d600c73c0 0x7f2d6015a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d56a00640 1 -- 172.21.15.146:0/1872861072 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d600c16f0 con 0x7f2d60154fa0 2024-09-16T01:53:44.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.643+0000 7f2d5fe00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2d6015e7f0 0x7f2d601f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.644+0000 7f2d56a00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d60154fa0 0x7f2d60157870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2d480097d0 tx=0x7f2d480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.644+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48037040 con 0x7f2d60154fa0 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.644+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d48004230 con 0x7f2d60154fa0 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.644+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003610 con 0x7f2d60154fa0 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.644+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d600c21b0 con 0x7f2d60154fa0 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.644+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6015b1b0 con 0x7f2d60154fa0 2024-09-16T01:53:44.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.645+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d60071d30 con 0x7f2d60154fa0 2024-09-16T01:53:44.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.646+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2d480037b0 con 0x7f2d60154fa0 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.647+0000 7f2d57e00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2d3c06c2b0 0x7f2d3c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.647+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5886+0+0 (secure 0 0 0) 0x7f2d480ac420 con 0x7f2d60154fa0 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.647+0000 7f2d57e00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f2d3c074710 0x7f2d3c076bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.647+0000 7f2d5f400640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2d3c06c2b0 0x7f2d3c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.647+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d48007e80 con 0x7f2d3c074710 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.647+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f2d480ac7d0 con 0x7f2d60154fa0 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.648+0000 7f2d5fe00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f2d3c074710 0x7f2d3c076bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.648+0000 7f2d5f400640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2d3c06c2b0 0x7f2d3c06e770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2d50000f60 tx=0x7f2d50007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.648+0000 7f2d5fe00640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f2d3c074710 0x7f2d3c076bb0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.649+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== osd.0 v2:172.21.15.146:6824/61440064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d48007e80 con 0x7f2d3c074710 2024-09-16T01:53:44.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.662+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 --> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2d60152eb0 con 0x7f2d3c074710 2024-09-16T01:53:44.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d57e00640 1 -- 172.21.15.146:0/1872861072 <== osd.0 v2:172.21.15.146:6824/61440064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2d60152eb0 con 0x7f2d3c074710 2024-09-16T01:53:44.665 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:44.665 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:44.665 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:44.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f2d3c074710 msgr2=0x7f2d3c076bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.146:6824/61440064,v1:172.21.15.146:6825/61440064] conn(0x7f2d3c074710 0x7f2d3c076bb0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2d3c06c2b0 msgr2=0x7f2d3c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f2d3c06c2b0 0x7f2d3c06e770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2d50000f60 tx=0x7f2d50007d50 comp rx=0 tx=0).stop 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d60154fa0 msgr2=0x7f2d60157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 --2- 172.21.15.146:0/1872861072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d60154fa0 0x7f2d60157870 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2d480097d0 tx=0x7f2d480046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d5fe00640 1 -- 172.21.15.146:0/1872861072 reap_dead start 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 shutdown_connections 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 >> 172.21.15.146:0/1872861072 conn(0x7f2d600cc0b0 msgr2=0x7f2d6006e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 shutdown_connections 2024-09-16T01:53:44.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.663+0000 7f2d66000640 1 -- 172.21.15.146:0/1872861072 wait complete. 2024-09-16T01:53:44.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T01:53:44.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.832+0000 7f39c0e00640 1 Processor -- start 2024-09-16T01:53:44.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.832+0000 7f39c0e00640 1 -- start start 2024-09-16T01:53:44.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39c0e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc0c6bb0 0x7f39bc0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39c0e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39bc072f50 con 0x7f39bc152850 2024-09-16T01:53:44.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39c0e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39bc15cb50 con 0x7f39bc0c58c0 2024-09-16T01:53:44.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39c0e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc073210 con 0x7f39bc0c6bb0 2024-09-16T01:53:44.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b9e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f39bc152850 0x7f39bc158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44254/0 (socket says 172.21.15.146:44254) 2024-09-16T01:53:44.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b9e00640 1 -- 172.21.15.146:0/683325990 learned_addr learned my addr 172.21.15.146:0/683325990 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b9400640 1 --2- 172.21.15.146:0/683325990 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc0c6bb0 0x7f39bc0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b8a00640 1 -- 172.21.15.146:0/683325990 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3725334774 0 0) 0x7f39bc072f50 con 0x7f39bc152850 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b8a00640 1 -- 172.21.15.146:0/683325990 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39a0003520 con 0x7f39bc152850 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b8a00640 1 -- 172.21.15.146:0/683325990 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000418126 0 0) 0x7f39bc15cb50 con 0x7f39bc0c58c0 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b9400640 1 -- 172.21.15.146:0/683325990 >> v1:172.21.15.161:6789/0 conn(0x7f39bc0c58c0 legacy=0x7f39bc162ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b9400640 1 -- 172.21.15.146:0/683325990 >> v1:172.21.15.146:6789/0 conn(0x7f39bc152850 legacy=0x7f39bc158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.834+0000 7f39b9400640 1 -- 172.21.15.146:0/683325990 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc15bd90 con 0x7f39bc0c6bb0 2024-09-16T01:53:44.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.835+0000 7f39b9400640 1 --2- 172.21.15.146:0/683325990 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc0c6bb0 0x7f39bc0c9700 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f39a80097d0 tx=0x7f39a802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10cc6993751d5205 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39b8a00640 1 -- 172.21.15.146:0/683325990 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8037070 con 0x7f39bc0c6bb0 2024-09-16T01:53:44.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39b8a00640 1 -- 172.21.15.146:0/683325990 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a8032440 con 0x7f39bc0c6bb0 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39b8a00640 1 -- 172.21.15.146:0/683325990 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a80327f0 con 0x7f39bc0c6bb0 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 -- 172.21.15.146:0/683325990 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc0c6bb0 msgr2=0x7f39bc0c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 --2- 172.21.15.146:0/683325990 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc0c6bb0 0x7f39bc0c9700 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f39a80097d0 tx=0x7f39a802a210 comp rx=0 tx=0).stop 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 -- 172.21.15.146:0/683325990 shutdown_connections 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 --2- 172.21.15.146:0/683325990 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc0c6bb0 0x7f39bc0c9700 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 -- 172.21.15.146:0/683325990 >> 172.21.15.146:0/683325990 conn(0x7f39bc06f3c0 msgr2=0x7f39bc06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 -- 172.21.15.146:0/683325990 shutdown_connections 2024-09-16T01:53:44.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.836+0000 7f39c0e00640 1 -- 172.21.15.146:0/683325990 wait complete. 2024-09-16T01:53:44.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.838+0000 7f39c0e00640 1 Processor -- start 2024-09-16T01:53:44.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.838+0000 7f39c0e00640 1 -- start start 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39c0e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 0x7f39bc0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39c0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 0x7f39bc148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39c0e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc152850 0x7f39bc15ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39c0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc15c630 con 0x7f39bc0c6bb0 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39c0e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc15c350 con 0x7f39bc0c58c0 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39c0e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc070200 con 0x7f39bc152850 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39b9e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 0x7f39bc0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39b9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 0x7f39bc148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39b9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 0x7f39bc148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43068/0 (socket says 172.21.15.146:43068) 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39b9e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 0x7f39bc0ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55032/0 (socket says 172.21.15.146:55032) 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39b9400640 1 -- 172.21.15.146:0/1178867163 learned_addr learned my addr 172.21.15.146:0/1178867163 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:44.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.839+0000 7f39ba800640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc152850 0x7f39bc15ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9e00640 1 -- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc152850 msgr2=0x7f39bc15ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc152850 0x7f39bc15ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9e00640 1 -- 172.21.15.146:0/1178867163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 msgr2=0x7f39bc148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 0x7f39bc148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9e00640 1 -- 172.21.15.146:0/1178867163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc148fa0 con 0x7f39bc0c58c0 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9400640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 0x7f39bc148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:44.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b9e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 0x7f39bc0ccb40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c3c0 tx=0x7f39ac001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac017070 con 0x7f39bc0c58c0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39ac005ce0 con 0x7f39bc0c58c0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac007310 con 0x7f39bc0c58c0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39bc0c3750 con 0x7f39bc0c58c0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.840+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39bc0c3d10 con 0x7f39bc0c58c0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.841+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39bc0cd080 con 0x7f39bc0c58c0 2024-09-16T01:53:44.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.842+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f39ac01a070 con 0x7f39bc0c58c0 2024-09-16T01:53:44.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.843+0000 7f39b2000640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f399c06c2b0 0x7f399c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:44.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.843+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5886+0+0 (secure 0 0 0) 0x7f39ac08a810 con 0x7f39bc0c58c0 2024-09-16T01:53:44.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.843+0000 7f39b9400640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f399c06c2b0 0x7f399c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:44.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.844+0000 7f39b9400640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f399c06c2b0 0x7f399c06e770 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009720 tx=0x7f39a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:44.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:44.848+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f39ac0587e0 con 0x7f39bc0c58c0 2024-09-16T01:53:45.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.023+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f39bc148670 con 0x7f399c06c2b0 2024-09-16T01:53:45.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.023+0000 7f39b2000640 1 -- 172.21.15.146:0/1178867163 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f39a8003710 con 0x7f399c06c2b0 2024-09-16T01:53:45.025 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T01:53:45.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f399c06c2b0 msgr2=0x7f399c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f399c06c2b0 0x7f399c06e770 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009720 tx=0x7f39a802e000 comp rx=0 tx=0).stop 2024-09-16T01:53:45.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 msgr2=0x7f39bc0ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 0x7f39bc0ccb40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00c3c0 tx=0x7f39ac001e70 comp rx=0 tx=0).stop 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 shutdown_connections 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f399c06c2b0 0x7f399c06e770 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f39bc152850 0x7f39bc15ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39bc0c6bb0 0x7f39bc148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 --2- 172.21.15.146:0/1178867163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39bc0c58c0 0x7f39bc0ccb40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 >> 172.21.15.146:0/1178867163 conn(0x7f39bc06f3c0 msgr2=0x7f39bc14e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 shutdown_connections 2024-09-16T01:53:45.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.026+0000 7f39c0e00640 1 -- 172.21.15.146:0/1178867163 wait complete. 2024-09-16T01:53:45.044 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T01:53:45.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:45.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.203+0000 7f4b02600640 1 Processor -- start 2024-09-16T01:53:45.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.203+0000 7f4b02600640 1 -- start start 2024-09-16T01:53:45.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b02600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4afc154ab0 0x7f4afc15a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b02600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0cd900 0x7f4afc15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b02600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4afc0c5000 0x7f4afc0cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b02600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f4afc0b55d0 con 0x7f4afc154ab0 2024-09-16T01:53:45.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b02600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4afc0c5540 con 0x7f4afc0c5000 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b02600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4afc154e90 con 0x7f4afc0cd900 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b01200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4afc154ab0 0x7f4afc15a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b01200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4afc154ab0 0x7f4afc15a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43082/0 (socket says 172.21.15.146:43082) 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b01200640 1 -- 172.21.15.146:0/983983430 learned_addr learned my addr 172.21.15.146:0/983983430 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4afbe00640 1 --2- 172.21.15.146:0/983983430 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0cd900 0x7f4afc15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.204+0000 7f4b01c00640 1 --2- 172.21.15.146:0/983983430 >> v2:172.21.15.161:3300/0 conn(0x7f4afc0c5000 0x7f4afc0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.205+0000 7f4b01200640 1 -- 172.21.15.146:0/983983430 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0cd900 msgr2=0x7f4afc15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.205+0000 7f4b01200640 1 --2- 172.21.15.146:0/983983430 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0cd900 0x7f4afc15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.205+0000 7f4b01200640 1 -- 172.21.15.146:0/983983430 >> v2:172.21.15.161:3300/0 conn(0x7f4afc0c5000 msgr2=0x7f4afc0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:45.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.205+0000 7f4b01200640 1 --2- 172.21.15.146:0/983983430 >> v2:172.21.15.161:3300/0 conn(0x7f4afc0c5000 0x7f4afc0cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.205+0000 7f4b01200640 1 -- 172.21.15.146:0/983983430 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc15ddd0 con 0x7f4afc154ab0 2024-09-16T01:53:45.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.206+0000 7f4b01200640 1 --2- 172.21.15.146:0/983983430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc15a8c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0b64c0 tx=0x7f4af002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63b2cd4f50ebcaf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.206+0000 7f4afb400640 1 -- 172.21.15.146:0/983983430 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0037070 con 0x7f4afc154ab0 2024-09-16T01:53:45.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.206+0000 7f4afb400640 1 -- 172.21.15.146:0/983983430 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af0032440 con 0x7f4afc154ab0 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.206+0000 7f4afb400640 1 -- 172.21.15.146:0/983983430 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0032810 con 0x7f4afc154ab0 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 -- 172.21.15.146:0/983983430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 msgr2=0x7f4afc15a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 --2- 172.21.15.146:0/983983430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc15a8c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0b64c0 tx=0x7f4af002a270 comp rx=0 tx=0).stop 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 -- 172.21.15.146:0/983983430 shutdown_connections 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 --2- 172.21.15.146:0/983983430 >> v2:172.21.15.161:3300/0 conn(0x7f4afc0c5000 0x7f4afc0cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 --2- 172.21.15.146:0/983983430 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0cd900 0x7f4afc15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 --2- 172.21.15.146:0/983983430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc15a8c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 -- 172.21.15.146:0/983983430 >> 172.21.15.146:0/983983430 conn(0x7f4afc067950 msgr2=0x7f4afc067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 -- 172.21.15.146:0/983983430 shutdown_connections 2024-09-16T01:53:45.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.207+0000 7f4b02600640 1 -- 172.21.15.146:0/983983430 wait complete. 2024-09-16T01:53:45.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.208+0000 7f4b02600640 1 Processor -- start 2024-09-16T01:53:45.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.208+0000 7f4b02600640 1 -- start start 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.209+0000 7f4b02600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0c5000 0x7f4afc1f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.209+0000 7f4b02600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4afc0cd900 0x7f4afc1f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.209+0000 7f4b02600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.209+0000 7f4b02600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc15e670 con 0x7f4afc154ab0 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.209+0000 7f4b02600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc15e390 con 0x7f4afc0cd900 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.209+0000 7f4b02600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4afc15e950 con 0x7f4afc0c5000 2024-09-16T01:53:45.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43088/0 (socket says 172.21.15.146:43088) 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 -- 172.21.15.146:0/3617715012 learned_addr learned my addr 172.21.15.146:0/3617715012 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01200640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0c5000 0x7f4afc1f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4afbe00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4afc0cd900 0x7f4afc1f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 -- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0c5000 msgr2=0x7f4afc1f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4afc0c5000 0x7f4afc1f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 -- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4afc0cd900 msgr2=0x7f4afc1f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4afc0cd900 0x7f4afc1f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 -- 172.21.15.146:0/3617715012 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc1f0410 con 0x7f4afc154ab0 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.210+0000 7f4b01c00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc0ce990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4af400ef00 tx=0x7f4af400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.211+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4018020 con 0x7f4afc154ab0 2024-09-16T01:53:45.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.211+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af400bd30 con 0x7f4afc154ab0 2024-09-16T01:53:45.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.211+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4014640 con 0x7f4afc154ab0 2024-09-16T01:53:45.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.211+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc1e7480 con 0x7f4afc154ab0 2024-09-16T01:53:45.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.211+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4afc1f0680 con 0x7f4afc154ab0 2024-09-16T01:53:45.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.212+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4afc1f0a80 con 0x7f4afc154ab0 2024-09-16T01:53:45.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.213+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4af401b020 con 0x7f4afc154ab0 2024-09-16T01:53:45.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.213+0000 7f4af8c00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4adc06c2b0 0x7f4adc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.214+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 5886+0+0 (secure 0 0 0) 0x7f4af408bd00 con 0x7f4afc154ab0 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.214+0000 7f4af8c00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4adc074710 0x7f4adc076bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.214+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4af401ae70 con 0x7f4adc074710 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.214+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=156) ==== 24+0+0 (secure 0 0 0) 0x7f4af408e320 con 0x7f4afc154ab0 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.214+0000 7f4b01200640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4adc06c2b0 0x7f4adc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.214+0000 7f4afbe00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4adc074710 0x7f4adc076bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.215+0000 7f4afbe00640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4adc074710 0x7f4adc076bb0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.215+0000 7f4b01200640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4adc06c2b0 0x7f4adc06e770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4af0034000 tx=0x7f4af002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.215+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== osd.1 v2:172.21.15.146:6808/2375201615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4af401ae70 con 0x7f4adc074710 2024-09-16T01:53:45.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.228+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 --> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4afc0c63a0 con 0x7f4adc074710 2024-09-16T01:53:45.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4af8c00640 1 -- 172.21.15.146:0/3617715012 <== osd.1 v2:172.21.15.146:6808/2375201615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4afc0c63a0 con 0x7f4adc074710 2024-09-16T01:53:45.231 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:45.231 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:45.231 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:45.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4adc074710 msgr2=0x7f4adc076bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:6808/2375201615,v1:172.21.15.146:6809/2375201615] conn(0x7f4adc074710 0x7f4adc076bb0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4adc06c2b0 msgr2=0x7f4adc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f4adc06c2b0 0x7f4adc06e770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4af0034000 tx=0x7f4af002e000 comp rx=0 tx=0).stop 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 msgr2=0x7f4afc0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 --2- 172.21.15.146:0/3617715012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4afc154ab0 0x7f4afc0ce990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4af400ef00 tx=0x7f4af400b480 comp rx=0 tx=0).stop 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b01c00640 1 -- 172.21.15.146:0/3617715012 reap_dead start 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 shutdown_connections 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 >> 172.21.15.146:0/3617715012 conn(0x7f4afc067950 msgr2=0x7f4afc0c5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 shutdown_connections 2024-09-16T01:53:45.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.229+0000 7f4b02600640 1 -- 172.21.15.146:0/3617715012 wait complete. 2024-09-16T01:53:45.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T01:53:45.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.398+0000 7fad77200640 1 Processor -- start 2024-09-16T01:53:45.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.398+0000 7fad77200640 1 -- start start 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad77200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad70164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad77200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad70072f50 con 0x7fad700c6bb0 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad77200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad7015cb50 con 0x7fad70152850 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad77200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fad70073210 con 0x7fad700c9700 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad6f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fad700c6bb0 0x7fad700c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44270/0 (socket says 172.21.15.146:44270) 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad6f400640 1 -- 172.21.15.146:0/1179004148 learned_addr learned my addr 172.21.15.146:0/1179004148 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:45.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad75200640 1 --2- 172.21.15.146:0/1179004148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad70164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad75200640 1 -- 172.21.15.146:0/1179004148 >> v1:172.21.15.161:6789/0 conn(0x7fad70152850 legacy=0x7fad70158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad75200640 1 -- 172.21.15.146:0/1179004148 >> v1:172.21.15.146:6789/0 conn(0x7fad700c6bb0 legacy=0x7fad700c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad75200640 1 -- 172.21.15.146:0/1179004148 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7015bd90 con 0x7fad700c9700 2024-09-16T01:53:45.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.400+0000 7fad6ea00640 1 -- 172.21.15.146:0/1179004148 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1929997800 0 0) 0x7fad70072f50 con 0x7fad700c6bb0 2024-09-16T01:53:45.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.401+0000 7fad75200640 1 --2- 172.21.15.146:0/1179004148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad70164000 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad70073390 tx=0x7fad6802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a4f3270c1f094d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.401+0000 7fad6ea00640 1 -- 172.21.15.146:0/1179004148 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68007740 con 0x7fad700c9700 2024-09-16T01:53:45.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad6ea00640 1 -- 172.21.15.146:0/1179004148 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad680078e0 con 0x7fad700c9700 2024-09-16T01:53:45.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad6ea00640 1 -- 172.21.15.146:0/1179004148 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68007cb0 con 0x7fad700c9700 2024-09-16T01:53:45.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 -- 172.21.15.146:0/1179004148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 msgr2=0x7fad70164000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 --2- 172.21.15.146:0/1179004148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad70164000 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad70073390 tx=0x7fad6802a6d0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 -- 172.21.15.146:0/1179004148 shutdown_connections 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 --2- 172.21.15.146:0/1179004148 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad70164000 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 -- 172.21.15.146:0/1179004148 >> 172.21.15.146:0/1179004148 conn(0x7fad7006f3c0 msgr2=0x7fad7006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 -- 172.21.15.146:0/1179004148 shutdown_connections 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.402+0000 7fad77200640 1 -- 172.21.15.146:0/1179004148 wait complete. 2024-09-16T01:53:45.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.404+0000 7fad77200640 1 Processor -- start 2024-09-16T01:53:45.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.404+0000 7fad77200640 1 -- start start 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad77200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad700c6bb0 0x7fad700cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad77200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad701eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad77200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 0x7fad70146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad77200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fad7015c630 con 0x7fad70152850 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad77200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad7015c350 con 0x7fad700c6bb0 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad77200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fad70070200 con 0x7fad700c9700 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad75200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 0x7fad70146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad701eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad700c6bb0 0x7fad700cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad75200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 0x7fad70146630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43102/0 (socket says 172.21.15.146:43102) 2024-09-16T01:53:45.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad75200640 1 -- 172.21.15.146:0/1829666114 learned_addr learned my addr 172.21.15.146:0/1829666114 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6f400640 1 -- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad700c6bb0 msgr2=0x7fad700cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6f400640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad700c6bb0 0x7fad700cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6f400640 1 -- 172.21.15.146:0/1829666114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 msgr2=0x7fad70146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6f400640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 0x7fad70146630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6f400640 1 -- 172.21.15.146:0/1829666114 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad700cc780 con 0x7fad700c9700 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad75200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 0x7fad70146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.405+0000 7fad6fe00640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad700c6bb0 0x7fad700cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.406+0000 7fad6f400640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad701eecb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c410 tx=0x7fad5c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.406+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c0074e0 con 0x7fad700c9700 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.406+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5c0047e0 con 0x7fad700c9700 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.406+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c004bd0 con 0x7fad700c9700 2024-09-16T01:53:45.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.406+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad700cca60 con 0x7fad700c9700 2024-09-16T01:53:45.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.406+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fad700cb380 con 0x7fad700c9700 2024-09-16T01:53:45.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.407+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad701451a0 con 0x7fad700c9700 2024-09-16T01:53:45.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.408+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fad5c011070 con 0x7fad700c9700 2024-09-16T01:53:45.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.409+0000 7fad67e00640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fad5006c2e0 0x7fad5006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.409+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(156..156 src has 1..156) ==== 5886+0+0 (secure 0 0 0) 0x7fad5c08b070 con 0x7fad700c9700 2024-09-16T01:53:45.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.409+0000 7fad6fe00640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fad5006c2e0 0x7fad5006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.410+0000 7fad6fe00640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fad5006c2e0 0x7fad5006e7a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fad60000f60 tx=0x7fad60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.412+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fad701451a0 con 0x7fad700c9700 2024-09-16T01:53:45.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.589+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fad700718a0 con 0x7fad5006c2e0 2024-09-16T01:53:45.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.590+0000 7fad67e00640 1 -- 172.21.15.146:0/1829666114 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fad700718a0 con 0x7fad5006c2e0 2024-09-16T01:53:45.592 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fad5006c2e0 msgr2=0x7fad5006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fad5006c2e0 0x7fad5006e7a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fad60000f60 tx=0x7fad60009040 comp rx=0 tx=0).stop 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 msgr2=0x7fad701eecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad701eecb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c410 tx=0x7fad5c00c8d0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 shutdown_connections 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fad5006c2e0 0x7fad5006e7a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad70152850 0x7fad70146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fad700c9700 0x7fad701eecb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 --2- 172.21.15.146:0/1829666114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad700c6bb0 0x7fad700cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 >> 172.21.15.146:0/1829666114 conn(0x7fad7006f3c0 msgr2=0x7fad700c8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 shutdown_connections 2024-09-16T01:53:45.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.592+0000 7fad77200640 1 -- 172.21.15.146:0/1829666114 wait complete. 2024-09-16T01:53:45.612 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T01:53:45.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:45.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.768+0000 7f0365000640 1 Processor -- start 2024-09-16T01:53:45.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.768+0000 7f0365000640 1 -- start start 2024-09-16T01:53:45.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.769+0000 7f0365000640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f036006c710 0x7f03600c6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.769+0000 7f0365000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f036006fd80 0x7f0360156e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.769+0000 7f0365000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0360071a00 con 0x7f0360072d80 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.769+0000 7f0365000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f0360071ce0 con 0x7f036006fd80 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.769+0000 7f0365000640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0360071fc0 con 0x7f036006c710 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f036006fd80 0x7f0360156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f036006c710 0x7f03600c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f036006c710 0x7f03600c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55060/0 (socket says 172.21.15.146:55060) 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f036006fd80 0x7f0360156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43104/0 (socket says 172.21.15.146:43104) 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035fe00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0360072d80 0x7f036006caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57330/0 (socket says 172.21.15.146:57330) 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 -- 172.21.15.146:0/3051812094 learned_addr learned my addr 172.21.15.146:0/3051812094 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 -- 172.21.15.146:0/3051812094 >> v2:172.21.15.146:3300/0 conn(0x7f036006fd80 msgr2=0x7f0360156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 --2- 172.21.15.146:0/3051812094 >> v2:172.21.15.146:3300/0 conn(0x7f036006fd80 0x7f0360156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 -- 172.21.15.146:0/3051812094 >> v1:172.21.15.161:6789/0 conn(0x7f0360072d80 legacy=0x7f036006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.770+0000 7f035f400640 1 -- 172.21.15.146:0/3051812094 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03600722a0 con 0x7f036006c710 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.771+0000 7f035f400640 1 --2- 172.21.15.146:0/3051812094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f036006c710 0x7f03600c6820 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f03600ccfe0 tx=0x7f035002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb660dcb6491ecbd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.771+0000 7f035e000640 1 -- 172.21.15.146:0/3051812094 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350037070 con 0x7f036006c710 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.771+0000 7f035e000640 1 -- 172.21.15.146:0/3051812094 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0350032440 con 0x7f036006c710 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.771+0000 7f035e000640 1 -- 172.21.15.146:0/3051812094 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350032810 con 0x7f036006c710 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 -- 172.21.15.146:0/3051812094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f036006c710 msgr2=0x7f03600c6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 --2- 172.21.15.146:0/3051812094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f036006c710 0x7f03600c6820 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f03600ccfe0 tx=0x7f035002a270 comp rx=0 tx=0).stop 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 -- 172.21.15.146:0/3051812094 shutdown_connections 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 --2- 172.21.15.146:0/3051812094 >> v2:172.21.15.146:3300/0 conn(0x7f036006fd80 0x7f0360156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 --2- 172.21.15.146:0/3051812094 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f036006c710 0x7f03600c6820 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 -- 172.21.15.146:0/3051812094 >> 172.21.15.146:0/3051812094 conn(0x7f036006f160 msgr2=0x7f036006f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 -- 172.21.15.146:0/3051812094 shutdown_connections 2024-09-16T01:53:45.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.772+0000 7f0365000640 1 -- 172.21.15.146:0/3051812094 wait complete. 2024-09-16T01:53:45.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.774+0000 7f0365000640 1 Processor -- start 2024-09-16T01:53:45.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.774+0000 7f0365000640 1 -- start start 2024-09-16T01:53:45.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f0365000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f036006c710 0x7f03600abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f0365000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f036006fd80 0x7f03600aa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f0365000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0360072d80 0x7f0360165060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f0365000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0360073230 con 0x7f036006c710 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f0365000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0360072860 con 0x7f0360072d80 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f0365000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f036006d2e0 con 0x7f036006fd80 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f036006c710 0x7f03600abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f036006c710 0x7f03600abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43106/0 (socket says 172.21.15.146:43106) 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 -- 172.21.15.146:0/2644602917 learned_addr learned my addr 172.21.15.146:0/2644602917 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035ea00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f036006fd80 0x7f03600aa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035fe00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0360072d80 0x7f0360165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 -- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f036006fd80 msgr2=0x7f03600aa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f036006fd80 0x7f03600aa6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 -- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0360072d80 msgr2=0x7f0360165060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0360072d80 0x7f0360165060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 -- 172.21.15.146:0/2644602917 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03600c5910 con 0x7f036006c710 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035fe00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0360072d80 0x7f0360165060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.775+0000 7f035f400640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f036006c710 0x7f03600abc40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f035002a2b0 tx=0x7f03500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.776+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350037040 con 0x7f036006c710 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.776+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0350042070 con 0x7f036006c710 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.776+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03600ab0a0 con 0x7f036006c710 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.776+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350030880 con 0x7f036006c710 2024-09-16T01:53:45.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.776+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0360164860 con 0x7f036006c710 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.777+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f03601f13f0 con 0x7f036006c710 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.778+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0350030a20 con 0x7f036006c710 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f0357400640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f034006c2b0 0x7f034006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f03500abc40 con 0x7f036006c710 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f035ea00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f034006c2b0 0x7f034006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f0357400640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0340074710 0x7f0340076bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0340077280 con 0x7f0340074710 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7f03500ae340 con 0x7f036006c710 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.779+0000 7f035fe00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0340074710 0x7f0340076bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.780+0000 7f035ea00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f034006c2b0 0x7f034006e770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0360050280 tx=0x7f034c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.780+0000 7f035fe00640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0340074710 0x7f0340076bb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.781+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== osd.2 v2:172.21.15.146:6800/1818492127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0340077280 con 0x7f0340074710 2024-09-16T01:53:45.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 --> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f03600ccd30 con 0x7f0340074710 2024-09-16T01:53:45.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0357400640 1 -- 172.21.15.146:0/2644602917 <== osd.2 v2:172.21.15.146:6800/1818492127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f03600ccd30 con 0x7f0340074710 2024-09-16T01:53:45.796 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:45.796 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:45.796 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:45.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0340074710 msgr2=0x7f0340076bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:6800/1818492127,v1:172.21.15.146:6801/1818492127] conn(0x7f0340074710 0x7f0340076bb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f034006c2b0 msgr2=0x7f034006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f034006c2b0 0x7f034006e770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0360050280 tx=0x7f034c009190 comp rx=0 tx=0).stop 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f036006c710 msgr2=0x7f03600abc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 --2- 172.21.15.146:0/2644602917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f036006c710 0x7f03600abc40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f035002a2b0 tx=0x7f03500046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f035fe00640 1 -- 172.21.15.146:0/2644602917 reap_dead start 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 shutdown_connections 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.794+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 >> 172.21.15.146:0/2644602917 conn(0x7f036006f160 msgr2=0x7f03600caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.795+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 shutdown_connections 2024-09-16T01:53:45.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.795+0000 7f0365000640 1 -- 172.21.15.146:0/2644602917 wait complete. 2024-09-16T01:53:45.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T01:53:45.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.964+0000 7f9d2da00640 1 Processor -- start 2024-09-16T01:53:45.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.965+0000 7f9d2da00640 1 -- start start 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d2da00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9d28151a90 0x7f9d280c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d2da00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d280c6410 0x7f9d28158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d2da00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d28151e70 con 0x7f9d280c5280 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d2da00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f9d280c8130 con 0x7f9d280c6410 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d2da00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9d280c8410 con 0x7f9d28151a90 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d1e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d280c6410 0x7f9d28158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d27400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9d280c5280 0x7f9d2815aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44286/0 (socket says 172.21.15.146:44286) 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d1e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d280c6410 0x7f9d28158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43108/0 (socket says 172.21.15.146:43108) 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d1e000640 1 -- 172.21.15.146:0/2133786236 learned_addr learned my addr 172.21.15.146:0/2133786236 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.966+0000 7f9d26a00640 1 --2- 172.21.15.146:0/2133786236 >> v2:172.21.15.161:3300/0 conn(0x7f9d28151a90 0x7f9d280c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.967+0000 7f9d1e000640 1 -- 172.21.15.146:0/2133786236 >> v2:172.21.15.161:3300/0 conn(0x7f9d28151a90 msgr2=0x7f9d280c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.967+0000 7f9d1e000640 1 --2- 172.21.15.146:0/2133786236 >> v2:172.21.15.161:3300/0 conn(0x7f9d28151a90 0x7f9d280c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.967+0000 7f9d1e000640 1 -- 172.21.15.146:0/2133786236 >> v1:172.21.15.146:6789/0 conn(0x7f9d280c5280 legacy=0x7f9d2815aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.967+0000 7f9d1e000640 1 -- 172.21.15.146:0/2133786236 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d280c86f0 con 0x7f9d280c6410 2024-09-16T01:53:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.967+0000 7f9d26a00640 1 --2- 172.21.15.146:0/2133786236 >> v2:172.21.15.161:3300/0 conn(0x7f9d28151a90 0x7f9d280c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:45.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.967+0000 7f9d26000640 1 -- 172.21.15.146:0/2133786236 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528757361 0 0) 0x7f9d28151e70 con 0x7f9d280c5280 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d1e000640 1 --2- 172.21.15.146:0/2133786236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28158040 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9d28069950 tx=0x7f9d1002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10204b9a3005fab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d26000640 1 -- 172.21.15.146:0/2133786236 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d10037070 con 0x7f9d280c6410 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d26000640 1 -- 172.21.15.146:0/2133786236 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d10032440 con 0x7f9d280c6410 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d26000640 1 -- 172.21.15.146:0/2133786236 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d10032810 con 0x7f9d280c6410 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 -- 172.21.15.146:0/2133786236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 msgr2=0x7f9d28158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 --2- 172.21.15.146:0/2133786236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28158040 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9d28069950 tx=0x7f9d1002a270 comp rx=0 tx=0).stop 2024-09-16T01:53:45.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 -- 172.21.15.146:0/2133786236 shutdown_connections 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 --2- 172.21.15.146:0/2133786236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28158040 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 --2- 172.21.15.146:0/2133786236 >> v2:172.21.15.161:3300/0 conn(0x7f9d28151a90 0x7f9d280c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 -- 172.21.15.146:0/2133786236 >> 172.21.15.146:0/2133786236 conn(0x7f9d2806f3c0 msgr2=0x7f9d2806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 -- 172.21.15.146:0/2133786236 shutdown_connections 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.968+0000 7f9d2da00640 1 -- 172.21.15.146:0/2133786236 wait complete. 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.970+0000 7f9d2da00640 1 Processor -- start 2024-09-16T01:53:45.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.970+0000 7f9d2da00640 1 -- start start 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d2da00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 0x7f9d281f0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d2da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d2da00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d28151a90 0x7f9d281545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d2da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d28157b20 con 0x7f9d280c6410 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d2da00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9d28157840 con 0x7f9d280c5280 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d2da00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9d280c3bf0 con 0x7f9d28151a90 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d1e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d26a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 0x7f9d281f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d27400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d28151a90 0x7f9d281545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d26a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 0x7f9d281f0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55092/0 (socket says 172.21.15.146:55092) 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.971+0000 7f9d26a00640 1 -- 172.21.15.146:0/1749072343 learned_addr learned my addr 172.21.15.146:0/1749072343 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1e000640 1 -- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d28151a90 msgr2=0x7f9d281545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1e000640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d28151a90 0x7f9d281545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1e000640 1 -- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 msgr2=0x7f9d281f0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1e000640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 0x7f9d281f0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1e000640 1 -- 172.21.15.146:0/1749072343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d281f0f30 con 0x7f9d280c6410 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d26a00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 0x7f9d281f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1e000640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28154090 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9d10031000 tx=0x7f9d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d10037040 con 0x7f9d280c6410 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d10004030 con 0x7f9d280c6410 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d10004400 con 0x7f9d280c6410 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d281f1210 con 0x7f9d280c6410 2024-09-16T01:53:45.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.972+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d28153890 con 0x7f9d280c6410 2024-09-16T01:53:45.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.973+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d281f1730 con 0x7f9d280c6410 2024-09-16T01:53:45.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.974+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9d1003a070 con 0x7f9d280c6410 2024-09-16T01:53:45.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.975+0000 7f9d1f400640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d0406c2b0 0x7f9d0406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:45.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.975+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f9d100ab9b0 con 0x7f9d280c6410 2024-09-16T01:53:45.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.975+0000 7f9d26a00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d0406c2b0 0x7f9d0406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:45.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.976+0000 7f9d26a00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d0406c2b0 0x7f9d0406e770 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9d180095e0 tx=0x7f9d18009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:45.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:45.980+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9d281f1730 con 0x7f9d280c6410 2024-09-16T01:53:46.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.152+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f9d28153680 con 0x7f9d0406c2b0 2024-09-16T01:53:46.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.153+0000 7f9d1f400640 1 -- 172.21.15.146:0/1749072343 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9d28153680 con 0x7f9d0406c2b0 2024-09-16T01:53:46.155 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d0406c2b0 msgr2=0x7f9d0406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d0406c2b0 0x7f9d0406e770 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9d180095e0 tx=0x7f9d18009190 comp rx=0 tx=0).stop 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 msgr2=0x7f9d28154090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28154090 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9d10031000 tx=0x7f9d100046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 shutdown_connections 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9d0406c2b0 0x7f9d0406e770 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9d28151a90 0x7f9d281545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280c6410 0x7f9d28154090 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 --2- 172.21.15.146:0/1749072343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9d280c5280 0x7f9d281f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.155+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 >> 172.21.15.146:0/1749072343 conn(0x7f9d2806f3c0 msgr2=0x7f9d280c7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.156+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 shutdown_connections 2024-09-16T01:53:46.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.156+0000 7f9d2da00640 1 -- 172.21.15.146:0/1749072343 wait complete. 2024-09-16T01:53:46.174 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T01:53:46.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:46.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.332+0000 7ffa03e00640 1 Processor -- start 2024-09-16T01:53:46.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.333+0000 7ffa03e00640 1 -- start start 2024-09-16T01:53:46.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ffa03e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffa0415b390 0x7ffa04158d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ffa03e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415b7c0 0x7ffa04159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ffa03e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa04072990 con 0x7ffa0415afb0 2024-09-16T01:53:46.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ffa03e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7ffa04073860 con 0x7ffa0415b390 2024-09-16T01:53:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ffa03e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ffa04072c50 con 0x7ffa0415b7c0 2024-09-16T01:53:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ff9fa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffa0415b390 0x7ffa04158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ff9fa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffa0415b390 0x7ffa04158d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43126/0 (socket says 172.21.15.146:43126) 2024-09-16T01:53:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ff9fa000640 1 -- 172.21.15.146:0/2929592303 learned_addr learned my addr 172.21.15.146:0/2929592303 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:46.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.334+0000 7ffa03400640 1 --2- 172.21.15.146:0/2929592303 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415b7c0 0x7ffa04159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.335+0000 7ff9fa000640 1 -- 172.21.15.146:0/2929592303 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415b7c0 msgr2=0x7ffa04159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.335+0000 7ff9fa000640 1 --2- 172.21.15.146:0/2929592303 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415b7c0 0x7ffa04159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.335+0000 7ff9fa000640 1 -- 172.21.15.146:0/2929592303 >> v1:172.21.15.161:6789/0 conn(0x7ffa0415afb0 legacy=0x7ffa04072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.335+0000 7ffa02000640 1 -- 172.21.15.146:0/2929592303 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179948559 0 0) 0x7ffa04072990 con 0x7ffa0415afb0 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.335+0000 7ff9fa000640 1 -- 172.21.15.146:0/2929592303 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa04072f30 con 0x7ffa0415b390 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.336+0000 7ff9fa000640 1 --2- 172.21.15.146:0/2929592303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b390 0x7ffa04158d40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63f3bbd69dcfb97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.336+0000 7ffa02000640 1 -- 172.21.15.146:0/2929592303 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec037070 con 0x7ffa0415b390 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.336+0000 7ffa02000640 1 -- 172.21.15.146:0/2929592303 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec02ae50 con 0x7ffa0415b390 2024-09-16T01:53:46.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.336+0000 7ffa02000640 1 -- 172.21.15.146:0/2929592303 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec032860 con 0x7ffa0415b390 2024-09-16T01:53:46.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.336+0000 7ffa03e00640 1 -- 172.21.15.146:0/2929592303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b390 msgr2=0x7ffa04158d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.336+0000 7ffa03e00640 1 --2- 172.21.15.146:0/2929592303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b390 0x7ffa04158d40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec02a300 comp rx=0 tx=0).stop 2024-09-16T01:53:46.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.337+0000 7ffa03e00640 1 -- 172.21.15.146:0/2929592303 shutdown_connections 2024-09-16T01:53:46.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.337+0000 7ffa03e00640 1 --2- 172.21.15.146:0/2929592303 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415b7c0 0x7ffa04159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.337+0000 7ffa03e00640 1 --2- 172.21.15.146:0/2929592303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b390 0x7ffa04158d40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.337+0000 7ffa03e00640 1 -- 172.21.15.146:0/2929592303 >> 172.21.15.146:0/2929592303 conn(0x7ffa0406f160 msgr2=0x7ffa0406f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.337+0000 7ffa03e00640 1 -- 172.21.15.146:0/2929592303 shutdown_connections 2024-09-16T01:53:46.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.337+0000 7ffa03e00640 1 -- 172.21.15.146:0/2929592303 wait complete. 2024-09-16T01:53:46.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.338+0000 7ffa03e00640 1 Processor -- start 2024-09-16T01:53:46.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.338+0000 7ffa03e00640 1 -- start start 2024-09-16T01:53:46.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415afb0 0x7ffa04155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa0415b390 0x7ffa04152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b7c0 0x7ffa040c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0406fc80 con 0x7ffa0415b7c0 2024-09-16T01:53:46.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffa040734f0 con 0x7ffa0415b390 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0406fe00 con 0x7ffa0415afb0 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b7c0 0x7ffa040c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415afb0 0x7ffa04155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415afb0 0x7ffa04155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:46356/0 (socket says 172.21.15.146:46356) 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ff9fa000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa0415b390 0x7ffa04152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa02a00640 1 -- 172.21.15.146:0/1067715477 learned_addr learned my addr 172.21.15.146:0/1067715477 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.339+0000 7ffa03400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b7c0 0x7ffa040c0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43136/0 (socket says 172.21.15.146:43136) 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa02a00640 1 -- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa0415b390 msgr2=0x7ffa04152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa02a00640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa0415b390 0x7ffa04152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa02a00640 1 -- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b7c0 msgr2=0x7ffa040c0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa02a00640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b7c0 0x7ffa040c0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa02a00640 1 -- 172.21.15.146:0/1067715477 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa040c2a10 con 0x7ffa0415afb0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa03400640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa0415b7c0 0x7ffa040c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa02a00640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415afb0 0x7ffa04155760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400c410 tx=0x7ff9f400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4017070 con 0x7ffa0415afb0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9f40052d0 con 0x7ffa0415afb0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa040c2c90 con 0x7ffa0415afb0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4003810 con 0x7ffa0415afb0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.340+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041f0650 con 0x7ffa0415afb0 2024-09-16T01:53:46.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.341+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa04072100 con 0x7ffa0415afb0 2024-09-16T01:53:46.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.342+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff9f401a070 con 0x7ffa0415afb0 2024-09-16T01:53:46.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.343+0000 7ff9fb400640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff9e006c2b0 0x7ff9e006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.343+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7ff9f408afe0 con 0x7ffa0415afb0 2024-09-16T01:53:46.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.343+0000 7ff9fb400640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff9e00746b0 0x7ff9e0076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.343+0000 7ff9fa000640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff9e006c2b0 0x7ff9e006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.343+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9e0077220 con 0x7ff9e00746b0 2024-09-16T01:53:46.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.343+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7ff9f408b430 con 0x7ffa0415afb0 2024-09-16T01:53:46.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.344+0000 7ffa03400640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff9e00746b0 0x7ff9e0076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.344+0000 7ff9fa000640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff9e006c2b0 0x7ff9e006e770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec034000 tx=0x7ff9ec02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.344+0000 7ffa03400640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff9e00746b0 0x7ff9e0076b50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.345+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== osd.3 v2:172.21.15.146:6816/1633804394 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9e0077220 con 0x7ff9e00746b0 2024-09-16T01:53:46.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.358+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 --> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffa041505a0 con 0x7ff9e00746b0 2024-09-16T01:53:46.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.358+0000 7ff9fb400640 1 -- 172.21.15.146:0/1067715477 <== osd.3 v2:172.21.15.146:6816/1633804394 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffa041505a0 con 0x7ff9e00746b0 2024-09-16T01:53:46.361 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:46.361 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:46.361 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.358+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff9e00746b0 msgr2=0x7ff9e0076b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.358+0000 7ffa03e00640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.146:6816/1633804394,v1:172.21.15.146:6817/1633804394] conn(0x7ff9e00746b0 0x7ff9e0076b50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff9e006c2b0 msgr2=0x7ff9e006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff9e006c2b0 0x7ff9e006e770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec034000 tx=0x7ff9ec02e000 comp rx=0 tx=0).stop 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415afb0 msgr2=0x7ffa04155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 --2- 172.21.15.146:0/1067715477 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffa0415afb0 0x7ffa04155760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400c410 tx=0x7ff9f400c8d0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03400640 1 -- 172.21.15.146:0/1067715477 reap_dead start 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 shutdown_connections 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 >> 172.21.15.146:0/1067715477 conn(0x7ffa0406f160 msgr2=0x7ffa0414fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 shutdown_connections 2024-09-16T01:53:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.359+0000 7ffa03e00640 1 -- 172.21.15.146:0/1067715477 wait complete. 2024-09-16T01:53:46.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T01:53:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.530+0000 7f6f83200640 1 Processor -- start 2024-09-16T01:53:46.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.530+0000 7f6f83200640 1 -- start start 2024-09-16T01:53:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.531+0000 7f6f83200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f840c6b70 0x7f6f8415b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f83200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f840cc040 0x7f6f840cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f83200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8406d140 con 0x7f6f840c75a0 2024-09-16T01:53:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f83200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6f8415b880 con 0x7f6f840c6b70 2024-09-16T01:53:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f83200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6f840cab20 con 0x7f6f840cc040 2024-09-16T01:53:46.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f840c6b70 0x7f6f8415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6f840c75a0 0x7f6f840cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44298/0 (socket says 172.21.15.146:44298) 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81400640 1 -- 172.21.15.146:0/670670201 learned_addr learned my addr 172.21.15.146:0/670670201 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f82800640 1 --2- 172.21.15.146:0/670670201 >> v2:172.21.15.161:3300/0 conn(0x7f6f840cc040 0x7f6f840cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81e00640 1 -- 172.21.15.146:0/670670201 >> v2:172.21.15.161:3300/0 conn(0x7f6f840cc040 msgr2=0x7f6f840cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81e00640 1 --2- 172.21.15.146:0/670670201 >> v2:172.21.15.161:3300/0 conn(0x7f6f840cc040 0x7f6f840cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81e00640 1 -- 172.21.15.146:0/670670201 >> v1:172.21.15.146:6789/0 conn(0x7f6f840c75a0 legacy=0x7f6f840cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.532+0000 7f6f81e00640 1 -- 172.21.15.146:0/670670201 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8406d420 con 0x7f6f840c6b70 2024-09-16T01:53:46.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.533+0000 7f6f81e00640 1 --2- 172.21.15.146:0/670670201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840c6b70 0x7f6f8415b340 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6f8406ee20 tx=0x7f6f7402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad22459795876313 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.533+0000 7f6f80a00640 1 -- 172.21.15.146:0/670670201 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f74037070 con 0x7f6f840c6b70 2024-09-16T01:53:46.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f80a00640 1 -- 172.21.15.146:0/670670201 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f74032440 con 0x7f6f840c6b70 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f80a00640 1 -- 172.21.15.146:0/670670201 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7402ae60 con 0x7f6f840c6b70 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 -- 172.21.15.146:0/670670201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840c6b70 msgr2=0x7f6f8415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 --2- 172.21.15.146:0/670670201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840c6b70 0x7f6f8415b340 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6f8406ee20 tx=0x7f6f7402a110 comp rx=0 tx=0).stop 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 -- 172.21.15.146:0/670670201 shutdown_connections 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 --2- 172.21.15.146:0/670670201 >> v2:172.21.15.161:3300/0 conn(0x7f6f840cc040 0x7f6f840cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 --2- 172.21.15.146:0/670670201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840c6b70 0x7f6f8415b340 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 -- 172.21.15.146:0/670670201 >> 172.21.15.146:0/670670201 conn(0x7f6f840ccc20 msgr2=0x7f6f840cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 -- 172.21.15.146:0/670670201 shutdown_connections 2024-09-16T01:53:46.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.534+0000 7f6f83200640 1 -- 172.21.15.146:0/670670201 wait complete. 2024-09-16T01:53:46.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.536+0000 7f6f83200640 1 Processor -- start 2024-09-16T01:53:46.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.536+0000 7f6f83200640 1 -- start start 2024-09-16T01:53:46.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f83200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f840c6b70 0x7f6f84153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f83200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f840c75a0 0x7f6f8415dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f83200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 0x7f6f840c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f83200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6f840cb050 con 0x7f6f840cc040 2024-09-16T01:53:46.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f83200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f840cad70 con 0x7f6f840c6b70 2024-09-16T01:53:46.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f83200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6f840cbc80 con 0x7f6f840c75a0 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 0x7f6f840c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 0x7f6f840c17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43164/0 (socket says 172.21.15.146:43164) 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 -- 172.21.15.146:0/1197452731 learned_addr learned my addr 172.21.15.146:0/1197452731 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f81e00640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f840c6b70 0x7f6f84153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f81400640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f840c75a0 0x7f6f8415dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 -- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f840c75a0 msgr2=0x7f6f8415dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f840c75a0 0x7f6f8415dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 -- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f840c6b70 msgr2=0x7f6f84153ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f840c6b70 0x7f6f84153ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.537+0000 7f6f82800640 1 -- 172.21.15.146:0/1197452731 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8415c130 con 0x7f6f840cc040 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.538+0000 7f6f82800640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 0x7f6f840c17b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009dd0 tx=0x7f6f7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.538+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c018020 con 0x7f6f840cc040 2024-09-16T01:53:46.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.538+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7c014480 con 0x7f6f840cc040 2024-09-16T01:53:46.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.538+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8415c410 con 0x7f6f840cc040 2024-09-16T01:53:46.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.538+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c014870 con 0x7f6f840cc040 2024-09-16T01:53:46.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.538+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8415d490 con 0x7f6f840cc040 2024-09-16T01:53:46.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.539+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f8406cd20 con 0x7f6f840cc040 2024-09-16T01:53:46.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.540+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6f7c01b020 con 0x7f6f840cc040 2024-09-16T01:53:46.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.541+0000 7f6f7a000640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6f6406c2b0 0x7f6f6406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.541+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f6f7c08c380 con 0x7f6f840cc040 2024-09-16T01:53:46.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.541+0000 7f6f81e00640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6f6406c2b0 0x7f6f6406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.542+0000 7f6f81e00640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6f6406c2b0 0x7f6f6406e770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6f74034000 tx=0x7f6f7402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.545+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6f7c022020 con 0x7f6f840cc040 2024-09-16T01:53:46.723 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.721+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f6f8415bee0 con 0x7f6f6406c2b0 2024-09-16T01:53:46.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.722+0000 7f6f7a000640 1 -- 172.21.15.146:0/1197452731 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6f8415bee0 con 0x7f6f6406c2b0 2024-09-16T01:53:46.724 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.724+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6f6406c2b0 msgr2=0x7f6f6406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.724+0000 7f6f83200640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6f6406c2b0 0x7f6f6406e770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6f74034000 tx=0x7f6f7402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.724+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 msgr2=0x7f6f840c17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.724+0000 7f6f83200640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 0x7f6f840c17b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009dd0 tx=0x7f6f7c00c490 comp rx=0 tx=0).stop 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 shutdown_connections 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6f6406c2b0 0x7f6f6406e770 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840cc040 0x7f6f840c17b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f840c75a0 0x7f6f8415dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 --2- 172.21.15.146:0/1197452731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f840c6b70 0x7f6f84153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 >> 172.21.15.146:0/1197452731 conn(0x7f6f840ccc20 msgr2=0x7f6f840c9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 shutdown_connections 2024-09-16T01:53:46.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.725+0000 7f6f83200640 1 -- 172.21.15.146:0/1197452731 wait complete. 2024-09-16T01:53:46.743 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T01:53:46.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:46.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.901+0000 7fc011e00640 1 Processor -- start 2024-09-16T01:53:46.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.901+0000 7fc011e00640 1 -- start start 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.902+0000 7fc011e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc00c152cc0 0x7fc00c0c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc011e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00c1530a0 con 0x7fc00c0c3da0 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc011e00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc00c0c2d70 con 0x7fc00c06e9e0 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc011e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc00c159960 con 0x7fc00c152cc0 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc00aa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc00c152cc0 0x7fc00c0c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc00b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc00c0c3da0 0x7fc00c0c4210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44306/0 (socket says 172.21.15.146:44306) 2024-09-16T01:53:46.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc00b400640 1 -- 172.21.15.146:0/1385717720 learned_addr learned my addr 172.21.15.146:0/1385717720 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:46.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc00aa00640 1 -- 172.21.15.146:0/1385717720 >> v1:172.21.15.161:6789/0 conn(0x7fc00c06e9e0 legacy=0x7fc00c06ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc00aa00640 1 -- 172.21.15.146:0/1385717720 >> v1:172.21.15.146:6789/0 conn(0x7fc00c0c3da0 legacy=0x7fc00c0c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc009600640 1 -- 172.21.15.146:0/1385717720 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519701428 0 0) 0x7fc00c0c2d70 con 0x7fc00c06e9e0 2024-09-16T01:53:46.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.903+0000 7fc00aa00640 1 -- 172.21.15.146:0/1385717720 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c0c8550 con 0x7fc00c152cc0 2024-09-16T01:53:46.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.904+0000 7fc00aa00640 1 --2- 172.21.15.146:0/1385717720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c152cc0 0x7fc00c0c6220 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0097a0 tx=0x7fbffc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402cf7240f7824d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc009600640 1 -- 172.21.15.146:0/1385717720 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc037070 con 0x7fc00c152cc0 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc009600640 1 -- 172.21.15.146:0/1385717720 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbffc02ae50 con 0x7fc00c152cc0 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc009600640 1 -- 172.21.15.146:0/1385717720 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc032660 con 0x7fc00c152cc0 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 -- 172.21.15.146:0/1385717720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c152cc0 msgr2=0x7fc00c0c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 --2- 172.21.15.146:0/1385717720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c152cc0 0x7fc00c0c6220 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0097a0 tx=0x7fbffc02a1c0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 -- 172.21.15.146:0/1385717720 shutdown_connections 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 --2- 172.21.15.146:0/1385717720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c152cc0 0x7fc00c0c6220 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 -- 172.21.15.146:0/1385717720 >> 172.21.15.146:0/1385717720 conn(0x7fc00c0cc0b0 msgr2=0x7fc00c0cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 -- 172.21.15.146:0/1385717720 shutdown_connections 2024-09-16T01:53:46.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.905+0000 7fc011e00640 1 -- 172.21.15.146:0/1385717720 wait complete. 2024-09-16T01:53:46.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.907+0000 7fc011e00640 1 Processor -- start 2024-09-16T01:53:46.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.907+0000 7fc011e00640 1 -- start start 2024-09-16T01:53:46.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc011e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c06e9e0 0x7fc00c1f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc011e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc00c0c3da0 0x7fc00c153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc011e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc00c152cc0 0x7fc00c150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc011e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c070320 con 0x7fc00c06e9e0 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc011e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c159680 con 0x7fc00c0c3da0 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc011e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc00c0c8210 con 0x7fc00c152cc0 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc00aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c06e9e0 0x7fc00c1f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc00aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c06e9e0 0x7fc00c1f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43194/0 (socket says 172.21.15.146:43194) 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc00aa00640 1 -- 172.21.15.146:0/54011096 learned_addr learned my addr 172.21.15.146:0/54011096 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc00b400640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc00c152cc0 0x7fc00c150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.908+0000 7fc00a000640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc00c0c3da0 0x7fc00c153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00aa00640 1 -- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc00c152cc0 msgr2=0x7fc00c150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00aa00640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc00c152cc0 0x7fc00c150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00aa00640 1 -- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc00c0c3da0 msgr2=0x7fc00c153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00aa00640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc00c0c3da0 0x7fc00c153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00aa00640 1 -- 172.21.15.146:0/54011096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c158450 con 0x7fc00c06e9e0 2024-09-16T01:53:46.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00a000640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc00c0c3da0 0x7fc00c153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc00aa00640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c06e9e0 0x7fc00c1f4990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbffc02a200 tx=0x7fbffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc037040 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbffc032e10 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c153e80 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc004270 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.909+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c071d90 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.910+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc00c0c6220 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.911+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbffc03a040 con 0x7fc00c06e9e0 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.912+0000 7fc003400640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbfec06c2b0 0x7fbfec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.912+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7fbffc0abb50 con 0x7fc00c06e9e0 2024-09-16T01:53:46.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.912+0000 7fc003400640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbfec074710 0x7fbfec076bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:46.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.912+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbffc043ec0 con 0x7fbfec074710 2024-09-16T01:53:46.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.912+0000 7fc00a000640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbfec06c2b0 0x7fbfec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.912+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7fbffc0b02c0 con 0x7fc00c06e9e0 2024-09-16T01:53:46.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.913+0000 7fc00b400640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbfec074710 0x7fbfec076bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:46.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.913+0000 7fc00a000640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbfec06c2b0 0x7fbfec06e770 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc00c06e790 tx=0x7fbff8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.913+0000 7fc00b400640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbfec074710 0x7fbfec076bb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:46.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.914+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== osd.4 v2:172.21.15.161:6800/154390265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbffc043ec0 con 0x7fbfec074710 2024-09-16T01:53:46.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.927+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 --> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc00c0c3970 con 0x7fbfec074710 2024-09-16T01:53:46.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc003400640 1 -- 172.21.15.146:0/54011096 <== osd.4 v2:172.21.15.161:6800/154390265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc00c0c3970 con 0x7fbfec074710 2024-09-16T01:53:46.930 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:46.930 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:46.930 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbfec074710 msgr2=0x7fbfec076bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6800/154390265,v1:172.21.15.161:6801/154390265] conn(0x7fbfec074710 0x7fbfec076bb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbfec06c2b0 msgr2=0x7fbfec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbfec06c2b0 0x7fbfec06e770 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc00c06e790 tx=0x7fbff8009040 comp rx=0 tx=0).stop 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c06e9e0 msgr2=0x7fc00c1f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 --2- 172.21.15.146:0/54011096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc00c06e9e0 0x7fc00c1f4990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbffc02a200 tx=0x7fbffc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc00b400640 1 -- 172.21.15.146:0/54011096 reap_dead start 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 shutdown_connections 2024-09-16T01:53:46.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 >> 172.21.15.146:0/54011096 conn(0x7fc00c0cc0b0 msgr2=0x7fc00c0c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:46.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 shutdown_connections 2024-09-16T01:53:46.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:46.928+0000 7fc011e00640 1 -- 172.21.15.146:0/54011096 wait complete. 2024-09-16T01:53:46.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T01:53:47.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.098+0000 7f9b44200640 1 Processor -- start 2024-09-16T01:53:47.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.098+0000 7f9b44200640 1 -- start start 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.099+0000 7f9b44200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c0c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.099+0000 7f9b44200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b3c0c6410 0x7f9b3c158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.099+0000 7f9b44200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b3c151e70 con 0x7f9b3c0c5280 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b44200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9b3c0c8130 con 0x7f9b3c0c6410 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b44200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c0c8410 con 0x7f9b3c151a90 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b42200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9b3c0c5280 0x7f9b3c15aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44320/0 (socket says 172.21.15.146:44320) 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b42200640 1 -- 172.21.15.146:0/4269455548 learned_addr learned my addr 172.21.15.146:0/4269455548 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b41800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c0c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b40e00640 1 --2- 172.21.15.146:0/4269455548 >> v2:172.21.15.161:3300/0 conn(0x7f9b3c0c6410 0x7f9b3c158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b41800640 1 -- 172.21.15.146:0/4269455548 >> v2:172.21.15.161:3300/0 conn(0x7f9b3c0c6410 msgr2=0x7f9b3c158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b41800640 1 --2- 172.21.15.146:0/4269455548 >> v2:172.21.15.161:3300/0 conn(0x7f9b3c0c6410 0x7f9b3c158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b41800640 1 -- 172.21.15.146:0/4269455548 >> v1:172.21.15.146:6789/0 conn(0x7f9b3c0c5280 legacy=0x7f9b3c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b3be00640 1 -- 172.21.15.146:0/4269455548 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831300713 0 0) 0x7f9b3c151e70 con 0x7f9b3c0c5280 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b41800640 1 -- 172.21.15.146:0/4269455548 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c0c86f0 con 0x7f9b3c151a90 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.100+0000 7f9b40e00640 1 --2- 172.21.15.146:0/4269455548 >> v2:172.21.15.161:3300/0 conn(0x7f9b3c0c6410 0x7f9b3c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:47.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.101+0000 7f9b41800640 1 --2- 172.21.15.146:0/4269455548 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c0c36b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009d90 tx=0x7f9b3002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2be31effe8b03e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b3be00640 1 -- 172.21.15.146:0/4269455548 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007740 con 0x7f9b3c151a90 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b3be00640 1 -- 172.21.15.146:0/4269455548 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b300078e0 con 0x7f9b3c151a90 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b3be00640 1 -- 172.21.15.146:0/4269455548 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007cb0 con 0x7f9b3c151a90 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 -- 172.21.15.146:0/4269455548 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 msgr2=0x7f9b3c0c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 --2- 172.21.15.146:0/4269455548 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c0c36b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009d90 tx=0x7f9b3002a660 comp rx=0 tx=0).stop 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 -- 172.21.15.146:0/4269455548 shutdown_connections 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 --2- 172.21.15.146:0/4269455548 >> v2:172.21.15.161:3300/0 conn(0x7f9b3c0c6410 0x7f9b3c158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 --2- 172.21.15.146:0/4269455548 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c0c36b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 -- 172.21.15.146:0/4269455548 >> 172.21.15.146:0/4269455548 conn(0x7f9b3c06f3c0 msgr2=0x7f9b3c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 -- 172.21.15.146:0/4269455548 shutdown_connections 2024-09-16T01:53:47.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.102+0000 7f9b44200640 1 -- 172.21.15.146:0/4269455548 wait complete. 2024-09-16T01:53:47.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.104+0000 7f9b44200640 1 Processor -- start 2024-09-16T01:53:47.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.104+0000 7f9b44200640 1 -- start start 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b44200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b3c0c5280 0x7f9b3c159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b44200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 0x7f9b3c1eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b44200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c1548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b44200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c157b20 con 0x7f9b3c0c5280 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b44200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c157840 con 0x7f9b3c0c6410 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b44200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c0c3bf0 con 0x7f9b3c151a90 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b41800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b3c0c5280 0x7f9b3c159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 0x7f9b3c1eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 0x7f9b3c1eebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55148/0 (socket says 172.21.15.146:55148) 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 -- 172.21.15.146:0/3601536636 learned_addr learned my addr 172.21.15.146:0/3601536636 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b41800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b3c0c5280 0x7f9b3c159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43208/0 (socket says 172.21.15.146:43208) 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b42200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c1548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 -- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 msgr2=0x7f9b3c1548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c1548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 -- 172.21.15.146:0/3601536636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b3c0c5280 msgr2=0x7f9b3c159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b3c0c5280 0x7f9b3c159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.105+0000 7f9b40e00640 1 -- 172.21.15.146:0/3601536636 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c15b1c0 con 0x7f9b3c0c6410 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.106+0000 7f9b40e00640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 0x7f9b3c1eebd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800ebb0 tx=0x7f9b2800c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.106+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28017070 con 0x7f9b3c0c6410 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.106+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b2800cc30 con 0x7f9b3c0c6410 2024-09-16T01:53:47.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.106+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28013470 con 0x7f9b3c0c6410 2024-09-16T01:53:47.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.106+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3c15b4a0 con 0x7f9b3c0c6410 2024-09-16T01:53:47.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.106+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c15ba60 con 0x7f9b3c0c6410 2024-09-16T01:53:47.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.107+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b3c0c79b0 con 0x7f9b3c0c6410 2024-09-16T01:53:47.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.108+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9b2801a070 con 0x7f9b3c0c6410 2024-09-16T01:53:47.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.109+0000 7f9b39600640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b1c0706c0 0x7f9b1c072b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.109+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f9b2808b810 con 0x7f9b3c0c6410 2024-09-16T01:53:47.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.109+0000 7f9b41800640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b1c0706c0 0x7f9b1c072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.110+0000 7f9b41800640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b1c0706c0 0x7f9b1c072b80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9b30034000 tx=0x7f9b3002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.114+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9b28059750 con 0x7f9b3c0c6410 2024-09-16T01:53:47.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.287+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f9b3c072a10 con 0x7f9b1c0706c0 2024-09-16T01:53:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.288+0000 7f9b39600640 1 -- 172.21.15.146:0/3601536636 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9b3c072a10 con 0x7f9b1c0706c0 2024-09-16T01:53:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T01:53:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b1c0706c0 msgr2=0x7f9b1c072b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b1c0706c0 0x7f9b1c072b80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9b30034000 tx=0x7f9b3002f000 comp rx=0 tx=0).stop 2024-09-16T01:53:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 msgr2=0x7f9b3c1eebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 0x7f9b3c1eebd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800ebb0 tx=0x7f9b2800c520 comp rx=0 tx=0).stop 2024-09-16T01:53:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 shutdown_connections 2024-09-16T01:53:47.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f9b1c0706c0 0x7f9b1c072b80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b3c151a90 0x7f9b3c1548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b3c0c6410 0x7f9b3c1eebd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 --2- 172.21.15.146:0/3601536636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b3c0c5280 0x7f9b3c159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 >> 172.21.15.146:0/3601536636 conn(0x7f9b3c06f3c0 msgr2=0x7f9b3c155970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:47.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 shutdown_connections 2024-09-16T01:53:47.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.291+0000 7f9b44200640 1 -- 172.21.15.146:0/3601536636 wait complete. 2024-09-16T01:53:47.308 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T01:53:47.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:47.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.466+0000 7f32dfe00640 1 Processor -- start 2024-09-16T01:53:47.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.466+0000 7f32dfe00640 1 -- start start 2024-09-16T01:53:47.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.467+0000 7f32dfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f32e015afb0 0x7f32e0072100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dfe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e0072990 con 0x7f32e015b390 2024-09-16T01:53:47.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dfe00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e0072c50 con 0x7f32e0159450 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f32e0073860 con 0x7f32e015afb0 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f32e015afb0 0x7f32e0072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32d6000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f32e015b390 0x7f32e0158d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44330/0 (socket says 172.21.15.146:44330) 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32d6000640 1 -- 172.21.15.146:0/4038046926 learned_addr learned my addr 172.21.15.146:0/4038046926 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32de000640 1 -- 172.21.15.146:0/4038046926 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964343659 0 0) 0x7f32e0072990 con 0x7f32e015b390 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dea00640 1 -- 172.21.15.146:0/4038046926 >> v1:172.21.15.161:6789/0 conn(0x7f32e0159450 legacy=0x7f32e0151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dea00640 1 -- 172.21.15.146:0/4038046926 >> v1:172.21.15.146:6789/0 conn(0x7f32e015b390 legacy=0x7f32e0158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.468+0000 7f32dea00640 1 -- 172.21.15.146:0/4038046926 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e0072f30 con 0x7f32e015afb0 2024-09-16T01:53:47.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.469+0000 7f32dea00640 1 --2- 172.21.15.146:0/4038046926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015afb0 0x7f32e0072100 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009720 tx=0x7f32d002a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcbbd321eb95e91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32de000640 1 -- 172.21.15.146:0/4038046926 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0037070 con 0x7f32e015afb0 2024-09-16T01:53:47.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32de000640 1 -- 172.21.15.146:0/4038046926 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d0032440 con 0x7f32e015afb0 2024-09-16T01:53:47.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32de000640 1 -- 172.21.15.146:0/4038046926 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d002ae60 con 0x7f32e015afb0 2024-09-16T01:53:47.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 -- 172.21.15.146:0/4038046926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015afb0 msgr2=0x7f32e0072100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 --2- 172.21.15.146:0/4038046926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015afb0 0x7f32e0072100 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009720 tx=0x7f32d002a150 comp rx=0 tx=0).stop 2024-09-16T01:53:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 -- 172.21.15.146:0/4038046926 shutdown_connections 2024-09-16T01:53:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 --2- 172.21.15.146:0/4038046926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015afb0 0x7f32e0072100 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 -- 172.21.15.146:0/4038046926 >> 172.21.15.146:0/4038046926 conn(0x7f32e006f160 msgr2=0x7f32e006f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 -- 172.21.15.146:0/4038046926 shutdown_connections 2024-09-16T01:53:47.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.470+0000 7f32dfe00640 1 -- 172.21.15.146:0/4038046926 wait complete. 2024-09-16T01:53:47.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.472+0000 7f32dfe00640 1 Processor -- start 2024-09-16T01:53:47.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.472+0000 7f32dfe00640 1 -- start start 2024-09-16T01:53:47.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32e0159450 0x7f32e00cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32e015afb0 0x7f32e014a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015b390 0x7f32e014af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0072640 con 0x7f32e015b390 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dfe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32e00734f0 con 0x7f32e0159450 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dfe00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f32e006fc80 con 0x7f32e015afb0 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32df400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015b390 0x7f32e014af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32df400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015b390 0x7f32e014af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43232/0 (socket says 172.21.15.146:43232) 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32df400640 1 -- 172.21.15.146:0/375064829 learned_addr learned my addr 172.21.15.146:0/375064829 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:47.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32dea00640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32e0159450 0x7f32e00cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.473+0000 7f32d6000640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32e015afb0 0x7f32e014a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32df400640 1 -- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32e015afb0 msgr2=0x7f32e014a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32df400640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32e015afb0 0x7f32e014a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32df400640 1 -- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32e0159450 msgr2=0x7f32e00cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32df400640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32e0159450 0x7f32e00cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32df400640 1 -- 172.21.15.146:0/375064829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e00ca1c0 con 0x7f32e015b390 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32df400640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015b390 0x7f32e014af20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c8c0 tx=0x7f32d800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8017070 con 0x7f32e015b390 2024-09-16T01:53:47.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d8003da0 con 0x7f32e015b390 2024-09-16T01:53:47.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e0144d90 con 0x7f32e015b390 2024-09-16T01:53:47.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8005290 con 0x7f32e015b390 2024-09-16T01:53:47.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.474+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e0149c20 con 0x7f32e015b390 2024-09-16T01:53:47.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.475+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f32e015b7c0 con 0x7f32e015b390 2024-09-16T01:53:47.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.476+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f32d801a070 con 0x7f32e015b390 2024-09-16T01:53:47.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.477+0000 7f32d7400640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f32bc06c2b0 0x7f32bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.477+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f32d808abd0 con 0x7f32e015b390 2024-09-16T01:53:47.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.477+0000 7f32d7400640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f32bc074710 0x7f32bc076bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.477+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32d8014ec0 con 0x7f32bc074710 2024-09-16T01:53:47.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.477+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7f32d808f2a0 con 0x7f32e015b390 2024-09-16T01:53:47.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.477+0000 7f32dea00640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f32bc06c2b0 0x7f32bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.478+0000 7f32d6000640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f32bc074710 0x7f32bc076bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.478+0000 7f32dea00640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f32bc06c2b0 0x7f32bc06e770 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f32d0034000 tx=0x7f32d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.478+0000 7f32d6000640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f32bc074710 0x7f32bc076bb0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.479+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== osd.5 v2:172.21.15.161:6824/356357279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f32d8014ec0 con 0x7f32bc074710 2024-09-16T01:53:47.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.492+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 --> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f32e0073cb0 con 0x7f32bc074710 2024-09-16T01:53:47.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.492+0000 7f32d7400640 1 -- 172.21.15.146:0/375064829 <== osd.5 v2:172.21.15.161:6824/356357279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f32e0073cb0 con 0x7f32bc074710 2024-09-16T01:53:47.495 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:47.496 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:47.496 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:47.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f32bc074710 msgr2=0x7f32bc076bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6824/356357279,v1:172.21.15.161:6825/356357279] conn(0x7f32bc074710 0x7f32bc076bb0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f32bc06c2b0 msgr2=0x7f32bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f32bc06c2b0 0x7f32bc06e770 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f32d0034000 tx=0x7f32d002e000 comp rx=0 tx=0).stop 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015b390 msgr2=0x7f32e014af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 --2- 172.21.15.146:0/375064829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32e015b390 0x7f32e014af20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c8c0 tx=0x7f32d800cd80 comp rx=0 tx=0).stop 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32df400640 1 -- 172.21.15.146:0/375064829 reap_dead start 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 shutdown_connections 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 >> 172.21.15.146:0/375064829 conn(0x7f32e006f160 msgr2=0x7f32e006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 shutdown_connections 2024-09-16T01:53:47.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.493+0000 7f32dfe00640 1 -- 172.21.15.146:0/375064829 wait complete. 2024-09-16T01:53:47.510 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T01:53:47.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.664+0000 7fabbae00640 1 Processor -- start 2024-09-16T01:53:47.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.664+0000 7fabbae00640 1 -- start start 2024-09-16T01:53:47.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.665+0000 7fabbae00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fabb406d270 0x7fabb41548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.665+0000 7fabbae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb4067ba0 con 0x7fabb406ce90 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.665+0000 7fabbae00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb4067e80 con 0x7fabb40c6840 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.665+0000 7fabbae00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fabb4067950 con 0x7fabb406d270 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.666+0000 7fabb3400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fabb406d270 0x7fabb41548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.666+0000 7fabb8e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fabb40c6840 0x7fabb40725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57344/0 (socket says 172.21.15.146:57344) 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.666+0000 7fabb8e00640 1 -- 172.21.15.146:0/945163916 learned_addr learned my addr 172.21.15.146:0/945163916 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.666+0000 7fabb3400640 1 -- 172.21.15.146:0/945163916 >> v1:172.21.15.161:6789/0 conn(0x7fabb40c6840 legacy=0x7fabb40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.666+0000 7fabb3400640 1 -- 172.21.15.146:0/945163916 >> v1:172.21.15.146:6789/0 conn(0x7fabb406ce90 legacy=0x7fabb415b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.666+0000 7fabb3400640 1 -- 172.21.15.146:0/945163916 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb406e400 con 0x7fabb406d270 2024-09-16T01:53:47.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.667+0000 7fabb3400640 1 --2- 172.21.15.146:0/945163916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406d270 0x7fabb41548b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7faba00097a0 tx=0x7faba002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=971c1c12907ffb64 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.669+0000 7fabb2a00640 1 -- 172.21.15.146:0/945163916 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0037070 con 0x7fabb406d270 2024-09-16T01:53:47.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.669+0000 7fabb2a00640 1 -- 172.21.15.146:0/945163916 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba0032440 con 0x7fabb406d270 2024-09-16T01:53:47.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.669+0000 7fabb2a00640 1 -- 172.21.15.146:0/945163916 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0032810 con 0x7fabb406d270 2024-09-16T01:53:47.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 -- 172.21.15.146:0/945163916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406d270 msgr2=0x7fabb41548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 --2- 172.21.15.146:0/945163916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406d270 0x7fabb41548b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7faba00097a0 tx=0x7faba002a230 comp rx=0 tx=0).stop 2024-09-16T01:53:47.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 -- 172.21.15.146:0/945163916 shutdown_connections 2024-09-16T01:53:47.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 --2- 172.21.15.146:0/945163916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406d270 0x7fabb41548b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 -- 172.21.15.146:0/945163916 >> 172.21.15.146:0/945163916 conn(0x7fabb406d8e0 msgr2=0x7fabb406dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:47.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 -- 172.21.15.146:0/945163916 shutdown_connections 2024-09-16T01:53:47.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.670+0000 7fabbae00640 1 -- 172.21.15.146:0/945163916 wait complete. 2024-09-16T01:53:47.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.672+0000 7fabbae00640 1 Processor -- start 2024-09-16T01:53:47.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.672+0000 7fabbae00640 1 -- start start 2024-09-16T01:53:47.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabbae00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406ce90 0x7fabb40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabbae00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 0x7fabb4147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabbae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabb40c6840 0x7fabb414b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabbae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fabb40756f0 con 0x7fabb40c6840 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabbae00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fabb406e9c0 con 0x7fabb406ce90 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabbae00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fabb4072cd0 con 0x7fabb406d270 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabb3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406ce90 0x7fabb40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabb3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 0x7fabb4147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabb8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabb40c6840 0x7fabb414b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabb3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 0x7fabb4147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:46418/0 (socket says 172.21.15.146:46418) 2024-09-16T01:53:47.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabb3400640 1 -- 172.21.15.146:0/2927993220 learned_addr learned my addr 172.21.15.146:0/2927993220 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.673+0000 7fabb3400640 1 -- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406ce90 msgr2=0x7fabb40c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabb3400640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406ce90 0x7fabb40c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabb3400640 1 -- 172.21.15.146:0/2927993220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabb40c6840 msgr2=0x7fabb414b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabb3400640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabb40c6840 0x7fabb414b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabb3400640 1 -- 172.21.15.146:0/2927993220 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb41f4bb0 con 0x7fabb406d270 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabb8e00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabb40c6840 0x7fabb414b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabb3400640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 0x7fabb4147090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faba0035040 tx=0x7faba00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0037040 con 0x7fabb406d270 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba0030770 con 0x7fabb406d270 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0030b80 con 0x7fabb406d270 2024-09-16T01:53:47.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb415b540 con 0x7fabb406d270 2024-09-16T01:53:47.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.674+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb41f40d0 con 0x7fabb406d270 2024-09-16T01:53:47.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.675+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb4071c50 con 0x7fabb406d270 2024-09-16T01:53:47.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.676+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faba0044020 con 0x7fabb406d270 2024-09-16T01:53:47.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.677+0000 7fababe00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab9406c2b0 0x7fab9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:47.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.677+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7faba00ab980 con 0x7fabb406d270 2024-09-16T01:53:47.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.677+0000 7fabb3e00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab9406c2b0 0x7fab9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:47.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.678+0000 7fabb3e00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab9406c2b0 0x7fab9406e770 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faba4001780 tx=0x7faba4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:47.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.682+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fabb4071c50 con 0x7fabb406d270 2024-09-16T01:53:47.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.854+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fabb415b2f0 con 0x7fab9406c2b0 2024-09-16T01:53:47.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.855+0000 7fababe00640 1 -- 172.21.15.146:0/2927993220 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fabb415b2f0 con 0x7fab9406c2b0 2024-09-16T01:53:47.857 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab9406c2b0 msgr2=0x7fab9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab9406c2b0 0x7fab9406e770 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7faba4001780 tx=0x7faba4009040 comp rx=0 tx=0).stop 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 msgr2=0x7fabb4147090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 0x7fabb4147090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7faba0035040 tx=0x7faba00046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 shutdown_connections 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fab9406c2b0 0x7fab9406e770 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabb40c6840 0x7fabb414b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fabb406d270 0x7fabb4147090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 --2- 172.21.15.146:0/2927993220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fabb406ce90 0x7fabb40c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:47.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.857+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 >> 172.21.15.146:0/2927993220 conn(0x7fabb406d8e0 msgr2=0x7fabb4158880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:47.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.858+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 shutdown_connections 2024-09-16T01:53:47.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:47.858+0000 7fabbae00640 1 -- 172.21.15.146:0/2927993220 wait complete. 2024-09-16T01:53:47.876 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T01:53:47.876 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:48.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.033+0000 7fbcea800640 1 Processor -- start 2024-09-16T01:53:48.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.033+0000 7fbcea800640 1 -- start start 2024-09-16T01:53:48.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcea800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4151900 0x7fbce40c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcea800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbce4072ea0 0x7fbce415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcea800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce4071b40 con 0x7fbce4151ce0 2024-09-16T01:53:48.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcea800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbce406f250 con 0x7fbce4072ea0 2024-09-16T01:53:48.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcea800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbce4073860 con 0x7fbce4151900 2024-09-16T01:53:48.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcdaa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbce4072ea0 0x7fbce415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbce3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4151900 0x7fbce40c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbce3e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbce4151ce0 0x7fbce40c5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57354/0 (socket says 172.21.15.146:57354) 2024-09-16T01:53:48.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbcdaa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbce4072ea0 0x7fbce415ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55188/0 (socket says 172.21.15.146:55188) 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.034+0000 7fbce3e00640 1 -- 172.21.15.146:0/3147298457 learned_addr learned my addr 172.21.15.146:0/3147298457 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.035+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4151900 msgr2=0x7fbce40c35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.035+0000 7fbcdaa00640 1 --2- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4151900 0x7fbce40c35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.035+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3147298457 >> v1:172.21.15.161:6789/0 conn(0x7fbce4151ce0 legacy=0x7fbce40c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.035+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3147298457 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce4071e20 con 0x7fbce4072ea0 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.035+0000 7fbce3400640 1 --2- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4151900 0x7fbce40c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:48.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.035+0000 7fbce2a00640 1 -- 172.21.15.146:0/3147298457 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2358518436 0 0) 0x7fbce4071b40 con 0x7fbce4151ce0 2024-09-16T01:53:48.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.036+0000 7fbcdaa00640 1 --2- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4072ea0 0x7fbce415ad70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097a0 tx=0x7fbccc02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=668fad212b219ffc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.036+0000 7fbce2a00640 1 -- 172.21.15.146:0/3147298457 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc037070 con 0x7fbce4072ea0 2024-09-16T01:53:48.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.036+0000 7fbce2a00640 1 -- 172.21.15.146:0/3147298457 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbccc02ae50 con 0x7fbce4072ea0 2024-09-16T01:53:48.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.036+0000 7fbce2a00640 1 -- 172.21.15.146:0/3147298457 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc032660 con 0x7fbce4072ea0 2024-09-16T01:53:48.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 -- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4072ea0 msgr2=0x7fbce415ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 --2- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4072ea0 0x7fbce415ad70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097a0 tx=0x7fbccc02a100 comp rx=0 tx=0).stop 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 -- 172.21.15.146:0/3147298457 shutdown_connections 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 --2- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4072ea0 0x7fbce415ad70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 --2- 172.21.15.146:0/3147298457 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4151900 0x7fbce40c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 -- 172.21.15.146:0/3147298457 >> 172.21.15.146:0/3147298457 conn(0x7fbce40cc0b0 msgr2=0x7fbce40cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 -- 172.21.15.146:0/3147298457 shutdown_connections 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.037+0000 7fbcea800640 1 -- 172.21.15.146:0/3147298457 wait complete. 2024-09-16T01:53:48.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.038+0000 7fbcea800640 1 Processor -- start 2024-09-16T01:53:48.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 -- start start 2024-09-16T01:53:48.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4072ea0 0x7fbce41e3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4151900 0x7fbce4145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce4151ce0 0x7fbce414a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbce4073cf0 con 0x7fbce4151ce0 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbce40c3bd0 con 0x7fbce4151900 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.039+0000 7fbcea800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbce40c5890 con 0x7fbce4072ea0 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4151900 0x7fbce4145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbce3400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4072ea0 0x7fbce41e3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4151900 0x7fbce4145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55194/0 (socket says 172.21.15.146:55194) 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbce3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce4151ce0 0x7fbce414a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3340906407 learned_addr learned my addr 172.21.15.146:0/3340906407 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:48.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4072ea0 msgr2=0x7fbce41e3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbce4072ea0 0x7fbce41e3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3340906407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce4151ce0 msgr2=0x7fbce414a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbce4151ce0 0x7fbce414a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 -- 172.21.15.146:0/3340906407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce41f12a0 con 0x7fbce4151900 2024-09-16T01:53:48.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.040+0000 7fbcdaa00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4151900 0x7fbce4145e40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbccc000c00 tx=0x7fbccc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.041+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc037040 con 0x7fbce4151900 2024-09-16T01:53:48.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.041+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbccc004030 con 0x7fbce4151900 2024-09-16T01:53:48.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.041+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc004400 con 0x7fbce4151900 2024-09-16T01:53:48.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.041+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce41f1540 con 0x7fbce4151900 2024-09-16T01:53:48.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.041+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce41f1b00 con 0x7fbce4151900 2024-09-16T01:53:48.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.041+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbce4146380 con 0x7fbce4151900 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.043+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbccc003610 con 0x7fbce4151900 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbcdbe00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc006c2e0 0x7fbcc006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7fbccc0acac0 con 0x7fbce4151900 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbce3400640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc006c2e0 0x7fbcc006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbcdbe00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbcc00746e0 0x7fbcc0076b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbccc036e50 con 0x7fbcc00746e0 2024-09-16T01:53:48.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7fbccc0ace70 con 0x7fbce4151900 2024-09-16T01:53:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbce3e00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbcc00746e0 0x7fbcc0076b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.044+0000 7fbce3400640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc006c2e0 0x7fbcc006e7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009560 tx=0x7fbcd4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.045+0000 7fbce3e00640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbcc00746e0 0x7fbcc0076b80 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.046+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== osd.6 v2:172.21.15.161:6808/3661001615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbccc036e50 con 0x7fbcc00746e0 2024-09-16T01:53:48.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.058+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 --> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbce41f1d40 con 0x7fbcc00746e0 2024-09-16T01:53:48.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcdbe00640 1 -- 172.21.15.146:0/3340906407 <== osd.6 v2:172.21.15.161:6808/3661001615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbce41f1d40 con 0x7fbcc00746e0 2024-09-16T01:53:48.061 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:48.061 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:48.061 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:48.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbcc00746e0 msgr2=0x7fbcc0076b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcea800640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6808/3661001615,v1:172.21.15.161:6809/3661001615] conn(0x7fbcc00746e0 0x7fbcc0076b80 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc006c2e0 msgr2=0x7fbcc006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcea800640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fbcc006c2e0 0x7fbcc006e7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4009560 tx=0x7fbcd4009190 comp rx=0 tx=0).stop 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4151900 msgr2=0x7fbce4145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbcea800640 1 --2- 172.21.15.146:0/3340906407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbce4151900 0x7fbce4145e40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbccc000c00 tx=0x7fbccc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.059+0000 7fbce3e00640 1 -- 172.21.15.146:0/3340906407 reap_dead start 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.060+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 shutdown_connections 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.060+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 >> 172.21.15.146:0/3340906407 conn(0x7fbce40cc0b0 msgr2=0x7fbce40c5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.060+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 shutdown_connections 2024-09-16T01:53:48.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.060+0000 7fbcea800640 1 -- 172.21.15.146:0/3340906407 wait complete. 2024-09-16T01:53:48.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T01:53:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.228+0000 7f0477a00640 1 Processor -- start 2024-09-16T01:53:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.228+0000 7f0477a00640 1 -- start start 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.229+0000 7f0477a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04700c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0477a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f04700c7a60 0x7f047006dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0477a00640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0470155110 con 0x7f0470154d30 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0477a00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f04700c6bb0 con 0x7f04700c7a60 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0477a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0470073140 con 0x7f04700c7f30 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0475a00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f04700c7a60 0x7f047006dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0467e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04700c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0475000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0470154d30 0x7f04700c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:57360/0 (socket says 172.21.15.146:57360) 2024-09-16T01:53:48.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0475000640 1 -- 172.21.15.146:0/1779259249 learned_addr learned my addr 172.21.15.146:0/1779259249 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:48.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0467e00640 1 -- 172.21.15.146:0/1779259249 >> v2:172.21.15.161:3300/0 conn(0x7f04700c7a60 msgr2=0x7f047006dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0467e00640 1 --2- 172.21.15.146:0/1779259249 >> v2:172.21.15.161:3300/0 conn(0x7f04700c7a60 0x7f047006dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0467e00640 1 -- 172.21.15.146:0/1779259249 >> v1:172.21.15.161:6789/0 conn(0x7f0470154d30 legacy=0x7f04700c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f046fe00640 1 -- 172.21.15.146:0/1779259249 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786849703 0 0) 0x7f0470155110 con 0x7f0470154d30 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0467e00640 1 -- 172.21.15.146:0/1779259249 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04700c6e00 con 0x7f04700c7f30 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.230+0000 7f0475a00640 1 --2- 172.21.15.146:0/1779259249 >> v2:172.21.15.161:3300/0 conn(0x7f04700c7a60 0x7f047006dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.231+0000 7f0467e00640 1 --2- 172.21.15.146:0/1779259249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04700c9570 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f04580097d0 tx=0x7f045802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e305d53f1bd29c37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f046fe00640 1 -- 172.21.15.146:0/1779259249 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458037070 con 0x7f04700c7f30 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f046fe00640 1 -- 172.21.15.146:0/1779259249 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0458032440 con 0x7f04700c7f30 2024-09-16T01:53:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f046fe00640 1 -- 172.21.15.146:0/1779259249 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458032810 con 0x7f04700c7f30 2024-09-16T01:53:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 -- 172.21.15.146:0/1779259249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 msgr2=0x7f04700c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 --2- 172.21.15.146:0/1779259249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04700c9570 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f04580097d0 tx=0x7f045802a250 comp rx=0 tx=0).stop 2024-09-16T01:53:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 -- 172.21.15.146:0/1779259249 shutdown_connections 2024-09-16T01:53:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 --2- 172.21.15.146:0/1779259249 >> v2:172.21.15.161:3300/0 conn(0x7f04700c7a60 0x7f047006dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 --2- 172.21.15.146:0/1779259249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04700c9570 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 -- 172.21.15.146:0/1779259249 >> 172.21.15.146:0/1779259249 conn(0x7f047006f3c0 msgr2=0x7f047006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 -- 172.21.15.146:0/1779259249 shutdown_connections 2024-09-16T01:53:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.232+0000 7f0477a00640 1 -- 172.21.15.146:0/1779259249 wait complete. 2024-09-16T01:53:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.234+0000 7f0477a00640 1 Processor -- start 2024-09-16T01:53:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.234+0000 7f0477a00640 1 -- start start 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0477a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04701f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0477a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0470154d30 0x7f047014dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0477a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04700cc190 0x7f04700cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0477a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0470072e00 con 0x7f04700cc190 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0477a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0470072920 con 0x7f0470154d30 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0477a00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f04700718c0 con 0x7f04700c7f30 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0475000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04701f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0467e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0470154d30 0x7f047014dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0475a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04700cc190 0x7f04700cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0467e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0470154d30 0x7f047014dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55212/0 (socket says 172.21.15.146:55212) 2024-09-16T01:53:48.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.235+0000 7f0467e00640 1 -- 172.21.15.146:0/3431522553 learned_addr learned my addr 172.21.15.146:0/3431522553 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:48.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0475000640 1 -- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0470154d30 msgr2=0x7f047014dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0475000640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0470154d30 0x7f047014dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0475000640 1 -- 172.21.15.146:0/3431522553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04700cc190 msgr2=0x7f04700cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0475000640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04700cc190 0x7f04700cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0475000640 1 -- 172.21.15.146:0/3431522553 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04700ccb60 con 0x7f04700c7f30 2024-09-16T01:53:48.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0475000640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04701f0fd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f046000c410 tx=0x7f046000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04600074e0 con 0x7f04700c7f30 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04600047e0 con 0x7f04700c7f30 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460004bd0 con 0x7f04700c7f30 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f047014bca0 con 0x7f04700c7f30 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.236+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04700cb6b0 con 0x7f04700c7f30 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.237+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04700c9570 con 0x7f04700c7f30 2024-09-16T01:53:48.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.239+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0460011070 con 0x7f04700c7f30 2024-09-16T01:53:48.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.239+0000 7f046d600640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f044c06c2e0 0x7f044c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.239+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f046008af10 con 0x7f04700c7f30 2024-09-16T01:53:48.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.240+0000 7f0467e00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f044c06c2e0 0x7f044c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.240+0000 7f0467e00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f044c06c2e0 0x7f044c06e7a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f04700c4270 tx=0x7f045802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.244+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0460059030 con 0x7f04700c7f30 2024-09-16T01:53:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.420+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f047014ba50 con 0x7f044c06c2e0 2024-09-16T01:53:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.420+0000 7f046d600640 1 -- 172.21.15.146:0/3431522553 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f047014ba50 con 0x7f044c06c2e0 2024-09-16T01:53:48.422 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T01:53:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f044c06c2e0 msgr2=0x7f044c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f044c06c2e0 0x7f044c06e7a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f04700c4270 tx=0x7f045802e000 comp rx=0 tx=0).stop 2024-09-16T01:53:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 msgr2=0x7f04701f0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04701f0fd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f046000c410 tx=0x7f046000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 shutdown_connections 2024-09-16T01:53:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f044c06c2e0 0x7f044c06e7a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04700cc190 0x7f04700cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0470154d30 0x7f047014dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 --2- 172.21.15.146:0/3431522553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f04700c7f30 0x7f04701f0fd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.422+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 >> 172.21.15.146:0/3431522553 conn(0x7f047006f3c0 msgr2=0x7f0470153e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.423+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 shutdown_connections 2024-09-16T01:53:48.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.423+0000 7f0477a00640 1 -- 172.21.15.146:0/3431522553 wait complete. 2024-09-16T01:53:48.441 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T01:53:48.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:53:48.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.601+0000 7ff4a5e00640 1 Processor -- start 2024-09-16T01:53:48.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.601+0000 7ff4a5e00640 1 -- start start 2024-09-16T01:53:48.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5e00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4a0154ab0 0x7ff4a015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00cd900 0x7ff4a015ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a00c5540 con 0x7ff4a00c5000 2024-09-16T01:53:48.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5e00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a00b55d0 con 0x7ff4a0154ab0 2024-09-16T01:53:48.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a0154e90 con 0x7ff4a00cd900 2024-09-16T01:53:48.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff4a00c5000 0x7ff4a00cbb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44348/0 (socket says 172.21.15.146:44348) 2024-09-16T01:53:48.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.602+0000 7ff4a5400640 1 -- 172.21.15.146:0/74821515 learned_addr learned my addr 172.21.15.146:0/74821515 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:48.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff49fe00640 1 --2- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00cd900 0x7ff4a015ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff4a4a00640 1 --2- 172.21.15.146:0/74821515 >> v2:172.21.15.161:3300/0 conn(0x7ff4a0154ab0 0x7ff4a015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff49f400640 1 -- 172.21.15.146:0/74821515 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 186920808 0 0) 0x7ff4a00c5540 con 0x7ff4a00c5000 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff49f400640 1 -- 172.21.15.146:0/74821515 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff488003520 con 0x7ff4a00c5000 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff4a4a00640 1 -- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00cd900 msgr2=0x7ff4a015ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff4a4a00640 1 --2- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00cd900 0x7ff4a015ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff4a4a00640 1 -- 172.21.15.146:0/74821515 >> v1:172.21.15.146:6789/0 conn(0x7ff4a00c5000 legacy=0x7ff4a00cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.603+0000 7ff4a4a00640 1 -- 172.21.15.146:0/74821515 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a015ddd0 con 0x7ff4a0154ab0 2024-09-16T01:53:48.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.604+0000 7ff4a4a00640 1 --2- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a0154ab0 0x7ff4a015a8c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff4940097a0 tx=0x7ff49402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ede7a5fc51b3e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.604+0000 7ff49f400640 1 -- 172.21.15.146:0/74821515 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494037070 con 0x7ff4a0154ab0 2024-09-16T01:53:48.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.604+0000 7ff49f400640 1 -- 172.21.15.146:0/74821515 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff494032440 con 0x7ff4a0154ab0 2024-09-16T01:53:48.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.604+0000 7ff49f400640 1 -- 172.21.15.146:0/74821515 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494032810 con 0x7ff4a0154ab0 2024-09-16T01:53:48.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 -- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a0154ab0 msgr2=0x7ff4a015a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 --2- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a0154ab0 0x7ff4a015a8c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff4940097a0 tx=0x7ff49402a210 comp rx=0 tx=0).stop 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 -- 172.21.15.146:0/74821515 shutdown_connections 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 --2- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00cd900 0x7ff4a015ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 --2- 172.21.15.146:0/74821515 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a0154ab0 0x7ff4a015a8c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 -- 172.21.15.146:0/74821515 >> 172.21.15.146:0/74821515 conn(0x7ff4a0067950 msgr2=0x7ff4a0067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 -- 172.21.15.146:0/74821515 shutdown_connections 2024-09-16T01:53:48.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.605+0000 7ff4a5e00640 1 -- 172.21.15.146:0/74821515 wait complete. 2024-09-16T01:53:48.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.607+0000 7ff4a5e00640 1 Processor -- start 2024-09-16T01:53:48.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.607+0000 7ff4a5e00640 1 -- start start 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00c5000 0x7ff4a00c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a00cd900 0x7ff4a014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a0154ab0 0x7ff4a014afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a015e670 con 0x7ff4a0154ab0 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a015e390 con 0x7ff4a00cd900 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5e00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a015e950 con 0x7ff4a00c5000 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00c5000 0x7ff4a00c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a00cd900 0x7ff4a014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a00cd900 0x7ff4a014f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:55222/0 (socket says 172.21.15.146:55222) 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a4a00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00c5000 0x7ff4a00c7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:46452/0 (socket says 172.21.15.146:46452) 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 -- 172.21.15.146:0/366508683 learned_addr learned my addr 172.21.15.146:0/366508683 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:48.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a5400640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a0154ab0 0x7ff4a014afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 -- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00c5000 msgr2=0x7ff4a00c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00c5000 0x7ff4a00c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 -- 172.21.15.146:0/366508683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a0154ab0 msgr2=0x7ff4a014afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4a0154ab0 0x7ff4a014afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff49fe00640 1 -- 172.21.15.146:0/366508683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a014c3e0 con 0x7ff4a00cd900 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.608+0000 7ff4a4a00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4a00c5000 0x7ff4a00c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.609+0000 7ff49fe00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a00cd900 0x7ff4a014f7e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00e9b0 tx=0x7ff48c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.609+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c017070 con 0x7ff4a00cd900 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.609+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48c00c9a0 con 0x7ff4a00cd900 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.609+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a014c650 con 0x7ff4a00cd900 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.609+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00cd90 con 0x7ff4a00cd900 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.609+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a01f3b20 con 0x7ff4a00cd900 2024-09-16T01:53:48.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.610+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff4a00c63a0 con 0x7ff4a00cd900 2024-09-16T01:53:48.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.611+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff48c01a070 con 0x7ff4a00cd900 2024-09-16T01:53:48.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.612+0000 7ff49cc00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff48006c2e0 0x7ff48006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.612+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7ff48c08b770 con 0x7ff4a00cd900 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.612+0000 7ff49cc00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff480074740 0x7ff480076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.612+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4800772b0 con 0x7ff480074740 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.612+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7ff48c08bc00 con 0x7ff4a00cd900 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.612+0000 7ff4a4a00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff48006c2e0 0x7ff48006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.613+0000 7ff4a5400640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff480074740 0x7ff480076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.613+0000 7ff4a4a00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff48006c2e0 0x7ff48006e7a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff494034000 tx=0x7ff49402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.613+0000 7ff4a5400640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff480074740 0x7ff480076be0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.614+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== osd.7 v2:172.21.15.161:6816/1279166573 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4800772b0 con 0x7ff480074740 2024-09-16T01:53:48.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.627+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 --> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff4a0153d80 con 0x7ff480074740 2024-09-16T01:53:48.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff49cc00640 1 -- 172.21.15.146:0/366508683 <== osd.7 v2:172.21.15.161:6816/1279166573 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff4a0153d80 con 0x7ff480074740 2024-09-16T01:53:48.630 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-09-16T01:53:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:53:48.631 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-09-16T01:53:48.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff480074740 msgr2=0x7ff480076be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6816/1279166573,v1:172.21.15.161:6817/1279166573] conn(0x7ff480074740 0x7ff480076be0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff48006c2e0 msgr2=0x7ff48006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7ff48006c2e0 0x7ff48006e7a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff494034000 tx=0x7ff49402e000 comp rx=0 tx=0).stop 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a00cd900 msgr2=0x7ff4a014f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 --2- 172.21.15.146:0/366508683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4a00cd900 0x7ff4a014f7e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00e9b0 tx=0x7ff48c00ee70 comp rx=0 tx=0).stop 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5400640 1 -- 172.21.15.146:0/366508683 reap_dead start 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 shutdown_connections 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 >> 172.21.15.146:0/366508683 conn(0x7ff4a0067950 msgr2=0x7ff4a00cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 shutdown_connections 2024-09-16T01:53:48.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.628+0000 7ff4a5e00640 1 -- 172.21.15.146:0/366508683 wait complete. 2024-09-16T01:53:48.645 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T01:53:48.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.800+0000 7fe757200640 1 Processor -- start 2024-09-16T01:53:48.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.800+0000 7fe757200640 1 -- start start 2024-09-16T01:53:48.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.801+0000 7fe757200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c6b70 0x7fe75815b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.801+0000 7fe757200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75806d140 con 0x7fe7580c75a0 2024-09-16T01:53:48.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.801+0000 7fe757200640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75806d420 con 0x7fe7580cc040 2024-09-16T01:53:48.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.801+0000 7fe757200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe75815b880 con 0x7fe7580c6b70 2024-09-16T01:53:48.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.801+0000 7fe74fe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe7580c75a0 0x7fe7580cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44364/0 (socket says 172.21.15.146:44364) 2024-09-16T01:53:48.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.801+0000 7fe74fe00640 1 -- 172.21.15.146:0/3471416693 learned_addr learned my addr 172.21.15.146:0/3471416693 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:48.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755e00640 1 --2- 172.21.15.146:0/3471416693 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c6b70 0x7fe75815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008981238 0 0) 0x7fe75806d140 con 0x7fe7580c75a0 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe73c003520 con 0x7fe7580c75a0 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2986259012 0 0) 0x7fe75806d420 con 0x7fe7580cc040 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe75806d140 con 0x7fe7580cc040 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755e00640 1 -- 172.21.15.146:0/3471416693 >> v1:172.21.15.161:6789/0 conn(0x7fe7580cc040 legacy=0x7fe758165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755e00640 1 -- 172.21.15.146:0/3471416693 >> v1:172.21.15.146:6789/0 conn(0x7fe7580c75a0 legacy=0x7fe7580cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.802+0000 7fe755e00640 1 -- 172.21.15.146:0/3471416693 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75806d700 con 0x7fe7580c6b70 2024-09-16T01:53:48.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.803+0000 7fe755e00640 1 --2- 172.21.15.146:0/3471416693 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c6b70 0x7fe75815b340 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe7480097d0 tx=0x7fe74802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=838ead6889fcec32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.803+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748037070 con 0x7fe7580c6b70 2024-09-16T01:53:48.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.803+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe748032440 con 0x7fe7580c6b70 2024-09-16T01:53:48.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.803+0000 7fe755400640 1 -- 172.21.15.146:0/3471416693 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748032810 con 0x7fe7580c6b70 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 -- 172.21.15.146:0/3471416693 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c6b70 msgr2=0x7fe75815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 --2- 172.21.15.146:0/3471416693 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c6b70 0x7fe75815b340 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe7480097d0 tx=0x7fe74802a180 comp rx=0 tx=0).stop 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 -- 172.21.15.146:0/3471416693 shutdown_connections 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 --2- 172.21.15.146:0/3471416693 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c6b70 0x7fe75815b340 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 -- 172.21.15.146:0/3471416693 >> 172.21.15.146:0/3471416693 conn(0x7fe7580ccc20 msgr2=0x7fe7580cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 -- 172.21.15.146:0/3471416693 shutdown_connections 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.804+0000 7fe757200640 1 -- 172.21.15.146:0/3471416693 wait complete. 2024-09-16T01:53:48.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 Processor -- start 2024-09-16T01:53:48.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 -- start start 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7580c6b70 0x7fe7580c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c75a0 0x7fe7581567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 0x7fe758156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe7580cb050 con 0x7fe7580cc040 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe7580cab20 con 0x7fe7580c6b70 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.806+0000 7fe757200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe7580cbc80 con 0x7fe7580c75a0 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 0x7fe758156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 0x7fe758156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43278/0 (socket says 172.21.15.146:43278) 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 -- 172.21.15.146:0/973673208 learned_addr learned my addr 172.21.15.146:0/973673208 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe74fe00640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c75a0 0x7fe7581567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe755e00640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7580c6b70 0x7fe7580c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 -- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c75a0 msgr2=0x7fe7581567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c75a0 0x7fe7581567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 -- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7580c6b70 msgr2=0x7fe7580c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7580c6b70 0x7fe7580c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 -- 172.21.15.146:0/973673208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7581f0800 con 0x7fe7580cc040 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe755e00640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7580c6b70 0x7fe7580c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.807+0000 7fe756800640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 0x7fe758156ce0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe75000c7d0 tx=0x7fe75000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.808+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe750003b70 con 0x7fe7580cc040 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.808+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe750003d10 con 0x7fe7580cc040 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.808+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75000aa90 con 0x7fe7580cc040 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.808+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7581f0ad0 con 0x7fe7580cc040 2024-09-16T01:53:48.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.808+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7581f1090 con 0x7fe7580cc040 2024-09-16T01:53:48.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.809+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe75806cd20 con 0x7fe7580cc040 2024-09-16T01:53:48.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.810+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe75000ac30 con 0x7fe7580cc040 2024-09-16T01:53:48.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.811+0000 7fe74e000640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73406c2e0 0x7fe73406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:48.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.811+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7fe75008b490 con 0x7fe7580cc040 2024-09-16T01:53:48.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.811+0000 7fe755e00640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73406c2e0 0x7fe73406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:48.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.812+0000 7fe755e00640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73406c2e0 0x7fe73406e7a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe748034000 tx=0x7fe74802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:48.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.815+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe750059b50 con 0x7fe7580cc040 2024-09-16T01:53:48.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.990+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fe7580abcd0 con 0x7fe73406c2e0 2024-09-16T01:53:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.991+0000 7fe74e000640 1 -- 172.21.15.146:0/973673208 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe748032650 con 0x7fe73406c2e0 2024-09-16T01:53:48.994 INFO:teuthology.orchestra.run.smithi146.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T01:53:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73406c2e0 msgr2=0x7fe73406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73406c2e0 0x7fe73406e7a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe748034000 tx=0x7fe74802e000 comp rx=0 tx=0).stop 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 msgr2=0x7fe758156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 0x7fe758156ce0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe75000c7d0 tx=0x7fe75000cc90 comp rx=0 tx=0).stop 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 shutdown_connections 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fe73406c2e0 0x7fe73406e7a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7580cc040 0x7fe758156ce0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe7580c75a0 0x7fe7581567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 --2- 172.21.15.146:0/973673208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7580c6b70 0x7fe7580c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 >> 172.21.15.146:0/973673208 conn(0x7fe7580ccc20 msgr2=0x7fe75815a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 shutdown_connections 2024-09-16T01:53:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:48.993+0000 7fe757200640 1 -- 172.21.15.146:0/973673208 wait complete. 2024-09-16T01:53:49.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:53:49.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.165+0000 7fc506200640 1 Processor -- start 2024-09-16T01:53:49.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.165+0000 7fc506200640 1 -- start start 2024-09-16T01:53:49.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc506200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc5000c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:49.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc506200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5000c6410 0x7fc500158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:49.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc506200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc500151e70 con 0x7fc5000c5280 2024-09-16T01:53:49.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc506200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7fc5000c8130 con 0x7fc5000c6410 2024-09-16T01:53:49.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc506200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5000c8410 con 0x7fc500151a90 2024-09-16T01:53:49.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4ffe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc5000c5280 0x7fc50015aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:44368/0 (socket says 172.21.15.146:44368) 2024-09-16T01:53:49.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4f6a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5000c6410 0x7fc500158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:49.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4ffe00640 1 -- 172.21.15.146:0/1348226217 learned_addr learned my addr 172.21.15.146:0/1348226217 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:53:49.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4ff400640 1 --2- 172.21.15.146:0/1348226217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc5000c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:49.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4fea00640 1 -- 172.21.15.146:0/1348226217 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1820368612 0 0) 0x7fc500151e70 con 0x7fc5000c5280 2024-09-16T01:53:49.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4f6a00640 1 -- 172.21.15.146:0/1348226217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 msgr2=0x7fc5000c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:53:49.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4f6a00640 1 --2- 172.21.15.146:0/1348226217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc5000c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.167+0000 7fc4f6a00640 1 -- 172.21.15.146:0/1348226217 >> v1:172.21.15.146:6789/0 conn(0x7fc5000c5280 legacy=0x7fc50015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:49.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.168+0000 7fc4f6a00640 1 -- 172.21.15.146:0/1348226217 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5000c86f0 con 0x7fc5000c6410 2024-09-16T01:53:49.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.168+0000 7fc4f6a00640 1 --2- 172.21.15.146:0/1348226217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc500158040 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800ba40 tx=0x7fc4e802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d294b941f9cdbb92 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:49.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc4fea00640 1 -- 172.21.15.146:0/1348226217 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8032750 con 0x7fc5000c6410 2024-09-16T01:53:49.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc4fea00640 1 -- 172.21.15.146:0/1348226217 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e8030050 con 0x7fc5000c6410 2024-09-16T01:53:49.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc4fea00640 1 -- 172.21.15.146:0/1348226217 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8032b10 con 0x7fc5000c6410 2024-09-16T01:53:49.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 -- 172.21.15.146:0/1348226217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 msgr2=0x7fc500158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:49.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 --2- 172.21.15.146:0/1348226217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc500158040 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800ba40 tx=0x7fc4e802c850 comp rx=0 tx=0).stop 2024-09-16T01:53:49.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 -- 172.21.15.146:0/1348226217 shutdown_connections 2024-09-16T01:53:49.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 --2- 172.21.15.146:0/1348226217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc500158040 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 --2- 172.21.15.146:0/1348226217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc5000c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 -- 172.21.15.146:0/1348226217 >> 172.21.15.146:0/1348226217 conn(0x7fc50006f3c0 msgr2=0x7fc50006f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:49.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 -- 172.21.15.146:0/1348226217 shutdown_connections 2024-09-16T01:53:49.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.169+0000 7fc506200640 1 -- 172.21.15.146:0/1348226217 wait complete. 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.171+0000 7fc506200640 1 Processor -- start 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.171+0000 7fc506200640 1 -- start start 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc506200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5000c5280 0x7fc5001e3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc506200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc5001e1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc506200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc50014a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc506200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc500157b20 con 0x7fc5000c6410 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc506200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc500157840 con 0x7fc5000c5280 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc506200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5000c3bf0 con 0x7fc500151a90 2024-09-16T01:53:49.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc4f6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc5001e1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc4f6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc5001e1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43296/0 (socket says 172.21.15.146:43296) 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc4f6a00640 1 -- 172.21.15.146:0/665825542 learned_addr learned my addr 172.21.15.146:0/665825542 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc4ff400640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5000c5280 0x7fc5001e3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.172+0000 7fc4ffe00640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc50014a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f6a00640 1 -- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 msgr2=0x7fc50014a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f6a00640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc50014a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f6a00640 1 -- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5000c5280 msgr2=0x7fc5001e3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f6a00640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5000c5280 0x7fc5001e3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f6a00640 1 -- 172.21.15.146:0/665825542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5001f4100 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4ffe00640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc50014a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f6a00640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc5001e1210 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800bb60 tx=0x7fc4e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80349d0 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e8044070 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8034dd0 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5001f43e0 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.173+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5001f49a0 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.174+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5000c71f0 con 0x7fc5000c6410 2024-09-16T01:53:49.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.175+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc4e803c040 con 0x7fc5000c6410 2024-09-16T01:53:49.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.176+0000 7fc4f7e00640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc4dc06c2b0 0x7fc4dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:53:49.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.176+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7fc4e80ac270 con 0x7fc5000c6410 2024-09-16T01:53:49.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.176+0000 7fc4ff400640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc4dc06c2b0 0x7fc4dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:53:49.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.177+0000 7fc4ff400640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc4dc06c2b0 0x7fc4dc06e770 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009aa0 tx=0x7fc4f0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:53:49.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.180+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc4e807a800 con 0x7fc5000c6410 2024-09-16T01:53:49.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.352+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc500155ce0 con 0x7fc4dc06c2b0 2024-09-16T01:53:49.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.358+0000 7fc4f7e00640 1 -- 172.21.15.146:0/665825542 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252915 (secure 0 0 0) 0x7fc500155ce0 con 0x7fc4dc06c2b0 2024-09-16T01:53:49.363 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:53:49.364 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:53:49.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc4dc06c2b0 msgr2=0x7fc4dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:49.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc4dc06c2b0 0x7fc4dc06e770 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009aa0 tx=0x7fc4f0009190 comp rx=0 tx=0).stop 2024-09-16T01:53:49.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 msgr2=0x7fc5001e1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:53:49.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc5001e1210 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800bb60 tx=0x7fc4e80046c0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 shutdown_connections 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fc4dc06c2b0 0x7fc4dc06e770 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc500151a90 0x7fc50014a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5000c6410 0x7fc5001e1210 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 --2- 172.21.15.146:0/665825542 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5000c5280 0x7fc5001e3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 >> 172.21.15.146:0/665825542 conn(0x7fc50006f3c0 msgr2=0x7fc5000c7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 shutdown_connections 2024-09-16T01:53:49.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:53:49.362+0000 7fc506200640 1 -- 172.21.15.146:0/665825542 wait complete. 2024-09-16T01:53:49.382 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T01:53:48.576181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":989271,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9803,"num_write_kb":25168843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4746,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11764,"ondisk_log_size":11764,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289952,"kb_used_data":15460,"kb_used_omap":12,"kb_used_meta":274291,"kb_avail":754684768,"statfs":{"total":773094113280,"available":772797202432,"internally_reserved":0,"allocated":15831040,"data_stored":9160965,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":280874544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.926257"},"pg_stats":[{"pgid":"3.3f","version":"152'167","reported_seq":503,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281838+0000","last_change":"2024-09-16T01:53:22.525372+0000","last_active":"2024-09-16T01:53:46.281838+0000","last_peered":"2024-09-16T01:53:46.281838+0000","last_clean":"2024-09-16T01:53:46.281838+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:46.281838+0000","last_undegraded":"2024-09-16T01:53:46.281838+0000","last_fullsized":"2024-09-16T01:53:46.281838+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"152'152","reported_seq":435,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510571+0000","last_change":"2024-09-16T01:48:58.825609+0000","last_active":"2024-09-16T01:53:45.510571+0000","last_peered":"2024-09-16T01:53:45.510571+0000","last_clean":"2024-09-16T01:53:45.510571+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:45.510571+0000","last_undegraded":"2024-09-16T01:53:45.510571+0000","last_fullsized":"2024-09-16T01:53:45.510571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:32.275503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"152'178","reported_seq":816,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510485+0000","last_change":"2024-09-16T01:53:23.518810+0000","last_active":"2024-09-16T01:53:45.510485+0000","last_peered":"2024-09-16T01:53:45.510485+0000","last_clean":"2024-09-16T01:53:45.510485+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:53:45.510485+0000","last_undegraded":"2024-09-16T01:53:45.510485+0000","last_fullsized":"2024-09-16T01:53:45.510485+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"152'193","reported_seq":601,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.280685+0000","last_change":"2024-09-16T01:53:32.511261+0000","last_active":"2024-09-16T01:53:46.280685+0000","last_peered":"2024-09-16T01:53:46.280685+0000","last_clean":"2024-09-16T01:53:46.280685+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:46.280685+0000","last_undegraded":"2024-09-16T01:53:46.280685+0000","last_fullsized":"2024-09-16T01:53:46.280685+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"152'174","reported_seq":537,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281213+0000","last_change":"2024-09-16T01:53:22.521371+0000","last_active":"2024-09-16T01:53:46.281213+0000","last_peered":"2024-09-16T01:53:46.281213+0000","last_clean":"2024-09-16T01:53:46.281213+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:46.281213+0000","last_undegraded":"2024-09-16T01:53:46.281213+0000","last_fullsized":"2024-09-16T01:53:46.281213+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"152'162","reported_seq":592,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.280566+0000","last_change":"2024-09-16T01:53:23.521468+0000","last_active":"2024-09-16T01:53:46.280566+0000","last_peered":"2024-09-16T01:53:46.280566+0000","last_clean":"2024-09-16T01:53:46.280566+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:46.280566+0000","last_undegraded":"2024-09-16T01:53:46.280566+0000","last_fullsized":"2024-09-16T01:53:46.280566+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"152'188","reported_seq":649,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281152+0000","last_change":"2024-09-16T01:53:22.523352+0000","last_active":"2024-09-16T01:53:46.281152+0000","last_peered":"2024-09-16T01:53:46.281152+0000","last_clean":"2024-09-16T01:53:46.281152+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:46.281152+0000","last_undegraded":"2024-09-16T01:53:46.281152+0000","last_fullsized":"2024-09-16T01:53:46.281152+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"152'204","reported_seq":741,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510211+0000","last_change":"2024-09-16T01:53:22.523335+0000","last_active":"2024-09-16T01:53:45.510211+0000","last_peered":"2024-09-16T01:53:45.510211+0000","last_clean":"2024-09-16T01:53:45.510211+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:53:45.510211+0000","last_undegraded":"2024-09-16T01:53:45.510211+0000","last_fullsized":"2024-09-16T01:53:45.510211+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:53.183089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"152'178","reported_seq":461,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243156+0000","last_change":"2024-09-16T01:48:58.836308+0000","last_active":"2024-09-16T01:53:46.243156+0000","last_peered":"2024-09-16T01:53:46.243156+0000","last_clean":"2024-09-16T01:53:46.243156+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:46.243156+0000","last_undegraded":"2024-09-16T01:53:46.243156+0000","last_fullsized":"2024-09-16T01:53:46.243156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"152'190","reported_seq":581,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281821+0000","last_change":"2024-09-16T01:53:22.526050+0000","last_active":"2024-09-16T01:53:46.281821+0000","last_peered":"2024-09-16T01:53:46.281821+0000","last_clean":"2024-09-16T01:53:46.281821+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:46.281821+0000","last_undegraded":"2024-09-16T01:53:46.281821+0000","last_fullsized":"2024-09-16T01:53:46.281821+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"152'214","reported_seq":628,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281274+0000","last_change":"2024-09-16T01:53:23.519544+0000","last_active":"2024-09-16T01:53:46.281274+0000","last_peered":"2024-09-16T01:53:46.281274+0000","last_clean":"2024-09-16T01:53:46.281274+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:46.281274+0000","last_undegraded":"2024-09-16T01:53:46.281274+0000","last_fullsized":"2024-09-16T01:53:46.281274+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"152'166","reported_seq":373,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281228+0000","last_change":"2024-09-16T01:53:22.519665+0000","last_active":"2024-09-16T01:53:46.281228+0000","last_peered":"2024-09-16T01:53:46.281228+0000","last_clean":"2024-09-16T01:53:46.281228+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:46.281228+0000","last_undegraded":"2024-09-16T01:53:46.281228+0000","last_fullsized":"2024-09-16T01:53:46.281228+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"152'177","reported_seq":552,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507920+0000","last_change":"2024-09-16T01:53:34.533587+0000","last_active":"2024-09-16T01:53:45.507920+0000","last_peered":"2024-09-16T01:53:45.507920+0000","last_clean":"2024-09-16T01:53:45.507920+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:45.507920+0000","last_undegraded":"2024-09-16T01:53:45.507920+0000","last_fullsized":"2024-09-16T01:53:45.507920+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:15:15.453058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"152'172","reported_seq":506,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507448+0000","last_change":"2024-09-16T01:51:19.358897+0000","last_active":"2024-09-16T01:53:45.507448+0000","last_peered":"2024-09-16T01:53:45.507448+0000","last_clean":"2024-09-16T01:53:45.507448+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:45.507448+0000","last_undegraded":"2024-09-16T01:53:45.507448+0000","last_fullsized":"2024-09-16T01:53:45.507448+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:13.148367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"152'177","reported_seq":773,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.513779+0000","last_change":"2024-09-16T01:53:45.513779+0000","last_active":"2024-09-16T01:53:45.513779+0000","last_peered":"2024-09-16T01:53:45.513779+0000","last_clean":"2024-09-16T01:53:45.513779+0000","last_became_active":"2024-09-16T01:53:45.513061+0000","last_became_peered":"2024-09-16T01:53:45.513061+0000","last_unstale":"2024-09-16T01:53:45.513779+0000","last_undegraded":"2024-09-16T01:53:45.513779+0000","last_fullsized":"2024-09-16T01:53:45.513779+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:29.386876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"152'162","reported_seq":445,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.506833+0000","last_change":"2024-09-16T01:48:58.831097+0000","last_active":"2024-09-16T01:53:45.506833+0000","last_peered":"2024-09-16T01:53:45.506833+0000","last_clean":"2024-09-16T01:53:45.506833+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:45.506833+0000","last_undegraded":"2024-09-16T01:53:45.506833+0000","last_fullsized":"2024-09-16T01:53:45.506833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"152'176","reported_seq":657,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507447+0000","last_change":"2024-09-16T01:53:23.526130+0000","last_active":"2024-09-16T01:53:45.507447+0000","last_peered":"2024-09-16T01:53:45.507447+0000","last_clean":"2024-09-16T01:53:45.507447+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:53:45.507447+0000","last_undegraded":"2024-09-16T01:53:45.507447+0000","last_fullsized":"2024-09-16T01:53:45.507447+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:23:02.127294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"152'179","reported_seq":595,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512484+0000","last_change":"2024-09-16T01:53:36.563883+0000","last_active":"2024-09-16T01:53:45.512484+0000","last_peered":"2024-09-16T01:53:45.512484+0000","last_clean":"2024-09-16T01:53:45.512484+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:45.512484+0000","last_undegraded":"2024-09-16T01:53:45.512484+0000","last_fullsized":"2024-09-16T01:53:45.512484+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"152'205","reported_seq":668,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512470+0000","last_change":"2024-09-16T01:53:36.554778+0000","last_active":"2024-09-16T01:53:45.512470+0000","last_peered":"2024-09-16T01:53:45.512470+0000","last_clean":"2024-09-16T01:53:45.512470+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:45.512470+0000","last_undegraded":"2024-09-16T01:53:45.512470+0000","last_fullsized":"2024-09-16T01:53:45.512470+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:16.261589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"152'179","reported_seq":607,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281127+0000","last_change":"2024-09-16T01:53:23.524759+0000","last_active":"2024-09-16T01:53:46.281127+0000","last_peered":"2024-09-16T01:53:46.281127+0000","last_clean":"2024-09-16T01:53:46.281127+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:46.281127+0000","last_undegraded":"2024-09-16T01:53:46.281127+0000","last_fullsized":"2024-09-16T01:53:46.281127+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"152'154","reported_seq":472,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949220+0000","last_change":"2024-09-16T01:53:23.520715+0000","last_active":"2024-09-16T01:53:43.949220+0000","last_peered":"2024-09-16T01:53:43.949220+0000","last_clean":"2024-09-16T01:53:43.949220+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:43.949220+0000","last_undegraded":"2024-09-16T01:53:43.949220+0000","last_fullsized":"2024-09-16T01:53:43.949220+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:45.628132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"152'199","reported_seq":841,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243033+0000","last_change":"2024-09-16T01:53:22.524407+0000","last_active":"2024-09-16T01:53:46.243033+0000","last_peered":"2024-09-16T01:53:46.243033+0000","last_clean":"2024-09-16T01:53:46.243033+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:46.243033+0000","last_undegraded":"2024-09-16T01:53:46.243033+0000","last_fullsized":"2024-09-16T01:53:46.243033+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"152'150","reported_seq":433,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510286+0000","last_change":"2024-09-16T01:48:58.835392+0000","last_active":"2024-09-16T01:53:45.510286+0000","last_peered":"2024-09-16T01:53:45.510286+0000","last_clean":"2024-09-16T01:53:45.510286+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:45.510286+0000","last_undegraded":"2024-09-16T01:53:45.510286+0000","last_fullsized":"2024-09-16T01:53:45.510286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:20.196122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"152'207","reported_seq":490,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.506833+0000","last_change":"2024-09-16T01:48:58.822946+0000","last_active":"2024-09-16T01:53:45.506833+0000","last_peered":"2024-09-16T01:53:45.506833+0000","last_clean":"2024-09-16T01:53:45.506833+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:45.506833+0000","last_undegraded":"2024-09-16T01:53:45.506833+0000","last_fullsized":"2024-09-16T01:53:45.506833+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:16.486446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"152'160","reported_seq":443,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510683+0000","last_change":"2024-09-16T01:48:58.821263+0000","last_active":"2024-09-16T01:53:45.510683+0000","last_peered":"2024-09-16T01:53:45.510683+0000","last_clean":"2024-09-16T01:53:45.510683+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:45.510683+0000","last_undegraded":"2024-09-16T01:53:45.510683+0000","last_fullsized":"2024-09-16T01:53:45.510683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:01.140541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"152'217","reported_seq":501,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.280825+0000","last_change":"2024-09-16T01:48:58.839123+0000","last_active":"2024-09-16T01:53:46.280825+0000","last_peered":"2024-09-16T01:53:46.280825+0000","last_clean":"2024-09-16T01:53:46.280825+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:46.280825+0000","last_undegraded":"2024-09-16T01:53:46.280825+0000","last_fullsized":"2024-09-16T01:53:46.280825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"152'166","reported_seq":680,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.508460+0000","last_change":"2024-09-16T01:52:49.495340+0000","last_active":"2024-09-16T01:53:45.508460+0000","last_peered":"2024-09-16T01:53:45.508460+0000","last_clean":"2024-09-16T01:53:45.508460+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:45.508460+0000","last_undegraded":"2024-09-16T01:53:45.508460+0000","last_fullsized":"2024-09-16T01:53:45.508460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:51.303049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"152'152","reported_seq":498,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507901+0000","last_change":"2024-09-16T01:53:34.533524+0000","last_active":"2024-09-16T01:53:45.507901+0000","last_peered":"2024-09-16T01:53:45.507901+0000","last_clean":"2024-09-16T01:53:45.507901+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:45.507901+0000","last_undegraded":"2024-09-16T01:53:45.507901+0000","last_fullsized":"2024-09-16T01:53:45.507901+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:57.738639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"152'134","reported_seq":700,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510537+0000","last_change":"2024-09-16T01:52:47.047065+0000","last_active":"2024-09-16T01:53:45.510537+0000","last_peered":"2024-09-16T01:53:45.510537+0000","last_clean":"2024-09-16T01:53:45.510537+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:53:45.510537+0000","last_undegraded":"2024-09-16T01:53:45.510537+0000","last_fullsized":"2024-09-16T01:53:45.510537+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:37.139004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"152'153","reported_seq":436,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.506968+0000","last_change":"2024-09-16T01:48:58.827156+0000","last_active":"2024-09-16T01:53:45.506968+0000","last_peered":"2024-09-16T01:53:45.506968+0000","last_clean":"2024-09-16T01:53:45.506968+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:45.506968+0000","last_undegraded":"2024-09-16T01:53:45.506968+0000","last_fullsized":"2024-09-16T01:53:45.506968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:31.858449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"152'158","reported_seq":580,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512541+0000","last_change":"2024-09-16T01:53:36.563260+0000","last_active":"2024-09-16T01:53:45.512541+0000","last_peered":"2024-09-16T01:53:45.512541+0000","last_clean":"2024-09-16T01:53:45.512541+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:45.512541+0000","last_undegraded":"2024-09-16T01:53:45.512541+0000","last_fullsized":"2024-09-16T01:53:45.512541+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:31.539361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"152'180","reported_seq":552,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512188+0000","last_change":"2024-09-16T01:53:36.551939+0000","last_active":"2024-09-16T01:53:45.512188+0000","last_peered":"2024-09-16T01:53:45.512188+0000","last_clean":"2024-09-16T01:53:45.512188+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:45.512188+0000","last_undegraded":"2024-09-16T01:53:45.512188+0000","last_fullsized":"2024-09-16T01:53:45.512188+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:52:12.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"152'149","reported_seq":432,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243037+0000","last_change":"2024-09-16T01:48:58.820714+0000","last_active":"2024-09-16T01:53:46.243037+0000","last_peered":"2024-09-16T01:53:46.243037+0000","last_clean":"2024-09-16T01:53:46.243037+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:46.243037+0000","last_undegraded":"2024-09-16T01:53:46.243037+0000","last_fullsized":"2024-09-16T01:53:46.243037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":289,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510673+0000","last_change":"2024-09-16T01:48:57.808272+0000","last_active":"2024-09-16T01:53:45.510673+0000","last_peered":"2024-09-16T01:53:45.510673+0000","last_clean":"2024-09-16T01:53:45.510673+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:45.510673+0000","last_undegraded":"2024-09-16T01:53:45.510673+0000","last_fullsized":"2024-09-16T01:53:45.510673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:37.755267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"152'162","reported_seq":776,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949235+0000","last_change":"2024-09-16T01:53:22.520820+0000","last_active":"2024-09-16T01:53:43.949235+0000","last_peered":"2024-09-16T01:53:43.949235+0000","last_clean":"2024-09-16T01:53:43.949235+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:43.949235+0000","last_undegraded":"2024-09-16T01:53:43.949235+0000","last_fullsized":"2024-09-16T01:53:43.949235+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:19:51.234044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'7","reported_seq":317,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507086+0000","last_change":"2024-09-16T01:53:19.532463+0000","last_active":"2024-09-16T01:53:45.507086+0000","last_peered":"2024-09-16T01:53:45.507086+0000","last_clean":"2024-09-16T01:53:45.507086+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:45.507086+0000","last_undegraded":"2024-09-16T01:53:45.507086+0000","last_fullsized":"2024-09-16T01:53:45.507086+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:52.176146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'3","reported_seq":285,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507437+0000","last_change":"2024-09-16T01:53:34.532869+0000","last_active":"2024-09-16T01:53:45.507437+0000","last_peered":"2024-09-16T01:53:45.507437+0000","last_clean":"2024-09-16T01:53:45.507437+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:45.507437+0000","last_undegraded":"2024-09-16T01:53:45.507437+0000","last_fullsized":"2024-09-16T01:53:45.507437+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:13.148781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"152'190","reported_seq":472,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510348+0000","last_change":"2024-09-16T01:48:58.819565+0000","last_active":"2024-09-16T01:53:45.510348+0000","last_peered":"2024-09-16T01:53:45.510348+0000","last_clean":"2024-09-16T01:53:45.510348+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:45.510348+0000","last_undegraded":"2024-09-16T01:53:45.510348+0000","last_fullsized":"2024-09-16T01:53:45.510348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:52:59.454264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'3","reported_seq":287,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.506961+0000","last_change":"2024-09-16T01:48:57.810142+0000","last_active":"2024-09-16T01:53:45.506961+0000","last_peered":"2024-09-16T01:53:45.506961+0000","last_clean":"2024-09-16T01:53:45.506961+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:45.506961+0000","last_undegraded":"2024-09-16T01:53:45.506961+0000","last_fullsized":"2024-09-16T01:53:45.506961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:03.325089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"152'187","reported_seq":434,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949089+0000","last_change":"2024-09-16T01:53:32.513845+0000","last_active":"2024-09-16T01:53:43.949089+0000","last_peered":"2024-09-16T01:53:43.949089+0000","last_clean":"2024-09-16T01:53:43.949089+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:43.949089+0000","last_undegraded":"2024-09-16T01:53:43.949089+0000","last_fullsized":"2024-09-16T01:53:43.949089+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:02.738813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"65'5","reported_seq":328,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243098+0000","last_change":"2024-09-16T01:53:20.522781+0000","last_active":"2024-09-16T01:53:46.243098+0000","last_peered":"2024-09-16T01:53:46.243098+0000","last_clean":"2024-09-16T01:53:46.243098+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:46.243098+0000","last_undegraded":"2024-09-16T01:53:46.243098+0000","last_fullsized":"2024-09-16T01:53:46.243098+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"152'251","reported_seq":547,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510442+0000","last_change":"2024-09-16T01:52:45.567223+0000","last_active":"2024-09-16T01:53:45.510442+0000","last_peered":"2024-09-16T01:53:45.510442+0000","last_clean":"2024-09-16T01:53:45.510442+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:45.510442+0000","last_undegraded":"2024-09-16T01:53:45.510442+0000","last_fullsized":"2024-09-16T01:53:45.510442+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:40.956815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"22'4","reported_seq":61,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281812+0000","last_change":"2024-09-16T01:53:19.579965+0000","last_active":"2024-09-16T01:53:46.281812+0000","last_peered":"2024-09-16T01:53:46.281812+0000","last_clean":"2024-09-16T01:53:46.281812+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:46.281812+0000","last_undegraded":"2024-09-16T01:53:46.281812+0000","last_fullsized":"2024-09-16T01:53:46.281812+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"152'160","reported_seq":442,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510852+0000","last_change":"2024-09-16T01:48:58.838741+0000","last_active":"2024-09-16T01:53:45.510852+0000","last_peered":"2024-09-16T01:53:45.510852+0000","last_clean":"2024-09-16T01:53:45.510852+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:45.510852+0000","last_undegraded":"2024-09-16T01:53:45.510852+0000","last_fullsized":"2024-09-16T01:53:45.510852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"133'11","reported_seq":170,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512421+0000","last_change":"2024-09-16T01:53:36.557008+0000","last_active":"2024-09-16T01:53:45.512421+0000","last_peered":"2024-09-16T01:53:45.512421+0000","last_clean":"2024-09-16T01:53:45.512421+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:45.512421+0000","last_undegraded":"2024-09-16T01:53:45.512421+0000","last_fullsized":"2024-09-16T01:53:45.512421+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:54:04.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'168","reported_seq":451,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507256+0000","last_change":"2024-09-16T01:48:58.838665+0000","last_active":"2024-09-16T01:53:45.507256+0000","last_peered":"2024-09-16T01:53:45.507256+0000","last_clean":"2024-09-16T01:53:45.507256+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:45.507256+0000","last_undegraded":"2024-09-16T01:53:45.507256+0000","last_fullsized":"2024-09-16T01:53:45.507256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:57:22.419652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"152'178","reported_seq":506,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.280650+0000","last_change":"2024-09-16T01:53:23.518694+0000","last_active":"2024-09-16T01:53:46.280650+0000","last_peered":"2024-09-16T01:53:46.280650+0000","last_clean":"2024-09-16T01:53:46.280650+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:46.280650+0000","last_undegraded":"2024-09-16T01:53:46.280650+0000","last_fullsized":"2024-09-16T01:53:46.280650+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391576,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"152'176","reported_seq":544,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281486+0000","last_change":"2024-09-16T01:53:22.524097+0000","last_active":"2024-09-16T01:53:46.281486+0000","last_peered":"2024-09-16T01:53:46.281486+0000","last_clean":"2024-09-16T01:53:46.281486+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:46.281486+0000","last_undegraded":"2024-09-16T01:53:46.281486+0000","last_fullsized":"2024-09-16T01:53:46.281486+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'9","reported_seq":274,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512245+0000","last_change":"2024-09-16T01:53:36.559516+0000","last_active":"2024-09-16T01:53:45.512245+0000","last_peered":"2024-09-16T01:53:45.512245+0000","last_clean":"2024-09-16T01:53:45.512245+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:45.512245+0000","last_undegraded":"2024-09-16T01:53:45.512245+0000","last_fullsized":"2024-09-16T01:53:45.512245+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:58.881979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"152'177","reported_seq":430,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507576+0000","last_change":"2024-09-16T01:53:34.529911+0000","last_active":"2024-09-16T01:53:45.507576+0000","last_peered":"2024-09-16T01:53:45.507576+0000","last_clean":"2024-09-16T01:53:45.507576+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:45.507576+0000","last_undegraded":"2024-09-16T01:53:45.507576+0000","last_fullsized":"2024-09-16T01:53:45.507576+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:29:37.770006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'5","reported_seq":368,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507143+0000","last_change":"2024-09-16T01:52:46.625454+0000","last_active":"2024-09-16T01:53:45.507143+0000","last_peered":"2024-09-16T01:53:45.507143+0000","last_clean":"2024-09-16T01:53:45.507143+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:53:45.507143+0000","last_undegraded":"2024-09-16T01:53:45.507143+0000","last_fullsized":"2024-09-16T01:53:45.507143+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:31.104424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"152'201","reported_seq":728,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507088+0000","last_change":"2024-09-16T01:53:23.521939+0000","last_active":"2024-09-16T01:53:45.507088+0000","last_peered":"2024-09-16T01:53:45.507088+0000","last_clean":"2024-09-16T01:53:45.507088+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:53:45.507088+0000","last_undegraded":"2024-09-16T01:53:45.507088+0000","last_fullsized":"2024-09-16T01:53:45.507088+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:55.352896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"152'7","reported_seq":64,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281055+0000","last_change":"2024-09-16T01:53:19.550015+0000","last_active":"2024-09-16T01:53:46.281055+0000","last_peered":"2024-09-16T01:53:46.281055+0000","last_clean":"2024-09-16T01:53:46.281055+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:46.281055+0000","last_undegraded":"2024-09-16T01:53:46.281055+0000","last_fullsized":"2024-09-16T01:53:46.281055+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"152'197","reported_seq":481,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243182+0000","last_change":"2024-09-16T01:48:58.822258+0000","last_active":"2024-09-16T01:53:46.243182+0000","last_peered":"2024-09-16T01:53:46.243182+0000","last_clean":"2024-09-16T01:53:46.243182+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:46.243182+0000","last_undegraded":"2024-09-16T01:53:46.243182+0000","last_fullsized":"2024-09-16T01:53:46.243182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"133'18","reported_seq":66,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949080+0000","last_change":"2024-09-16T01:53:20.541056+0000","last_active":"2024-09-16T01:53:43.949080+0000","last_peered":"2024-09-16T01:53:43.949080+0000","last_clean":"2024-09-16T01:53:43.949080+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:43.949080+0000","last_undegraded":"2024-09-16T01:53:43.949080+0000","last_fullsized":"2024-09-16T01:53:43.949080+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:21.564935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"152'215","reported_seq":498,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.506976+0000","last_change":"2024-09-16T01:48:58.838973+0000","last_active":"2024-09-16T01:53:45.506976+0000","last_peered":"2024-09-16T01:53:45.506976+0000","last_clean":"2024-09-16T01:53:45.506976+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:45.506976+0000","last_undegraded":"2024-09-16T01:53:45.506976+0000","last_fullsized":"2024-09-16T01:53:45.506976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:01.453478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"153'4","reported_seq":62,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281682+0000","last_change":"2024-09-16T01:53:19.563306+0000","last_active":"2024-09-16T01:53:46.281682+0000","last_peered":"2024-09-16T01:53:46.281682+0000","last_clean":"2024-09-16T01:53:46.281682+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:46.281682+0000","last_undegraded":"2024-09-16T01:53:46.281682+0000","last_fullsized":"2024-09-16T01:53:46.281682+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"152'175","reported_seq":663,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.280676+0000","last_change":"2024-09-16T01:53:22.527907+0000","last_active":"2024-09-16T01:53:46.280676+0000","last_peered":"2024-09-16T01:53:46.280676+0000","last_clean":"2024-09-16T01:53:46.280676+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:46.280676+0000","last_undegraded":"2024-09-16T01:53:46.280676+0000","last_fullsized":"2024-09-16T01:53:46.280676+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'5","reported_seq":54,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949405+0000","last_change":"2024-09-16T01:53:20.547257+0000","last_active":"2024-09-16T01:53:43.949405+0000","last_peered":"2024-09-16T01:53:43.949405+0000","last_clean":"2024-09-16T01:53:43.949405+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:43.949405+0000","last_undegraded":"2024-09-16T01:53:43.949405+0000","last_fullsized":"2024-09-16T01:53:43.949405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:04.416789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"152'162","reported_seq":543,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281707+0000","last_change":"2024-09-16T01:53:23.527787+0000","last_active":"2024-09-16T01:53:46.281707+0000","last_peered":"2024-09-16T01:53:46.281707+0000","last_clean":"2024-09-16T01:53:46.281707+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:53:46.281707+0000","last_undegraded":"2024-09-16T01:53:46.281707+0000","last_fullsized":"2024-09-16T01:53:46.281707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":288,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510618+0000","last_change":"2024-09-16T01:48:57.818410+0000","last_active":"2024-09-16T01:53:45.510618+0000","last_peered":"2024-09-16T01:53:45.510618+0000","last_clean":"2024-09-16T01:53:45.510618+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:45.510618+0000","last_undegraded":"2024-09-16T01:53:45.510618+0000","last_fullsized":"2024-09-16T01:53:45.510618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:30.439316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"152'165","reported_seq":787,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281385+0000","last_change":"2024-09-16T01:53:45.454388+0000","last_active":"2024-09-16T01:53:46.281385+0000","last_peered":"2024-09-16T01:53:46.281385+0000","last_clean":"2024-09-16T01:53:46.281385+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:46.281385+0000","last_undegraded":"2024-09-16T01:53:46.281385+0000","last_fullsized":"2024-09-16T01:53:46.281385+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'165","last_scrub_stamp":"2024-09-16T01:53:45.454296+0000","last_deep_scrub":"152'165","last_deep_scrub_stamp":"2024-09-16T01:53:45.454296+0000","last_clean_scrub_stamp":"2024-09-16T01:53:45.454296+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:36:12.315196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'5","reported_seq":273,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.508127+0000","last_change":"2024-09-16T01:53:34.533293+0000","last_active":"2024-09-16T01:53:45.508127+0000","last_peered":"2024-09-16T01:53:45.508127+0000","last_clean":"2024-09-16T01:53:45.508127+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:45.508127+0000","last_undegraded":"2024-09-16T01:53:45.508127+0000","last_fullsized":"2024-09-16T01:53:45.508127+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:17:39.949872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"152'175","reported_seq":458,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507382+0000","last_change":"2024-09-16T01:48:58.824839+0000","last_active":"2024-09-16T01:53:45.507382+0000","last_peered":"2024-09-16T01:53:45.507382+0000","last_clean":"2024-09-16T01:53:45.507382+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:45.507382+0000","last_undegraded":"2024-09-16T01:53:45.507382+0000","last_fullsized":"2024-09-16T01:53:45.507382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:28.964273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":442,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243003+0000","last_change":"2024-09-16T01:48:48.116302+0000","last_active":"2024-09-16T01:53:46.243003+0000","last_peered":"2024-09-16T01:53:46.243003+0000","last_clean":"2024-09-16T01:53:46.243003+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:46.243003+0000","last_undegraded":"2024-09-16T01:53:46.243003+0000","last_fullsized":"2024-09-16T01:53:46.243003+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:46.750991+0000","last_clean_scrub_stamp":"2024-09-16T01:48:46.750991+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'6","reported_seq":291,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507189+0000","last_change":"2024-09-16T01:48:57.819938+0000","last_active":"2024-09-16T01:53:45.507189+0000","last_peered":"2024-09-16T01:53:45.507189+0000","last_clean":"2024-09-16T01:53:45.507189+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:45.507189+0000","last_undegraded":"2024-09-16T01:53:45.507189+0000","last_fullsized":"2024-09-16T01:53:45.507189+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:31:40.213743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"152'183","reported_seq":606,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949224+0000","last_change":"2024-09-16T01:53:22.524529+0000","last_active":"2024-09-16T01:53:43.949224+0000","last_peered":"2024-09-16T01:53:43.949224+0000","last_clean":"2024-09-16T01:53:43.949224+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:43.949224+0000","last_undegraded":"2024-09-16T01:53:43.949224+0000","last_fullsized":"2024-09-16T01:53:43.949224+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:37:39.441997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":289,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.506993+0000","last_change":"2024-09-16T01:48:57.816412+0000","last_active":"2024-09-16T01:53:45.506993+0000","last_peered":"2024-09-16T01:53:45.506993+0000","last_clean":"2024-09-16T01:53:45.506993+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:45.506993+0000","last_undegraded":"2024-09-16T01:53:45.506993+0000","last_fullsized":"2024-09-16T01:53:45.506993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:19.343342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"152'165","reported_seq":418,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507323+0000","last_change":"2024-09-16T01:53:34.532478+0000","last_active":"2024-09-16T01:53:45.507323+0000","last_peered":"2024-09-16T01:53:45.507323+0000","last_clean":"2024-09-16T01:53:45.507323+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:45.507323+0000","last_undegraded":"2024-09-16T01:53:45.507323+0000","last_fullsized":"2024-09-16T01:53:45.507323+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:58.403570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"152'13","reported_seq":322,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512534+0000","last_change":"2024-09-16T01:53:36.554302+0000","last_active":"2024-09-16T01:53:45.512534+0000","last_peered":"2024-09-16T01:53:45.512534+0000","last_clean":"2024-09-16T01:53:45.512534+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:45.512534+0000","last_undegraded":"2024-09-16T01:53:45.512534+0000","last_fullsized":"2024-09-16T01:53:45.512534+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:40.430016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"152'165","reported_seq":665,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510284+0000","last_change":"2024-09-16T01:52:47.725186+0000","last_active":"2024-09-16T01:53:45.510284+0000","last_peered":"2024-09-16T01:53:45.510284+0000","last_clean":"2024-09-16T01:53:45.510284+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:53:45.510284+0000","last_undegraded":"2024-09-16T01:53:45.510284+0000","last_fullsized":"2024-09-16T01:53:45.510284+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:32.111367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'5","reported_seq":290,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507190+0000","last_change":"2024-09-16T01:48:57.818207+0000","last_active":"2024-09-16T01:53:45.507190+0000","last_peered":"2024-09-16T01:53:45.507190+0000","last_clean":"2024-09-16T01:53:45.507190+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:45.507190+0000","last_undegraded":"2024-09-16T01:53:45.507190+0000","last_fullsized":"2024-09-16T01:53:45.507190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:46.678809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"152'177","reported_seq":561,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281373+0000","last_change":"2024-09-16T01:53:23.527347+0000","last_active":"2024-09-16T01:53:46.281373+0000","last_peered":"2024-09-16T01:53:46.281373+0000","last_clean":"2024-09-16T01:53:46.281373+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:53:46.281373+0000","last_undegraded":"2024-09-16T01:53:46.281373+0000","last_fullsized":"2024-09-16T01:53:46.281373+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":286,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510440+0000","last_change":"2024-09-16T01:48:57.805713+0000","last_active":"2024-09-16T01:53:45.510440+0000","last_peered":"2024-09-16T01:53:45.510440+0000","last_clean":"2024-09-16T01:53:45.510440+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:45.510440+0000","last_undegraded":"2024-09-16T01:53:45.510440+0000","last_fullsized":"2024-09-16T01:53:45.510440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:27.185418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"152'160","reported_seq":452,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.376058+0000","last_change":"2024-09-16T01:53:46.376096+0000","last_active":"2024-09-16T01:53:46.376058+0000","last_peered":"2024-09-16T01:53:46.376058+0000","last_clean":"2024-09-16T01:53:46.376058+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:46.376058+0000","last_undegraded":"2024-09-16T01:53:46.376058+0000","last_fullsized":"2024-09-16T01:53:46.376058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'160","last_scrub_stamp":"2024-09-16T01:53:46.375956+0000","last_deep_scrub":"152'160","last_deep_scrub_stamp":"2024-09-16T01:53:46.375956+0000","last_clean_scrub_stamp":"2024-09-16T01:53:46.375956+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:15.973803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'4","reported_seq":129,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281532+0000","last_change":"2024-09-16T01:53:20.534486+0000","last_active":"2024-09-16T01:53:46.281532+0000","last_peered":"2024-09-16T01:53:46.281532+0000","last_clean":"2024-09-16T01:53:46.281532+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:46.281532+0000","last_undegraded":"2024-09-16T01:53:46.281532+0000","last_fullsized":"2024-09-16T01:53:46.281532+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"152'162","reported_seq":415,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.508015+0000","last_change":"2024-09-16T01:53:34.539129+0000","last_active":"2024-09-16T01:53:45.508015+0000","last_peered":"2024-09-16T01:53:45.508015+0000","last_clean":"2024-09-16T01:53:45.508015+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:45.508015+0000","last_undegraded":"2024-09-16T01:53:45.508015+0000","last_fullsized":"2024-09-16T01:53:45.508015+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:17.044571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"153'26","reported_seq":65,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949450+0000","last_change":"2024-09-16T01:53:20.586608+0000","last_active":"2024-09-16T01:53:43.949450+0000","last_peered":"2024-09-16T01:53:43.949450+0000","last_clean":"2024-09-16T01:53:43.949450+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:43.949450+0000","last_undegraded":"2024-09-16T01:53:43.949450+0000","last_fullsized":"2024-09-16T01:53:43.949450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:55.614045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"152'166","reported_seq":477,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949431+0000","last_change":"2024-09-16T01:53:23.514351+0000","last_active":"2024-09-16T01:53:43.949431+0000","last_peered":"2024-09-16T01:53:43.949431+0000","last_clean":"2024-09-16T01:53:43.949431+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:43.949431+0000","last_undegraded":"2024-09-16T01:53:43.949431+0000","last_fullsized":"2024-09-16T01:53:43.949431+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:14.373245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"130'8","reported_seq":323,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507551+0000","last_change":"2024-09-16T01:53:19.543649+0000","last_active":"2024-09-16T01:53:45.507551+0000","last_peered":"2024-09-16T01:53:45.507551+0000","last_clean":"2024-09-16T01:53:45.507551+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:53:45.507551+0000","last_undegraded":"2024-09-16T01:53:45.507551+0000","last_fullsized":"2024-09-16T01:53:45.507551+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:14:12.126459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'183","reported_seq":554,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949576+0000","last_change":"2024-09-16T01:53:23.518054+0000","last_active":"2024-09-16T01:53:43.949576+0000","last_peered":"2024-09-16T01:53:43.949576+0000","last_clean":"2024-09-16T01:53:43.949576+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:43.949576+0000","last_undegraded":"2024-09-16T01:53:43.949576+0000","last_fullsized":"2024-09-16T01:53:43.949576+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:50.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":59,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.280621+0000","last_change":"2024-09-16T01:53:19.575999+0000","last_active":"2024-09-16T01:53:46.280621+0000","last_peered":"2024-09-16T01:53:46.280621+0000","last_clean":"2024-09-16T01:53:46.280621+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:46.280621+0000","last_undegraded":"2024-09-16T01:53:46.280621+0000","last_fullsized":"2024-09-16T01:53:46.280621+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"152'162","reported_seq":597,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507087+0000","last_change":"2024-09-16T01:53:23.514324+0000","last_active":"2024-09-16T01:53:45.507087+0000","last_peered":"2024-09-16T01:53:45.507087+0000","last_clean":"2024-09-16T01:53:45.507087+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:53:45.507087+0000","last_undegraded":"2024-09-16T01:53:45.507087+0000","last_fullsized":"2024-09-16T01:53:45.507087+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:23:27.575957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'7","reported_seq":64,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949491+0000","last_change":"2024-09-16T01:53:20.576951+0000","last_active":"2024-09-16T01:53:43.949491+0000","last_peered":"2024-09-16T01:53:43.949491+0000","last_clean":"2024-09-16T01:53:43.949491+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:43.949491+0000","last_undegraded":"2024-09-16T01:53:43.949491+0000","last_fullsized":"2024-09-16T01:53:43.949491+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:20.021875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"152'173","reported_seq":509,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949064+0000","last_change":"2024-09-16T01:53:23.517332+0000","last_active":"2024-09-16T01:53:43.949064+0000","last_peered":"2024-09-16T01:53:43.949064+0000","last_clean":"2024-09-16T01:53:43.949064+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:43.949064+0000","last_undegraded":"2024-09-16T01:53:43.949064+0000","last_fullsized":"2024-09-16T01:53:43.949064+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:15.929317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":258,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507696+0000","last_change":"2024-09-16T01:53:34.538675+0000","last_active":"2024-09-16T01:53:45.507696+0000","last_peered":"2024-09-16T01:53:45.507696+0000","last_clean":"2024-09-16T01:53:45.507696+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:45.507696+0000","last_undegraded":"2024-09-16T01:53:45.507696+0000","last_fullsized":"2024-09-16T01:53:45.507696+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:37.937332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"152'164","reported_seq":642,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.512307+0000","last_change":"2024-09-16T01:53:36.554614+0000","last_active":"2024-09-16T01:53:45.512307+0000","last_peered":"2024-09-16T01:53:45.512307+0000","last_clean":"2024-09-16T01:53:45.512307+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:45.512307+0000","last_undegraded":"2024-09-16T01:53:45.512307+0000","last_fullsized":"2024-09-16T01:53:45.512307+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:23.435521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"65'5","reported_seq":290,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507141+0000","last_change":"2024-09-16T01:48:57.819971+0000","last_active":"2024-09-16T01:53:45.507141+0000","last_peered":"2024-09-16T01:53:45.507141+0000","last_clean":"2024-09-16T01:53:45.507141+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:45.507141+0000","last_undegraded":"2024-09-16T01:53:45.507141+0000","last_fullsized":"2024-09-16T01:53:45.507141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:55.244024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"152'154","reported_seq":437,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507028+0000","last_change":"2024-09-16T01:48:58.826537+0000","last_active":"2024-09-16T01:53:45.507028+0000","last_peered":"2024-09-16T01:53:45.507028+0000","last_clean":"2024-09-16T01:53:45.507028+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:45.507028+0000","last_undegraded":"2024-09-16T01:53:45.507028+0000","last_fullsized":"2024-09-16T01:53:45.507028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.569435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"152'205","reported_seq":727,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.507364+0000","last_change":"2024-09-16T01:53:22.527742+0000","last_active":"2024-09-16T01:53:45.507364+0000","last_peered":"2024-09-16T01:53:45.507364+0000","last_clean":"2024-09-16T01:53:45.507364+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:53:45.507364+0000","last_undegraded":"2024-09-16T01:53:45.507364+0000","last_fullsized":"2024-09-16T01:53:45.507364+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:12:17.944070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"157'8","reported_seq":74,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.476521+0000","last_change":"2024-09-16T01:53:46.476569+0000","last_active":"2024-09-16T01:53:46.476521+0000","last_peered":"2024-09-16T01:53:46.476521+0000","last_clean":"2024-09-16T01:53:46.476521+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:46.476521+0000","last_undegraded":"2024-09-16T01:53:46.476521+0000","last_fullsized":"2024-09-16T01:53:46.476521+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'8","last_scrub_stamp":"2024-09-16T01:53:46.476442+0000","last_deep_scrub":"157'8","last_deep_scrub_stamp":"2024-09-16T01:53:46.476442+0000","last_clean_scrub_stamp":"2024-09-16T01:53:46.476442+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:52.399904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"152'162","reported_seq":637,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.243034+0000","last_change":"2024-09-16T01:53:22.521982+0000","last_active":"2024-09-16T01:53:46.243034+0000","last_peered":"2024-09-16T01:53:46.243034+0000","last_clean":"2024-09-16T01:53:46.243034+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:46.243034+0000","last_undegraded":"2024-09-16T01:53:46.243034+0000","last_fullsized":"2024-09-16T01:53:46.243034+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"65'13","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281382+0000","last_change":"2024-09-16T01:53:19.559597+0000","last_active":"2024-09-16T01:53:46.281382+0000","last_peered":"2024-09-16T01:53:46.281382+0000","last_clean":"2024-09-16T01:53:46.281382+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:46.281382+0000","last_undegraded":"2024-09-16T01:53:46.281382+0000","last_fullsized":"2024-09-16T01:53:46.281382+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"152'160","reported_seq":443,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510667+0000","last_change":"2024-09-16T01:48:58.820363+0000","last_active":"2024-09-16T01:53:45.510667+0000","last_peered":"2024-09-16T01:53:45.510667+0000","last_clean":"2024-09-16T01:53:45.510667+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:45.510667+0000","last_undegraded":"2024-09-16T01:53:45.510667+0000","last_fullsized":"2024-09-16T01:53:45.510667+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:18.450204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":317,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510856+0000","last_change":"2024-09-16T01:52:46.544387+0000","last_active":"2024-09-16T01:53:45.510856+0000","last_peered":"2024-09-16T01:53:45.510856+0000","last_clean":"2024-09-16T01:53:45.510856+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:45.510856+0000","last_undegraded":"2024-09-16T01:53:45.510856+0000","last_fullsized":"2024-09-16T01:53:45.510856+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:56.731696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"152'181","reported_seq":464,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:45.510672+0000","last_change":"2024-09-16T01:48:58.835463+0000","last_active":"2024-09-16T01:53:45.510672+0000","last_peered":"2024-09-16T01:53:45.510672+0000","last_clean":"2024-09-16T01:53:45.510672+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:45.510672+0000","last_undegraded":"2024-09-16T01:53:45.510672+0000","last_fullsized":"2024-09-16T01:53:45.510672+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:57.787391+0000","last_clean_scrub_stamp":"2024-09-16T01:48:57.787391+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:55.034551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'13","reported_seq":62,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-16T01:53:43.949360+0000","last_change":"2024-09-16T01:53:20.557667+0000","last_active":"2024-09-16T01:53:43.949360+0000","last_peered":"2024-09-16T01:53:43.949360+0000","last_clean":"2024-09-16T01:53:43.949360+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:43.949360+0000","last_undegraded":"2024-09-16T01:53:43.949360+0000","last_fullsized":"2024-09-16T01:53:43.949360+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:48:56.780448+0000","last_clean_scrub_stamp":"2024-09-16T01:48:56.780448+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:05:38.159129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11255,"ondisk_log_size":11255,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398903,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":428,"num_write_kb":603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797806,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":392,"ondisk_log_size":392,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509505,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1916928,"data_stored":905386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509505,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41752,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330088,"statfs":{"total":96636764160,"available":96594010112,"internally_reserved":0,"allocated":2297856,"data_stored":1294132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509504,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1953792,"data_stored":1488064,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509505,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1916928,"data_stored":905318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542208,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1908736,"data_stored":897841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542208,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1888256,"data_stored":898662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542208,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":1487981,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41448,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330392,"statfs":{"total":96636764160,"available":96594321408,"internally_reserved":0,"allocated":1986560,"data_stored":1283581,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393414,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:53:49.388 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.388 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.388 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.388 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.389 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.390 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.391 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.392 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.393 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.394 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T01:48:46.750991+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.395 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.396 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.397 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-16T01:48:57.787391+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-09-16T01:48:56.780448+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=48, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=53, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:53:49.398 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T01:53:57.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:53:57.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:54:09.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:54:09.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.563+0000 7f5ddb800640 1 Processor -- start 2024-09-16T01:54:09.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.563+0000 7f5ddb800640 1 -- start start 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.564+0000 7f5ddb800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5dd4152850 0x7f5dd4158670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.564+0000 7f5ddb800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd40c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.564+0000 7f5ddb800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd4072f50 con 0x7f5dd40c6bb0 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.564+0000 7f5ddb800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd4073210 con 0x7f5dd4152850 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.564+0000 7f5ddb800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd415cb50 con 0x7f5dd4158bb0 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5dd4152850 0x7f5dd4158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dcbe00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd40c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5dd4152850 0x7f5dd4158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58858/0 (socket says 172.21.15.146:58858) 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 -- 172.21.15.146:0/1518082568 learned_addr learned my addr 172.21.15.146:0/1518082568 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:09.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd9800640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5dd40c6bb0 0x7f5dd40c86f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:47880/0 (socket says 172.21.15.146:47880) 2024-09-16T01:54:09.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 -- 172.21.15.146:0/1518082568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 msgr2=0x7f5dd40c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:54:09.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 --2- 172.21.15.146:0/1518082568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd40c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 -- 172.21.15.146:0/1518082568 >> v1:172.21.15.161:6789/0 conn(0x7f5dd40c6bb0 legacy=0x7f5dd40c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd8e00640 1 -- 172.21.15.146:0/1518082568 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd415bd90 con 0x7f5dd4152850 2024-09-16T01:54:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.565+0000 7f5dd3e00640 1 -- 172.21.15.146:0/1518082568 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559010993 0 0) 0x7f5dd4072f50 con 0x7f5dd40c6bb0 2024-09-16T01:54:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.566+0000 7f5dd8e00640 1 --2- 172.21.15.146:0/1518082568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4158670 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009870 tx=0x7f5dc402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01e233954bf5cf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.566+0000 7f5dd3e00640 1 -- 172.21.15.146:0/1518082568 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4037070 con 0x7f5dd4152850 2024-09-16T01:54:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.566+0000 7f5dd3e00640 1 -- 172.21.15.146:0/1518082568 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc4032440 con 0x7f5dd4152850 2024-09-16T01:54:09.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.566+0000 7f5dd3e00640 1 -- 172.21.15.146:0/1518082568 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4032810 con 0x7f5dd4152850 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 -- 172.21.15.146:0/1518082568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 msgr2=0x7f5dd4158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1518082568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4158670 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009870 tx=0x7f5dc402a270 comp rx=0 tx=0).stop 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 -- 172.21.15.146:0/1518082568 shutdown_connections 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1518082568 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd40c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1518082568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4158670 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 -- 172.21.15.146:0/1518082568 >> 172.21.15.146:0/1518082568 conn(0x7f5dd406f3c0 msgr2=0x7f5dd406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 -- 172.21.15.146:0/1518082568 shutdown_connections 2024-09-16T01:54:09.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.567+0000 7f5ddb800640 1 -- 172.21.15.146:0/1518082568 wait complete. 2024-09-16T01:54:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 Processor -- start 2024-09-16T01:54:09.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 -- start start 2024-09-16T01:54:09.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dd40c6bb0 0x7f5dd40caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd414e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd415c630 con 0x7f5dd4152850 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd415c350 con 0x7f5dd40c6bb0 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5ddb800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd4070200 con 0x7f5dd4158bb0 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58874/0 (socket says 172.21.15.146:58874) 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 -- 172.21.15.146:0/1338024446 learned_addr learned my addr 172.21.15.146:0/1338024446 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dd8e00640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dd40c6bb0 0x7f5dd40caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dd9800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd414e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 -- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 msgr2=0x7f5dd414e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd414e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 -- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dd40c6bb0 msgr2=0x7f5dd40caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dd40c6bb0 0x7f5dd40caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.570+0000 7f5dcbe00640 1 -- 172.21.15.146:0/1338024446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd41f0c20 con 0x7f5dd4152850 2024-09-16T01:54:09.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.571+0000 7f5dcbe00640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4157600 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00e870 tx=0x7f5dbc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:09.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.571+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc017070 con 0x7f5dd4152850 2024-09-16T01:54:09.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.571+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc00c940 con 0x7f5dd4152850 2024-09-16T01:54:09.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.571+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd414d840 con 0x7f5dd4152850 2024-09-16T01:54:09.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.571+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc00cd30 con 0x7f5dd4152850 2024-09-16T01:54:09.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.571+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd414de00 con 0x7f5dd4152850 2024-09-16T01:54:09.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.572+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd40718a0 con 0x7f5dd4152850 2024-09-16T01:54:09.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.573+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5dbc01a070 con 0x7f5dd4152850 2024-09-16T01:54:09.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.574+0000 7f5dd1600640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5db006c2b0 0x7f5db006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.574+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f5dbc08b410 con 0x7f5dd4152850 2024-09-16T01:54:09.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.574+0000 7f5dd8e00640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5db006c2b0 0x7f5db006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.575+0000 7f5dd8e00640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5db006c2b0 0x7f5db006e770 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4034000 tx=0x7f5dc402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:09.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.578+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5dbc059350 con 0x7f5dd4152850 2024-09-16T01:54:09.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.749+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 --> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5dd40728f0 con 0x7f5db006c2b0 2024-09-16T01:54:09.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.758+0000 7f5dd1600640 1 -- 172.21.15.146:0/1338024446 <== mgr.4100 v2:172.21.15.161:6832/3835694012 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254608 (secure 0 0 0) 0x7f5dd40728f0 con 0x7f5db006c2b0 2024-09-16T01:54:09.762 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:54:09.763 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-09-16T01:54:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5db006c2b0 msgr2=0x7f5db006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5db006c2b0 0x7f5db006e770 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4034000 tx=0x7f5dc402e000 comp rx=0 tx=0).stop 2024-09-16T01:54:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 msgr2=0x7f5dd4157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4157600 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00e870 tx=0x7f5dbc00ed30 comp rx=0 tx=0).stop 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 shutdown_connections 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f5db006c2b0 0x7f5db006e770 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5dd4158bb0 0x7f5dd414e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd4152850 0x7f5dd4157600 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 --2- 172.21.15.146:0/1338024446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dd40c6bb0 0x7f5dd40caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 >> 172.21.15.146:0/1338024446 conn(0x7f5dd406f3c0 msgr2=0x7f5dd414f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 shutdown_connections 2024-09-16T01:54:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.761+0000 7f5ddb800640 1 -- 172.21.15.146:0/1338024446 wait complete. 2024-09-16T01:54:09.779 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-16T01:54:08.581318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":989271,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":120,"num_read_kb":104,"num_write":9803,"num_write_kb":25168843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4746,"num_bytes_recovered":9886615120,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11856,"ondisk_log_size":11856,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289964,"kb_used_data":15532,"kb_used_omap":12,"kb_used_meta":274291,"kb_avail":754684756,"statfs":{"total":773094113280,"available":772797190144,"internally_reserved":0,"allocated":15904768,"data_stored":9197550,"data_compressed":124339,"data_compressed_allocated":163840,"data_compressed_original":1536000,"omap_allocated":12752,"internal_metadata":280874544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003076"},"pg_stats":[{"pgid":"3.3f","version":"152'167","reported_seq":511,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:56.347092+0000","last_change":"2024-09-16T01:53:56.347116+0000","last_active":"2024-09-16T01:53:56.347092+0000","last_peered":"2024-09-16T01:53:56.347092+0000","last_clean":"2024-09-16T01:53:56.347092+0000","last_became_active":"2024-09-16T01:53:22.523320+0000","last_became_peered":"2024-09-16T01:53:22.523320+0000","last_unstale":"2024-09-16T01:53:56.347092+0000","last_undegraded":"2024-09-16T01:53:56.347092+0000","last_fullsized":"2024-09-16T01:53:56.347092+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'167","last_scrub_stamp":"2024-09-16T01:53:56.347011+0000","last_deep_scrub":"152'167","last_deep_scrub_stamp":"2024-09-16T01:53:56.347011+0000","last_clean_scrub_stamp":"2024-09-16T01:53:56.347011+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:10:10.985186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"152'152","reported_seq":444,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:55.500490+0000","last_change":"2024-09-16T01:53:55.500517+0000","last_active":"2024-09-16T01:53:55.500490+0000","last_peered":"2024-09-16T01:53:55.500490+0000","last_clean":"2024-09-16T01:53:55.500490+0000","last_became_active":"2024-09-16T01:48:58.825318+0000","last_became_peered":"2024-09-16T01:48:58.825318+0000","last_unstale":"2024-09-16T01:53:55.500490+0000","last_undegraded":"2024-09-16T01:53:55.500490+0000","last_fullsized":"2024-09-16T01:53:55.500490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'152","last_scrub_stamp":"2024-09-16T01:53:55.500401+0000","last_deep_scrub":"152'152","last_deep_scrub_stamp":"2024-09-16T01:53:55.500401+0000","last_clean_scrub_stamp":"2024-09-16T01:53:55.500401+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:30.730368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"152'178","reported_seq":825,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:03.508517+0000","last_change":"2024-09-16T01:54:03.508564+0000","last_active":"2024-09-16T01:54:03.508517+0000","last_peered":"2024-09-16T01:54:03.508517+0000","last_clean":"2024-09-16T01:54:03.508517+0000","last_became_active":"2024-09-16T01:53:23.518010+0000","last_became_peered":"2024-09-16T01:53:23.518010+0000","last_unstale":"2024-09-16T01:54:03.508517+0000","last_undegraded":"2024-09-16T01:54:03.508517+0000","last_fullsized":"2024-09-16T01:54:03.508517+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'178","last_scrub_stamp":"2024-09-16T01:54:03.508417+0000","last_deep_scrub":"152'178","last_deep_scrub_stamp":"2024-09-16T01:54:03.508417+0000","last_clean_scrub_stamp":"2024-09-16T01:54:03.508417+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:52.930166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":381681664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"152'193","reported_seq":609,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.241650+0000","last_change":"2024-09-16T01:53:52.241695+0000","last_active":"2024-09-16T01:53:52.241650+0000","last_peered":"2024-09-16T01:53:52.241650+0000","last_clean":"2024-09-16T01:53:52.241650+0000","last_became_active":"2024-09-16T01:53:32.510334+0000","last_became_peered":"2024-09-16T01:53:32.510334+0000","last_unstale":"2024-09-16T01:53:52.241650+0000","last_undegraded":"2024-09-16T01:53:52.241650+0000","last_fullsized":"2024-09-16T01:53:52.241650+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'193","last_scrub_stamp":"2024-09-16T01:53:52.241546+0000","last_deep_scrub":"152'193","last_deep_scrub_stamp":"2024-09-16T01:53:52.241546+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.241546+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:30.300718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"152'174","reported_seq":545,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.414802+0000","last_change":"2024-09-16T01:53:53.414826+0000","last_active":"2024-09-16T01:53:53.414802+0000","last_peered":"2024-09-16T01:53:53.414802+0000","last_clean":"2024-09-16T01:53:53.414802+0000","last_became_active":"2024-09-16T01:53:22.520262+0000","last_became_peered":"2024-09-16T01:53:22.520262+0000","last_unstale":"2024-09-16T01:53:53.414802+0000","last_undegraded":"2024-09-16T01:53:53.414802+0000","last_fullsized":"2024-09-16T01:53:53.414802+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'174","last_scrub_stamp":"2024-09-16T01:53:53.414716+0000","last_deep_scrub":"152'174","last_deep_scrub_stamp":"2024-09-16T01:53:53.414716+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.414716+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:28.644683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":348127232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"152'162","reported_seq":601,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.276459+0000","last_change":"2024-09-16T01:53:51.276459+0000","last_active":"2024-09-16T01:53:51.276459+0000","last_peered":"2024-09-16T01:53:51.276459+0000","last_clean":"2024-09-16T01:53:51.276459+0000","last_became_active":"2024-09-16T01:53:23.518494+0000","last_became_peered":"2024-09-16T01:53:23.518494+0000","last_unstale":"2024-09-16T01:53:51.276459+0000","last_undegraded":"2024-09-16T01:53:51.276459+0000","last_fullsized":"2024-09-16T01:53:51.276459+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:53:51.276317+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:53:51.276317+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.276317+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:20:18.031841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"152'188","reported_seq":657,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:54.417953+0000","last_change":"2024-09-16T01:53:54.417978+0000","last_active":"2024-09-16T01:53:54.417953+0000","last_peered":"2024-09-16T01:53:54.417953+0000","last_clean":"2024-09-16T01:53:54.417953+0000","last_became_active":"2024-09-16T01:53:22.522319+0000","last_became_peered":"2024-09-16T01:53:22.522319+0000","last_unstale":"2024-09-16T01:53:54.417953+0000","last_undegraded":"2024-09-16T01:53:54.417953+0000","last_fullsized":"2024-09-16T01:53:54.417953+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'188","last_scrub_stamp":"2024-09-16T01:53:54.417869+0000","last_deep_scrub":"152'188","last_deep_scrub_stamp":"2024-09-16T01:53:54.417869+0000","last_clean_scrub_stamp":"2024-09-16T01:53:54.417869+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:49.804514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":268435456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"152'204","reported_seq":751,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:02.473235+0000","last_change":"2024-09-16T01:54:02.473279+0000","last_active":"2024-09-16T01:54:02.473235+0000","last_peered":"2024-09-16T01:54:02.473235+0000","last_clean":"2024-09-16T01:54:02.473235+0000","last_became_active":"2024-09-16T01:53:22.522634+0000","last_became_peered":"2024-09-16T01:53:22.522634+0000","last_unstale":"2024-09-16T01:54:02.473235+0000","last_undegraded":"2024-09-16T01:54:02.473235+0000","last_fullsized":"2024-09-16T01:54:02.473235+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'204","last_scrub_stamp":"2024-09-16T01:54:02.473127+0000","last_deep_scrub":"152'204","last_deep_scrub_stamp":"2024-09-16T01:54:02.473127+0000","last_clean_scrub_stamp":"2024-09-16T01:54:02.473127+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:48:02.870046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"152'178","reported_seq":470,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.209673+0000","last_change":"2024-09-16T01:53:50.209673+0000","last_active":"2024-09-16T01:53:50.209673+0000","last_peered":"2024-09-16T01:53:50.209673+0000","last_clean":"2024-09-16T01:53:50.209673+0000","last_became_active":"2024-09-16T01:48:58.836000+0000","last_became_peered":"2024-09-16T01:48:58.836000+0000","last_unstale":"2024-09-16T01:53:50.209673+0000","last_undegraded":"2024-09-16T01:53:50.209673+0000","last_fullsized":"2024-09-16T01:53:50.209673+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'178","last_scrub_stamp":"2024-09-16T01:53:50.209569+0000","last_deep_scrub":"152'178","last_deep_scrub_stamp":"2024-09-16T01:53:50.209569+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.209569+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:00.746931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"152'190","reported_seq":589,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:57.298361+0000","last_change":"2024-09-16T01:53:57.298383+0000","last_active":"2024-09-16T01:53:57.298361+0000","last_peered":"2024-09-16T01:53:57.298361+0000","last_clean":"2024-09-16T01:53:57.298361+0000","last_became_active":"2024-09-16T01:53:22.524972+0000","last_became_peered":"2024-09-16T01:53:22.524972+0000","last_unstale":"2024-09-16T01:53:57.298361+0000","last_undegraded":"2024-09-16T01:53:57.298361+0000","last_fullsized":"2024-09-16T01:53:57.298361+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'190","last_scrub_stamp":"2024-09-16T01:53:57.298284+0000","last_deep_scrub":"152'190","last_deep_scrub_stamp":"2024-09-16T01:53:57.298284+0000","last_clean_scrub_stamp":"2024-09-16T01:53:57.298284+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:27:44.716222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"152'214","reported_seq":636,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:58.320803+0000","last_change":"2024-09-16T01:53:58.320828+0000","last_active":"2024-09-16T01:53:58.320803+0000","last_peered":"2024-09-16T01:53:58.320803+0000","last_clean":"2024-09-16T01:53:58.320803+0000","last_became_active":"2024-09-16T01:53:23.518245+0000","last_became_peered":"2024-09-16T01:53:23.518245+0000","last_unstale":"2024-09-16T01:53:58.320803+0000","last_undegraded":"2024-09-16T01:53:58.320803+0000","last_fullsized":"2024-09-16T01:53:58.320803+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'214","last_scrub_stamp":"2024-09-16T01:53:58.320723+0000","last_deep_scrub":"152'214","last_deep_scrub_stamp":"2024-09-16T01:53:58.320723+0000","last_clean_scrub_stamp":"2024-09-16T01:53:58.320723+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:12.599562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"152'166","reported_seq":381,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.415329+0000","last_change":"2024-09-16T01:53:52.415352+0000","last_active":"2024-09-16T01:53:52.415329+0000","last_peered":"2024-09-16T01:53:52.415329+0000","last_clean":"2024-09-16T01:53:52.415329+0000","last_became_active":"2024-09-16T01:53:22.518645+0000","last_became_peered":"2024-09-16T01:53:22.518645+0000","last_unstale":"2024-09-16T01:53:52.415329+0000","last_undegraded":"2024-09-16T01:53:52.415329+0000","last_fullsized":"2024-09-16T01:53:52.415329+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'166","last_scrub_stamp":"2024-09-16T01:53:52.415213+0000","last_deep_scrub":"152'166","last_deep_scrub_stamp":"2024-09-16T01:53:52.415213+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.415213+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:28.230995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":620756992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"152'177","reported_seq":562,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:56.019481+0000","last_change":"2024-09-16T01:53:56.019509+0000","last_active":"2024-09-16T01:53:56.019481+0000","last_peered":"2024-09-16T01:53:56.019481+0000","last_clean":"2024-09-16T01:53:56.019481+0000","last_became_active":"2024-09-16T01:53:34.532754+0000","last_became_peered":"2024-09-16T01:53:34.532754+0000","last_unstale":"2024-09-16T01:53:56.019481+0000","last_undegraded":"2024-09-16T01:53:56.019481+0000","last_fullsized":"2024-09-16T01:53:56.019481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'177","last_scrub_stamp":"2024-09-16T01:53:56.019399+0000","last_deep_scrub":"152'177","last_deep_scrub_stamp":"2024-09-16T01:53:56.019399+0000","last_clean_scrub_stamp":"2024-09-16T01:53:56.019399+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:50:26.424125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"152'172","reported_seq":515,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.070291+0000","last_change":"2024-09-16T01:53:49.070334+0000","last_active":"2024-09-16T01:53:49.070291+0000","last_peered":"2024-09-16T01:53:49.070291+0000","last_clean":"2024-09-16T01:53:49.070291+0000","last_became_active":"2024-09-16T01:51:18.924678+0000","last_became_peered":"2024-09-16T01:51:18.924678+0000","last_unstale":"2024-09-16T01:53:49.070291+0000","last_undegraded":"2024-09-16T01:53:49.070291+0000","last_fullsized":"2024-09-16T01:53:49.070291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'172","last_scrub_stamp":"2024-09-16T01:53:49.070193+0000","last_deep_scrub":"152'172","last_deep_scrub_stamp":"2024-09-16T01:53:49.070193+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.070193+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:49:59.607555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"152'177","reported_seq":782,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:04.511680+0000","last_change":"2024-09-16T01:54:04.511704+0000","last_active":"2024-09-16T01:54:04.511680+0000","last_peered":"2024-09-16T01:54:04.511680+0000","last_clean":"2024-09-16T01:54:04.511680+0000","last_became_active":"2024-09-16T01:53:45.513061+0000","last_became_peered":"2024-09-16T01:53:45.513061+0000","last_unstale":"2024-09-16T01:54:04.511680+0000","last_undegraded":"2024-09-16T01:54:04.511680+0000","last_fullsized":"2024-09-16T01:54:04.511680+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'177","last_scrub_stamp":"2024-09-16T01:54:04.511598+0000","last_deep_scrub":"152'177","last_deep_scrub_stamp":"2024-09-16T01:54:04.511598+0000","last_clean_scrub_stamp":"2024-09-16T01:54:04.511598+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:46.287930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"152'162","reported_seq":454,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:47.682762+0000","last_change":"2024-09-16T01:53:47.682798+0000","last_active":"2024-09-16T01:53:47.682762+0000","last_peered":"2024-09-16T01:53:47.682762+0000","last_clean":"2024-09-16T01:53:47.682762+0000","last_became_active":"2024-09-16T01:48:58.830786+0000","last_became_peered":"2024-09-16T01:48:58.830786+0000","last_unstale":"2024-09-16T01:53:47.682762+0000","last_undegraded":"2024-09-16T01:53:47.682762+0000","last_fullsized":"2024-09-16T01:53:47.682762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:53:47.682683+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:53:47.682683+0000","last_clean_scrub_stamp":"2024-09-16T01:53:47.682683+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:42.621582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"152'176","reported_seq":666,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:05.703558+0000","last_change":"2024-09-16T01:54:05.703595+0000","last_active":"2024-09-16T01:54:05.703558+0000","last_peered":"2024-09-16T01:54:05.703558+0000","last_clean":"2024-09-16T01:54:05.703558+0000","last_became_active":"2024-09-16T01:53:23.525296+0000","last_became_peered":"2024-09-16T01:53:23.525296+0000","last_unstale":"2024-09-16T01:54:05.703558+0000","last_undegraded":"2024-09-16T01:54:05.703558+0000","last_fullsized":"2024-09-16T01:54:05.703558+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'176","last_scrub_stamp":"2024-09-16T01:54:05.703459+0000","last_deep_scrub":"152'176","last_deep_scrub_stamp":"2024-09-16T01:54:05.703459+0000","last_clean_scrub_stamp":"2024-09-16T01:54:05.703459+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:48.596553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"152'179","reported_seq":604,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:56.102111+0000","last_change":"2024-09-16T01:53:56.102153+0000","last_active":"2024-09-16T01:53:56.102111+0000","last_peered":"2024-09-16T01:53:56.102111+0000","last_clean":"2024-09-16T01:53:56.102111+0000","last_became_active":"2024-09-16T01:53:36.563349+0000","last_became_peered":"2024-09-16T01:53:36.563349+0000","last_unstale":"2024-09-16T01:53:56.102111+0000","last_undegraded":"2024-09-16T01:53:56.102111+0000","last_fullsized":"2024-09-16T01:53:56.102111+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'179","last_scrub_stamp":"2024-09-16T01:53:56.102004+0000","last_deep_scrub":"152'179","last_deep_scrub_stamp":"2024-09-16T01:53:56.102004+0000","last_clean_scrub_stamp":"2024-09-16T01:53:56.102004+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:15.359184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":343932928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"152'205","reported_seq":677,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.102300+0000","last_change":"2024-09-16T01:53:52.102335+0000","last_active":"2024-09-16T01:53:52.102300+0000","last_peered":"2024-09-16T01:53:52.102300+0000","last_clean":"2024-09-16T01:53:52.102300+0000","last_became_active":"2024-09-16T01:53:36.553810+0000","last_became_peered":"2024-09-16T01:53:36.553810+0000","last_unstale":"2024-09-16T01:53:52.102300+0000","last_undegraded":"2024-09-16T01:53:52.102300+0000","last_fullsized":"2024-09-16T01:53:52.102300+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'205","last_scrub_stamp":"2024-09-16T01:53:52.102221+0000","last_deep_scrub":"152'205","last_deep_scrub_stamp":"2024-09-16T01:53:52.102221+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.102221+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:32.084836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"152'179","reported_seq":615,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:59.306677+0000","last_change":"2024-09-16T01:53:59.306701+0000","last_active":"2024-09-16T01:53:59.306677+0000","last_peered":"2024-09-16T01:53:59.306677+0000","last_clean":"2024-09-16T01:53:59.306677+0000","last_became_active":"2024-09-16T01:53:23.520976+0000","last_became_peered":"2024-09-16T01:53:23.520976+0000","last_unstale":"2024-09-16T01:53:59.306677+0000","last_undegraded":"2024-09-16T01:53:59.306677+0000","last_fullsized":"2024-09-16T01:53:59.306677+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'179","last_scrub_stamp":"2024-09-16T01:53:59.306594+0000","last_deep_scrub":"152'179","last_deep_scrub_stamp":"2024-09-16T01:53:59.306594+0000","last_clean_scrub_stamp":"2024-09-16T01:53:59.306594+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:43.715491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"152'154","reported_seq":485,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:58.285228+0000","last_change":"2024-09-16T01:53:58.285253+0000","last_active":"2024-09-16T01:53:58.285228+0000","last_peered":"2024-09-16T01:53:58.285228+0000","last_clean":"2024-09-16T01:53:58.285228+0000","last_became_active":"2024-09-16T01:53:23.519683+0000","last_became_peered":"2024-09-16T01:53:23.519683+0000","last_unstale":"2024-09-16T01:53:58.285228+0000","last_undegraded":"2024-09-16T01:53:58.285228+0000","last_fullsized":"2024-09-16T01:53:58.285228+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'154","last_scrub_stamp":"2024-09-16T01:53:58.285104+0000","last_deep_scrub":"152'154","last_deep_scrub_stamp":"2024-09-16T01:53:58.285104+0000","last_clean_scrub_stamp":"2024-09-16T01:53:58.285104+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:30:40.770396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"152'199","reported_seq":849,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.239327+0000","last_change":"2024-09-16T01:53:53.239392+0000","last_active":"2024-09-16T01:53:53.239327+0000","last_peered":"2024-09-16T01:53:53.239327+0000","last_clean":"2024-09-16T01:53:53.239327+0000","last_became_active":"2024-09-16T01:53:22.520646+0000","last_became_peered":"2024-09-16T01:53:22.520646+0000","last_unstale":"2024-09-16T01:53:53.239327+0000","last_undegraded":"2024-09-16T01:53:53.239327+0000","last_fullsized":"2024-09-16T01:53:53.239327+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'199","last_scrub_stamp":"2024-09-16T01:53:53.239211+0000","last_deep_scrub":"152'199","last_deep_scrub_stamp":"2024-09-16T01:53:53.239211+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.239211+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:55.924968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":377487360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"152'150","reported_seq":442,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:56.526037+0000","last_change":"2024-09-16T01:53:56.526079+0000","last_active":"2024-09-16T01:53:56.526037+0000","last_peered":"2024-09-16T01:53:56.526037+0000","last_clean":"2024-09-16T01:53:56.526037+0000","last_became_active":"2024-09-16T01:48:58.835098+0000","last_became_peered":"2024-09-16T01:48:58.835098+0000","last_unstale":"2024-09-16T01:53:56.526037+0000","last_undegraded":"2024-09-16T01:53:56.526037+0000","last_fullsized":"2024-09-16T01:53:56.526037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'150","last_scrub_stamp":"2024-09-16T01:53:56.525924+0000","last_deep_scrub":"152'150","last_deep_scrub_stamp":"2024-09-16T01:53:56.525924+0000","last_clean_scrub_stamp":"2024-09-16T01:53:56.525924+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:10.970827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"152'207","reported_seq":499,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:55.724193+0000","last_change":"2024-09-16T01:53:55.724219+0000","last_active":"2024-09-16T01:53:55.724193+0000","last_peered":"2024-09-16T01:53:55.724193+0000","last_clean":"2024-09-16T01:53:55.724193+0000","last_became_active":"2024-09-16T01:48:58.822636+0000","last_became_peered":"2024-09-16T01:48:58.822636+0000","last_unstale":"2024-09-16T01:53:55.724193+0000","last_undegraded":"2024-09-16T01:53:55.724193+0000","last_fullsized":"2024-09-16T01:53:55.724193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'207","last_scrub_stamp":"2024-09-16T01:53:55.724109+0000","last_deep_scrub":"152'207","last_deep_scrub_stamp":"2024-09-16T01:53:55.724109+0000","last_clean_scrub_stamp":"2024-09-16T01:53:55.724109+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:48.676260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"152'160","reported_seq":452,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:54.476150+0000","last_change":"2024-09-16T01:53:54.476191+0000","last_active":"2024-09-16T01:53:54.476150+0000","last_peered":"2024-09-16T01:53:54.476150+0000","last_clean":"2024-09-16T01:53:54.476150+0000","last_became_active":"2024-09-16T01:48:58.821062+0000","last_became_peered":"2024-09-16T01:48:58.821062+0000","last_unstale":"2024-09-16T01:53:54.476150+0000","last_undegraded":"2024-09-16T01:53:54.476150+0000","last_fullsized":"2024-09-16T01:53:54.476150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'160","last_scrub_stamp":"2024-09-16T01:53:54.476053+0000","last_deep_scrub":"152'160","last_deep_scrub_stamp":"2024-09-16T01:53:54.476053+0000","last_clean_scrub_stamp":"2024-09-16T01:53:54.476053+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:30.291836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"152'217","reported_seq":509,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:47.347093+0000","last_change":"2024-09-16T01:53:47.347133+0000","last_active":"2024-09-16T01:53:47.347093+0000","last_peered":"2024-09-16T01:53:47.347093+0000","last_clean":"2024-09-16T01:53:47.347093+0000","last_became_active":"2024-09-16T01:48:58.838878+0000","last_became_peered":"2024-09-16T01:48:58.838878+0000","last_unstale":"2024-09-16T01:53:47.347093+0000","last_undegraded":"2024-09-16T01:53:47.347093+0000","last_fullsized":"2024-09-16T01:53:47.347093+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'217","last_scrub_stamp":"2024-09-16T01:53:47.346999+0000","last_deep_scrub":"152'217","last_deep_scrub_stamp":"2024-09-16T01:53:47.346999+0000","last_clean_scrub_stamp":"2024-09-16T01:53:47.346999+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:48.012341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"152'166","reported_seq":689,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.065582+0000","last_change":"2024-09-16T01:53:50.065608+0000","last_active":"2024-09-16T01:53:50.065582+0000","last_peered":"2024-09-16T01:53:50.065582+0000","last_clean":"2024-09-16T01:53:50.065582+0000","last_became_active":"2024-09-16T01:52:49.494749+0000","last_became_peered":"2024-09-16T01:52:49.494749+0000","last_unstale":"2024-09-16T01:53:50.065582+0000","last_undegraded":"2024-09-16T01:53:50.065582+0000","last_fullsized":"2024-09-16T01:53:50.065582+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'166","last_scrub_stamp":"2024-09-16T01:53:50.065496+0000","last_deep_scrub":"152'166","last_deep_scrub_stamp":"2024-09-16T01:53:50.065496+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.065496+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:31.307102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":335544320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"152'152","reported_seq":507,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:55.011220+0000","last_change":"2024-09-16T01:53:55.011272+0000","last_active":"2024-09-16T01:53:55.011220+0000","last_peered":"2024-09-16T01:53:55.011220+0000","last_clean":"2024-09-16T01:53:55.011220+0000","last_became_active":"2024-09-16T01:53:34.532678+0000","last_became_peered":"2024-09-16T01:53:34.532678+0000","last_unstale":"2024-09-16T01:53:55.011220+0000","last_undegraded":"2024-09-16T01:53:55.011220+0000","last_fullsized":"2024-09-16T01:53:55.011220+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'152","last_scrub_stamp":"2024-09-16T01:53:55.011117+0000","last_deep_scrub":"152'152","last_deep_scrub_stamp":"2024-09-16T01:53:55.011117+0000","last_clean_scrub_stamp":"2024-09-16T01:53:55.011117+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:18:52.576622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"152'134","reported_seq":709,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:00.476315+0000","last_change":"2024-09-16T01:54:00.476358+0000","last_active":"2024-09-16T01:54:00.476315+0000","last_peered":"2024-09-16T01:54:00.476315+0000","last_clean":"2024-09-16T01:54:00.476315+0000","last_became_active":"2024-09-16T01:52:46.542553+0000","last_became_peered":"2024-09-16T01:52:46.542553+0000","last_unstale":"2024-09-16T01:54:00.476315+0000","last_undegraded":"2024-09-16T01:54:00.476315+0000","last_fullsized":"2024-09-16T01:54:00.476315+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'134","last_scrub_stamp":"2024-09-16T01:54:00.476215+0000","last_deep_scrub":"152'134","last_deep_scrub_stamp":"2024-09-16T01:54:00.476215+0000","last_clean_scrub_stamp":"2024-09-16T01:54:00.476215+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:36:45.470384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":360710144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"152'153","reported_seq":445,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.677870+0000","last_change":"2024-09-16T01:53:49.677890+0000","last_active":"2024-09-16T01:53:49.677870+0000","last_peered":"2024-09-16T01:53:49.677870+0000","last_clean":"2024-09-16T01:53:49.677870+0000","last_became_active":"2024-09-16T01:48:58.826881+0000","last_became_peered":"2024-09-16T01:48:58.826881+0000","last_unstale":"2024-09-16T01:53:49.677870+0000","last_undegraded":"2024-09-16T01:53:49.677870+0000","last_fullsized":"2024-09-16T01:53:49.677870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'153","last_scrub_stamp":"2024-09-16T01:53:49.677804+0000","last_deep_scrub":"152'153","last_deep_scrub_stamp":"2024-09-16T01:53:49.677804+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.677804+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:49.239976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"152'158","reported_seq":589,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:55.105154+0000","last_change":"2024-09-16T01:53:55.105198+0000","last_active":"2024-09-16T01:53:55.105154+0000","last_peered":"2024-09-16T01:53:55.105154+0000","last_clean":"2024-09-16T01:53:55.105154+0000","last_became_active":"2024-09-16T01:53:36.562698+0000","last_became_peered":"2024-09-16T01:53:36.562698+0000","last_unstale":"2024-09-16T01:53:55.105154+0000","last_undegraded":"2024-09-16T01:53:55.105154+0000","last_fullsized":"2024-09-16T01:53:55.105154+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'158","last_scrub_stamp":"2024-09-16T01:53:55.105054+0000","last_deep_scrub":"152'158","last_deep_scrub_stamp":"2024-09-16T01:53:55.105054+0000","last_clean_scrub_stamp":"2024-09-16T01:53:55.105054+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:04:36.261938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":310378496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"152'180","reported_seq":561,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.099295+0000","last_change":"2024-09-16T01:53:49.099321+0000","last_active":"2024-09-16T01:53:49.099295+0000","last_peered":"2024-09-16T01:53:49.099295+0000","last_clean":"2024-09-16T01:53:49.099295+0000","last_became_active":"2024-09-16T01:53:36.550952+0000","last_became_peered":"2024-09-16T01:53:36.550952+0000","last_unstale":"2024-09-16T01:53:49.099295+0000","last_undegraded":"2024-09-16T01:53:49.099295+0000","last_fullsized":"2024-09-16T01:53:49.099295+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'180","last_scrub_stamp":"2024-09-16T01:53:49.099211+0000","last_deep_scrub":"152'180","last_deep_scrub_stamp":"2024-09-16T01:53:49.099211+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.099211+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:15.504342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"152'149","reported_seq":441,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:48.244139+0000","last_change":"2024-09-16T01:53:48.244139+0000","last_active":"2024-09-16T01:53:48.244139+0000","last_peered":"2024-09-16T01:53:48.244139+0000","last_clean":"2024-09-16T01:53:48.244139+0000","last_became_active":"2024-09-16T01:48:58.820438+0000","last_became_peered":"2024-09-16T01:48:58.820438+0000","last_unstale":"2024-09-16T01:53:48.244139+0000","last_undegraded":"2024-09-16T01:53:48.244139+0000","last_fullsized":"2024-09-16T01:53:48.244139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'149","last_scrub_stamp":"2024-09-16T01:53:48.244048+0000","last_deep_scrub":"152'149","last_deep_scrub_stamp":"2024-09-16T01:53:48.244048+0000","last_clean_scrub_stamp":"2024-09-16T01:53:48.244048+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:09:37.259486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"157'7","reported_seq":301,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.506671+0000","last_change":"2024-09-16T01:53:50.506671+0000","last_active":"2024-09-16T01:53:50.506671+0000","last_peered":"2024-09-16T01:53:50.506671+0000","last_clean":"2024-09-16T01:53:50.506671+0000","last_became_active":"2024-09-16T01:48:57.807914+0000","last_became_peered":"2024-09-16T01:48:57.807914+0000","last_unstale":"2024-09-16T01:53:50.506671+0000","last_undegraded":"2024-09-16T01:53:50.506671+0000","last_fullsized":"2024-09-16T01:53:50.506671+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'7","last_scrub_stamp":"2024-09-16T01:53:50.506526+0000","last_deep_scrub":"157'7","last_deep_scrub_stamp":"2024-09-16T01:53:50.506526+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.506526+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:42.286116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"152'162","reported_seq":789,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.150280+0000","last_change":"2024-09-16T01:53:53.150306+0000","last_active":"2024-09-16T01:53:53.150280+0000","last_peered":"2024-09-16T01:53:53.150280+0000","last_clean":"2024-09-16T01:53:53.150280+0000","last_became_active":"2024-09-16T01:53:22.520180+0000","last_became_peered":"2024-09-16T01:53:22.520180+0000","last_unstale":"2024-09-16T01:53:53.150280+0000","last_undegraded":"2024-09-16T01:53:53.150280+0000","last_fullsized":"2024-09-16T01:53:53.150280+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:53:53.150198+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:53:53.150198+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.150198+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:24:34.524890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":486539264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"157'10","reported_seq":330,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:59.710816+0000","last_change":"2024-09-16T01:53:59.710816+0000","last_active":"2024-09-16T01:53:59.710816+0000","last_peered":"2024-09-16T01:53:59.710816+0000","last_clean":"2024-09-16T01:53:59.710816+0000","last_became_active":"2024-09-16T01:53:19.522667+0000","last_became_peered":"2024-09-16T01:53:19.522667+0000","last_unstale":"2024-09-16T01:53:59.710816+0000","last_undegraded":"2024-09-16T01:53:59.710816+0000","last_fullsized":"2024-09-16T01:53:59.710816+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'10","last_scrub_stamp":"2024-09-16T01:53:59.710711+0000","last_deep_scrub":"157'10","last_deep_scrub_stamp":"2024-09-16T01:53:59.710711+0000","last_clean_scrub_stamp":"2024-09-16T01:53:59.710711+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:56.217738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"157'5","reported_seq":297,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.048800+0000","last_change":"2024-09-16T01:53:53.048832+0000","last_active":"2024-09-16T01:53:53.048800+0000","last_peered":"2024-09-16T01:53:53.048800+0000","last_clean":"2024-09-16T01:53:53.048800+0000","last_became_active":"2024-09-16T01:53:34.532490+0000","last_became_peered":"2024-09-16T01:53:34.532490+0000","last_unstale":"2024-09-16T01:53:53.048800+0000","last_undegraded":"2024-09-16T01:53:53.048800+0000","last_fullsized":"2024-09-16T01:53:53.048800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'5","last_scrub_stamp":"2024-09-16T01:53:53.048715+0000","last_deep_scrub":"157'5","last_deep_scrub_stamp":"2024-09-16T01:53:53.048715+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.048715+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:48.444413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"152'190","reported_seq":482,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.466205+0000","last_change":"2024-09-16T01:53:52.466231+0000","last_active":"2024-09-16T01:53:52.466205+0000","last_peered":"2024-09-16T01:53:52.466205+0000","last_clean":"2024-09-16T01:53:52.466205+0000","last_became_active":"2024-09-16T01:48:58.819273+0000","last_became_peered":"2024-09-16T01:48:58.819273+0000","last_unstale":"2024-09-16T01:53:52.466205+0000","last_undegraded":"2024-09-16T01:53:52.466205+0000","last_fullsized":"2024-09-16T01:53:52.466205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'190","last_scrub_stamp":"2024-09-16T01:53:52.466117+0000","last_deep_scrub":"152'190","last_deep_scrub_stamp":"2024-09-16T01:53:52.466117+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.466117+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:36.136259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"157'4","reported_seq":298,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.715185+0000","last_change":"2024-09-16T01:53:50.715185+0000","last_active":"2024-09-16T01:53:50.715185+0000","last_peered":"2024-09-16T01:53:50.715185+0000","last_clean":"2024-09-16T01:53:50.715185+0000","last_became_active":"2024-09-16T01:48:57.809836+0000","last_became_peered":"2024-09-16T01:48:57.809836+0000","last_unstale":"2024-09-16T01:53:50.715185+0000","last_undegraded":"2024-09-16T01:53:50.715185+0000","last_fullsized":"2024-09-16T01:53:50.715185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'4","last_scrub_stamp":"2024-09-16T01:53:50.715029+0000","last_deep_scrub":"157'4","last_deep_scrub_stamp":"2024-09-16T01:53:50.715029+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.715029+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:34.385171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"152'187","reported_seq":447,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:59.291645+0000","last_change":"2024-09-16T01:53:59.291671+0000","last_active":"2024-09-16T01:53:59.291645+0000","last_peered":"2024-09-16T01:53:59.291645+0000","last_clean":"2024-09-16T01:53:59.291645+0000","last_became_active":"2024-09-16T01:53:32.512732+0000","last_became_peered":"2024-09-16T01:53:32.512732+0000","last_unstale":"2024-09-16T01:53:59.291645+0000","last_undegraded":"2024-09-16T01:53:59.291645+0000","last_fullsized":"2024-09-16T01:53:59.291645+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'187","last_scrub_stamp":"2024-09-16T01:53:59.291551+0000","last_deep_scrub":"152'187","last_deep_scrub_stamp":"2024-09-16T01:53:59.291551+0000","last_clean_scrub_stamp":"2024-09-16T01:53:59.291551+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:33.041948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"157'8","reported_seq":340,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.255877+0000","last_change":"2024-09-16T01:53:51.255877+0000","last_active":"2024-09-16T01:53:51.255877+0000","last_peered":"2024-09-16T01:53:51.255877+0000","last_clean":"2024-09-16T01:53:51.255877+0000","last_became_active":"2024-09-16T01:53:20.504571+0000","last_became_peered":"2024-09-16T01:53:20.504571+0000","last_unstale":"2024-09-16T01:53:51.255877+0000","last_undegraded":"2024-09-16T01:53:51.255877+0000","last_fullsized":"2024-09-16T01:53:51.255877+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'8","last_scrub_stamp":"2024-09-16T01:53:51.255764+0000","last_deep_scrub":"157'8","last_deep_scrub_stamp":"2024-09-16T01:53:51.255764+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.255764+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:32.497370+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"152'251","reported_seq":556,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:58.497463+0000","last_change":"2024-09-16T01:53:58.497487+0000","last_active":"2024-09-16T01:53:58.497463+0000","last_peered":"2024-09-16T01:53:58.497463+0000","last_clean":"2024-09-16T01:53:58.497463+0000","last_became_active":"2024-09-16T01:52:45.561828+0000","last_became_peered":"2024-09-16T01:52:45.561828+0000","last_unstale":"2024-09-16T01:53:58.497463+0000","last_undegraded":"2024-09-16T01:53:58.497463+0000","last_fullsized":"2024-09-16T01:53:58.497463+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'251","last_scrub_stamp":"2024-09-16T01:53:58.497379+0000","last_deep_scrub":"152'251","last_deep_scrub_stamp":"2024-09-16T01:53:58.497379+0000","last_clean_scrub_stamp":"2024-09-16T01:53:58.497379+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:58.406010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":260046848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"157'6","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.428924+0000","last_change":"2024-09-16T01:53:51.428959+0000","last_active":"2024-09-16T01:53:51.428924+0000","last_peered":"2024-09-16T01:53:51.428924+0000","last_clean":"2024-09-16T01:53:51.428924+0000","last_became_active":"2024-09-16T01:53:19.524265+0000","last_became_peered":"2024-09-16T01:53:19.524265+0000","last_unstale":"2024-09-16T01:53:51.428924+0000","last_undegraded":"2024-09-16T01:53:51.428924+0000","last_fullsized":"2024-09-16T01:53:51.428924+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'6","last_scrub_stamp":"2024-09-16T01:53:51.428822+0000","last_deep_scrub":"157'6","last_deep_scrub_stamp":"2024-09-16T01:53:51.428822+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.428822+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:24.558847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"152'160","reported_seq":452,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:57.524442+0000","last_change":"2024-09-16T01:53:57.524486+0000","last_active":"2024-09-16T01:53:57.524442+0000","last_peered":"2024-09-16T01:53:57.524442+0000","last_clean":"2024-09-16T01:53:57.524442+0000","last_became_active":"2024-09-16T01:48:58.838477+0000","last_became_peered":"2024-09-16T01:48:58.838477+0000","last_unstale":"2024-09-16T01:53:57.524442+0000","last_undegraded":"2024-09-16T01:53:57.524442+0000","last_fullsized":"2024-09-16T01:53:57.524442+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'160","last_scrub_stamp":"2024-09-16T01:53:57.524335+0000","last_deep_scrub":"152'160","last_deep_scrub_stamp":"2024-09-16T01:53:57.524335+0000","last_clean_scrub_stamp":"2024-09-16T01:53:57.524335+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:41.757784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"157'15","reported_seq":183,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.097165+0000","last_change":"2024-09-16T01:53:53.097200+0000","last_active":"2024-09-16T01:53:53.097165+0000","last_peered":"2024-09-16T01:53:53.097165+0000","last_clean":"2024-09-16T01:53:53.097165+0000","last_became_active":"2024-09-16T01:53:36.556486+0000","last_became_peered":"2024-09-16T01:53:36.556486+0000","last_unstale":"2024-09-16T01:53:53.097165+0000","last_undegraded":"2024-09-16T01:53:53.097165+0000","last_fullsized":"2024-09-16T01:53:53.097165+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'15","last_scrub_stamp":"2024-09-16T01:53:53.097082+0000","last_deep_scrub":"157'15","last_deep_scrub_stamp":"2024-09-16T01:53:53.097082+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.097082+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:42:09.465416+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":4032,"num_keys_recovered":60,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'168","reported_seq":460,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:48.710175+0000","last_change":"2024-09-16T01:53:48.710196+0000","last_active":"2024-09-16T01:53:48.710175+0000","last_peered":"2024-09-16T01:53:48.710175+0000","last_clean":"2024-09-16T01:53:48.710175+0000","last_became_active":"2024-09-16T01:48:58.838319+0000","last_became_peered":"2024-09-16T01:48:58.838319+0000","last_unstale":"2024-09-16T01:53:48.710175+0000","last_undegraded":"2024-09-16T01:53:48.710175+0000","last_fullsized":"2024-09-16T01:53:48.710175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'168","last_scrub_stamp":"2024-09-16T01:53:48.710104+0000","last_deep_scrub":"152'168","last_deep_scrub_stamp":"2024-09-16T01:53:48.710104+0000","last_clean_scrub_stamp":"2024-09-16T01:53:48.710104+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:40.489693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"157'182","reported_seq":518,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.318162+0000","last_change":"2024-09-16T01:53:50.318199+0000","last_active":"2024-09-16T01:53:50.318162+0000","last_peered":"2024-09-16T01:53:50.318162+0000","last_clean":"2024-09-16T01:53:50.318162+0000","last_became_active":"2024-09-16T01:53:23.518277+0000","last_became_peered":"2024-09-16T01:53:23.518277+0000","last_unstale":"2024-09-16T01:53:50.318162+0000","last_undegraded":"2024-09-16T01:53:50.318162+0000","last_fullsized":"2024-09-16T01:53:50.318162+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'182","last_scrub_stamp":"2024-09-16T01:53:50.318088+0000","last_deep_scrub":"157'182","last_deep_scrub_stamp":"2024-09-16T01:53:50.318088+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.318088+0000","objects_scrubbed":6,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:12.144744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391576,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":628902,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"152'176","reported_seq":552,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:55.385194+0000","last_change":"2024-09-16T01:53:55.385218+0000","last_active":"2024-09-16T01:53:55.385194+0000","last_peered":"2024-09-16T01:53:55.385194+0000","last_clean":"2024-09-16T01:53:55.385194+0000","last_became_active":"2024-09-16T01:53:22.519283+0000","last_became_peered":"2024-09-16T01:53:22.519283+0000","last_unstale":"2024-09-16T01:53:55.385194+0000","last_undegraded":"2024-09-16T01:53:55.385194+0000","last_fullsized":"2024-09-16T01:53:55.385194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'176","last_scrub_stamp":"2024-09-16T01:53:55.385086+0000","last_deep_scrub":"152'176","last_deep_scrub_stamp":"2024-09-16T01:53:55.385086+0000","last_clean_scrub_stamp":"2024-09-16T01:53:55.385086+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:39.618535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"157'15","reported_seq":289,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:54.093886+0000","last_change":"2024-09-16T01:53:54.093920+0000","last_active":"2024-09-16T01:53:54.093886+0000","last_peered":"2024-09-16T01:53:54.093886+0000","last_clean":"2024-09-16T01:53:54.093886+0000","last_became_active":"2024-09-16T01:53:36.559266+0000","last_became_peered":"2024-09-16T01:53:36.559266+0000","last_unstale":"2024-09-16T01:53:54.093886+0000","last_undegraded":"2024-09-16T01:53:54.093886+0000","last_fullsized":"2024-09-16T01:53:54.093886+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'15","last_scrub_stamp":"2024-09-16T01:53:54.093796+0000","last_deep_scrub":"157'15","last_deep_scrub_stamp":"2024-09-16T01:53:54.093796+0000","last_clean_scrub_stamp":"2024-09-16T01:53:54.093796+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:11.759464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":272,"num_keys_recovered":40,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"152'177","reported_seq":439,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.040084+0000","last_change":"2024-09-16T01:53:51.040109+0000","last_active":"2024-09-16T01:53:51.040084+0000","last_peered":"2024-09-16T01:53:51.040084+0000","last_clean":"2024-09-16T01:53:51.040084+0000","last_became_active":"2024-09-16T01:53:34.529089+0000","last_became_peered":"2024-09-16T01:53:34.529089+0000","last_unstale":"2024-09-16T01:53:51.040084+0000","last_undegraded":"2024-09-16T01:53:51.040084+0000","last_fullsized":"2024-09-16T01:53:51.040084+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'177","last_scrub_stamp":"2024-09-16T01:53:51.039995+0000","last_deep_scrub":"152'177","last_deep_scrub_stamp":"2024-09-16T01:53:51.039995+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.039995+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:07.294468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"157'8","reported_seq":380,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:01.714823+0000","last_change":"2024-09-16T01:54:01.714857+0000","last_active":"2024-09-16T01:54:01.714823+0000","last_peered":"2024-09-16T01:54:01.714823+0000","last_clean":"2024-09-16T01:54:01.714823+0000","last_became_active":"2024-09-16T01:52:46.561983+0000","last_became_peered":"2024-09-16T01:52:46.561983+0000","last_unstale":"2024-09-16T01:54:01.714823+0000","last_undegraded":"2024-09-16T01:54:01.714823+0000","last_fullsized":"2024-09-16T01:54:01.714823+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'8","last_scrub_stamp":"2024-09-16T01:54:01.714744+0000","last_deep_scrub":"157'8","last_deep_scrub_stamp":"2024-09-16T01:54:01.714744+0000","last_clean_scrub_stamp":"2024-09-16T01:54:01.714744+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:08.823499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"152'201","reported_seq":737,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:04.740785+0000","last_change":"2024-09-16T01:54:04.740811+0000","last_active":"2024-09-16T01:54:04.740785+0000","last_peered":"2024-09-16T01:54:04.740785+0000","last_clean":"2024-09-16T01:54:04.740785+0000","last_became_active":"2024-09-16T01:53:23.520052+0000","last_became_peered":"2024-09-16T01:53:23.520052+0000","last_unstale":"2024-09-16T01:54:04.740785+0000","last_undegraded":"2024-09-16T01:54:04.740785+0000","last_fullsized":"2024-09-16T01:54:04.740785+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'201","last_scrub_stamp":"2024-09-16T01:54:04.740696+0000","last_deep_scrub":"152'201","last_deep_scrub_stamp":"2024-09-16T01:54:04.740696+0000","last_clean_scrub_stamp":"2024-09-16T01:54:04.740696+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:11.818008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":213909504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"157'9","reported_seq":74,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:47.460274+0000","last_change":"2024-09-16T01:53:47.460299+0000","last_active":"2024-09-16T01:53:47.460274+0000","last_peered":"2024-09-16T01:53:47.460274+0000","last_clean":"2024-09-16T01:53:47.460274+0000","last_became_active":"2024-09-16T01:53:19.519660+0000","last_became_peered":"2024-09-16T01:53:19.519660+0000","last_unstale":"2024-09-16T01:53:47.460274+0000","last_undegraded":"2024-09-16T01:53:47.460274+0000","last_fullsized":"2024-09-16T01:53:47.460274+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'9","last_scrub_stamp":"2024-09-16T01:53:47.460212+0000","last_deep_scrub":"157'9","last_deep_scrub_stamp":"2024-09-16T01:53:47.460212+0000","last_clean_scrub_stamp":"2024-09-16T01:53:47.460212+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:42.399111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"152'197","reported_seq":489,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.253817+0000","last_change":"2024-09-16T01:53:49.253840+0000","last_active":"2024-09-16T01:53:49.253817+0000","last_peered":"2024-09-16T01:53:49.253817+0000","last_clean":"2024-09-16T01:53:49.253817+0000","last_became_active":"2024-09-16T01:48:58.821919+0000","last_became_peered":"2024-09-16T01:48:58.821919+0000","last_unstale":"2024-09-16T01:53:49.253817+0000","last_undegraded":"2024-09-16T01:53:49.253817+0000","last_fullsized":"2024-09-16T01:53:49.253817+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'197","last_scrub_stamp":"2024-09-16T01:53:49.253737+0000","last_deep_scrub":"152'197","last_deep_scrub_stamp":"2024-09-16T01:53:49.253737+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.253737+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:22.211187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"157'26","reported_seq":87,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:48.143891+0000","last_change":"2024-09-16T01:53:48.143943+0000","last_active":"2024-09-16T01:53:48.143891+0000","last_peered":"2024-09-16T01:53:48.143891+0000","last_clean":"2024-09-16T01:53:48.143891+0000","last_became_active":"2024-09-16T01:53:20.507900+0000","last_became_peered":"2024-09-16T01:53:20.507900+0000","last_unstale":"2024-09-16T01:53:48.143891+0000","last_undegraded":"2024-09-16T01:53:48.143891+0000","last_fullsized":"2024-09-16T01:53:48.143891+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'26","last_scrub_stamp":"2024-09-16T01:53:48.143793+0000","last_deep_scrub":"157'26","last_deep_scrub_stamp":"2024-09-16T01:53:48.143793+0000","last_clean_scrub_stamp":"2024-09-16T01:53:48.143793+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:10.563681+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":896,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"152'215","reported_seq":507,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:58.686417+0000","last_change":"2024-09-16T01:53:58.686442+0000","last_active":"2024-09-16T01:53:58.686417+0000","last_peered":"2024-09-16T01:53:58.686417+0000","last_clean":"2024-09-16T01:53:58.686417+0000","last_became_active":"2024-09-16T01:48:58.838699+0000","last_became_peered":"2024-09-16T01:48:58.838699+0000","last_unstale":"2024-09-16T01:53:58.686417+0000","last_undegraded":"2024-09-16T01:53:58.686417+0000","last_fullsized":"2024-09-16T01:53:58.686417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'215","last_scrub_stamp":"2024-09-16T01:53:58.686333+0000","last_deep_scrub":"152'215","last_deep_scrub_stamp":"2024-09-16T01:53:58.686333+0000","last_clean_scrub_stamp":"2024-09-16T01:53:58.686333+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:37.171788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"157'5","reported_seq":71,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.473369+0000","last_change":"2024-09-16T01:53:49.473410+0000","last_active":"2024-09-16T01:53:49.473369+0000","last_peered":"2024-09-16T01:53:49.473369+0000","last_clean":"2024-09-16T01:53:49.473369+0000","last_became_active":"2024-09-16T01:53:19.521956+0000","last_became_peered":"2024-09-16T01:53:19.521956+0000","last_unstale":"2024-09-16T01:53:49.473369+0000","last_undegraded":"2024-09-16T01:53:49.473369+0000","last_fullsized":"2024-09-16T01:53:49.473369+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'5","last_scrub_stamp":"2024-09-16T01:53:49.473279+0000","last_deep_scrub":"157'5","last_deep_scrub_stamp":"2024-09-16T01:53:49.473279+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.473279+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:49.035451+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"152'175","reported_seq":671,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.353403+0000","last_change":"2024-09-16T01:53:49.353443+0000","last_active":"2024-09-16T01:53:49.353403+0000","last_peered":"2024-09-16T01:53:49.353403+0000","last_clean":"2024-09-16T01:53:49.353403+0000","last_became_active":"2024-09-16T01:53:22.525155+0000","last_became_peered":"2024-09-16T01:53:22.525155+0000","last_unstale":"2024-09-16T01:53:49.353403+0000","last_undegraded":"2024-09-16T01:53:49.353403+0000","last_fullsized":"2024-09-16T01:53:49.353403+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'175","last_scrub_stamp":"2024-09-16T01:53:49.353322+0000","last_deep_scrub":"152'175","last_deep_scrub_stamp":"2024-09-16T01:53:49.353322+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.353322+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:56:14.634486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":171966464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"157'7","reported_seq":69,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.121583+0000","last_change":"2024-09-16T01:53:49.121642+0000","last_active":"2024-09-16T01:53:49.121583+0000","last_peered":"2024-09-16T01:53:49.121583+0000","last_clean":"2024-09-16T01:53:49.121583+0000","last_became_active":"2024-09-16T01:53:20.510366+0000","last_became_peered":"2024-09-16T01:53:20.510366+0000","last_unstale":"2024-09-16T01:53:49.121583+0000","last_undegraded":"2024-09-16T01:53:49.121583+0000","last_fullsized":"2024-09-16T01:53:49.121583+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'7","last_scrub_stamp":"2024-09-16T01:53:49.121475+0000","last_deep_scrub":"157'7","last_deep_scrub_stamp":"2024-09-16T01:53:49.121475+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.121475+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:08:48.597112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"152'162","reported_seq":551,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:01.218416+0000","last_change":"2024-09-16T01:54:01.218444+0000","last_active":"2024-09-16T01:54:01.218416+0000","last_peered":"2024-09-16T01:54:01.218416+0000","last_clean":"2024-09-16T01:54:01.218416+0000","last_became_active":"2024-09-16T01:53:23.526799+0000","last_became_peered":"2024-09-16T01:53:23.526799+0000","last_unstale":"2024-09-16T01:54:01.218416+0000","last_undegraded":"2024-09-16T01:54:01.218416+0000","last_fullsized":"2024-09-16T01:54:01.218416+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:54:01.218324+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:54:01.218324+0000","last_clean_scrub_stamp":"2024-09-16T01:54:01.218324+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:08.327080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":155189248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"157'4","reported_seq":298,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.473221+0000","last_change":"2024-09-16T01:53:51.473264+0000","last_active":"2024-09-16T01:53:51.473221+0000","last_peered":"2024-09-16T01:53:51.473221+0000","last_clean":"2024-09-16T01:53:51.473221+0000","last_became_active":"2024-09-16T01:48:57.818134+0000","last_became_peered":"2024-09-16T01:48:57.818134+0000","last_unstale":"2024-09-16T01:53:51.473221+0000","last_undegraded":"2024-09-16T01:53:51.473221+0000","last_fullsized":"2024-09-16T01:53:51.473221+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'4","last_scrub_stamp":"2024-09-16T01:53:51.473118+0000","last_deep_scrub":"157'4","last_deep_scrub_stamp":"2024-09-16T01:53:51.473118+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.473118+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:51.035289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"152'165","reported_seq":787,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.281385+0000","last_change":"2024-09-16T01:53:45.454388+0000","last_active":"2024-09-16T01:53:46.281385+0000","last_peered":"2024-09-16T01:53:46.281385+0000","last_clean":"2024-09-16T01:53:46.281385+0000","last_became_active":"2024-09-16T01:53:22.520192+0000","last_became_peered":"2024-09-16T01:53:22.520192+0000","last_unstale":"2024-09-16T01:53:46.281385+0000","last_undegraded":"2024-09-16T01:53:46.281385+0000","last_fullsized":"2024-09-16T01:53:46.281385+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'165","last_scrub_stamp":"2024-09-16T01:53:45.454296+0000","last_deep_scrub":"152'165","last_deep_scrub_stamp":"2024-09-16T01:53:45.454296+0000","last_clean_scrub_stamp":"2024-09-16T01:53:45.454296+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:36:12.315196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":482344960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"157'8","reported_seq":285,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:54.021948+0000","last_change":"2024-09-16T01:53:54.021983+0000","last_active":"2024-09-16T01:53:54.021948+0000","last_peered":"2024-09-16T01:53:54.021948+0000","last_clean":"2024-09-16T01:53:54.021948+0000","last_became_active":"2024-09-16T01:53:34.532684+0000","last_became_peered":"2024-09-16T01:53:34.532684+0000","last_unstale":"2024-09-16T01:53:54.021948+0000","last_undegraded":"2024-09-16T01:53:54.021948+0000","last_fullsized":"2024-09-16T01:53:54.021948+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'8","last_scrub_stamp":"2024-09-16T01:53:54.021855+0000","last_deep_scrub":"157'8","last_deep_scrub_stamp":"2024-09-16T01:53:54.021855+0000","last_clean_scrub_stamp":"2024-09-16T01:53:54.021855+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:18.379122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"152'175","reported_seq":467,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:56.730228+0000","last_change":"2024-09-16T01:53:56.730256+0000","last_active":"2024-09-16T01:53:56.730228+0000","last_peered":"2024-09-16T01:53:56.730228+0000","last_clean":"2024-09-16T01:53:56.730228+0000","last_became_active":"2024-09-16T01:48:58.824413+0000","last_became_peered":"2024-09-16T01:48:58.824413+0000","last_unstale":"2024-09-16T01:53:56.730228+0000","last_undegraded":"2024-09-16T01:53:56.730228+0000","last_fullsized":"2024-09-16T01:53:56.730228+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'175","last_scrub_stamp":"2024-09-16T01:53:56.730105+0000","last_deep_scrub":"152'175","last_deep_scrub_stamp":"2024-09-16T01:53:56.730105+0000","last_clean_scrub_stamp":"2024-09-16T01:53:56.730105+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:10:11.368280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"157'119","reported_seq":452,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:47.247653+0000","last_change":"2024-09-16T01:53:47.247681+0000","last_active":"2024-09-16T01:53:47.247653+0000","last_peered":"2024-09-16T01:53:47.247653+0000","last_clean":"2024-09-16T01:53:47.247653+0000","last_became_active":"2024-09-16T01:48:48.115165+0000","last_became_peered":"2024-09-16T01:48:48.115165+0000","last_unstale":"2024-09-16T01:53:47.247653+0000","last_undegraded":"2024-09-16T01:53:47.247653+0000","last_fullsized":"2024-09-16T01:53:47.247653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-09-16T01:53:47.247591+0000","last_deep_scrub":"157'119","last_deep_scrub_stamp":"2024-09-16T01:53:47.247591+0000","last_clean_scrub_stamp":"2024-09-16T01:53:47.247591+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:49:06.504772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'11","reported_seq":305,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.776408+0000","last_change":"2024-09-16T01:53:53.776442+0000","last_active":"2024-09-16T01:53:53.776408+0000","last_peered":"2024-09-16T01:53:53.776408+0000","last_clean":"2024-09-16T01:53:53.776408+0000","last_became_active":"2024-09-16T01:48:57.819692+0000","last_became_peered":"2024-09-16T01:48:57.819692+0000","last_unstale":"2024-09-16T01:53:53.776408+0000","last_undegraded":"2024-09-16T01:53:53.776408+0000","last_fullsized":"2024-09-16T01:53:53.776408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'11","last_scrub_stamp":"2024-09-16T01:53:53.776330+0000","last_deep_scrub":"157'11","last_deep_scrub_stamp":"2024-09-16T01:53:53.776330+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.776330+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:29.006297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"152'183","reported_seq":619,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:54.154873+0000","last_change":"2024-09-16T01:53:54.154904+0000","last_active":"2024-09-16T01:53:54.154873+0000","last_peered":"2024-09-16T01:53:54.154873+0000","last_clean":"2024-09-16T01:53:54.154873+0000","last_became_active":"2024-09-16T01:53:22.523558+0000","last_became_peered":"2024-09-16T01:53:22.523558+0000","last_unstale":"2024-09-16T01:53:54.154873+0000","last_undegraded":"2024-09-16T01:53:54.154873+0000","last_fullsized":"2024-09-16T01:53:54.154873+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'183","last_scrub_stamp":"2024-09-16T01:53:54.154777+0000","last_deep_scrub":"152'183","last_deep_scrub_stamp":"2024-09-16T01:53:54.154777+0000","last_clean_scrub_stamp":"2024-09-16T01:53:54.154777+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:11.926912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":218103808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"157'5","reported_seq":299,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.740837+0000","last_change":"2024-09-16T01:53:51.740863+0000","last_active":"2024-09-16T01:53:51.740837+0000","last_peered":"2024-09-16T01:53:51.740837+0000","last_clean":"2024-09-16T01:53:51.740837+0000","last_became_active":"2024-09-16T01:48:57.816211+0000","last_became_peered":"2024-09-16T01:48:57.816211+0000","last_unstale":"2024-09-16T01:53:51.740837+0000","last_undegraded":"2024-09-16T01:53:51.740837+0000","last_fullsized":"2024-09-16T01:53:51.740837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'5","last_scrub_stamp":"2024-09-16T01:53:51.740754+0000","last_deep_scrub":"157'5","last_deep_scrub_stamp":"2024-09-16T01:53:51.740754+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.740754+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:28:51.665369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"152'165","reported_seq":427,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.003450+0000","last_change":"2024-09-16T01:53:52.003477+0000","last_active":"2024-09-16T01:53:52.003450+0000","last_peered":"2024-09-16T01:53:52.003450+0000","last_clean":"2024-09-16T01:53:52.003450+0000","last_became_active":"2024-09-16T01:53:34.531916+0000","last_became_peered":"2024-09-16T01:53:34.531916+0000","last_unstale":"2024-09-16T01:53:52.003450+0000","last_undegraded":"2024-09-16T01:53:52.003450+0000","last_fullsized":"2024-09-16T01:53:52.003450+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'165","last_scrub_stamp":"2024-09-16T01:53:52.003365+0000","last_deep_scrub":"152'165","last_deep_scrub_stamp":"2024-09-16T01:53:52.003365+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.003365+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:54.689122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"157'16","reported_seq":334,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.059636+0000","last_change":"2024-09-16T01:53:50.059698+0000","last_active":"2024-09-16T01:53:50.059636+0000","last_peered":"2024-09-16T01:53:50.059636+0000","last_clean":"2024-09-16T01:53:50.059636+0000","last_became_active":"2024-09-16T01:53:36.553933+0000","last_became_peered":"2024-09-16T01:53:36.553933+0000","last_unstale":"2024-09-16T01:53:50.059636+0000","last_undegraded":"2024-09-16T01:53:50.059636+0000","last_fullsized":"2024-09-16T01:53:50.059636+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'16","last_scrub_stamp":"2024-09-16T01:53:50.059537+0000","last_deep_scrub":"157'16","last_deep_scrub_stamp":"2024-09-16T01:53:50.059537+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.059537+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:34:06.231272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1065,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"152'165","reported_seq":674,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:01.461289+0000","last_change":"2024-09-16T01:54:01.461329+0000","last_active":"2024-09-16T01:54:01.461289+0000","last_peered":"2024-09-16T01:54:01.461289+0000","last_clean":"2024-09-16T01:54:01.461289+0000","last_became_active":"2024-09-16T01:52:46.542575+0000","last_became_peered":"2024-09-16T01:52:46.542575+0000","last_unstale":"2024-09-16T01:54:01.461289+0000","last_undegraded":"2024-09-16T01:54:01.461289+0000","last_fullsized":"2024-09-16T01:54:01.461289+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'165","last_scrub_stamp":"2024-09-16T01:54:01.461152+0000","last_deep_scrub":"152'165","last_deep_scrub_stamp":"2024-09-16T01:54:01.461152+0000","last_clean_scrub_stamp":"2024-09-16T01:54:01.461152+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:15.739991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":293601280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"157'7","reported_seq":301,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.784670+0000","last_change":"2024-09-16T01:53:52.784702+0000","last_active":"2024-09-16T01:53:52.784670+0000","last_peered":"2024-09-16T01:53:52.784670+0000","last_clean":"2024-09-16T01:53:52.784670+0000","last_became_active":"2024-09-16T01:48:57.817938+0000","last_became_peered":"2024-09-16T01:48:57.817938+0000","last_unstale":"2024-09-16T01:53:52.784670+0000","last_undegraded":"2024-09-16T01:53:52.784670+0000","last_fullsized":"2024-09-16T01:53:52.784670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'7","last_scrub_stamp":"2024-09-16T01:53:52.784574+0000","last_deep_scrub":"157'7","last_deep_scrub_stamp":"2024-09-16T01:53:52.784574+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.784574+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:15:28.600356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"152'177","reported_seq":569,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:00.261318+0000","last_change":"2024-09-16T01:54:00.261344+0000","last_active":"2024-09-16T01:54:00.261318+0000","last_peered":"2024-09-16T01:54:00.261318+0000","last_clean":"2024-09-16T01:54:00.261318+0000","last_became_active":"2024-09-16T01:53:23.526356+0000","last_became_peered":"2024-09-16T01:53:23.526356+0000","last_unstale":"2024-09-16T01:54:00.261318+0000","last_undegraded":"2024-09-16T01:54:00.261318+0000","last_fullsized":"2024-09-16T01:54:00.261318+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'177","last_scrub_stamp":"2024-09-16T01:54:00.261193+0000","last_deep_scrub":"152'177","last_deep_scrub_stamp":"2024-09-16T01:54:00.261193+0000","last_clean_scrub_stamp":"2024-09-16T01:54:00.261193+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:15.590359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'1","reported_seq":295,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:49.480596+0000","last_change":"2024-09-16T01:53:49.480623+0000","last_active":"2024-09-16T01:53:49.480596+0000","last_peered":"2024-09-16T01:53:49.480596+0000","last_clean":"2024-09-16T01:53:49.480596+0000","last_became_active":"2024-09-16T01:48:57.803937+0000","last_became_peered":"2024-09-16T01:48:57.803937+0000","last_unstale":"2024-09-16T01:53:49.480596+0000","last_undegraded":"2024-09-16T01:53:49.480596+0000","last_fullsized":"2024-09-16T01:53:49.480596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'1","last_scrub_stamp":"2024-09-16T01:53:49.480509+0000","last_deep_scrub":"22'1","last_deep_scrub_stamp":"2024-09-16T01:53:49.480509+0000","last_clean_scrub_stamp":"2024-09-16T01:53:49.480509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:44.419409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"152'160","reported_seq":452,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.376058+0000","last_change":"2024-09-16T01:53:46.376096+0000","last_active":"2024-09-16T01:53:46.376058+0000","last_peered":"2024-09-16T01:53:46.376058+0000","last_clean":"2024-09-16T01:53:46.376058+0000","last_became_active":"2024-09-16T01:48:58.823837+0000","last_became_peered":"2024-09-16T01:48:58.823837+0000","last_unstale":"2024-09-16T01:53:46.376058+0000","last_undegraded":"2024-09-16T01:53:46.376058+0000","last_fullsized":"2024-09-16T01:53:46.376058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'160","last_scrub_stamp":"2024-09-16T01:53:46.375956+0000","last_deep_scrub":"152'160","last_deep_scrub_stamp":"2024-09-16T01:53:46.375956+0000","last_clean_scrub_stamp":"2024-09-16T01:53:46.375956+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:41:15.973803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"157'7","reported_seq":140,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:48.373382+0000","last_change":"2024-09-16T01:53:48.373408+0000","last_active":"2024-09-16T01:53:48.373382+0000","last_peered":"2024-09-16T01:53:48.373382+0000","last_clean":"2024-09-16T01:53:48.373382+0000","last_became_active":"2024-09-16T01:53:20.524637+0000","last_became_peered":"2024-09-16T01:53:20.524637+0000","last_unstale":"2024-09-16T01:53:48.373382+0000","last_undegraded":"2024-09-16T01:53:48.373382+0000","last_fullsized":"2024-09-16T01:53:48.373382+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'7","last_scrub_stamp":"2024-09-16T01:53:48.373319+0000","last_deep_scrub":"157'7","last_deep_scrub_stamp":"2024-09-16T01:53:48.373319+0000","last_clean_scrub_stamp":"2024-09-16T01:53:48.373319+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:33:48.768000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"152'162","reported_seq":424,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:58.084962+0000","last_change":"2024-09-16T01:53:58.085005+0000","last_active":"2024-09-16T01:53:58.084962+0000","last_peered":"2024-09-16T01:53:58.084962+0000","last_clean":"2024-09-16T01:53:58.084962+0000","last_became_active":"2024-09-16T01:53:34.538388+0000","last_became_peered":"2024-09-16T01:53:34.538388+0000","last_unstale":"2024-09-16T01:53:58.084962+0000","last_undegraded":"2024-09-16T01:53:58.084962+0000","last_fullsized":"2024-09-16T01:53:58.084962+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:53:58.084823+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:53:58.084823+0000","last_clean_scrub_stamp":"2024-09-16T01:53:58.084823+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:09.597682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"157'29","reported_seq":81,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.171007+0000","last_change":"2024-09-16T01:53:52.171054+0000","last_active":"2024-09-16T01:53:52.171007+0000","last_peered":"2024-09-16T01:53:52.171007+0000","last_clean":"2024-09-16T01:53:52.171007+0000","last_became_active":"2024-09-16T01:53:20.559517+0000","last_became_peered":"2024-09-16T01:53:20.559517+0000","last_unstale":"2024-09-16T01:53:52.171007+0000","last_undegraded":"2024-09-16T01:53:52.171007+0000","last_fullsized":"2024-09-16T01:53:52.171007+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'29","last_scrub_stamp":"2024-09-16T01:53:52.170906+0000","last_deep_scrub":"157'29","last_deep_scrub_stamp":"2024-09-16T01:53:52.170906+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.170906+0000","objects_scrubbed":4,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:19:53.374755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":5,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"152'166","reported_seq":490,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:55.192730+0000","last_change":"2024-09-16T01:53:55.192759+0000","last_active":"2024-09-16T01:53:55.192730+0000","last_peered":"2024-09-16T01:53:55.192730+0000","last_clean":"2024-09-16T01:53:55.192730+0000","last_became_active":"2024-09-16T01:53:23.513282+0000","last_became_peered":"2024-09-16T01:53:23.513282+0000","last_unstale":"2024-09-16T01:53:55.192730+0000","last_undegraded":"2024-09-16T01:53:55.192730+0000","last_fullsized":"2024-09-16T01:53:55.192730+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'166","last_scrub_stamp":"2024-09-16T01:53:55.192627+0000","last_deep_scrub":"152'166","last_deep_scrub_stamp":"2024-09-16T01:53:55.192627+0000","last_clean_scrub_stamp":"2024-09-16T01:53:55.192627+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:30:11.815907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"157'12","reported_seq":336,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:00.715788+0000","last_change":"2024-09-16T01:54:00.715822+0000","last_active":"2024-09-16T01:54:00.715788+0000","last_peered":"2024-09-16T01:54:00.715788+0000","last_clean":"2024-09-16T01:54:00.715788+0000","last_became_active":"2024-09-16T01:53:19.522696+0000","last_became_peered":"2024-09-16T01:53:19.522696+0000","last_unstale":"2024-09-16T01:54:00.715788+0000","last_undegraded":"2024-09-16T01:54:00.715788+0000","last_fullsized":"2024-09-16T01:54:00.715788+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'12","last_scrub_stamp":"2024-09-16T01:54:00.715710+0000","last_deep_scrub":"157'12","last_deep_scrub_stamp":"2024-09-16T01:54:00.715710+0000","last_clean_scrub_stamp":"2024-09-16T01:54:00.715710+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:16.044876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'183","reported_seq":567,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:57.274070+0000","last_change":"2024-09-16T01:53:57.274115+0000","last_active":"2024-09-16T01:53:57.274070+0000","last_peered":"2024-09-16T01:53:57.274070+0000","last_clean":"2024-09-16T01:53:57.274070+0000","last_became_active":"2024-09-16T01:53:23.516597+0000","last_became_peered":"2024-09-16T01:53:23.516597+0000","last_unstale":"2024-09-16T01:53:57.274070+0000","last_undegraded":"2024-09-16T01:53:57.274070+0000","last_fullsized":"2024-09-16T01:53:57.274070+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'183","last_scrub_stamp":"2024-09-16T01:53:57.273966+0000","last_deep_scrub":"152'183","last_deep_scrub_stamp":"2024-09-16T01:53:57.273966+0000","last_clean_scrub_stamp":"2024-09-16T01:53:57.273966+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:57.341800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":444596224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"157'3","reported_seq":68,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.436850+0000","last_change":"2024-09-16T01:53:50.436878+0000","last_active":"2024-09-16T01:53:50.436850+0000","last_peered":"2024-09-16T01:53:50.436850+0000","last_clean":"2024-09-16T01:53:50.436850+0000","last_became_active":"2024-09-16T01:53:19.522531+0000","last_became_peered":"2024-09-16T01:53:19.522531+0000","last_unstale":"2024-09-16T01:53:50.436850+0000","last_undegraded":"2024-09-16T01:53:50.436850+0000","last_fullsized":"2024-09-16T01:53:50.436850+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'3","last_scrub_stamp":"2024-09-16T01:53:50.436776+0000","last_deep_scrub":"157'3","last_deep_scrub_stamp":"2024-09-16T01:53:50.436776+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.436776+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:34.106917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"152'162","reported_seq":606,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:03.751328+0000","last_change":"2024-09-16T01:54:03.751353+0000","last_active":"2024-09-16T01:54:03.751328+0000","last_peered":"2024-09-16T01:54:03.751328+0000","last_clean":"2024-09-16T01:54:03.751328+0000","last_became_active":"2024-09-16T01:53:23.513534+0000","last_became_peered":"2024-09-16T01:53:23.513534+0000","last_unstale":"2024-09-16T01:54:03.751328+0000","last_undegraded":"2024-09-16T01:54:03.751328+0000","last_fullsized":"2024-09-16T01:54:03.751328+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:54:03.751202+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:54:03.751202+0000","last_clean_scrub_stamp":"2024-09-16T01:54:03.751202+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:29.197656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"157'11","reported_seq":81,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.149041+0000","last_change":"2024-09-16T01:53:51.149098+0000","last_active":"2024-09-16T01:53:51.149041+0000","last_peered":"2024-09-16T01:53:51.149041+0000","last_clean":"2024-09-16T01:53:51.149041+0000","last_became_active":"2024-09-16T01:53:20.529832+0000","last_became_peered":"2024-09-16T01:53:20.529832+0000","last_unstale":"2024-09-16T01:53:51.149041+0000","last_undegraded":"2024-09-16T01:53:51.149041+0000","last_fullsized":"2024-09-16T01:53:51.149041+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'11","last_scrub_stamp":"2024-09-16T01:53:51.148944+0000","last_deep_scrub":"157'11","last_deep_scrub_stamp":"2024-09-16T01:53:51.148944+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.148944+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:10.974111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"152'173","reported_seq":522,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:56.227486+0000","last_change":"2024-09-16T01:53:56.227516+0000","last_active":"2024-09-16T01:53:56.227486+0000","last_peered":"2024-09-16T01:53:56.227486+0000","last_clean":"2024-09-16T01:53:56.227486+0000","last_became_active":"2024-09-16T01:53:23.516256+0000","last_became_peered":"2024-09-16T01:53:23.516256+0000","last_unstale":"2024-09-16T01:53:56.227486+0000","last_undegraded":"2024-09-16T01:53:56.227486+0000","last_fullsized":"2024-09-16T01:53:56.227486+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'173","last_scrub_stamp":"2024-09-16T01:53:56.227401+0000","last_deep_scrub":"152'173","last_deep_scrub_stamp":"2024-09-16T01:53:56.227401+0000","last_clean_scrub_stamp":"2024-09-16T01:53:56.227401+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:05.797895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":532676608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"157'5","reported_seq":269,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:57.054458+0000","last_change":"2024-09-16T01:53:57.054517+0000","last_active":"2024-09-16T01:53:57.054458+0000","last_peered":"2024-09-16T01:53:57.054458+0000","last_clean":"2024-09-16T01:53:57.054458+0000","last_became_active":"2024-09-16T01:53:34.538330+0000","last_became_peered":"2024-09-16T01:53:34.538330+0000","last_unstale":"2024-09-16T01:53:57.054458+0000","last_undegraded":"2024-09-16T01:53:57.054458+0000","last_fullsized":"2024-09-16T01:53:57.054458+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'5","last_scrub_stamp":"2024-09-16T01:53:57.054364+0000","last_deep_scrub":"157'5","last_deep_scrub_stamp":"2024-09-16T01:53:57.054364+0000","last_clean_scrub_stamp":"2024-09-16T01:53:57.054364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:27.410338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"152'164","reported_seq":651,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:51.101063+0000","last_change":"2024-09-16T01:53:51.101089+0000","last_active":"2024-09-16T01:53:51.101063+0000","last_peered":"2024-09-16T01:53:51.101063+0000","last_clean":"2024-09-16T01:53:51.101063+0000","last_became_active":"2024-09-16T01:53:36.553774+0000","last_became_peered":"2024-09-16T01:53:36.553774+0000","last_unstale":"2024-09-16T01:53:51.101063+0000","last_undegraded":"2024-09-16T01:53:51.101063+0000","last_fullsized":"2024-09-16T01:53:51.101063+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'164","last_scrub_stamp":"2024-09-16T01:53:51.100983+0000","last_deep_scrub":"152'164","last_deep_scrub_stamp":"2024-09-16T01:53:51.100983+0000","last_clean_scrub_stamp":"2024-09-16T01:53:51.100983+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:20.062572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":339738624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"157'7","reported_seq":301,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:54.741860+0000","last_change":"2024-09-16T01:53:54.741893+0000","last_active":"2024-09-16T01:53:54.741860+0000","last_peered":"2024-09-16T01:53:54.741860+0000","last_clean":"2024-09-16T01:53:54.741860+0000","last_became_active":"2024-09-16T01:48:57.819711+0000","last_became_peered":"2024-09-16T01:48:57.819711+0000","last_unstale":"2024-09-16T01:53:54.741860+0000","last_undegraded":"2024-09-16T01:53:54.741860+0000","last_fullsized":"2024-09-16T01:53:54.741860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'7","last_scrub_stamp":"2024-09-16T01:53:54.741775+0000","last_deep_scrub":"157'7","last_deep_scrub_stamp":"2024-09-16T01:53:54.741775+0000","last_clean_scrub_stamp":"2024-09-16T01:53:54.741775+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:50.128421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"152'154","reported_seq":446,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:57.716480+0000","last_change":"2024-09-16T01:53:57.716505+0000","last_active":"2024-09-16T01:53:57.716480+0000","last_peered":"2024-09-16T01:53:57.716480+0000","last_clean":"2024-09-16T01:53:57.716480+0000","last_became_active":"2024-09-16T01:48:58.826286+0000","last_became_peered":"2024-09-16T01:48:58.826286+0000","last_unstale":"2024-09-16T01:53:57.716480+0000","last_undegraded":"2024-09-16T01:53:57.716480+0000","last_fullsized":"2024-09-16T01:53:57.716480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'154","last_scrub_stamp":"2024-09-16T01:53:57.716385+0000","last_deep_scrub":"152'154","last_deep_scrub_stamp":"2024-09-16T01:53:57.716385+0000","last_clean_scrub_stamp":"2024-09-16T01:53:57.716385+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:27:45.134323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"152'205","reported_seq":737,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:54:02.704159+0000","last_change":"2024-09-16T01:54:02.704186+0000","last_active":"2024-09-16T01:54:02.704159+0000","last_peered":"2024-09-16T01:54:02.704159+0000","last_clean":"2024-09-16T01:54:02.704159+0000","last_became_active":"2024-09-16T01:53:22.526372+0000","last_became_peered":"2024-09-16T01:53:22.526372+0000","last_unstale":"2024-09-16T01:54:02.704159+0000","last_undegraded":"2024-09-16T01:54:02.704159+0000","last_fullsized":"2024-09-16T01:54:02.704159+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'205","last_scrub_stamp":"2024-09-16T01:54:02.704074+0000","last_deep_scrub":"152'205","last_deep_scrub_stamp":"2024-09-16T01:54:02.704074+0000","last_clean_scrub_stamp":"2024-09-16T01:54:02.704074+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:51.687995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":188743680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"157'8","reported_seq":74,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:46.476521+0000","last_change":"2024-09-16T01:53:46.476569+0000","last_active":"2024-09-16T01:53:46.476521+0000","last_peered":"2024-09-16T01:53:46.476521+0000","last_clean":"2024-09-16T01:53:46.476521+0000","last_became_active":"2024-09-16T01:53:19.518875+0000","last_became_peered":"2024-09-16T01:53:19.518875+0000","last_unstale":"2024-09-16T01:53:46.476521+0000","last_undegraded":"2024-09-16T01:53:46.476521+0000","last_fullsized":"2024-09-16T01:53:46.476521+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'8","last_scrub_stamp":"2024-09-16T01:53:46.476442+0000","last_deep_scrub":"157'8","last_deep_scrub_stamp":"2024-09-16T01:53:46.476442+0000","last_clean_scrub_stamp":"2024-09-16T01:53:46.476442+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:52.399904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"152'162","reported_seq":646,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:52.222411+0000","last_change":"2024-09-16T01:53:52.222411+0000","last_active":"2024-09-16T01:53:52.222411+0000","last_peered":"2024-09-16T01:53:52.222411+0000","last_clean":"2024-09-16T01:53:52.222411+0000","last_became_active":"2024-09-16T01:53:22.521217+0000","last_became_peered":"2024-09-16T01:53:22.521217+0000","last_unstale":"2024-09-16T01:53:52.222411+0000","last_undegraded":"2024-09-16T01:53:52.222411+0000","last_fullsized":"2024-09-16T01:53:52.222411+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'162","last_scrub_stamp":"2024-09-16T01:53:52.222310+0000","last_deep_scrub":"152'162","last_deep_scrub_stamp":"2024-09-16T01:53:52.222310+0000","last_clean_scrub_stamp":"2024-09-16T01:53:52.222310+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:08.476783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"157'20","reported_seq":86,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:48.474756+0000","last_change":"2024-09-16T01:53:48.474790+0000","last_active":"2024-09-16T01:53:48.474756+0000","last_peered":"2024-09-16T01:53:48.474756+0000","last_clean":"2024-09-16T01:53:48.474756+0000","last_became_active":"2024-09-16T01:53:19.521110+0000","last_became_peered":"2024-09-16T01:53:19.521110+0000","last_unstale":"2024-09-16T01:53:48.474756+0000","last_undegraded":"2024-09-16T01:53:48.474756+0000","last_fullsized":"2024-09-16T01:53:48.474756+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'20","last_scrub_stamp":"2024-09-16T01:53:48.474695+0000","last_deep_scrub":"157'20","last_deep_scrub_stamp":"2024-09-16T01:53:48.474695+0000","last_clean_scrub_stamp":"2024-09-16T01:53:48.474695+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:40.254285+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"152'160","reported_seq":452,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:53.434175+0000","last_change":"2024-09-16T01:53:53.434228+0000","last_active":"2024-09-16T01:53:53.434175+0000","last_peered":"2024-09-16T01:53:53.434175+0000","last_clean":"2024-09-16T01:53:53.434175+0000","last_became_active":"2024-09-16T01:48:58.820058+0000","last_became_peered":"2024-09-16T01:48:58.820058+0000","last_unstale":"2024-09-16T01:53:53.434175+0000","last_undegraded":"2024-09-16T01:53:53.434175+0000","last_fullsized":"2024-09-16T01:53:53.434175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'160","last_scrub_stamp":"2024-09-16T01:53:53.434079+0000","last_deep_scrub":"152'160","last_deep_scrub_stamp":"2024-09-16T01:53:53.434079+0000","last_clean_scrub_stamp":"2024-09-16T01:53:53.434079+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:26.564104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":326,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:59.513976+0000","last_change":"2024-09-16T01:53:59.514021+0000","last_active":"2024-09-16T01:53:59.513976+0000","last_peered":"2024-09-16T01:53:59.513976+0000","last_clean":"2024-09-16T01:53:59.513976+0000","last_became_active":"2024-09-16T01:52:46.543970+0000","last_became_peered":"2024-09-16T01:52:46.543970+0000","last_unstale":"2024-09-16T01:53:59.513976+0000","last_undegraded":"2024-09-16T01:53:59.513976+0000","last_fullsized":"2024-09-16T01:53:59.513976+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'2","last_scrub_stamp":"2024-09-16T01:53:59.513826+0000","last_deep_scrub":"22'2","last_deep_scrub_stamp":"2024-09-16T01:53:59.513826+0000","last_clean_scrub_stamp":"2024-09-16T01:53:59.513826+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:06.399795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"152'181","reported_seq":473,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:48.466391+0000","last_change":"2024-09-16T01:53:48.466442+0000","last_active":"2024-09-16T01:53:48.466391+0000","last_peered":"2024-09-16T01:53:48.466391+0000","last_clean":"2024-09-16T01:53:48.466391+0000","last_became_active":"2024-09-16T01:48:58.835145+0000","last_became_peered":"2024-09-16T01:48:58.835145+0000","last_unstale":"2024-09-16T01:53:48.466391+0000","last_undegraded":"2024-09-16T01:53:48.466391+0000","last_fullsized":"2024-09-16T01:53:48.466391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'181","last_scrub_stamp":"2024-09-16T01:53:48.466282+0000","last_deep_scrub":"152'181","last_deep_scrub_stamp":"2024-09-16T01:53:48.466282+0000","last_clean_scrub_stamp":"2024-09-16T01:53:48.466282+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:54.389744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"157'19","reported_seq":81,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T01:53:50.136864+0000","last_change":"2024-09-16T01:53:50.136905+0000","last_active":"2024-09-16T01:53:50.136864+0000","last_peered":"2024-09-16T01:53:50.136864+0000","last_clean":"2024-09-16T01:53:50.136864+0000","last_became_active":"2024-09-16T01:53:20.525723+0000","last_became_peered":"2024-09-16T01:53:20.525723+0000","last_unstale":"2024-09-16T01:53:50.136864+0000","last_undegraded":"2024-09-16T01:53:50.136864+0000","last_fullsized":"2024-09-16T01:53:50.136864+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'19","last_scrub_stamp":"2024-09-16T01:53:50.136767+0000","last_deep_scrub":"157'19","last_deep_scrub_stamp":"2024-09-16T01:53:50.136767+0000","last_clean_scrub_stamp":"2024-09-16T01:53:50.136767+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:33.723071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":9885974528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11255,"ondisk_log_size":11255,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":398903,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":428,"num_write_kb":603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":640592,"num_keys_recovered":146,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":733184,"data_stored":797806,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},"log_size":482,"ondisk_log_size":482,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":15,"seq":64424509509,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37216,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334624,"statfs":{"total":96636764160,"available":96598654976,"internally_reserved":0,"allocated":1916928,"data_stored":905386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509509,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41748,"kb_used_data":2244,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330092,"statfs":{"total":96636764160,"available":96594014208,"internally_reserved":0,"allocated":2297856,"data_stored":1294132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509508,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1966080,"data_stored":1494158,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509509,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1916928,"data_stored":905318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542212,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1921024,"data_stored":903935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542212,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":904756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1974272,"data_stored":1494075,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2011136,"data_stored":1295790,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32420,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":391756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":393414,"data_compressed":59499,"data_compressed_allocated":73728,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:54:09.786 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T01:54:09.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.971+0000 7f49fe800640 1 Processor -- start 2024-09-16T01:54:09.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.971+0000 7f49fe800640 1 -- start start 2024-09-16T01:54:09.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.972+0000 7f49fe800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49f8154ac0 0x7f49f815b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.972+0000 7f49fe800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49f815bc50 0x7f49f815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.972+0000 7f49fe800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49f8154ea0 con 0x7f49f806d820 2024-09-16T01:54:09.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.972+0000 7f49fe800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f49f815b6d0 con 0x7f49f815bc50 2024-09-16T01:54:09.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.972+0000 7f49fe800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f49f815c5c0 con 0x7f49f8154ac0 2024-09-16T01:54:09.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fd400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f49f8154ac0 0x7f49f815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49f815bc50 0x7f49f815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fca00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f49f806d820 0x7f49f80d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38782/0 (socket says 172.21.15.146:38782) 2024-09-16T01:54:09.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fca00640 1 -- 172.21.15.146:0/637210557 learned_addr learned my addr 172.21.15.146:0/637210557 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:54:09.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fde00640 1 -- 172.21.15.146:0/637210557 >> v2:172.21.15.161:3300/0 conn(0x7f49f8154ac0 msgr2=0x7f49f815b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:54:09.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fde00640 1 --2- 172.21.15.146:0/637210557 >> v2:172.21.15.161:3300/0 conn(0x7f49f8154ac0 0x7f49f815b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fde00640 1 -- 172.21.15.146:0/637210557 >> v1:172.21.15.146:6789/0 conn(0x7f49f806d820 legacy=0x7f49f80d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fde00640 1 -- 172.21.15.146:0/637210557 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f80c5340 con 0x7f49f815bc50 2024-09-16T01:54:09.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.973+0000 7f49fd400640 1 --2- 172.21.15.146:0/637210557 >> v2:172.21.15.161:3300/0 conn(0x7f49f8154ac0 0x7f49f815b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:54:09.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.974+0000 7f49fde00640 1 --2- 172.21.15.146:0/637210557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f815bc50 0x7f49f815c080 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f49f80d0790 tx=0x7f49f002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3520a3cd112208 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:09.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.974+0000 7f49f7e00640 1 -- 172.21.15.146:0/637210557 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0007740 con 0x7f49f815bc50 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49f7e00640 1 -- 172.21.15.146:0/637210557 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f00078e0 con 0x7f49f815bc50 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49f7e00640 1 -- 172.21.15.146:0/637210557 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0007cb0 con 0x7f49f815bc50 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 -- 172.21.15.146:0/637210557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f815bc50 msgr2=0x7f49f815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 --2- 172.21.15.146:0/637210557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f815bc50 0x7f49f815c080 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f49f80d0790 tx=0x7f49f002a710 comp rx=0 tx=0).stop 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 -- 172.21.15.146:0/637210557 shutdown_connections 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 --2- 172.21.15.146:0/637210557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f815bc50 0x7f49f815c080 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 --2- 172.21.15.146:0/637210557 >> v2:172.21.15.161:3300/0 conn(0x7f49f8154ac0 0x7f49f815b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 -- 172.21.15.146:0/637210557 >> 172.21.15.146:0/637210557 conn(0x7f49f806e400 msgr2=0x7f49f806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 -- 172.21.15.146:0/637210557 shutdown_connections 2024-09-16T01:54:09.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.975+0000 7f49fe800640 1 -- 172.21.15.146:0/637210557 wait complete. 2024-09-16T01:54:09.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.977+0000 7f49fe800640 1 Processor -- start 2024-09-16T01:54:09.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.977+0000 7f49fe800640 1 -- start start 2024-09-16T01:54:09.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fe800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 0x7f49f814a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fe800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49f8154ac0 0x7f49f80cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fe800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49f815bc50 0x7f49f80cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fe800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f49f80c5be0 con 0x7f49f806d820 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fe800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49f80c5900 con 0x7f49f815bc50 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fe800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f49f80c5ec0 con 0x7f49f8154ac0 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 0x7f49f814a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 0x7f49f814a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58900/0 (socket says 172.21.15.146:58900) 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fd400640 1 -- 172.21.15.146:0/1443126985 learned_addr learned my addr 172.21.15.146:0/1443126985 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:09.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fca00640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49f8154ac0 0x7f49f80cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.978+0000 7f49fde00640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49f815bc50 0x7f49f80cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fd400640 1 -- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49f8154ac0 msgr2=0x7f49f80cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fd400640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49f8154ac0 0x7f49f80cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fd400640 1 -- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49f815bc50 msgr2=0x7f49f80cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fd400640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49f815bc50 0x7f49f80cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fd400640 1 -- 172.21.15.146:0/1443126985 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f80cf3f0 con 0x7f49f806d820 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fd400640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 0x7f49f814a630 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00ec10 tx=0x7f49ec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec017070 con 0x7f49f806d820 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ec00cc30 con 0x7f49f806d820 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec013470 con 0x7f49f806d820 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f815de50 con 0x7f49f806d820 2024-09-16T01:54:09.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.979+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f81f0b70 con 0x7f49f806d820 2024-09-16T01:54:09.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.981+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49f80c95d0 con 0x7f49f806d820 2024-09-16T01:54:09.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.982+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f49ec01a070 con 0x7f49f806d820 2024-09-16T01:54:09.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.982+0000 7f49f5600640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49d806c2b0 0x7f49d806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:09.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.983+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f49ec08b750 con 0x7f49f806d820 2024-09-16T01:54:09.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.983+0000 7f49fca00640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49d806c2b0 0x7f49d806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:09.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.983+0000 7f49fca00640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49d806c2b0 0x7f49d806e770 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f49f80d10a0 tx=0x7f49e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:09.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:09.991+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f49ec059ea0 con 0x7f49f806d820 2024-09-16T01:54:10.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.174+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f49f80c61a0 con 0x7f49f806d820 2024-09-16T01:54:10.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.178+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f49ec0597a0 con 0x7f49f806d820 2024-09-16T01:54:10.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.178+0000 7f49f5600640 1 -- 172.21.15.146:0/1443126985 <== mon.0 v2:172.21.15.146:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f49ec05db50 con 0x7f49f806d820 2024-09-16T01:54:10.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49d806c2b0 msgr2=0x7f49d806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49d806c2b0 0x7f49d806e770 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f49f80d10a0 tx=0x7f49e4009040 comp rx=0 tx=0).stop 2024-09-16T01:54:10.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 msgr2=0x7f49f814a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 0x7f49f814a630 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f49ec00ec10 tx=0x7f49ec00c520 comp rx=0 tx=0).stop 2024-09-16T01:54:10.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 shutdown_connections 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f49d806c2b0 0x7f49d806e770 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49f815bc50 0x7f49f80cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49f8154ac0 0x7f49f80cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 --2- 172.21.15.146:0/1443126985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49f806d820 0x7f49f814a630 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 >> 172.21.15.146:0/1443126985 conn(0x7f49f806e400 msgr2=0x7f49f81507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 shutdown_connections 2024-09-16T01:54:10.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.181+0000 7f49fe800640 1 -- 172.21.15.146:0/1443126985 wait complete. 2024-09-16T01:54:10.199 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:54:10.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:54:10.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.356+0000 7f7e3f400640 1 Processor -- start 2024-09-16T01:54:10.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.357+0000 7f7e3f400640 1 -- start start 2024-09-16T01:54:10.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e3f400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7e3815e040 0x7f7e380c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e3f400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e3806c950 con 0x7f7e38154d00 2024-09-16T01:54:10.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e3f400640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e380707e0 con 0x7f7e38071010 2024-09-16T01:54:10.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e3f400640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7e38070a60 con 0x7f7e3815e040 2024-09-16T01:54:10.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e3ca00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7e38154d00 0x7f7e380c8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38796/0 (socket says 172.21.15.146:38796) 2024-09-16T01:54:10.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e3ca00640 1 -- 172.21.15.146:0/231489351 learned_addr learned my addr 172.21.15.146:0/231489351 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:54:10.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.358+0000 7f7e37e00640 1 --2- 172.21.15.146:0/231489351 >> v2:172.21.15.161:3300/0 conn(0x7f7e3815e040 0x7f7e380c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120820627 0 0) 0x7f7e3806c950 con 0x7f7e38154d00 2024-09-16T01:54:10.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e20003520 con 0x7f7e38154d00 2024-09-16T01:54:10.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189030967 0 0) 0x7f7e30001490 con 0x7f7e38071010 2024-09-16T01:54:10.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e3806c950 con 0x7f7e38071010 2024-09-16T01:54:10.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37e00640 1 -- 172.21.15.146:0/231489351 >> v1:172.21.15.161:6789/0 conn(0x7f7e38071010 legacy=0x7f7e380cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37e00640 1 -- 172.21.15.146:0/231489351 >> v1:172.21.15.146:6789/0 conn(0x7f7e38154d00 legacy=0x7f7e380c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.359+0000 7f7e37e00640 1 -- 172.21.15.146:0/231489351 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3806c390 con 0x7f7e3815e040 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.360+0000 7f7e37e00640 1 --2- 172.21.15.146:0/231489351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380c7bc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c5806441ca8f728 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.360+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24037070 con 0x7f7e3815e040 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.360+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7e24032440 con 0x7f7e3815e040 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.360+0000 7f7e37400640 1 -- 172.21.15.146:0/231489351 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24032830 con 0x7f7e3815e040 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 -- 172.21.15.146:0/231489351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 msgr2=0x7f7e380c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 --2- 172.21.15.146:0/231489351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380c7bc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402a230 comp rx=0 tx=0).stop 2024-09-16T01:54:10.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 -- 172.21.15.146:0/231489351 shutdown_connections 2024-09-16T01:54:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 --2- 172.21.15.146:0/231489351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380c7bc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 -- 172.21.15.146:0/231489351 >> 172.21.15.146:0/231489351 conn(0x7f7e38067950 msgr2=0x7f7e38067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 -- 172.21.15.146:0/231489351 shutdown_connections 2024-09-16T01:54:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.361+0000 7f7e3f400640 1 -- 172.21.15.146:0/231489351 wait complete. 2024-09-16T01:54:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.362+0000 7f7e3f400640 1 Processor -- start 2024-09-16T01:54:10.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 -- start start 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7e38071010 0x7f7e3815bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 0x7f7e381505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7e380c75a0 con 0x7f7e38154d00 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38070d40 con 0x7f7e3815e040 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.363+0000 7f7e3f400640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7e381550e0 con 0x7f7e38071010 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 0x7f7e381505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e3ca00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7e38071010 0x7f7e3815bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e3d400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 0x7f7e381505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58914/0 (socket says 172.21.15.146:58914) 2024-09-16T01:54:10.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 -- 172.21.15.146:0/2439248288 learned_addr learned my addr 172.21.15.146:0/2439248288 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 -- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7e38071010 msgr2=0x7f7e3815bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7e38071010 0x7f7e3815bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 -- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 msgr2=0x7f7e380cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 -- 172.21.15.146:0/2439248288 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3815d980 con 0x7f7e38154d00 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.364+0000 7f7e37e00640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 0x7f7e381505e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7e240098c0 tx=0x7f7e240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.365+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24041020 con 0x7f7e38154d00 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.365+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7e24030680 con 0x7f7e38154d00 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.365+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24037040 con 0x7f7e38154d00 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.365+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3815dbf0 con 0x7f7e38154d00 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.365+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3815cb80 con 0x7f7e38154d00 2024-09-16T01:54:10.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.366+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e3815d360 con 0x7f7e38154d00 2024-09-16T01:54:10.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.367+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7e24044020 con 0x7f7e38154d00 2024-09-16T01:54:10.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.368+0000 7f7e34c00640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e1806c2b0 0x7f7e1806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.368+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f7e240abbd0 con 0x7f7e38154d00 2024-09-16T01:54:10.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.368+0000 7f7e3ca00640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e1806c2b0 0x7f7e1806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.369+0000 7f7e3ca00640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e1806c2b0 0x7f7e1806e770 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c000f60 tx=0x7f7e2c007d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.374+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7e3815d360 con 0x7f7e38154d00 2024-09-16T01:54:10.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.600+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e380c7bc0 con 0x7f7e38154d00 2024-09-16T01:54:10.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.600+0000 7f7e34c00640 1 -- 172.21.15.146:0/2439248288 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6960 (secure 0 0 0) 0x7f7e2404b030 con 0x7f7e38154d00 2024-09-16T01:54:10.602 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:54:10.602 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":16,"btime":"2024-09-16T01:49:26:945057+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:26.945053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4256,4259,4277,4288,4291]},"id":1}]} 2024-09-16T01:54:10.602 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 16 2024-09-16T01:54:10.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e1806c2b0 msgr2=0x7f7e1806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e1806c2b0 0x7f7e1806e770 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c000f60 tx=0x7f7e2c007d70 comp rx=0 tx=0).stop 2024-09-16T01:54:10.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 msgr2=0x7f7e381505e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 0x7f7e381505e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7e240098c0 tx=0x7f7e240046c0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 shutdown_connections 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f7e1806c2b0 0x7f7e1806e770 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e3815e040 0x7f7e380cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e38154d00 0x7f7e381505e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 --2- 172.21.15.146:0/2439248288 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7e38071010 0x7f7e3815bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 >> 172.21.15.146:0/2439248288 conn(0x7f7e38067950 msgr2=0x7f7e38067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 shutdown_connections 2024-09-16T01:54:10.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.603+0000 7f7e3f400640 1 -- 172.21.15.146:0/2439248288 wait complete. 2024-09-16T01:54:10.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:54:10.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.779+0000 7f2662000640 1 Processor -- start 2024-09-16T01:54:10.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.780+0000 7f2662000640 1 -- start start 2024-09-16T01:54:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f2662000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f265c06cfa0 0x7f265c0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f2662000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265c15af00 con 0x7f265c15aa70 2024-09-16T01:54:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f2662000640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265c15b1e0 con 0x7f265c0c5eb0 2024-09-16T01:54:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f2662000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f265c06dbd0 con 0x7f265c06cfa0 2024-09-16T01:54:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f2661600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f265c06cfa0 0x7f265c0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f265be00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f265c15aa70 0x7f265c06c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38806/0 (socket says 172.21.15.146:38806) 2024-09-16T01:54:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.781+0000 7f265be00640 1 -- 172.21.15.146:0/2064240765 learned_addr learned my addr 172.21.15.146:0/2064240765 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.782+0000 7f2661600640 1 -- 172.21.15.146:0/2064240765 >> v1:172.21.15.161:6789/0 conn(0x7f265c0c5eb0 legacy=0x7f265c154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.782+0000 7f2661600640 1 -- 172.21.15.146:0/2064240765 >> v1:172.21.15.146:6789/0 conn(0x7f265c15aa70 legacy=0x7f265c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.782+0000 7f265b400640 1 -- 172.21.15.146:0/2064240765 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2668633504 0 0) 0x7f265c15b1e0 con 0x7f265c0c5eb0 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.782+0000 7f2661600640 1 -- 172.21.15.146:0/2064240765 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c15b4c0 con 0x7f265c06cfa0 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f2661600640 1 --2- 172.21.15.146:0/2064240765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c06cfa0 0x7f265c0cac30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2654009d90 tx=0x7f265402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbac0754077d8c19 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f265b400640 1 -- 172.21.15.146:0/2064240765 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654007740 con 0x7f265c06cfa0 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f265b400640 1 -- 172.21.15.146:0/2064240765 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f26540078e0 con 0x7f265c06cfa0 2024-09-16T01:54:10.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f265b400640 1 -- 172.21.15.146:0/2064240765 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654007cd0 con 0x7f265c06cfa0 2024-09-16T01:54:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f2662000640 1 -- 172.21.15.146:0/2064240765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c06cfa0 msgr2=0x7f265c0cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f2662000640 1 --2- 172.21.15.146:0/2064240765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c06cfa0 0x7f265c0cac30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2654009d90 tx=0x7f265402a680 comp rx=0 tx=0).stop 2024-09-16T01:54:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f2662000640 1 -- 172.21.15.146:0/2064240765 shutdown_connections 2024-09-16T01:54:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f2662000640 1 --2- 172.21.15.146:0/2064240765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c06cfa0 0x7f265c0cac30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.783+0000 7f2662000640 1 -- 172.21.15.146:0/2064240765 >> 172.21.15.146:0/2064240765 conn(0x7f265c067b30 msgr2=0x7f265c067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.784+0000 7f2662000640 1 -- 172.21.15.146:0/2064240765 shutdown_connections 2024-09-16T01:54:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.784+0000 7f2662000640 1 -- 172.21.15.146:0/2064240765 wait complete. 2024-09-16T01:54:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.785+0000 7f2662000640 1 Processor -- start 2024-09-16T01:54:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.785+0000 7f2662000640 1 -- start start 2024-09-16T01:54:10.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2662000640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f265c06cfa0 0x7f265c14cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2662000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f265c0c5eb0 0x7f265c1e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2662000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 0x7f265c1e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2662000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f265c155020 con 0x7f265c1e4470 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2662000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f265c06c370 con 0x7f265c0c5eb0 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2662000640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f265c1551a0 con 0x7f265c06cfa0 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2661600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 0x7f265c1e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2660c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f265c06cfa0 0x7f265c14cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f265be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f265c0c5eb0 0x7f265c1e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2661600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 0x7f265c1e5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58932/0 (socket says 172.21.15.146:58932) 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2660c00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f265c06cfa0 0x7f265c14cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43310/0 (socket says 172.21.15.146:43310) 2024-09-16T01:54:10.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f2661600640 1 -- 172.21.15.146:0/1653334017 learned_addr learned my addr 172.21.15.146:0/1653334017 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.786+0000 7f265be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f265c0c5eb0 0x7f265c1e5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:51962/0 (socket says 172.21.15.146:51962) 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2661600640 1 -- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f265c06cfa0 msgr2=0x7f265c14cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2661600640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f265c06cfa0 0x7f265c14cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2661600640 1 -- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f265c0c5eb0 msgr2=0x7f265c1e5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2661600640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f265c0c5eb0 0x7f265c1e5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2661600640 1 -- 172.21.15.146:0/1653334017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c1e4980 con 0x7f265c1e4470 2024-09-16T01:54:10.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2661600640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 0x7f265c1e5ff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2654009d90 tx=0x7f2654004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265403f070 con 0x7f265c1e4470 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2654031c90 con 0x7f265c1e4470 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654003bd0 con 0x7f265c1e4470 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.787+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265c1e4c60 con 0x7f265c1e4470 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.788+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265c153240 con 0x7f265c1e4470 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.789+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f265c1e5180 con 0x7f265c1e4470 2024-09-16T01:54:10.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.790+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2654004030 con 0x7f265c1e4470 2024-09-16T01:54:10.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.790+0000 7f2658c00640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f263c06c2b0 0x7f263c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:10.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.791+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f26540ac940 con 0x7f265c1e4470 2024-09-16T01:54:10.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.791+0000 7f2660c00640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f263c06c2b0 0x7f263c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:10.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.791+0000 7f2660c00640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f263c06c2b0 0x7f263c06e770 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2650000f60 tx=0x7f2650009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:10.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:10.795+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f265c1e5180 con 0x7f265c1e4470 2024-09-16T01:54:11.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.013+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f265c0c6290 con 0x7f265c1e4470 2024-09-16T01:54:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.014+0000 7f2658c00640 1 -- 172.21.15.146:0/1653334017 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6960 (secure 0 0 0) 0x7f265407aa40 con 0x7f265c1e4470 2024-09-16T01:54:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:54:11.015 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":16,"btime":"2024-09-16T01:49:26:945057+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6836/396091672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":396091672},{"type":"v1","addr":"172.21.15.161:6836","nonce":396091672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:48:58.207792+0000","modified":"2024-09-16T01:49:26.945053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4291,"mds_1":4288,"mds_2":4277,"mds_3":4259,"mds_4":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"d","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.161:6837/3568385147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6835","nonce":3568385147},{"type":"v1","addr":"172.21.15.161:6837","nonce":3568385147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.161:6839/2836611107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2836611107},{"type":"v1","addr":"172.21.15.161:6839","nonce":2836611107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4277":{"gid":4277,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.146:6837/835076388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":835076388},{"type":"v1","addr":"172.21.15.146:6837","nonce":835076388}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"a","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6833/1022834380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1022834380},{"type":"v1","addr":"172.21.15.146:6833","nonce":1022834380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6836/1807355541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1807355541},{"type":"v1","addr":"172.21.15.146:6836","nonce":1807355541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4291,"qdb_cluster":[4256,4259,4277,4288,4291]},"id":1}]} 2024-09-16T01:54:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 16 2024-09-16T01:54:11.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f263c06c2b0 msgr2=0x7f263c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f263c06c2b0 0x7f263c06e770 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2650000f60 tx=0x7f2650009040 comp rx=0 tx=0).stop 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 msgr2=0x7f265c1e5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 0x7f265c1e5ff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2654009d90 tx=0x7f2654004a40 comp rx=0 tx=0).stop 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 shutdown_connections 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f263c06c2b0 0x7f263c06e770 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f265c1e4470 0x7f265c1e5ff0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f265c0c5eb0 0x7f265c1e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 --2- 172.21.15.146:0/1653334017 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f265c06cfa0 0x7f265c14cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 >> 172.21.15.146:0/1653334017 conn(0x7f265c067b30 msgr2=0x7f265c0ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 shutdown_connections 2024-09-16T01:54:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.016+0000 7f2662000640 1 -- 172.21.15.146:0/1653334017 wait complete. 2024-09-16T01:54:11.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:54:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.190+0000 7fda05800640 1 Processor -- start 2024-09-16T01:54:11.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.190+0000 7fda05800640 1 -- start start 2024-09-16T01:54:11.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.191+0000 7fda05800640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda000b3790 0x7fda000b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.191+0000 7fda05800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda00155430 con 0x7fda000d0bf0 2024-09-16T01:54:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.191+0000 7fda05800640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda0015aea0 con 0x7fda000d0fd0 2024-09-16T01:54:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.191+0000 7fda05800640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fda0015b180 con 0x7fda000b3790 2024-09-16T01:54:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9fea00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fda000d0bf0 0x7fda000c9fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38822/0 (socket says 172.21.15.146:38822) 2024-09-16T01:54:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9fea00640 1 -- 172.21.15.146:0/3583665986 learned_addr learned my addr 172.21.15.146:0/3583665986 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:54:11.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9ff400640 1 --2- 172.21.15.146:0/3583665986 >> v2:172.21.15.161:3300/0 conn(0x7fda000b3790 0x7fda000b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9fe000640 1 -- 172.21.15.146:0/3583665986 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220766916 0 0) 0x7fda00155430 con 0x7fda000d0bf0 2024-09-16T01:54:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9fe000640 1 -- 172.21.15.146:0/3583665986 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e4003520 con 0x7fda000d0bf0 2024-09-16T01:54:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9ff400640 1 -- 172.21.15.146:0/3583665986 >> v1:172.21.15.161:6789/0 conn(0x7fda000d0fd0 legacy=0x7fda000b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9ff400640 1 -- 172.21.15.146:0/3583665986 >> v1:172.21.15.146:6789/0 conn(0x7fda000d0bf0 legacy=0x7fda000c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.192+0000 7fd9ff400640 1 -- 172.21.15.146:0/3583665986 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0015b460 con 0x7fda000b3790 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.193+0000 7fd9ff400640 1 --2- 172.21.15.146:0/3583665986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda000b3f20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009d90 tx=0x7fd9f802a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d62e399410d6ccf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.193+0000 7fd9fe000640 1 -- 172.21.15.146:0/3583665986 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8037070 con 0x7fda000b3790 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.193+0000 7fd9fe000640 1 -- 172.21.15.146:0/3583665986 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd9f8007740 con 0x7fda000b3790 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.193+0000 7fd9fe000640 1 -- 172.21.15.146:0/3583665986 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8007b30 con 0x7fda000b3790 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 -- 172.21.15.146:0/3583665986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 msgr2=0x7fda000b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 --2- 172.21.15.146:0/3583665986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda000b3f20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009d90 tx=0x7fd9f802a780 comp rx=0 tx=0).stop 2024-09-16T01:54:11.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 -- 172.21.15.146:0/3583665986 shutdown_connections 2024-09-16T01:54:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 --2- 172.21.15.146:0/3583665986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda000b3f20 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 -- 172.21.15.146:0/3583665986 >> 172.21.15.146:0/3583665986 conn(0x7fda0006c900 msgr2=0x7fda0006cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 -- 172.21.15.146:0/3583665986 shutdown_connections 2024-09-16T01:54:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.194+0000 7fda05800640 1 -- 172.21.15.146:0/3583665986 wait complete. 2024-09-16T01:54:11.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.196+0000 7fda05800640 1 Processor -- start 2024-09-16T01:54:11.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.196+0000 7fda05800640 1 -- start start 2024-09-16T01:54:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.196+0000 7fda05800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda00156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fda05800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda000d0bf0 0x7fda000c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fda05800640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda000d0fd0 0x7fda001e6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fda05800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fda000ca6f0 con 0x7fda000d0bf0 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fda05800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda0015ba20 con 0x7fda000b3790 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fda05800640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fda000ca870 con 0x7fda000d0fd0 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9f7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda000d0bf0 0x7fda000c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9fea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda00156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9ff400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda000d0fd0 0x7fda001e6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9f7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda000d0bf0 0x7fda000c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58936/0 (socket says 172.21.15.146:58936) 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9f7e00640 1 -- 172.21.15.146:0/885701563 learned_addr learned my addr 172.21.15.146:0/885701563 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9fea00640 1 -- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda000d0fd0 msgr2=0x7fda001e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9fea00640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda000d0fd0 0x7fda001e6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9fea00640 1 -- 172.21.15.146:0/885701563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda000d0bf0 msgr2=0x7fda000c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9fea00640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda000d0bf0 0x7fda000c6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.197+0000 7fd9fea00640 1 -- 172.21.15.146:0/885701563 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda001e7990 con 0x7fda000b3790 2024-09-16T01:54:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.198+0000 7fd9fea00640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda00156350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000c460 tx=0x7fd9f000c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.198+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0017070 con 0x7fda000b3790 2024-09-16T01:54:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.198+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd9f0005ce0 con 0x7fda000b3790 2024-09-16T01:54:11.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.198+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda001e56b0 con 0x7fda000b3790 2024-09-16T01:54:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.198+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0007290 con 0x7fda000b3790 2024-09-16T01:54:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.198+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda000ca460 con 0x7fda000b3790 2024-09-16T01:54:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.199+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0015edc0 con 0x7fda000b3790 2024-09-16T01:54:11.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.200+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd9f001a070 con 0x7fda000b3790 2024-09-16T01:54:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.201+0000 7fd9f6a00640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd9dc06c2e0 0x7fd9dc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.201+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7fd9f008a8c0 con 0x7fda000b3790 2024-09-16T01:54:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.201+0000 7fd9f7e00640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd9dc06c2e0 0x7fd9dc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.201+0000 7fd9f7e00640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd9dc06c2e0 0x7fd9dc06e7a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fda000c98a0 tx=0x7fd9e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.205+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd9f0058800 con 0x7fda000b3790 2024-09-16T01:54:11.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.388+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda000c8870 con 0x7fda000b3790 2024-09-16T01:54:11.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.390+0000 7fd9f6a00640 1 -- 172.21.15.146:0/885701563 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) ==== 74+0+16986 (secure 0 0 0) 0x7fd9f005cb00 con 0x7fda000b3790 2024-09-16T01:54:11.391 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-16T01:54:11.393 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":157,"fsid":"4c90eeb9-6ff9-4365-a19f-bfe8d622ae2d","created":"2024-09-16T01:48:34.399638+0000","modified":"2024-09-16T01:53:45.498531+0000","last_up_change":"2024-09-16T01:48:45.748664+0000","last_in_change":"2024-09-16T01:53:19.491406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:48:46.570355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:48:56.008811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:48:57.207121+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7758ddf-75ff-43c7-abae-3647b170f7c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6825","nonce":61440064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6827","nonce":61440064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6831","nonce":61440064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":61440064},{"type":"v1","addr":"172.21.15.146:6829","nonce":61440064}]},"public_addr":"172.21.15.146:6825/61440064","cluster_addr":"172.21.15.146:6827/61440064","heartbeat_back_addr":"172.21.15.146:6831/61440064","heartbeat_front_addr":"172.21.15.146:6829/61440064","state":["exists","up"]},{"osd":1,"uuid":"a065e77e-26f0-4245-abe8-e36c2fb4436b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6809","nonce":2375201615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6811","nonce":2375201615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6815","nonce":2375201615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2375201615},{"type":"v1","addr":"172.21.15.146:6813","nonce":2375201615}]},"public_addr":"172.21.15.146:6809/2375201615","cluster_addr":"172.21.15.146:6811/2375201615","heartbeat_back_addr":"172.21.15.146:6815/2375201615","heartbeat_front_addr":"172.21.15.146:6813/2375201615","state":["exists","up"]},{"osd":2,"uuid":"fefb6d8f-6799-4344-a977-85ab983ec0cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6801","nonce":1818492127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6803","nonce":1818492127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6807","nonce":1818492127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1818492127},{"type":"v1","addr":"172.21.15.146:6805","nonce":1818492127}]},"public_addr":"172.21.15.146:6801/1818492127","cluster_addr":"172.21.15.146:6803/1818492127","heartbeat_back_addr":"172.21.15.146:6807/1818492127","heartbeat_front_addr":"172.21.15.146:6805/1818492127","state":["exists","up"]},{"osd":3,"uuid":"9d1a2f96-efcb-4a73-b5db-ec5e2a63122a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6817","nonce":1633804394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6819","nonce":1633804394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6823","nonce":1633804394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":1633804394},{"type":"v1","addr":"172.21.15.146:6821","nonce":1633804394}]},"public_addr":"172.21.15.146:6817/1633804394","cluster_addr":"172.21.15.146:6819/1633804394","heartbeat_back_addr":"172.21.15.146:6823/1633804394","heartbeat_front_addr":"172.21.15.146:6821/1633804394","state":["exists","up"]},{"osd":4,"uuid":"b2d6ad94-1f9b-4cbb-a567-6bdbbe413d5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6801","nonce":154390265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6803","nonce":154390265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6807","nonce":154390265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":154390265},{"type":"v1","addr":"172.21.15.161:6805","nonce":154390265}]},"public_addr":"172.21.15.161:6801/154390265","cluster_addr":"172.21.15.161:6803/154390265","heartbeat_back_addr":"172.21.15.161:6807/154390265","heartbeat_front_addr":"172.21.15.161:6805/154390265","state":["exists","up"]},{"osd":5,"uuid":"11562dd5-d537-4d88-a282-e8687caebad6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6825","nonce":356357279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6827","nonce":356357279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6831","nonce":356357279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":356357279},{"type":"v1","addr":"172.21.15.161:6829","nonce":356357279}]},"public_addr":"172.21.15.161:6825/356357279","cluster_addr":"172.21.15.161:6827/356357279","heartbeat_back_addr":"172.21.15.161:6831/356357279","heartbeat_front_addr":"172.21.15.161:6829/356357279","state":["exists","up"]},{"osd":6,"uuid":"b31adf01-6768-49cf-a232-127b9d0d2dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6809","nonce":3661001615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6811","nonce":3661001615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6815","nonce":3661001615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":3661001615},{"type":"v1","addr":"172.21.15.161:6813","nonce":3661001615}]},"public_addr":"172.21.15.161:6809/3661001615","cluster_addr":"172.21.15.161:6811/3661001615","heartbeat_back_addr":"172.21.15.161:6815/3661001615","heartbeat_front_addr":"172.21.15.161:6813/3661001615","state":["exists","up"]},{"osd":7,"uuid":"67c95c28-b600-430c-9474-a902cf46a73b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6817","nonce":1279166573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6819","nonce":1279166573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6823","nonce":1279166573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":1279166573},{"type":"v1","addr":"172.21.15.161:6821","nonce":1279166573}]},"public_addr":"172.21.15.161:6817/1279166573","cluster_addr":"172.21.15.161:6819/1279166573","heartbeat_back_addr":"172.21.15.161:6823/1279166573","heartbeat_front_addr":"172.21.15.161:6821/1279166573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.135308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.290750+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.712744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.988678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:42.198114+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.085332+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.394321+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:48:43.236983+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":2,"to":5}]},{"pgid":"3.20","mappings":[{"from":2,"to":7}]},{"pgid":"3.2d","mappings":[{"from":3,"to":7}]},{"pgid":"3.31","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd9dc06c2e0 msgr2=0x7fd9dc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd9dc06c2e0 0x7fd9dc06e7a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fda000c98a0 tx=0x7fd9e8009040 comp rx=0 tx=0).stop 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 msgr2=0x7fda00156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda00156350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd9f000c460 tx=0x7fd9f000c920 comp rx=0 tx=0).stop 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 shutdown_connections 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd9dc06c2e0 0x7fd9dc06e7a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fda000d0fd0 0x7fda001e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda000d0bf0 0x7fda000c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 --2- 172.21.15.146:0/885701563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda000b3790 0x7fda00156350 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 >> 172.21.15.146:0/885701563 conn(0x7fda0006c900 msgr2=0x7fda000c9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:11.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.392+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 shutdown_connections 2024-09-16T01:54:11.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.393+0000 7fda05800640 1 -- 172.21.15.146:0/885701563 wait complete. 2024-09-16T01:54:11.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:54:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.566+0000 7f6c4fc00640 1 Processor -- start 2024-09-16T01:54:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.566+0000 7f6c4fc00640 1 -- start start 2024-09-16T01:54:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.567+0000 7f6c4fc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6c48151a90 0x7f6c480c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.567+0000 7f6c4fc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c48158040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.567+0000 7f6c4fc00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6c480c5280 0x7f6c480c5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.567+0000 7f6c4fc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6c48151e70 con 0x7f6c48151a90 2024-09-16T01:54:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.567+0000 7f6c4fc00640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6c480c8130 con 0x7f6c480c5280 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.567+0000 7f6c4fc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6c480c8410 con 0x7f6c480c6410 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6c48151a90 0x7f6c480c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6c48151a90 0x7f6c480c36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58938/0 (socket says 172.21.15.146:58938) 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c47e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c48158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 -- 172.21.15.146:0/3598611319 learned_addr learned my addr 172.21.15.146:0/3598611319 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4dc00640 1 --2- 172.21.15.146:0/3598611319 >> v2:172.21.15.161:3300/0 conn(0x7f6c480c5280 0x7f6c480c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 -- 172.21.15.146:0/3598611319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 msgr2=0x7f6c48158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 --2- 172.21.15.146:0/3598611319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c48158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 -- 172.21.15.146:0/3598611319 >> v2:172.21.15.161:3300/0 conn(0x7f6c480c5280 msgr2=0x7f6c480c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 --2- 172.21.15.146:0/3598611319 >> v2:172.21.15.161:3300/0 conn(0x7f6c480c5280 0x7f6c480c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4d200640 1 -- 172.21.15.146:0/3598611319 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c480c86f0 con 0x7f6c48151a90 2024-09-16T01:54:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c47e00640 1 --2- 172.21.15.146:0/3598611319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c48158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:54:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.568+0000 7f6c4dc00640 1 --2- 172.21.15.146:0/3598611319 >> v2:172.21.15.161:3300/0 conn(0x7f6c480c5280 0x7f6c480c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:54:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.569+0000 7f6c4d200640 1 --2- 172.21.15.146:0/3598611319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c480c36b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6c4815bd40 tx=0x7f6c3c02a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f96b0e0fe27ce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.569+0000 7f6c47400640 1 -- 172.21.15.146:0/3598611319 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c037070 con 0x7f6c48151a90 2024-09-16T01:54:11.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.569+0000 7f6c47400640 1 -- 172.21.15.146:0/3598611319 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6c3c032440 con 0x7f6c48151a90 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.569+0000 7f6c47400640 1 -- 172.21.15.146:0/3598611319 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c032830 con 0x7f6c48151a90 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 -- 172.21.15.146:0/3598611319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 msgr2=0x7f6c480c36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/3598611319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c480c36b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6c4815bd40 tx=0x7f6c3c02a1d0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 -- 172.21.15.146:0/3598611319 shutdown_connections 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/3598611319 >> v2:172.21.15.161:3300/0 conn(0x7f6c480c5280 0x7f6c480c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/3598611319 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c48158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/3598611319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c480c36b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 -- 172.21.15.146:0/3598611319 >> 172.21.15.146:0/3598611319 conn(0x7f6c4806f3c0 msgr2=0x7f6c4806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 -- 172.21.15.146:0/3598611319 shutdown_connections 2024-09-16T01:54:11.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.570+0000 7f6c4fc00640 1 -- 172.21.15.146:0/3598611319 wait complete. 2024-09-16T01:54:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.572+0000 7f6c4fc00640 1 Processor -- start 2024-09-16T01:54:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.572+0000 7f6c4fc00640 1 -- start start 2024-09-16T01:54:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4fc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c480c5280 0x7f6c48156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4fc00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c481ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c481ec810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4fc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6c48157b20 con 0x7f6c48151a90 2024-09-16T01:54:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4fc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6c48157840 con 0x7f6c480c5280 2024-09-16T01:54:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4fc00640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6c480c3bf0 con 0x7f6c480c6410 2024-09-16T01:54:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c481ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c481ec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4d200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c480c5280 0x7f6c48156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c481ec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43322/0 (socket says 172.21.15.146:43322) 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 -- 172.21.15.146:0/172710580 learned_addr learned my addr 172.21.15.146:0/172710580 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 -- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c480c5280 msgr2=0x7f6c48156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c480c5280 0x7f6c48156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 -- 172.21.15.146:0/172710580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 msgr2=0x7f6c481ec810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c481ec810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c47e00640 1 -- 172.21.15.146:0/172710580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c481498c0 con 0x7f6c480c6410 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.573+0000 7f6c4dc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c481ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.574+0000 7f6c47e00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c481ec2d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400ebe0 tx=0x7f6c34009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.574+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34017070 con 0x7f6c480c6410 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.574+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6c3400ca00 con 0x7f6c480c6410 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.574+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400cdf0 con 0x7f6c480c6410 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.574+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4814ad50 con 0x7f6c480c6410 2024-09-16T01:54:11.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.574+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c481efcc0 con 0x7f6c480c6410 2024-09-16T01:54:11.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.575+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c480c71f0 con 0x7f6c480c6410 2024-09-16T01:54:11.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.576+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6c3401a070 con 0x7f6c480c6410 2024-09-16T01:54:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.577+0000 7f6c44c00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c2806c2b0 0x7f6c2806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.577+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(157..157 src has 1..157) ==== 5886+0+0 (secure 0 0 0) 0x7f6c3408b6d0 con 0x7f6c480c6410 2024-09-16T01:54:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.577+0000 7f6c4d200640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c2806c2b0 0x7f6c2806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.578+0000 7f6c4d200640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c2806c2b0 0x7f6c2806e770 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c034000 tx=0x7f6c3c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.582+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6c340596a0 con 0x7f6c480c6410 2024-09-16T01:54:11.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.791+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6c480c32c0 con 0x7f6c480c6410 2024-09-16T01:54:11.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.800+0000 7f6c44c00640 1 -- 172.21.15.146:0/172710580 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f6c3405d9a0 con 0x7f6c480c6410 2024-09-16T01:54:11.801 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:54:11.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c2806c2b0 msgr2=0x7f6c2806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c2806c2b0 0x7f6c2806e770 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c034000 tx=0x7f6c3c02e000 comp rx=0 tx=0).stop 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 msgr2=0x7f6c481ec2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c481ec2d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400ebe0 tx=0x7f6c34009e00 comp rx=0 tx=0).stop 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 shutdown_connections 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6c2806c2b0 0x7f6c2806e770 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c48151a90 0x7f6c481ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6c480c6410 0x7f6c481ec2d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 --2- 172.21.15.146:0/172710580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c480c5280 0x7f6c48156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 >> 172.21.15.146:0/172710580 conn(0x7f6c4806f3c0 msgr2=0x7f6c48155420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 shutdown_connections 2024-09-16T01:54:11.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.802+0000 7f6c4fc00640 1 -- 172.21.15.146:0/172710580 wait complete. 2024-09-16T01:54:11.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:54:11.839 INFO:tasks.ceph.mds.d.smithi161.stdout:starting mds.d at 2024-09-16T01:54:11.839 INFO:tasks.ceph.mds.f.smithi161.stdout:starting mds.f at 2024-09-16T01:54:11.865 INFO:tasks.ceph.mds.e.smithi146.stdout:starting mds.e at 2024-09-16T01:54:11.865 INFO:tasks.ceph.mds.c.smithi146.stdout:starting mds.c at 2024-09-16T01:54:11.868 INFO:tasks.ceph.mds.a.smithi146.stdout:starting mds.a at 2024-09-16T01:54:11.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.988+0000 7f4045200640 1 Processor -- start 2024-09-16T01:54:11.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.989+0000 7f4045200640 1 -- start start 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4045200640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4040152760 0x7f40400c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4045200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40400c6cc0 0x7f40400c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4045200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4040158890 con 0x7f404006fee0 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4045200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f40400c8530 con 0x7f40400c6cc0 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4045200640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4040158b00 con 0x7f4040152760 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4035400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40400c6cc0 0x7f40400c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4035400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40400c6cc0 0x7f40400c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59014/0 (socket says 172.21.15.146:59014) 2024-09-16T01:54:11.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f403e800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f404006fee0 0x7f404015bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:38828/0 (socket says 172.21.15.146:38828) 2024-09-16T01:54:11.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4035400640 1 -- 172.21.15.146:0/2818162754 learned_addr learned my addr 172.21.15.146:0/2818162754 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:11.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f403de00640 1 --2- 172.21.15.146:0/2818162754 >> v2:172.21.15.161:3300/0 conn(0x7f4040152760 0x7f40400c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4035400640 1 -- 172.21.15.146:0/2818162754 >> v2:172.21.15.161:3300/0 conn(0x7f4040152760 msgr2=0x7f40400c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4035400640 1 --2- 172.21.15.146:0/2818162754 >> v2:172.21.15.161:3300/0 conn(0x7f4040152760 0x7f40400c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.990+0000 7f4035400640 1 -- 172.21.15.146:0/2818162754 >> v1:172.21.15.146:6789/0 conn(0x7f404006fee0 legacy=0x7f404015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.991+0000 7f4035400640 1 -- 172.21.15.146:0/2818162754 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4040158de0 con 0x7f40400c6cc0 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.991+0000 7f403d400640 1 -- 172.21.15.146:0/2818162754 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756535680 0 0) 0x7f4040158890 con 0x7f404006fee0 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.991+0000 7f403de00640 1 --2- 172.21.15.146:0/2818162754 >> v2:172.21.15.161:3300/0 conn(0x7f4040152760 0x7f40400c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4035400640 1 --2- 172.21.15.146:0/2818162754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40400c6cc0 0x7f40400c5f10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f404006d830 tx=0x7f402802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e296eafa39e194d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f403d400640 1 -- 172.21.15.146:0/2818162754 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028037070 con 0x7f40400c6cc0 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f403d400640 1 -- 172.21.15.146:0/2818162754 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f402802ae50 con 0x7f40400c6cc0 2024-09-16T01:54:11.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f403d400640 1 -- 172.21.15.146:0/2818162754 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028032880 con 0x7f40400c6cc0 2024-09-16T01:54:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 -- 172.21.15.146:0/2818162754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40400c6cc0 msgr2=0x7f40400c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 --2- 172.21.15.146:0/2818162754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40400c6cc0 0x7f40400c5f10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f404006d830 tx=0x7f402802a300 comp rx=0 tx=0).stop 2024-09-16T01:54:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 -- 172.21.15.146:0/2818162754 shutdown_connections 2024-09-16T01:54:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 --2- 172.21.15.146:0/2818162754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40400c6cc0 0x7f40400c5f10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 --2- 172.21.15.146:0/2818162754 >> v2:172.21.15.161:3300/0 conn(0x7f4040152760 0x7f40400c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 -- 172.21.15.146:0/2818162754 >> 172.21.15.146:0/2818162754 conn(0x7f404006e670 msgr2=0x7f404006ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 -- 172.21.15.146:0/2818162754 shutdown_connections 2024-09-16T01:54:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.992+0000 7f4045200640 1 -- 172.21.15.146:0/2818162754 wait complete. 2024-09-16T01:54:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.994+0000 7f4045200640 1 Processor -- start 2024-09-16T01:54:11.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.994+0000 7f4045200640 1 -- start start 2024-09-16T01:54:11.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4045200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404006fee0 0x7f40401e27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4045200640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 0x7f40401e2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4045200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4040152760 0x7f40400cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4045200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4040152b40 con 0x7f404006fee0 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4045200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f40401593a0 con 0x7f4040152760 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4045200640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4040152cc0 con 0x7f40400c6cc0 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f403de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404006fee0 0x7f40401e27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4035400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 0x7f40401e2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f403e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4040152760 0x7f40400cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4035400640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 0x7f40401e2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:43354/0 (socket says 172.21.15.146:43354) 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.995+0000 7f4035400640 1 -- 172.21.15.146:0/382593422 learned_addr learned my addr 172.21.15.146:0/382593422 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:11.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4035400640 1 -- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4040152760 msgr2=0x7f40400cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4035400640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4040152760 0x7f40400cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4035400640 1 -- 172.21.15.146:0/382593422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404006fee0 msgr2=0x7f40401e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4035400640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404006fee0 0x7f40401e27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4035400640 1 -- 172.21.15.146:0/382593422 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40401e1d30 con 0x7f40400c6cc0 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f403de00640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404006fee0 0x7f40401e27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4035400640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 0x7f40401e2fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f402802a340 tx=0x7f40280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028041070 con 0x7f40400c6cc0 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f402803c680 con 0x7f40400c6cc0 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40401e36c0 con 0x7f40400c6cc0 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028037040 con 0x7f40400c6cc0 2024-09-16T01:54:11.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.996+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40401e1fd0 con 0x7f40400c6cc0 2024-09-16T01:54:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.997+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f404006f450 con 0x7f40400c6cc0 2024-09-16T01:54:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.998+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4028044070 con 0x7f40400c6cc0 2024-09-16T01:54:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.999+0000 7f4036a00640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f401c06c2b0 0x7f401c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.999+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 5 ==== osd_map(158..158 src has 1..158) ==== 6316+0+0 (secure 0 0 0) 0x7f40280ac3a0 con 0x7f40400c6cc0 2024-09-16T01:54:12.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:11.999+0000 7f403de00640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f401c06c2b0 0x7f401c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:12.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.000+0000 7f403de00640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f401c06c2b0 0x7f401c06e770 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4030009560 tx=0x7f4030009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:12.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.003+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f404006f450 con 0x7f40400c6cc0 2024-09-16T01:54:12.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.212+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f40400c72c0 con 0x7f40400c6cc0 2024-09-16T01:54:12.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.800+0000 7f4036a00640 1 -- 172.21.15.146:0/382593422 <== mon.2 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7f402807a250 con 0x7f40400c6cc0 2024-09-16T01:54:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f401c06c2b0 msgr2=0x7f401c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f401c06c2b0 0x7f401c06e770 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4030009560 tx=0x7f4030009190 comp rx=0 tx=0).stop 2024-09-16T01:54:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 msgr2=0x7f40401e2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 0x7f40401e2fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f402802a340 tx=0x7f40280046c0 comp rx=0 tx=0).stop 2024-09-16T01:54:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 shutdown_connections 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f401c06c2b0 0x7f401c06e770 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4040152760 0x7f40400cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40400c6cc0 0x7f40401e2fa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 --2- 172.21.15.146:0/382593422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404006fee0 0x7f40401e27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 >> 172.21.15.146:0/382593422 conn(0x7f404006e670 msgr2=0x7f404014ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 shutdown_connections 2024-09-16T01:54:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:12.803+0000 7f4045200640 1 -- 172.21.15.146:0/382593422 wait complete. 2024-09-16T01:54:12.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:54:13.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.018+0000 7fd6aa600640 1 Processor -- start 2024-09-16T01:54:13.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.018+0000 7fd6aa600640 1 -- start start 2024-09-16T01:54:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6aa600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6a4151900 0x7fd6a40c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6aa600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a415b480 con 0x7fd6a4072ea0 2024-09-16T01:54:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6aa600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a4071b40 con 0x7fd6a4151ce0 2024-09-16T01:54:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6aa600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd6a406f250 con 0x7fd6a4151900 2024-09-16T01:54:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6a3400640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6a4151900 0x7fd6a40c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:13.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6a3e00640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd6a4151ce0 0x7fd6a40c5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:35664/0 (socket says 172.21.15.146:35664) 2024-09-16T01:54:13.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.019+0000 7fd6a3e00640 1 -- 172.21.15.146:0/3436551479 learned_addr learned my addr 172.21.15.146:0/3436551479 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:54:13.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.020+0000 7fd6a3400640 1 -- 172.21.15.146:0/3436551479 >> v1:172.21.15.161:6789/0 conn(0x7fd6a4151ce0 legacy=0x7fd6a40c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:13.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.020+0000 7fd6a3400640 1 -- 172.21.15.146:0/3436551479 >> v1:172.21.15.146:6789/0 conn(0x7fd6a4072ea0 legacy=0x7fd6a415ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:13.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.020+0000 7fd6a3400640 1 -- 172.21.15.146:0/3436551479 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a4071e20 con 0x7fd6a4151900 2024-09-16T01:54:13.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.020+0000 7fd6a2a00640 1 -- 172.21.15.146:0/3436551479 <== mon.0 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83986459 0 0) 0x7fd6a415b480 con 0x7fd6a4072ea0 2024-09-16T01:54:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.021+0000 7fd6a3400640 1 --2- 172.21.15.146:0/3436551479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c35b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd6940097a0 tx=0x7fd69402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=817359a8723b32e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.021+0000 7fd6a2a00640 1 -- 172.21.15.146:0/3436551479 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694037070 con 0x7fd6a4151900 2024-09-16T01:54:13.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.021+0000 7fd6a2a00640 1 -- 172.21.15.146:0/3436551479 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd694032440 con 0x7fd6a4151900 2024-09-16T01:54:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.021+0000 7fd6a2a00640 1 -- 172.21.15.146:0/3436551479 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694032830 con 0x7fd6a4151900 2024-09-16T01:54:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.022+0000 7fd6aa600640 1 -- 172.21.15.146:0/3436551479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 msgr2=0x7fd6a40c35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:13.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.022+0000 7fd6aa600640 1 --2- 172.21.15.146:0/3436551479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c35b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd6940097a0 tx=0x7fd69402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:54:13.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.022+0000 7fd6aa600640 1 -- 172.21.15.146:0/3436551479 shutdown_connections 2024-09-16T01:54:13.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.022+0000 7fd6aa600640 1 --2- 172.21.15.146:0/3436551479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c35b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:13.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.022+0000 7fd6aa600640 1 -- 172.21.15.146:0/3436551479 >> 172.21.15.146:0/3436551479 conn(0x7fd6a40cc0b0 msgr2=0x7fd6a40cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:13.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.023+0000 7fd6aa600640 1 -- 172.21.15.146:0/3436551479 shutdown_connections 2024-09-16T01:54:13.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.023+0000 7fd6aa600640 1 -- 172.21.15.146:0/3436551479 wait complete. 2024-09-16T01:54:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.024+0000 7fd6aa600640 1 Processor -- start 2024-09-16T01:54:13.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.024+0000 7fd6aa600640 1 -- start start 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6aa600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 0x7fd6a41f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6aa600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6aa600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6a4151ce0 0x7fd6a40c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6aa600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a40c3bd0 con 0x7fd6a4072ea0 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6aa600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a4073860 con 0x7fd6a4151900 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6aa600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a4073cf0 con 0x7fd6a4151ce0 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6a3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 0x7fd6a41f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6a3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 0x7fd6a41f1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:52428/0 (socket says 172.21.15.146:52428) 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.025+0000 7fd6a3400640 1 -- 172.21.15.146:0/419904205 learned_addr learned my addr 172.21.15.146:0/419904205 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd69aa00640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:13.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3e00640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6a4151ce0 0x7fd6a40c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3400640 1 -- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6a4151ce0 msgr2=0x7fd6a40c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3400640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6a4151ce0 0x7fd6a40c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3400640 1 -- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 msgr2=0x7fd6a40c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3400640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:13.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3400640 1 -- 172.21.15.146:0/419904205 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a4141230 con 0x7fd6a4072ea0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6a3400640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 0x7fd6a41f1c60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd694035040 tx=0x7fd6940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694037040 con 0x7fd6a4072ea0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd694042070 con 0x7fd6a4072ea0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a41f2650 con 0x7fd6a4072ea0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.026+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694030c10 con 0x7fd6a4072ea0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.027+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a4140f50 con 0x7fd6a4072ea0 2024-09-16T01:54:13.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.028+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a406e950 con 0x7fd6a4072ea0 2024-09-16T01:54:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.029+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd694047460 con 0x7fd6a4072ea0 2024-09-16T01:54:13.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.029+0000 7fd69be00640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd68006c2b0 0x7fd68006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.030+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 6316+0+0 (secure 0 0 0) 0x7fd6940ac4a0 con 0x7fd6a4072ea0 2024-09-16T01:54:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.030+0000 7fd69aa00640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd68006c2b0 0x7fd68006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:13.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.030+0000 7fd69aa00640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd68006c2b0 0x7fd68006e770 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd6a413cb60 tx=0x7fd68c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:13.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.034+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd6a4141230 con 0x7fd6a4072ea0 2024-09-16T01:54:13.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:13.208+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd6a41503e0 con 0x7fd6a4072ea0 2024-09-16T01:54:14.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.196+0000 7fd69be00640 1 -- 172.21.15.146:0/419904205 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v159) ==== 181+0+0 (secure 0 0 0) 0x7fd69407a880 con 0x7fd6a4072ea0 2024-09-16T01:54:14.197 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd68006c2b0 msgr2=0x7fd68006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd68006c2b0 0x7fd68006e770 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd6a413cb60 tx=0x7fd68c009040 comp rx=0 tx=0).stop 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 msgr2=0x7fd6a41f1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 0x7fd6a41f1c60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd694035040 tx=0x7fd6940046c0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 shutdown_connections 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7fd68006c2b0 0x7fd68006e770 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6a4151ce0 0x7fd6a40c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6a4151900 0x7fd6a40c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 --2- 172.21.15.146:0/419904205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6a4072ea0 0x7fd6a41f1c60 secure :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd694035040 tx=0x7fd6940046c0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.200+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 >> 172.21.15.146:0/419904205 conn(0x7fd6a40cc0b0 msgr2=0x7fd6a40cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.201+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 shutdown_connections 2024-09-16T01:54:14.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.201+0000 7fd6aa600640 1 -- 172.21.15.146:0/419904205 wait complete. 2024-09-16T01:54:14.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:54:14.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.382+0000 7f6b31600640 1 Processor -- start 2024-09-16T01:54:14.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.382+0000 7f6b31600640 1 -- start start 2024-09-16T01:54:14.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.383+0000 7f6b31600640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6b2c152cc0 0x7f6b2c0c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:14.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.383+0000 7f6b31600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6b2c0c3da0 0x7f6b2c0c4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.383+0000 7f6b31600640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2c1530a0 con 0x7f6b2c06e9e0 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.383+0000 7f6b31600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 -- 0x7f6b2c159960 con 0x7f6b2c0c3da0 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.383+0000 7f6b31600640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6b2c0c2d70 con 0x7f6b2c152cc0 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2aa00640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6b2c152cc0 0x7f6b2c0c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2a000640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6b2c06e9e0 0x7f6b2c06ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.146:35666/0 (socket says 172.21.15.146:35666) 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2a000640 1 -- 172.21.15.146:0/1341350178 learned_addr learned my addr 172.21.15.146:0/1341350178 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-16T01:54:14.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2b400640 1 --2- 172.21.15.146:0/1341350178 >> v2:172.21.15.146:3300/0 conn(0x7f6b2c0c3da0 0x7f6b2c0c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:14.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2aa00640 1 -- 172.21.15.146:0/1341350178 >> v2:172.21.15.146:3300/0 conn(0x7f6b2c0c3da0 msgr2=0x7f6b2c0c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:54:14.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2aa00640 1 --2- 172.21.15.146:0/1341350178 >> v2:172.21.15.146:3300/0 conn(0x7f6b2c0c3da0 0x7f6b2c0c4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2aa00640 1 -- 172.21.15.146:0/1341350178 >> v1:172.21.15.161:6789/0 conn(0x7f6b2c06e9e0 legacy=0x7f6b2c06ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:14.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b29600640 1 -- 172.21.15.146:0/1341350178 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676732876 0 0) 0x7f6b2c1530a0 con 0x7f6b2c06e9e0 2024-09-16T01:54:14.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.384+0000 7f6b2aa00640 1 -- 172.21.15.146:0/1341350178 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2c0c8550 con 0x7f6b2c152cc0 2024-09-16T01:54:14.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.385+0000 7f6b2aa00640 1 --2- 172.21.15.146:0/1341350178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c0c6220 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c074060 tx=0x7f6b1c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68460947fb003a04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:14.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.385+0000 7f6b29600640 1 -- 172.21.15.146:0/1341350178 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c037070 con 0x7f6b2c152cc0 2024-09-16T01:54:14.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.385+0000 7f6b29600640 1 -- 172.21.15.146:0/1341350178 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6b1c032440 con 0x7f6b2c152cc0 2024-09-16T01:54:14.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.385+0000 7f6b29600640 1 -- 172.21.15.146:0/1341350178 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c032830 con 0x7f6b2c152cc0 2024-09-16T01:54:14.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 -- 172.21.15.146:0/1341350178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 msgr2=0x7f6b2c0c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:14.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 --2- 172.21.15.146:0/1341350178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c0c6220 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c074060 tx=0x7f6b1c02a230 comp rx=0 tx=0).stop 2024-09-16T01:54:14.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 -- 172.21.15.146:0/1341350178 shutdown_connections 2024-09-16T01:54:14.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 --2- 172.21.15.146:0/1341350178 >> v2:172.21.15.146:3300/0 conn(0x7f6b2c0c3da0 0x7f6b2c0c4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 --2- 172.21.15.146:0/1341350178 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c0c6220 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 -- 172.21.15.146:0/1341350178 >> 172.21.15.146:0/1341350178 conn(0x7f6b2c0cc0b0 msgr2=0x7f6b2c0cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:14.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 -- 172.21.15.146:0/1341350178 shutdown_connections 2024-09-16T01:54:14.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.386+0000 7f6b31600640 1 -- 172.21.15.146:0/1341350178 wait complete. 2024-09-16T01:54:14.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.388+0000 7f6b31600640 1 Processor -- start 2024-09-16T01:54:14.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.388+0000 7f6b31600640 1 -- start start 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b31600640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6b2c06e9e0 0x7f6b2c14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b31600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b2c0c3da0 0x7f6b2c151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b31600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c1f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b31600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2c070320 con 0x7f6b2c0c3da0 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b31600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2c159680 con 0x7f6b2c152cc0 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b31600640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6b2c0c8210 con 0x7f6b2c06e9e0 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6b2c06e9e0 0x7f6b2c14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c1f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:14.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b2c0c3da0 0x7f6b2c151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2aa00640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6b2c06e9e0 0x7f6b2c14e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.146:55334/0 (socket says 172.21.15.146:55334) 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c1f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.146:54448/0 (socket says 172.21.15.146:54448) 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2aa00640 1 -- 172.21.15.146:0/4059652994 learned_addr learned my addr 172.21.15.146:0/4059652994 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 -- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6b2c06e9e0 msgr2=0x7f6b2c14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6b2c06e9e0 0x7f6b2c14e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 -- 172.21.15.146:0/4059652994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b2c0c3da0 msgr2=0x7f6b2c151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b2c0c3da0 0x7f6b2c151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.389+0000 7f6b2b400640 1 -- 172.21.15.146:0/4059652994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2c1f0db0 con 0x7f6b2c152cc0 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.390+0000 7f6b2b400640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c1f4230 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009d90 tx=0x7f6b2400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.390+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24018020 con 0x7f6b2c152cc0 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.390+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6b24014480 con 0x7f6b2c152cc0 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.390+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2c14edc0 con 0x7f6b2c152cc0 2024-09-16T01:54:14.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.390+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24014890 con 0x7f6b2c152cc0 2024-09-16T01:54:14.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.390+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b2c1f1330 con 0x7f6b2c152cc0 2024-09-16T01:54:14.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.391+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b2c04ffb0 con 0x7f6b2c152cc0 2024-09-16T01:54:14.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.392+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6b2401b020 con 0x7f6b2c152cc0 2024-09-16T01:54:14.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.393+0000 7f6b22a00640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6b0c06c2e0 0x7f6b0c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:54:14.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.393+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 5849+0+0 (secure 0 0 0) 0x7f6b2408c2d0 con 0x7f6b2c152cc0 2024-09-16T01:54:14.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.393+0000 7f6b2aa00640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6b0c06c2e0 0x7f6b0c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:54:14.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.394+0000 7f6b2aa00640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6b0c06c2e0 0x7f6b0c06e7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c02a270 tx=0x7f6b1c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:54:14.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.397+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6b2c04ffb0 con 0x7f6b2c152cc0 2024-09-16T01:54:14.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:14.579+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6b2c0c6220 con 0x7f6b2c152cc0 2024-09-16T01:54:15.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.205+0000 7f6b22a00640 1 -- 172.21.15.146:0/4059652994 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v160) ==== 169+0+0 (secure 0 0 0) 0x7f6b24022020 con 0x7f6b2c152cc0 2024-09-16T01:54:15.206 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' removed 2024-09-16T01:54:15.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6b0c06c2e0 msgr2=0x7f6b0c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:15.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6b0c06c2e0 0x7f6b0c06e7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c02a270 tx=0x7f6b1c02e000 comp rx=0 tx=0).stop 2024-09-16T01:54:15.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 msgr2=0x7f6b2c1f4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:54:15.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c1f4230 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009d90 tx=0x7f6b2400c490 comp rx=0 tx=0).stop 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 shutdown_connections 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:6832/3835694012,v1:172.21.15.161:6833/3835694012] conn(0x7f6b0c06c2e0 0x7f6b0c06e7a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6b2c152cc0 0x7f6b2c1f4230 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b2c0c3da0 0x7f6b2c151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 --2- 172.21.15.146:0/4059652994 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6b2c06e9e0 0x7f6b2c14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 >> 172.21.15.146:0/4059652994 conn(0x7f6b2c0cc0b0 msgr2=0x7f6b2c156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 shutdown_connections 2024-09-16T01:54:15.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-16T01:54:15.209+0000 7f6b31600640 1 -- 172.21.15.146:0/4059652994 wait complete. 2024-09-16T01:54:15.229 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T01:54:15.229 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:54:15.229 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.244 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:54:15.244 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:54:15.244 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.298 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:54:15.298 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-09-16T01:54:15.298 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.353 INFO:tasks.ceph.mds.e:Stopped 2024-09-16T01:54:15.353 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:54:15.353 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.364 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:54:15.364 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:54:15.364 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.416 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:54:15.416 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-09-16T01:54:15.416 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.467 INFO:tasks.ceph.mds.f:Stopped 2024-09-16T01:54:15.467 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T01:54:15.467 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T01:54:15.467 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.519 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T01:54:15.519 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T01:54:15.520 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.574 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T01:54:15.574 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T01:54:15.575 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.633 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T01:54:15.633 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T01:54:15.633 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-09-16T01:54:15.703 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T01:54:15.703 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T01:54:15.703 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.744 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T01:54:15.744 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T01:54:15.744 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.817 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T01:54:15.817 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T01:54:15.817 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.889 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T01:54:15.889 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T01:54:15.889 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:15.960 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T01:54:15.960 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T01:54:15.960 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T01:54:15.960 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:16.020 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T01:54:16.020 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T01:54:16.020 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:16.050 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T01:54:16.050 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T01:54:16.050 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T01:54:16.050 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:16.070 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T01:54:16.070 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T01:54:16.070 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:16.103 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T01:54:16.103 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T01:54:16.103 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:54:16.155 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T01:54:16.156 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T01:54:16.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v MDS_SLOW_METADATA_IO | egrep -v MDS_TRIM | head -n 1 2024-09-16T01:54:16.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:54:16.260 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T01:54:16.607 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:54:16.607 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T01:54:16.782 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:54:16.782 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T01:54:16.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:54:16.884 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T01:54:17.018 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:54:17.018 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T01:54:17.204 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:54:17.204 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T01:54:17.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:54:17.283 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T01:54:17.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:54:17.340 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T01:54:17.390 INFO:tasks.ceph:Archiving mon data... 2024-09-16T01:54:17.391 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/data/mon.a.tgz 2024-09-16T01:54:17.392 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-09-16T01:54:17.427 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.3tQiyyYYIq 2024-09-16T01:54:17.428 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3tQiyyYYIq 2024-09-16T01:54:17.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.3tQiyyYYIq 2024-09-16T01:54:17.727 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.3tQiyyYYIq is 471KB 2024-09-16T01:54:17.819 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.3tQiyyYYIq 2024-09-16T01:54:17.854 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/data/mon.b.tgz 2024-09-16T01:54:17.855 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-09-16T01:54:17.874 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.uD10PQDCff 2024-09-16T01:54:17.875 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uD10PQDCff 2024-09-16T01:54:18.089 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.uD10PQDCff 2024-09-16T01:54:18.172 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.uD10PQDCff is 668KB 2024-09-16T01:54:18.275 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.uD10PQDCff 2024-09-16T01:54:18.290 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/data/mon.c.tgz 2024-09-16T01:54:18.290 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-09-16T01:54:18.345 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.T2lpuu4yP8 2024-09-16T01:54:18.346 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.T2lpuu4yP8 2024-09-16T01:54:18.544 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.T2lpuu4yP8 2024-09-16T01:54:18.629 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.T2lpuu4yP8 is 668KB 2024-09-16T01:54:18.703 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.T2lpuu4yP8 2024-09-16T01:54:18.718 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T01:54:18.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T01:54:18.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T01:54:18.820 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T01:54:18.838 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T01:54:18.839 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T01:54:18.840 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/remote/smithi146/crash 2024-09-16T01:54:18.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T01:54:18.921 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/remote/smithi161/crash 2024-09-16T01:54:18.922 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T01:54:18.946 INFO:tasks.ceph:Shutting down logrotate 2024-09-16T01:54:18.946 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:54:18.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:54:19.044 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-16T01:54:19.099 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-16T01:54:19.124 INFO:tasks.ceph:Compressing logs... 2024-09-16T01:54:19.124 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:54:19.128 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:54:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64797.log 2024-09-16T01:54:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T01:54:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47915.log 2024-09-16T01:54:19.180 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64797.log.gz 2024-09-16T01:54:19.181 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40487.log 2024-09-16T01:54:19.182 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2024-09-16T01:54:19.182 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47915.log.gz 2024-09-16T01:54:19.183 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60086.log 2024-09-16T01:54:19.184 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40487.log.gz 2024-09-16T01:54:19.184 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65962.log 2024-09-16T01:54:19.184 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2024-09-16T01:54:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57988.log 2024-09-16T01:54:19.186 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60086.log.gz 2024-09-16T01:54:19.186 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49675.log 2024-09-16T01:54:19.186 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65962.log.gz 2024-09-16T01:54:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48400.log 2024-09-16T01:54:19.188 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57988.log.gz 2024-09-16T01:54:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41248.log 2024-09-16T01:54:19.188 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49675.log.gz 2024-09-16T01:54:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44609.log 2024-09-16T01:54:19.192 INFO:teuthology.orchestra.run.smithi146.stderr: 90.5%/var/log/ceph/ceph-client.admin.41248.log: -- replaced with /var/log/ceph/ceph-client.admin.48400.log.gz 2024-09-16T01:54:19.192 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52906.log 2024-09-16T01:54:19.192 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41248.log.gz 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49490.log 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.44609.log.gz 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59688.log 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.52906.log.gz 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63596.log 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49490.log.gz 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45645.log 2024-09-16T01:54:19.193 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59688.log.gz 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36912.log 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.36912.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36912.log.gz 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37083.log 2024-09-16T01:54:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.37083.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36952.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37083.log.gz 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37203.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.36952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37032.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36952.log.gz 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37203.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37203.log.gz 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37032.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9%/var/log/ceph/ceph-osd.6.log: -- replaced with /var/log/ceph/ceph-client.admin.37032.log.gz 2024-09-16T01:54:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47898.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.63596.log.gz 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66677.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45645.log.gz 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47898.log.gz 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66677.log.gz 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62736.log 2024-09-16T01:54:19.199 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53209.log 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66344.log 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mds.d.log: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T01:54:19.200 INFO:teuthology.orchestra.run.smithi161.stderr: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T01:54:19.201 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62736.log.gz 2024-09-16T01:54:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51297.log 2024-09-16T01:54:19.202 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53209.log.gz 2024-09-16T01:54:19.202 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67646.log 2024-09-16T01:54:19.202 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51297.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66344.log.gz 2024-09-16T01:54:19.202 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67222.log 2024-09-16T01:54:19.202 INFO:teuthology.orchestra.run.smithi146.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.51297.log.gz 2024-09-16T01:54:19.203 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42571.log 2024-09-16T01:54:19.203 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67646.log.gz 2024-09-16T01:54:19.203 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64641.log 2024-09-16T01:54:19.204 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67222.log.gz 2024-09-16T01:54:19.204 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68360.log 2024-09-16T01:54:19.204 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42571.log.gz 2024-09-16T01:54:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2024-09-16T01:54:19.205 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64641.log.gz 2024-09-16T01:54:19.206 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48159.log 2024-09-16T01:54:19.206 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68360.log.gz 2024-09-16T01:54:19.207 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62554.log 2024-09-16T01:54:19.207 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2024-09-16T01:54:19.208 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56184.log 2024-09-16T01:54:19.208 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48159.log.gz 2024-09-16T01:54:19.208 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51476.log 2024-09-16T01:54:19.209 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.62554.log.gz 2024-09-16T01:54:19.209 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37123.log 2024-09-16T01:54:19.210 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64433.log 2024-09-16T01:54:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36992.log 2024-09-16T01:54:19.210 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37123.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37123.log.gz 2024-09-16T01:54:19.211 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51476.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.56184.log.gz 2024-09-16T01:54:19.211 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68597.log 2024-09-16T01:54:19.211 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51476.log.gz 2024-09-16T01:54:19.211 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60214.log 2024-09-16T01:54:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37163.log 2024-09-16T01:54:19.212 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.36992.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36992.log.gz 2024-09-16T01:54:19.213 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64433.log.gz 2024-09-16T01:54:19.213 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58093.log 2024-09-16T01:54:19.213 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68597.log.gz 2024-09-16T01:54:19.213 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T01:54:19.213 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37163.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37163.log.gz 2024-09-16T01:54:19.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2024-09-16T01:54:19.214 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60214.log.gz 2024-09-16T01:54:19.214 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57364.log 2024-09-16T01:54:19.214 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58093.log.gz 2024-09-16T01:54:19.215 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2024-09-16T01:54:19.215 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2024-09-16T01:54:19.216 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-09-16T01:54:19.216 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44587.log 2024-09-16T01:54:19.216 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41590.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.57364.log.gz 2024-09-16T01:54:19.217 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51317.log 2024-09-16T01:54:19.217 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2024-09-16T01:54:19.217 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-09-16T01:54:19.217 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44587.log.gz 2024-09-16T01:54:19.218 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43428.log 2024-09-16T01:54:19.218 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51317.log.gz 2024-09-16T01:54:19.219 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65022.log 2024-09-16T01:54:19.219 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-09-16T01:54:19.220 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51079.log 2024-09-16T01:54:19.220 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.43428.log.gz 2024-09-16T01:54:19.221 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53864.log 2024-09-16T01:54:19.221 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65022.log.gz 2024-09-16T01:54:19.222 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51079.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T01:54:19.222 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51079.log.gz 2024-09-16T01:54:19.223 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68066.log 2024-09-16T01:54:19.223 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53864.log.gz 2024-09-16T01:54:19.224 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58012.log 2024-09-16T01:54:19.224 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42505.log 2024-09-16T01:54:19.225 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68066.log.gz 2024-09-16T01:54:19.226 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61988.log 2024-09-16T01:54:19.226 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58012.log.gz 2024-09-16T01:54:19.228 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-16T01:54:19.229 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53868.log 2024-09-16T01:54:19.229 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61988.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42505.log.gz 2024-09-16T01:54:19.229 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49614.log 2024-09-16T01:54:19.229 INFO:teuthology.orchestra.run.smithi146.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61988.log.gz 2024-09-16T01:54:19.229 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61744.log 2024-09-16T01:54:19.229 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.53868.log.gz 2024-09-16T01:54:19.230 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49614.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49614.log.gz 2024-09-16T01:54:19.230 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-09-16T01:54:19.231 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54811.log 2024-09-16T01:54:19.231 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61744.log.gz 2024-09-16T01:54:19.232 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67380.log 2024-09-16T01:54:19.232 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-09-16T01:54:19.233 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41587.log 2024-09-16T01:54:19.233 INFO:teuthology.orchestra.run.smithi146.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.54811.log.gz 2024-09-16T01:54:19.234 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60840.log 2024-09-16T01:54:19.234 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67380.log.gz 2024-09-16T01:54:19.235 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41587.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41587.log.gz 2024-09-16T01:54:19.235 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66353.log 2024-09-16T01:54:19.236 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47342.log 2024-09-16T01:54:19.236 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60840.log.gz 2024-09-16T01:54:19.237 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47065.log 2024-09-16T01:54:19.237 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66353.log.gz 2024-09-16T01:54:19.238 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45543.log 2024-09-16T01:54:19.238 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47342.log.gz 2024-09-16T01:54:19.238 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46604.log 2024-09-16T01:54:19.239 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47065.log.gz 2024-09-16T01:54:19.240 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-09-16T01:54:19.240 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45543.log.gz 2024-09-16T01:54:19.241 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49780.log 2024-09-16T01:54:19.241 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46604.log.gz 2024-09-16T01:54:19.242 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.39769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67264.log 2024-09-16T01:54:19.242 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gz 2024-09-16T01:54:19.242 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60677.log 2024-09-16T01:54:19.243 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49780.log.gz 2024-09-16T01:54:19.243 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41597.log 2024-09-16T01:54:19.244 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67264.log.gz 2024-09-16T01:54:19.245 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56950.log 2024-09-16T01:54:19.245 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60677.log.gz 2024-09-16T01:54:19.245 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48005.log 2024-09-16T01:54:19.245 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41597.log.gz 2024-09-16T01:54:19.246 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66130.log 2024-09-16T01:54:19.246 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56950.log.gz 2024-09-16T01:54:19.247 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58681.log 2024-09-16T01:54:19.248 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2024-09-16T01:54:19.248 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66130.log: gzip -5 --verbose 88.2% -- -- replaced with /var/log/ceph/ceph-client.admin.66130.log.gz 2024-09-16T01:54:19.248 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.56436.log 2024-09-16T01:54:19.249 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67730.log 2024-09-16T01:54:19.249 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58681.log.gz 2024-09-16T01:54:19.250 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2024-09-16T01:54:19.250 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56436.log.gz 2024-09-16T01:54:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52744.log 2024-09-16T01:54:19.251 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67730.log.gz 2024-09-16T01:54:19.252 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67898.log 2024-09-16T01:54:19.252 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2024-09-16T01:54:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57463.log 2024-09-16T01:54:19.253 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52744.log.gz 2024-09-16T01:54:19.254 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67898.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67898.log.gz 2024-09-16T01:54:19.254 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54558.log 2024-09-16T01:54:19.255 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42103.log 2024-09-16T01:54:19.255 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57463.log.gz 2024-09-16T01:54:19.256 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53251.log 2024-09-16T01:54:19.256 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54558.log.gz 2024-09-16T01:54:19.257 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45346.log 2024-09-16T01:54:19.257 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42103.log.gz 2024-09-16T01:54:19.258 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53251.log: gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53251.log.gz -5 2024-09-16T01:54:19.258 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49428.log 2024-09-16T01:54:19.258 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40034.log 2024-09-16T01:54:19.259 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.45346.log.gz 2024-09-16T01:54:19.259 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49428.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49428.log.gz 2024-09-16T01:54:19.260 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68108.log 2024-09-16T01:54:19.261 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40034.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.40034.log.gz 2024-09-16T01:54:19.261 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63132.log 2024-09-16T01:54:19.261 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64033.log 87.6% 2024-09-16T01:54:19.262 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68108.log.gz 2024-09-16T01:54:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mds.f.log: 92.5% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-09-16T01:54:19.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46701.log 2024-09-16T01:54:19.263 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63132.log.gz 2024-09-16T01:54:19.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-09-16T01:54:19.264 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64033.log.gz 2024-09-16T01:54:19.264 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42373.log 2024-09-16T01:54:19.265 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46701.log.gz 2024-09-16T01:54:19.265 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.39884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55623.log 2024-09-16T01:54:19.265 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-09-16T01:54:19.266 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65309.log 2024-09-16T01:54:19.267 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42373.log.gz 2024-09-16T01:54:19.267 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66340.log 2024-09-16T01:54:19.268 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55623.log.gz 2024-09-16T01:54:19.268 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55971.log 2024-09-16T01:54:19.268 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65309.log.gz 2024-09-16T01:54:19.269 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66340.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66340.log.gz 2024-09-16T01:54:19.270 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55733.log 2024-09-16T01:54:19.270 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42061.log 2024-09-16T01:54:19.271 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55971.log.gz 2024-09-16T01:54:19.271 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53103.log 2024-09-16T01:54:19.272 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.55733.log.gz 2024-09-16T01:54:19.272 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51811.log 2024-09-16T01:54:19.272 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42061.log.gz 2024-09-16T01:54:19.273 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66352.log 2024-09-16T01:54:19.273 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53103.log.gz 2024-09-16T01:54:19.274 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56660.log 2024-09-16T01:54:19.274 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51811.log.gz 2024-09-16T01:54:19.275 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53334.log 2024-09-16T01:54:19.275 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66352.log.gz 2024-09-16T01:54:19.276 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52468.log 2024-09-16T01:54:19.276 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.56660.log.gz 2024-09-16T01:54:19.277 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67025.log 2024-09-16T01:54:19.277 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53334.log.gz 2024-09-16T01:54:19.278 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68823.log 2024-09-16T01:54:19.278 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52468.log.gz 2024-09-16T01:54:19.279 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67520.log 2024-09-16T01:54:19.279 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67025.log.gz 2024-09-16T01:54:19.280 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59974.log 2024-09-16T01:54:19.280 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68823.log.gz 2024-09-16T01:54:19.281 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66685.log 2024-09-16T01:54:19.281 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67520.log.gz 2024-09-16T01:54:19.281 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63090.log 2024-09-16T01:54:19.282 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59974.log.gz 2024-09-16T01:54:19.282 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41589.log 2024-09-16T01:54:19.282 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66685.log.gz 2024-09-16T01:54:19.283 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67026.log 2024-09-16T01:54:19.283 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63090.log.gz 2024-09-16T01:54:19.284 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53955.log 2024-09-16T01:54:19.284 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41589.log.gz 2024-09-16T01:54:19.285 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49303.log 2024-09-16T01:54:19.285 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67026.log.gz 2024-09-16T01:54:19.286 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62088.log 2024-09-16T01:54:19.286 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53955.log.gz 2024-09-16T01:54:19.287 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55307.log 2024-09-16T01:54:19.287 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49303.log.gz 2024-09-16T01:54:19.288 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68192.log 2024-09-16T01:54:19.288 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62088.log.gz 2024-09-16T01:54:19.289 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55307.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55307.log.gz 2024-09-16T01:54:19.289 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59664.log 2024-09-16T01:54:19.290 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50697.log 2024-09-16T01:54:19.290 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68192.log.gz 2024-09-16T01:54:19.290 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59664.log: gzip -5 88.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.59664.log.gz 2024-09-16T01:54:19.291 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.40310.log 2024-09-16T01:54:19.291 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45088.log 2024-09-16T01:54:19.292 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.50697.log.gz 2024-09-16T01:54:19.293 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40310.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40310.log.gz 2024-09-16T01:54:19.293 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59514.log 2024-09-16T01:54:19.293 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64036.log 2024-09-16T01:54:19.294 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.45088.log.gz 2024-09-16T01:54:19.295 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58498.log 2024-09-16T01:54:19.295 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.59514.log.gz 2024-09-16T01:54:19.295 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67024.log 2024-09-16T01:54:19.296 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64036.log.gz 2024-09-16T01:54:19.296 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50858.log 2024-09-16T01:54:19.296 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58498.log.gz 2024-09-16T01:54:19.297 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67024.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.67024.log.gz 2024-09-16T01:54:19.297 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph.audit.log 2024-09-16T01:54:19.298 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50858.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.50858.log.gz 2024-09-16T01:54:19.298 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43240.log 2024-09-16T01:54:19.299 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46201.log 2024-09-16T01:54:19.301 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43240.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43240.log.gz 2024-09-16T01:54:19.301 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48103.log 2024-09-16T01:54:19.301 INFO:teuthology.orchestra.run.smithi146.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T01:54:19.301 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50575.log 2024-09-16T01:54:19.302 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46201.log.gz 2024-09-16T01:54:19.302 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49125.log 2024-09-16T01:54:19.302 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48103.log.gz 2024-09-16T01:54:19.303 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46462.log 2024-09-16T01:54:19.303 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50575.log.gz 2024-09-16T01:54:19.304 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40739.log 2024-09-16T01:54:19.304 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49125.log.gz 2024-09-16T01:54:19.305 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46462.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46462.log.gz 2024-09-16T01:54:19.305 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47526.log 2024-09-16T01:54:19.306 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59120.log 2024-09-16T01:54:19.306 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40739.log.gz 2024-09-16T01:54:19.307 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55069.log 2024-09-16T01:54:19.307 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.47526.log.gz 2024-09-16T01:54:19.308 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53537.log 2024-09-16T01:54:19.308 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59120.log.gz 2024-09-16T01:54:19.309 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41595.log 2024-09-16T01:54:19.310 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.55069.log.gz 2024-09-16T01:54:19.310 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53537.log: gzip -5 88.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.53537.log.gz 2024-09-16T01:54:19.310 INFO:teuthology.orchestra.run.smithi146.stderr: -- /var/log/ceph/ceph-client.admin.50417.log 2024-09-16T01:54:19.311 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41595.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.41595.log.gz 2024-09-16T01:54:19.311 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54340.log 2024-09-16T01:54:19.312 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44172.log 2024-09-16T01:54:19.312 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.50417.log.gz 2024-09-16T01:54:19.313 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2024-09-16T01:54:19.313 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54340.log.gz 2024-09-16T01:54:19.314 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67604.log 2024-09-16T01:54:19.315 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44172.log.gz 2024-09-16T01:54:19.315 INFO:teuthology.orchestra.run.smithi161.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T01:54:19.317 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41255.log 2024-09-16T01:54:19.317 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-09-16T01:54:19.317 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65524.log 2024-09-16T01:54:19.317 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67604.log.gz 2024-09-16T01:54:19.318 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-09-16T01:54:19.318 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41255.log.gz 2024-09-16T01:54:19.320 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-09-16T01:54:19.320 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65524.log.gz 2024-09-16T01:54:19.320 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60146.log 2024-09-16T01:54:19.320 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-09-16T01:54:19.320 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.39986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61119.log 2024-09-16T01:54:19.320 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-09-16T01:54:19.321 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62653.log 2024-09-16T01:54:19.321 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60146.log.gz 2024-09-16T01:54:19.321 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61119.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61119.log.gz -5 2024-09-16T01:54:19.321 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58622.log 2024-09-16T01:54:19.322 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45805.log 2024-09-16T01:54:19.322 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62653.log.gz 2024-09-16T01:54:19.323 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67940.log 2024-09-16T01:54:19.323 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.58622.log.gz 2024-09-16T01:54:19.324 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65566.log 2024-09-16T01:54:19.324 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45805.log.gz 2024-09-16T01:54:19.325 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47330.log 2024-09-16T01:54:19.325 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67940.log.gz 2024-09-16T01:54:19.326 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46689.log 2024-09-16T01:54:19.326 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65566.log.gz 2024-09-16T01:54:19.327 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47330.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47330.log.gz 2024-09-16T01:54:19.327 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58980.log 2024-09-16T01:54:19.328 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49366.log 2024-09-16T01:54:19.328 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46689.log.gz 2024-09-16T01:54:19.329 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41256.log 2024-09-16T01:54:19.329 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58980.log.gz 2024-09-16T01:54:19.330 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35859.log 2024-09-16T01:54:19.330 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49366.log.gz 2024-09-16T01:54:19.331 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46808.log 2024-09-16T01:54:19.331 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41256.log.gz 2024-09-16T01:54:19.332 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.35859.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.35859.log.gz --verbose 2024-09-16T01:54:19.332 INFO:teuthology.orchestra.run.smithi146.stderr: -- /var/log/ceph/ceph-client.admin.44349.log 2024-09-16T01:54:19.332 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61481.log 2024-09-16T01:54:19.333 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46808.log.gz 2024-09-16T01:54:19.333 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44349.log: gzip -5 --verbose 87.9% -- -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-09-16T01:54:19.334 INFO:teuthology.orchestra.run.smithi146.stderr: /var/log/ceph/ceph-client.admin.60139.log 2024-09-16T01:54:19.334 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49973.log 2024-09-16T01:54:19.335 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61481.log.gz 2024-09-16T01:54:19.335 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60139.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60139.log.gz 2024-09-16T01:54:19.336 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61866.log 2024-09-16T01:54:19.336 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49973.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49973.log.gz 2024-09-16T01:54:19.337 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2024-09-16T01:54:19.337 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66684.log 2024-09-16T01:54:19.337 INFO:teuthology.orchestra.run.smithi146.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61866.log.gz 2024-09-16T01:54:19.338 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43765.log 2024-09-16T01:54:19.338 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2024-09-16T01:54:19.339 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-09-16T01:54:19.339 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66684.log.gz 2024-09-16T01:54:19.340 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44904.log 2024-09-16T01:54:19.340 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43765.log.gz 2024-09-16T01:54:19.341 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60463.log 87.7% 2024-09-16T01:54:19.341 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2024-09-16T01:54:19.342 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61948.log 2024-09-16T01:54:19.342 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44904.log.gz 2024-09-16T01:54:19.343 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50794.log 2024-09-16T01:54:19.343 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60463.log.gz 2024-09-16T01:54:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61948.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61948.log.gz 2024-09-16T01:54:19.344 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65104.log 2024-09-16T01:54:19.345 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56871.log 2024-09-16T01:54:19.345 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50794.log.gz 2024-09-16T01:54:19.346 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68318.log 2024-09-16T01:54:19.346 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65104.log.gz 2024-09-16T01:54:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42637.log 2024-09-16T01:54:19.347 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56871.log.gz 2024-09-16T01:54:19.347 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43567.log 2024-09-16T01:54:19.348 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68318.log.gz 2024-09-16T01:54:19.348 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60942.log 2024-09-16T01:54:19.349 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42637.log.gz 2024-09-16T01:54:19.349 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43567.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.53435.log -- replaced with /var/log/ceph/ceph-client.admin.43567.log.gz 2024-09-16T01:54:19.350 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-16T01:54:19.351 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36124.log 2024-09-16T01:54:19.351 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60942.log.gz 2024-09-16T01:54:19.351 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53435.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53435.log.gz 2024-09-16T01:54:19.351 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53166.log 2024-09-16T01:54:19.352 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36124.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36124.log.gzgzip 2024-09-16T01:54:19.352 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65836.log 2024-09-16T01:54:19.353 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49799.log 2024-09-16T01:54:19.353 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53166.log.gz 2024-09-16T01:54:19.354 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-09-16T01:54:19.354 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65836.log.gz 2024-09-16T01:54:19.355 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66345.log 2024-09-16T01:54:19.355 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49799.log.gz 2024-09-16T01:54:19.356 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65482.log 2024-09-16T01:54:19.356 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-09-16T01:54:19.357 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47234.log 2024-09-16T01:54:19.357 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66345.log.gz 2024-09-16T01:54:19.358 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-09-16T01:54:19.358 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65482.log.gz 2024-09-16T01:54:19.359 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65267.log 2024-09-16T01:54:19.359 INFO:teuthology.orchestra.run.smithi146.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.47234.log.gz 2024-09-16T01:54:19.360 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60297.log 2024-09-16T01:54:19.360 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-09-16T01:54:19.360 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65267.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-16T01:54:19.361 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65267.log.gz 2024-09-16T01:54:19.361 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52186.log 2024-09-16T01:54:19.362 INFO:teuthology.orchestra.run.smithi146.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60297.log.gz 2024-09-16T01:54:19.362 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-09-16T01:54:19.363 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52186.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52186.log.gz 2024-09-16T01:54:19.364 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68640.log 2024-09-16T01:54:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40909.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-09-16T01:54:19.365 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50985.log 2024-09-16T01:54:19.366 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48771.log 2024-09-16T01:54:19.366 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3%/var/log/ceph/ceph-client.admin.50985.log: -- replaced with /var/log/ceph/ceph-client.admin.68640.log.gz 2024-09-16T01:54:19.367 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63174.log 2024-09-16T01:54:19.367 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2024-09-16T01:54:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59267.log 2024-09-16T01:54:19.368 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.48771.log.gz 2024-09-16T01:54:19.369 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52269.log 2024-09-16T01:54:19.369 INFO:teuthology.orchestra.run.smithi146.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63174.log.gz 2024-09-16T01:54:19.370 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49701.log 2024-09-16T01:54:19.371 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59267.log.gz 2024-09-16T01:54:19.372 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51984.log 2024-09-16T01:54:19.372 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.52269.log.gz 2024-09-16T01:54:19.373 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53294.log 2024-09-16T01:54:19.373 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49701.log.gz 2024-09-16T01:54:19.374 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51753.log 2024-09-16T01:54:19.375 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53294.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.51984.log.gz 2024-09-16T01:54:19.375 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53294.log.gz 2024-09-16T01:54:19.375 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63576.log 2024-09-16T01:54:19.376 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66690.log 2024-09-16T01:54:19.377 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.51753.log.gz 2024-09-16T01:54:19.377 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67772.log 2024-09-16T01:54:19.378 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63576.log.gz 2024-09-16T01:54:19.378 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66671.log 2024-09-16T01:54:19.379 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66690.log.gz 2024-09-16T01:54:19.379 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64756.log 2024-09-16T01:54:19.379 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67772.log.gz 2024-09-16T01:54:19.381 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66671.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66671.log.gz 2024-09-16T01:54:19.381 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59062.log 2024-09-16T01:54:19.381 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42289.log 2024-09-16T01:54:19.382 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64756.log.gz 2024-09-16T01:54:19.383 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42739.log 2024-09-16T01:54:19.383 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.59062.log.gz 2024-09-16T01:54:19.384 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65878.log 2024-09-16T01:54:19.384 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42289.log.gz 2024-09-16T01:54:19.385 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65608.log 2024-09-16T01:54:19.385 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65878.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42739.log.gz 2024-09-16T01:54:19.386 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.42145.log -- replaced with /var/log/ceph/ceph-client.admin.65878.log.gz 2024-09-16T01:54:19.386 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-16T01:54:19.388 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62269.log 2024-09-16T01:54:19.388 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65608.log.gz 2024-09-16T01:54:19.388 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42145.log: gzip -5 --verbose 88.1% -- -- replaced with /var/log/ceph/ceph-client.admin.42145.log.gz /var/log/ceph/ceph-client.admin.57322.log 2024-09-16T01:54:19.389 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-16T01:54:19.389 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65794.log 2024-09-16T01:54:19.390 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57322.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.62269.log.gz 2024-09-16T01:54:19.391 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60563.log 87.5% 2024-09-16T01:54:19.391 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57322.log.gz 2024-09-16T01:54:19.392 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61735.log 2024-09-16T01:54:19.392 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65794.log.gz 2024-09-16T01:54:19.393 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60563.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T01:54:19.393 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60563.log.gz 2024-09-16T01:54:19.394 INFO:teuthology.orchestra.run.smithi146.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-16T01:54:19.394 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67856.log 2024-09-16T01:54:19.395 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61735.log.gz 2024-09-16T01:54:19.395 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59442.log 2024-09-16T01:54:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65225.log 2024-09-16T01:54:19.396 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67856.log.gz 2024-09-16T01:54:19.398 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59442.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59442.log.gz 2024-09-16T01:54:19.398 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49552.log 2024-09-16T01:54:19.398 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60402.log 2024-09-16T01:54:19.399 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65225.log.gz 2024-09-16T01:54:19.400 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67306.log 2024-09-16T01:54:19.400 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49552.log.gz 2024-09-16T01:54:19.401 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60402.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60402.log.gz 2024-09-16T01:54:19.401 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48559.log 2024-09-16T01:54:19.402 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68234.log 2024-09-16T01:54:19.402 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67306.log: /var/log/ceph/ceph-client.admin.48559.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67306.log.gz 2024-09-16T01:54:19.403 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48559.log.gz 2024-09-16T01:54:19.403 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65440.log 2024-09-16T01:54:19.404 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.68234.log: /var/log/ceph/ceph-client.admin.67015.log 2024-09-16T01:54:19.405 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68234.log.gz 2024-09-16T01:54:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65440.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65440.log.gz 2024-09-16T01:54:19.405 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62611.log 2024-09-16T01:54:19.406 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-09-16T01:54:19.406 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67015.log.gz 2024-09-16T01:54:19.407 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62611.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62611.log.gz 2024-09-16T01:54:19.407 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48461.log 2024-09-16T01:54:19.408 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41491.log: -- /var/log/ceph/ceph-client.admin.66673.log 2024-09-16T01:54:19.409 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-09-16T01:54:19.409 INFO:teuthology.orchestra.run.smithi161.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T01:54:19.410 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48461.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48461.log.gz 2024-09-16T01:54:19.410 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61583.log 2024-09-16T01:54:19.410 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43563.log/var/log/ceph/ceph-client.admin.66673.log: 2024-09-16T01:54:19.411 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66673.log.gz 2024-09-16T01:54:19.411 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61583.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61583.log.gz 2024-09-16T01:54:19.411 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54730.log 2024-09-16T01:54:19.412 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log/var/log/ceph/ceph-client.admin.43563.log: 2024-09-16T01:54:19.413 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54730.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.43563.log.gz 2024-09-16T01:54:19.413 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54730.log.gz 2024-09-16T01:54:19.414 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66922.log 2024-09-16T01:54:19.414 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46305.log 2024-09-16T01:54:19.415 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.66922.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66922.log.gz 2024-09-16T01:54:19.415 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55630.log 2024-09-16T01:54:19.416 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43366.log 2024-09-16T01:54:19.417 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46305.log: /var/log/ceph/ceph-client.admin.55630.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46305.log.gz 2024-09-16T01:54:19.417 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64436.log 2024-09-16T01:54:19.418 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55630.log.gz 2024-09-16T01:54:19.418 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2024-09-16T01:54:19.419 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43366.log.gz 2024-09-16T01:54:19.420 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-09-16T01:54:19.420 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64436.log.gz 2024-09-16T01:54:19.421 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57280.log 2024-09-16T01:54:19.421 INFO:teuthology.orchestra.run.smithi146.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2024-09-16T01:54:19.422 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2024-09-16T01:54:19.422 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-09-16T01:54:19.423 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65920.log 2024-09-16T01:54:19.423 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57280.log.gz 2024-09-16T01:54:19.424 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41250.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2024-09-16T01:54:19.424 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63319.log 2024-09-16T01:54:19.425 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68276.log 2024-09-16T01:54:19.426 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65920.log.gz 2024-09-16T01:54:19.426 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62267.log 2024-09-16T01:54:19.427 INFO:teuthology.orchestra.run.smithi146.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.63319.log.gz 2024-09-16T01:54:19.427 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43324.log 2024-09-16T01:54:19.428 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68276.log.gz 2024-09-16T01:54:19.429 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52387.log 2024-09-16T01:54:19.429 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62267.log.gz 2024-09-16T01:54:19.430 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43324.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43324.log.gz 2024-09-16T01:54:19.430 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66342.log 2024-09-16T01:54:19.433 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42016.log 2024-09-16T01:54:19.433 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52387.log.gz 2024-09-16T01:54:19.433 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49954.log 2024-09-16T01:54:19.433 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66342.log.gz 2024-09-16T01:54:19.433 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42016.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42016.log.gz 2024-09-16T01:54:19.433 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57198.log 2024-09-16T01:54:19.434 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61370.log 2024-09-16T01:54:19.435 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57198.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49954.log.gz 2024-09-16T01:54:19.435 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57198.log.gz 2024-09-16T01:54:19.435 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61664.log 2024-09-16T01:54:19.436 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2024-09-16T01:54:19.437 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61664.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61370.log.gz 2024-09-16T01:54:19.437 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45791.log 2024-09-16T01:54:19.438 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.61664.log.gz 2024-09-16T01:54:19.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43282.log 2024-09-16T01:54:19.439 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2024-09-16T01:54:19.440 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60783.log 2024-09-16T01:54:19.440 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43282.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45791.log.gz 2024-09-16T01:54:19.441 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65351.log 2024-09-16T01:54:19.441 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43282.log.gz 2024-09-16T01:54:19.442 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.60783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-09-16T01:54:19.442 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60783.log.gz 2024-09-16T01:54:19.443 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66689.log 2024-09-16T01:54:19.443 INFO:teuthology.orchestra.run.smithi146.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65351.log.gz 2024-09-16T01:54:19.445 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64114.log 2024-09-16T01:54:19.445 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-09-16T01:54:19.445 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47705.log 2024-09-16T01:54:19.446 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66689.log.gz 2024-09-16T01:54:19.447 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64114.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2024-09-16T01:54:19.447 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T01:54:19.447 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59601.log 2024-09-16T01:54:19.449 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.y.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.47705.log.gz 2024-09-16T01:54:19.449 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54190.log 2024-09-16T01:54:19.450 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66046.log 2024-09-16T01:54:19.450 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59601.log.gz 2024-09-16T01:54:19.451 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68505.log 2024-09-16T01:54:19.451 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54190.log.gz 2024-09-16T01:54:19.453 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66046.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T01:54:19.453 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66046.log.gz 2024-09-16T01:54:19.454 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36217.log 2024-09-16T01:54:19.454 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68505.log.gz 2024-09-16T01:54:19.455 INFO:teuthology.orchestra.run.smithi146.stderr: 93.7%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T01:54:19.455 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44451.log 2024-09-16T01:54:19.456 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64321.log 2024-09-16T01:54:19.457 INFO:teuthology.orchestra.run.smithi146.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36217.log.gz 2024-09-16T01:54:19.458 INFO:teuthology.orchestra.run.smithi146.stderr: 85.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T01:54:19.458 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52687.log 2024-09-16T01:54:19.458 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44451.log.gz 2024-09-16T01:54:19.459 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58843.log 2024-09-16T01:54:19.459 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.64321.log.gz 2024-09-16T01:54:19.460 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.52687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67018.log 2024-09-16T01:54:19.460 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.52687.log.gz 2024-09-16T01:54:19.461 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-09-16T01:54:19.461 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.58843.log.gz 2024-09-16T01:54:19.462 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58149.log 2024-09-16T01:54:19.463 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67018.log.gz 2024-09-16T01:54:19.463 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56191.log 2024-09-16T01:54:19.464 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-09-16T01:54:19.465 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49028.log 2024-09-16T01:54:19.465 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.58149.log.gz 2024-09-16T01:54:19.466 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-09-16T01:54:19.466 INFO:teuthology.orchestra.run.smithi146.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56191.log.gz 2024-09-16T01:54:19.467 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47583.log 2024-09-16T01:54:19.467 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49028.log.gz 2024-09-16T01:54:19.468 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67562.log 2024-09-16T01:54:19.469 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-09-16T01:54:19.469 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.47583.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.47583.log.gz 2024-09-16T01:54:19.469 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-09-16T01:54:19.470 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46184.log 2024-09-16T01:54:19.471 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67562.log.gz 2024-09-16T01:54:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63816.log 2024-09-16T01:54:19.472 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-09-16T01:54:19.473 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46184.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.58582.log -- replaced with /var/log/ceph/ceph-client.admin.46184.log.gz 2024-09-16T01:54:19.473 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-16T01:54:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-09-16T01:54:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58582.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.63816.log.gz 2024-09-16T01:54:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61299.log 2024-09-16T01:54:19.475 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58582.log.gz 2024-09-16T01:54:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40268.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-09-16T01:54:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57393.log 2024-09-16T01:54:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54989.log 2024-09-16T01:54:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61299.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61299.log.gz 2024-09-16T01:54:19.478 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57393.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29817.log 2024-09-16T01:54:19.478 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57393.log.gz 2024-09-16T01:54:19.479 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41886.log 2024-09-16T01:54:19.479 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54989.log.gz 2024-09-16T01:54:19.480 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.tmp-client.admin.29817.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29817.log.gz 2024-09-16T01:54:19.480 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63035.log 2024-09-16T01:54:19.481 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66349.log 2024-09-16T01:54:19.482 INFO:teuthology.orchestra.run.smithi161.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T01:54:19.482 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41886.log: 89.4%/var/log/ceph/ceph-client.admin.63035.log: -- replaced with /var/log/ceph/ceph-client.admin.41886.log.gz 2024-09-16T01:54:19.483 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45512.log 2024-09-16T01:54:19.483 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.63035.log.gz 2024-09-16T01:54:19.483 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56140.log 2024-09-16T01:54:19.484 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66349.log.gz 2024-09-16T01:54:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45512.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45512.log.gz 2024-09-16T01:54:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62154.log 2024-09-16T01:54:19.486 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65650.log 2024-09-16T01:54:19.486 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56140.log.gz 2024-09-16T01:54:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43076.log 2024-09-16T01:54:19.487 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62154.log.gz 2024-09-16T01:54:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65650.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65650.log.gz 2024-09-16T01:54:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2024-09-16T01:54:19.489 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.0.43076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43076.log.gz 2024-09-16T01:54:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65393.log 2024-09-16T01:54:19.491 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59907.log 2024-09-16T01:54:19.491 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2024-09-16T01:54:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55470.log 2024-09-16T01:54:19.492 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65393.log.gz 2024-09-16T01:54:19.493 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.59907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2024-09-16T01:54:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55470.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.59907.log.gz 2024-09-16T01:54:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-09-16T01:54:19.494 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.55470.log.gz 2024-09-16T01:54:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68553.log 2024-09-16T01:54:19.496 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2024-09-16T01:54:19.497 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40571.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57671.log 2024-09-16T01:54:19.497 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-09-16T01:54:19.497 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-09-16T01:54:19.498 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68553.log.gz 2024-09-16T01:54:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44847.log 2024-09-16T01:54:19.499 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57671.log.gz 2024-09-16T01:54:19.500 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40697.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2024-09-16T01:54:19.500 INFO:teuthology.orchestra.run.smithi146.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65692.log 2024-09-16T01:54:19.501 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.44847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57079.log 2024-09-16T01:54:19.501 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65692.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.44847.log.gz 2024-09-16T01:54:19.502 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48708.log 2024-09-16T01:54:19.502 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65692.log.gz 2024-09-16T01:54:19.503 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43848.log 2024-09-16T01:54:19.503 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.57079.log.gz 2024-09-16T01:54:19.504 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66688.log 2024-09-16T01:54:19.505 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43848.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48708.log.gz 2024-09-16T01:54:19.505 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43848.log.gz 2024-09-16T01:54:19.506 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68461.log 2024-09-16T01:54:19.507 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66688.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66688.log.gz 2024-09-16T01:54:19.507 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40655.log 2024-09-16T01:54:19.508 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-09-16T01:54:19.509 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68461.log.gz 2024-09-16T01:54:19.510 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49207.log 2024-09-16T01:54:19.510 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40655.log.gz 2024-09-16T01:54:19.510 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64560.log 2024-09-16T01:54:19.511 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-09-16T01:54:19.512 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.49207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2024-09-16T01:54:19.512 INFO:teuthology.orchestra.run.smithi146.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.49207.log.gz 2024-09-16T01:54:19.513 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62815.log 2024-09-16T01:54:19.513 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64560.log.gz 2024-09-16T01:54:19.514 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48602.log 2024-09-16T01:54:19.514 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2024-09-16T01:54:19.515 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62472.log 2024-09-16T01:54:19.516 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.62815.log.gz 2024-09-16T01:54:19.516 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53640.log 2024-09-16T01:54:19.516 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48602.log.gz 2024-09-16T01:54:19.517 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57772.log 2024-09-16T01:54:19.518 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62472.log.gz 2024-09-16T01:54:19.518 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62043.log 2024-09-16T01:54:19.519 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.53640.log.gz 2024-09-16T01:54:19.519 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65734.log 2024-09-16T01:54:19.520 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.62043.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.57772.log.gz 2024-09-16T01:54:19.521 INFO:teuthology.orchestra.run.smithi146.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62043.log.gz -5 2024-09-16T01:54:19.521 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54117.log 2024-09-16T01:54:19.522 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.65734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2024-09-16T01:54:19.522 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65734.log.gz 2024-09-16T01:54:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48645.log 2024-09-16T01:54:19.523 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54117.log.gz 2024-09-16T01:54:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50315.log 2024-09-16T01:54:19.524 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2024-09-16T01:54:19.525 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48645.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48645.log.gz 2024-09-16T01:54:19.525 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40436.log 2024-09-16T01:54:19.526 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64939.log 2024-09-16T01:54:19.527 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50315.log.gz 2024-09-16T01:54:19.528 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40436.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-09-16T01:54:19.528 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-09-16T01:54:19.529 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64939.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64939.log.gz 2024-09-16T01:54:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45265.log 2024-09-16T01:54:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40910.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-09-16T01:54:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54972.log 2024-09-16T01:54:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.45265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63735.log 2024-09-16T01:54:19.532 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45265.log.gz 2024-09-16T01:54:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68150.log 2024-09-16T01:54:19.533 INFO:teuthology.orchestra.run.smithi146.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54972.log.gz 2024-09-16T01:54:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54405.log 2024-09-16T01:54:19.534 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63735.log.gz 2024-09-16T01:54:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61200.log 2024-09-16T01:54:19.536 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68150.log.gz 2024-09-16T01:54:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.54405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63474.log 2024-09-16T01:54:19.536 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54405.log.gz 2024-09-16T01:54:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61200.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61200.log.gz 2024-09-16T01:54:19.538 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56729.log 2024-09-16T01:54:19.538 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.63474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66088.log 2024-09-16T01:54:19.539 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63474.log.gz 2024-09-16T01:54:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67022.log 2024-09-16T01:54:19.540 INFO:teuthology.orchestra.run.smithi146.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56729.log.gz 2024-09-16T01:54:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56424.log 2024-09-16T01:54:19.541 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66088.log.gz 2024-09-16T01:54:19.542 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57588.log 2024-09-16T01:54:19.542 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67022.log.gz 2024-09-16T01:54:19.543 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68024.log 2024-09-16T01:54:19.543 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.56424.log.gz 2024-09-16T01:54:19.544 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51933.log 2024-09-16T01:54:19.545 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57588.log.gz 2024-09-16T01:54:19.545 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68420.log 2024-09-16T01:54:19.546 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68024.log.gz 2024-09-16T01:54:19.547 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51933.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51933.log.gz 2024-09-16T01:54:19.547 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-09-16T01:54:19.547 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2024-09-16T01:54:19.548 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68420.log: 88.1%/var/log/ceph/ceph-mds.e.log: -- replaced with /var/log/ceph/ceph-client.admin.68420.log.gz 2024-09-16T01:54:19.549 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39679.log 2024-09-16T01:54:19.550 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51573.log 2024-09-16T01:54:19.550 INFO:teuthology.orchestra.run.smithi146.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2024-09-16T01:54:19.551 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.39679.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39679.log.gz 2024-09-16T01:54:19.552 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57505.log 2024-09-16T01:54:19.552 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.51573.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51573.log.gz 2024-09-16T01:54:19.553 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56282.log 2024-09-16T01:54:19.554 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.57505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66354.log 2024-09-16T01:54:19.554 INFO:teuthology.orchestra.run.smithi146.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57505.log.gz 2024-09-16T01:54:19.556 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56282.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56282.log.gz 2024-09-16T01:54:19.556 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50218.log 2024-09-16T01:54:19.557 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66354.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66354.log.gz 2024-09-16T01:54:19.557 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58231.log 2024-09-16T01:54:19.558 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.50218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64237.log 2024-09-16T01:54:19.559 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50218.log.gz 2024-09-16T01:54:19.560 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58231.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58231.log.gz 2024-09-16T01:54:19.560 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43564.log 2024-09-16T01:54:19.561 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64237.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64237.log.gz 2024-09-16T01:54:19.561 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68736.log 2024-09-16T01:54:19.563 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.43564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67814.log 2024-09-16T01:54:19.563 INFO:teuthology.orchestra.run.smithi146.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43564.log.gz 2024-09-16T01:54:19.564 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68736.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68736.log.gz 2024-09-16T01:54:19.565 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46083.log 2024-09-16T01:54:19.566 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67814.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67814.log.gz 2024-09-16T01:54:19.566 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55870.log 2024-09-16T01:54:19.567 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.46083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-09-16T01:54:19.567 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46083.log.gz 2024-09-16T01:54:19.569 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55870.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55870.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-09-16T01:54:19.569 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-16T01:54:19.569 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40919.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz 2024-09-16T01:54:19.570 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T01:54:19.571 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.40088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66004.log 2024-09-16T01:54:19.571 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-09-16T01:54:19.572 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42439.log 2024-09-16T01:54:19.574 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66004.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66004.log.gz 2024-09-16T01:54:19.574 INFO:teuthology.orchestra.run.smithi146.stderr:gzip/var/log/ceph/ceph-client.admin.42439.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48517.log 2024-09-16T01:54:19.576 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42439.log.gz 2024-09-16T01:54:19.576 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-16T01:54:19.577 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48517.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48517.log.gz 2024-09-16T01:54:19.578 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59202.log 2024-09-16T01:54:19.580 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.59202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2024-09-16T01:54:19.582 INFO:teuthology.orchestra.run.smithi146.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59202.log.gz 2024-09-16T01:54:19.583 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53377.log 2024-09-16T01:54:19.585 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.55250.log: 92.2% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-09-16T01:54:19.586 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2024-09-16T01:54:19.587 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42331.log 2024-09-16T01:54:19.587 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53377.log.gz 2024-09-16T01:54:19.588 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66172.log 2024-09-16T01:54:19.589 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.42331.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42331.log.gz 2024-09-16T01:54:19.590 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58400.log 2024-09-16T01:54:19.590 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.66172.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66172.log.gz 2024-09-16T01:54:19.591 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60321.log 2024-09-16T01:54:19.593 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.58400.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.58400.log.gz 2024-09-16T01:54:19.593 INFO:teuthology.orchestra.run.smithi146.stderr:gzip/var/log/ceph/ceph-client.admin.60321.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53043.log 2024-09-16T01:54:19.594 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60321.log.gz 2024-09-16T01:54:19.595 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67011.log 2024-09-16T01:54:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.53043.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53043.log.gz 2024-09-16T01:54:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.67011.log: /var/log/ceph/ceph-client.admin.48905.log 2024-09-16T01:54:19.597 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67011.log.gz 2024-09-16T01:54:19.598 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64642.log 2024-09-16T01:54:19.600 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.48905.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48905.log.gz 2024-09-16T01:54:19.601 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61340.log 2024-09-16T01:54:19.601 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.64642.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64642.log.gz 2024-09-16T01:54:19.602 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67688.log 2024-09-16T01:54:19.603 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.61340.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61340.log.gz 2024-09-16T01:54:19.604 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67982.log 2024-09-16T01:54:19.605 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67688.log.gz 2024-09-16T01:54:19.606 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41844.log 2024-09-16T01:54:19.608 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.67982.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67982.log.gz 2024-09-16T01:54:19.608 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68781.log 2024-09-16T01:54:19.608 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.41844.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41844.log.gz 2024-09-16T01:54:19.609 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56787.log 2024-09-16T01:54:19.610 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.68781.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68781.log.gz 2024-09-16T01:54:19.612 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.56787.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56787.log.gz 2024-09-16T01:54:19.759 INFO:teuthology.orchestra.run.smithi146.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-16T01:54:19.904 INFO:teuthology.orchestra.run.smithi146.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T01:54:20.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f has been restored 2024-09-16T01:54:20.872 INFO:teuthology.orchestra.run.smithi161.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T01:54:22.929 INFO:teuthology.orchestra.run.smithi161.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T01:54:23.125 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T01:54:24.294 INFO:teuthology.orchestra.run.smithi161.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T01:54:24.531 INFO:teuthology.orchestra.run.smithi161.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T01:54:24.533 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-09-16T01:54:24.533 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m5.354s 2024-09-16T01:54:24.533 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m15.955s 2024-09-16T01:54:24.533 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.581s 2024-09-16T01:54:24.909 INFO:teuthology.orchestra.run.smithi146.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T01:54:27.479 INFO:teuthology.orchestra.run.smithi146.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T01:54:27.585 INFO:teuthology.orchestra.run.smithi146.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T01:54:27.589 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-16T01:54:27.589 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m8.432s 2024-09-16T01:54:27.589 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m24.339s 2024-09-16T01:54:27.589 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m3.204s 2024-09-16T01:54:27.590 INFO:tasks.ceph:Archiving logs... 2024-09-16T01:54:27.590 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/remote/smithi146/log 2024-09-16T01:54:27.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:54:31.511 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/remote/smithi161/log 2024-09-16T01:54:31.512 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:54:33.900 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T01:54:33.910 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T01:54:33.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:54:33.916 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:54:34.023 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T01:54:34.032 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T01:54:34.033 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:54:34.037 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:54:34.054 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:54:34.059 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:54:34.059 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-09-16T01:54:34.059 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 -432us[ -437us] +/- 50ms 2024-09-16T01:54:34.059 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 +373us[ +373us] +/- 62ms 2024-09-16T01:54:34.059 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +325us[ +325us] +/- 86ms 2024-09-16T01:54:34.059 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:54:34.069 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:54:34.076 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:54:34.076 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-09-16T01:54:34.076 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 -452us[ -457us] +/- 50ms 2024-09-16T01:54:34.076 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 +313us[ +308us] +/- 62ms 2024-09-16T01:54:34.077 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +283us[ +279us] +/- 86ms 2024-09-16T01:54:34.077 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:54:34.077 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T01:54:34.088 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T01:54:34.088 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T01:54:34.124 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:54:34.263 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:54:34.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:54:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:54:34.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:54:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:54:34.435 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T01:54:34.445 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T01:54:34.468 INFO:teuthology.task.internal:Duration was 791.091299 seconds 2024-09-16T01:54:34.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T01:54:34.502 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T01:54:34.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:54:34.506 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:54:34.548 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:54:34.589 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:54:35.085 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T01:54:35.085 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-09-16T01:54:35.085 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:54:35.134 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-09-16T01:54:35.135 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:54:35.163 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T01:54:35.164 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:54:35.169 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:54:35.232 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T01:54:35.232 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:54:35.237 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:54:35.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T01:54:35.422 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T01:54:35.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:54:35.479 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:54:35.505 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T01:54:35.515 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:54:35.519 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:54:35.568 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-09-16T01:54:35.587 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-09-16T01:54:35.617 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:54:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:54:35.653 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:54:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:54:35.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T01:54:35.687 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T01:54:35.689 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/remote/smithi146 2024-09-16T01:54:35.689 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:54:35.761 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906490/remote/smithi161 2024-09-16T01:54:35.761 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:54:35.796 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T01:54:35.796 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:54:35.801 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:54:35.848 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T01:54:35.864 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T01:54:35.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T01:54:35.897 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T01:54:35.897 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:54:35.902 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:54:35.918 INFO:teuthology.orchestra.run.smithi161.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:54 /home/ubuntu/cephtest 2024-09-16T01:54:35.933 INFO:teuthology.orchestra.run.smithi146.stdout: 393233 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:54 /home/ubuntu/cephtest 2024-09-16T01:54:35.938 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T01:54:35.951 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T01:54:36.020 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} duration: 791.0912988185883 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T01:54:36.020 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:54:36.110 INFO:teuthology.run:pass